2024-09-16T04:28:35.878 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T04:28:35.900 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T04:28:35.982 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906031 branch: squid description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/many workloads/pool-create-delete} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906031' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon client ping interval: 4 mon client ping timeout: 12 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - slow request - overall HEALTH_ sha1: 8921c8ef947807b717db041d85f0761b7914d5fc ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 8921c8ef947807b717db041d85f0761b7914d5fc selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: ad35db06e8924912f01e32b35576297c0f1352b4 owner: scheduled_teuthology@teuthology priority: 700 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 5616 sha1: 8921c8ef947807b717db041d85f0761b7914d5fc sleep_before_teardown: 0 subset: 15927/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ad35db06e8924912f01e32b35576297c0f1352b4 targets: smithi046.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNMLqXwawDOhk5uLrKgnesDWCzWHSdcfDOfa/3p9Posol0v5qa0BAvoS1qWIDkZRfdk/yXM9lBElVBQSehJO1jQ= smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE+GRyLy3+p2Go8uZ+tF/JqOLdZIsCS79Rnej/qSyoEotA1p9KvZcCEfdhMrhw62EGFDhE8bwRlwvsBmk5AOsEc= tasks: - install: null - ceph: null - mon_thrash: freeze_mon_duration: 20 freeze_mon_probability: 10 revive_delay: 20 thrash_delay: 1 thrash_many: true - exec: client.0: - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_21:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T04:28:35.983 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa; will attempt to use it 2024-09-16T04:28:35.983 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks 2024-09-16T04:28:35.984 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T04:28:35.985 INFO:teuthology.task.internal:Checking packages... 2024-09-16T04:28:36.007 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8921c8ef947807b717db041d85f0761b7914d5fc' 2024-09-16T04:28:36.007 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T04:28:36.007 INFO:teuthology.packaging:ref: None 2024-09-16T04:28:36.007 INFO:teuthology.packaging:tag: None 2024-09-16T04:28:36.007 INFO:teuthology.packaging:branch: squid 2024-09-16T04:28:36.007 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:28:36.007 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-16T04:28:36.209 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-328.gad35db06 2024-09-16T04:28:36.211 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T04:28:36.218 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T04:28:36.218 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T04:28:36.273 INFO:teuthology.task.internal:Saving configuration 2024-09-16T04:28:36.290 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T04:28:36.303 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T04:28:36.328 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906031', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 04:23:43.535494', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNMLqXwawDOhk5uLrKgnesDWCzWHSdcfDOfa/3p9Posol0v5qa0BAvoS1qWIDkZRfdk/yXM9lBElVBQSehJO1jQ='} 2024-09-16T04:28:36.351 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906031', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 04:23:43.534134', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE+GRyLy3+p2Go8uZ+tF/JqOLdZIsCS79Rnej/qSyoEotA1p9KvZcCEfdhMrhw62EGFDhE8bwRlwvsBmk5AOsEc='} 2024-09-16T04:28:36.351 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T04:28:36.361 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-16T04:28:36.362 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-16T04:28:36.362 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T04:28:36.492 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9314d6fac0>, signals=[15]) 2024-09-16T04:28:36.493 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T04:28:36.509 INFO:teuthology.task.internal:Opening connections... 2024-09-16T04:28:36.510 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com 2024-09-16T04:28:36.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:28:36.595 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T04:28:36.595 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:28:36.672 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T04:28:36.679 DEBUG:teuthology.orchestra.run.smithi046:> uname -m 2024-09-16T04:28:36.698 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2024-09-16T04:28:36.698 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release 2024-09-16T04:28:36.757 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="CentOS Stream" 2024-09-16T04:28:36.757 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="9" 2024-09-16T04:28:36.757 INFO:teuthology.orchestra.run.smithi046.stdout:ID="centos" 2024-09-16T04:28:36.758 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE="rhel fedora" 2024-09-16T04:28:36.758 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="9" 2024-09-16T04:28:36.758 INFO:teuthology.orchestra.run.smithi046.stdout:PLATFORM_ID="platform:el9" 2024-09-16T04:28:36.758 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T04:28:36.758 INFO:teuthology.orchestra.run.smithi046.stdout:ANSI_COLOR="0;31" 2024-09-16T04:28:36.758 INFO:teuthology.orchestra.run.smithi046.stdout:LOGO="fedora-logo-icon" 2024-09-16T04:28:36.758 INFO:teuthology.orchestra.run.smithi046.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T04:28:36.758 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://centos.org/" 2024-09-16T04:28:36.758 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T04:28:36.758 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T04:28:36.759 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T04:28:36.759 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2024-09-16T04:28:36.782 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2024-09-16T04:28:36.800 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2024-09-16T04:28:36.800 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2024-09-16T04:28:36.857 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2024-09-16T04:28:36.858 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="9" 2024-09-16T04:28:36.858 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2024-09-16T04:28:36.858 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2024-09-16T04:28:36.858 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="9" 2024-09-16T04:28:36.858 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el9" 2024-09-16T04:28:36.858 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T04:28:36.858 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2024-09-16T04:28:36.858 INFO:teuthology.orchestra.run.smithi114.stdout:LOGO="fedora-logo-icon" 2024-09-16T04:28:36.858 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T04:28:36.858 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2024-09-16T04:28:36.858 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T04:28:36.859 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T04:28:36.859 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T04:28:36.859 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2024-09-16T04:28:36.882 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T04:28:36.889 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T04:28:36.918 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T04:28:36.918 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest 2024-09-16T04:28:36.921 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2024-09-16T04:28:36.938 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T04:28:36.945 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T04:28:36.945 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph) 2024-09-16T04:28:36.978 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2024-09-16T04:28:37.005 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T04:28:37.058 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T04:28:37.072 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T04:28:37.072 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T04:28:37.073 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T04:28:37.073 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T04:28:37.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:28:37.090 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-09-16T04:28:37.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:28:37.091 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-09-16T04:28:37.147 INFO:teuthology.orchestra.run.smithi046.stdout:5.14.0-503.el9.x86_64 2024-09-16T04:28:37.147 INFO:teuthology.task.kernel:Running kernel on smithi046: 5.14.0-503.el9.x86_64 2024-09-16T04:28:37.147 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum install -y kernel 2024-09-16T04:28:37.150 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-503.el9.x86_64 2024-09-16T04:28:37.150 INFO:teuthology.task.kernel:Running kernel on smithi114: 5.14.0-503.el9.x86_64 2024-09-16T04:28:37.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum install -y kernel 2024-09-16T04:28:40.554 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.2 MB 00:01 2024-09-16T04:28:40.645 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.2 MB 00:01 2024-09-16T04:28:42.372 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 31 MB/s | 20 MB 00:00 2024-09-16T04:28:43.405 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 12 MB/s | 20 MB 00:01 2024-09-16T04:28:47.132 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-16T04:28:47.922 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 6.5 MB 00:00 2024-09-16T04:28:48.657 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 19 kB 00:00 2024-09-16T04:28:49.590 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 19 kB 00:00 2024-09-16T04:28:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-16T04:28:50.643 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-16T04:28:55.359 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-16T04:28:55.983 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-16T04:28:57.448 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T04:28:57.449 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T04:28:57.532 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:28:57.539 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:28:57.539 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-09-16T04:28:57.539 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:28:57.539 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-09-16T04:28:57.539 INFO:teuthology.orchestra.run.smithi046.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T04:28:57.539 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-09-16T04:28:57.540 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T04:28:57.540 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T04:28:57.540 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T04:28:57.540 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:28:57.540 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-09-16T04:28:57.540 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:28:57.540 INFO:teuthology.orchestra.run.smithi046.stdout:Install 4 Packages 2024-09-16T04:28:57.540 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:28:57.541 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 79 M 2024-09-16T04:28:57.541 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 126 M 2024-09-16T04:28:57.541 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-09-16T04:28:57.851 INFO:teuthology.orchestra.run.smithi046.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 78 kB/s | 23 kB 00:00 2024-09-16T04:28:57.997 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T04:28:57.997 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T04:28:58.084 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:28:58.090 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:28:58.090 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-09-16T04:28:58.090 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:28:58.090 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T04:28:58.090 INFO:teuthology.orchestra.run.smithi114.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T04:28:58.090 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-16T04:28:58.090 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T04:28:58.090 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T04:28:58.090 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T04:28:58.091 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:28:58.091 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T04:28:58.091 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:28:58.091 INFO:teuthology.orchestra.run.smithi114.stdout:Install 4 Packages 2024-09-16T04:28:58.091 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:28:58.091 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 79 M 2024-09-16T04:28:58.091 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 126 M 2024-09-16T04:28:58.091 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T04:28:58.801 INFO:teuthology.orchestra.run.smithi114.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 39 kB/s | 23 kB 00:00 2024-09-16T04:28:59.601 INFO:teuthology.orchestra.run.smithi114.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-16T04:29:00.285 INFO:teuthology.orchestra.run.smithi114.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 17 MB/s | 35 MB 00:02 2024-09-16T04:29:01.060 INFO:teuthology.orchestra.run.smithi114.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 13 MB/s | 29 MB 00:02 2024-09-16T04:29:01.060 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:29:01.060 INFO:teuthology.orchestra.run.smithi114.stdout:Total 27 MB/s | 79 MB 00:02 2024-09-16T04:29:01.750 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T04:29:02.116 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T04:29:02.116 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T04:29:03.073 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T04:29:03.074 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T04:29:05.537 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T04:29:06.215 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T04:29:06.301 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T04:29:09.093 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T04:29:09.220 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T04:29:11.532 INFO:teuthology.orchestra.run.smithi046.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 1.1 MB/s | 16 MB 00:13 2024-09-16T04:29:12.284 INFO:teuthology.orchestra.run.smithi046.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 2.3 MB/s | 35 MB 00:14 2024-09-16T04:29:12.801 INFO:teuthology.orchestra.run.smithi046.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 1.9 MB/s | 29 MB 00:14 2024-09-16T04:29:12.801 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:29:12.801 INFO:teuthology.orchestra.run.smithi046.stdout:Total 5.2 MB/s | 79 MB 00:15 2024-09-16T04:29:13.512 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-09-16T04:29:13.907 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-09-16T04:29:13.908 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-09-16T04:29:14.050 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T04:29:14.087 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:29:14.889 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-09-16T04:29:14.889 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-09-16T04:29:17.467 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-09-16T04:29:18.135 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T04:29:18.417 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T04:29:18.947 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:29:21.251 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T04:29:21.387 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T04:29:26.518 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T04:29:26.563 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:29:31.706 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:29:57.341 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:29:57.380 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:29:57.724 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:29:57.724 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T04:29:57.724 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T04:29:57.725 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T04:29:58.610 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:29:58.611 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:29:58.611 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T04:29:58.611 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T04:29:58.611 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T04:29:58.611 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T04:29:58.611 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T04:29:58.611 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:29:58.611 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:29:58.956 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T04:29:59.045 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T04:29:59.045 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T04:29:59.046 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T04:29:59.046 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T04:29:59.046 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T04:29:59.046 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T04:29:59.046 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi114.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T04:29:59.046 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T04:29:59.105 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T04:29:59.105 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T04:29:59.106 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -qi grub2-tools 2024-09-16T04:29:59.137 INFO:teuthology.orchestra.run.smithi114.stdout:Name : grub2-tools 2024-09-16T04:29:59.138 INFO:teuthology.orchestra.run.smithi114.stdout:Epoch : 1 2024-09-16T04:29:59.138 INFO:teuthology.orchestra.run.smithi114.stdout:Version : 2.06 2024-09-16T04:29:59.138 INFO:teuthology.orchestra.run.smithi114.stdout:Release : 82.el9 2024-09-16T04:29:59.138 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: x86_64 2024-09-16T04:29:59.138 INFO:teuthology.orchestra.run.smithi114.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T04:29:59.138 INFO:teuthology.orchestra.run.smithi114.stdout:Group : Unspecified 2024-09-16T04:29:59.138 INFO:teuthology.orchestra.run.smithi114.stdout:Size : 8274670 2024-09-16T04:29:59.138 INFO:teuthology.orchestra.run.smithi114.stdout:License : GPLv3+ 2024-09-16T04:29:59.138 INFO:teuthology.orchestra.run.smithi114.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T04:29:59.138 INFO:teuthology.orchestra.run.smithi114.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T04:29:59.138 INFO:teuthology.orchestra.run.smithi114.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T04:29:59.138 INFO:teuthology.orchestra.run.smithi114.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T04:29:59.138 INFO:teuthology.orchestra.run.smithi114.stdout:Packager : builder@centos.org 2024-09-16T04:29:59.139 INFO:teuthology.orchestra.run.smithi114.stdout:Vendor : CentOS 2024-09-16T04:29:59.139 INFO:teuthology.orchestra.run.smithi114.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T04:29:59.139 INFO:teuthology.orchestra.run.smithi114.stdout:Summary : Support tools for GRUB. 2024-09-16T04:29:59.139 INFO:teuthology.orchestra.run.smithi114.stdout:Description : 2024-09-16T04:29:59.139 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:29:59.139 INFO:teuthology.orchestra.run.smithi114.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T04:29:59.139 INFO:teuthology.orchestra.run.smithi114.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T04:29:59.139 INFO:teuthology.orchestra.run.smithi114.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T04:29:59.139 INFO:teuthology.orchestra.run.smithi114.stdout:hardware devices. 2024-09-16T04:29:59.139 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:29:59.139 INFO:teuthology.orchestra.run.smithi114.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T04:29:59.140 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T04:29:59.140 INFO:teuthology.task.kernel:Updating grub on smithi114 to boot 5.14.0-505.el9.x86_64 2024-09-16T04:29:59.140 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T04:29:59.751 INFO:teuthology.orchestra.run.smithi114.stderr:Generating grub configuration file ... 2024-09-16T04:30:01.006 INFO:teuthology.orchestra.run.smithi114.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T04:30:01.037 INFO:teuthology.orchestra.run.smithi114.stderr:done 2024-09-16T04:30:01.040 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-09-16T04:30:01.055 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.6cPdamSLph 2024-09-16T04:30:01.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6cPdamSLph 2024-09-16T04:30:01.123 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.6cPdamSLph 2024-09-16T04:30:01.281 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.6cPdamSLph is 6KB 2024-09-16T04:30:01.330 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.6cPdamSLph 2024-09-16T04:30:01.345 DEBUG:teuthology.orchestra.run.smithi114:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T04:30:01.409 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T04:30:01.409 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T04:30:01.409 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T04:30:01.409 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T04:30:01.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T04:30:01.582 DEBUG:teuthology.orchestra.run.smithi114:> sudo shutdown -r now 2024-09-16T04:30:10.409 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:30:10.450 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:30:10.840 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:30:10.840 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T04:30:10.840 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T04:30:10.840 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T04:30:11.666 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:30:11.666 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:30:11.666 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-09-16T04:30:11.666 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T04:30:11.666 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T04:30:11.666 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T04:30:11.666 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T04:30:11.666 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:30:11.666 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:30:12.042 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T04:30:12.127 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T04:30:12.128 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T04:30:12.128 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T04:30:12.128 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T04:30:12.128 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T04:30:12.128 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T04:30:12.128 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi046.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T04:30:12.128 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T04:30:12.187 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T04:30:12.187 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T04:30:12.187 DEBUG:teuthology.orchestra.run.smithi046:> sudo rpm -qi grub2-tools 2024-09-16T04:30:12.224 INFO:teuthology.orchestra.run.smithi046.stdout:Name : grub2-tools 2024-09-16T04:30:12.225 INFO:teuthology.orchestra.run.smithi046.stdout:Epoch : 1 2024-09-16T04:30:12.225 INFO:teuthology.orchestra.run.smithi046.stdout:Version : 2.06 2024-09-16T04:30:12.225 INFO:teuthology.orchestra.run.smithi046.stdout:Release : 82.el9 2024-09-16T04:30:12.225 INFO:teuthology.orchestra.run.smithi046.stdout:Architecture: x86_64 2024-09-16T04:30:12.225 INFO:teuthology.orchestra.run.smithi046.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T04:30:12.225 INFO:teuthology.orchestra.run.smithi046.stdout:Group : Unspecified 2024-09-16T04:30:12.225 INFO:teuthology.orchestra.run.smithi046.stdout:Size : 8274670 2024-09-16T04:30:12.225 INFO:teuthology.orchestra.run.smithi046.stdout:License : GPLv3+ 2024-09-16T04:30:12.225 INFO:teuthology.orchestra.run.smithi046.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T04:30:12.225 INFO:teuthology.orchestra.run.smithi046.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T04:30:12.225 INFO:teuthology.orchestra.run.smithi046.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T04:30:12.225 INFO:teuthology.orchestra.run.smithi046.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T04:30:12.226 INFO:teuthology.orchestra.run.smithi046.stdout:Packager : builder@centos.org 2024-09-16T04:30:12.226 INFO:teuthology.orchestra.run.smithi046.stdout:Vendor : CentOS 2024-09-16T04:30:12.226 INFO:teuthology.orchestra.run.smithi046.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T04:30:12.226 INFO:teuthology.orchestra.run.smithi046.stdout:Summary : Support tools for GRUB. 2024-09-16T04:30:12.226 INFO:teuthology.orchestra.run.smithi046.stdout:Description : 2024-09-16T04:30:12.226 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:30:12.226 INFO:teuthology.orchestra.run.smithi046.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T04:30:12.226 INFO:teuthology.orchestra.run.smithi046.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T04:30:12.227 INFO:teuthology.orchestra.run.smithi046.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T04:30:12.227 INFO:teuthology.orchestra.run.smithi046.stdout:hardware devices. 2024-09-16T04:30:12.227 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:30:12.227 INFO:teuthology.orchestra.run.smithi046.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T04:30:12.228 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T04:30:12.228 INFO:teuthology.task.kernel:Updating grub on smithi046 to boot 5.14.0-505.el9.x86_64 2024-09-16T04:30:12.228 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T04:30:12.885 INFO:teuthology.orchestra.run.smithi046.stderr:Generating grub configuration file ... 2024-09-16T04:30:14.272 INFO:teuthology.orchestra.run.smithi046.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T04:30:14.305 INFO:teuthology.orchestra.run.smithi046.stderr:done 2024-09-16T04:30:14.308 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-09-16T04:30:14.323 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.hyXBLLa3TF 2024-09-16T04:30:14.323 DEBUG:teuthology.orchestra.run.smithi046:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.hyXBLLa3TF 2024-09-16T04:30:14.390 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.hyXBLLa3TF 2024-09-16T04:30:14.534 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.hyXBLLa3TF is 6KB 2024-09-16T04:30:14.582 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.hyXBLLa3TF 2024-09-16T04:30:14.596 DEBUG:teuthology.orchestra.run.smithi046:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T04:30:14.658 INFO:teuthology.orchestra.run.smithi046.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T04:30:14.658 INFO:teuthology.orchestra.run.smithi046.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T04:30:14.659 INFO:teuthology.orchestra.run.smithi046.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T04:30:14.659 INFO:teuthology.orchestra.run.smithi046.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T04:30:14.660 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T04:30:14.813 DEBUG:teuthology.orchestra.run.smithi046:> sudo shutdown -r now 2024-09-16T04:30:31.601 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T04:30:31.601 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-16T04:30:31.602 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:30:44.828 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T04:30:44.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-09-16T04:30:44.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:31:19.357 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-09-16T04:31:28.365 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-09-16T04:31:28.366 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:31:31.607 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T04:31:40.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-16T04:31:40.618 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:31:41.047 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-09-16T04:31:41.518 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-16T04:31:41.518 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-16T04:31:41.518 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-09-16T04:31:41.533 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-505.el9.x86_64 2024-09-16T04:31:41.533 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T04:31:41.533 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T04:31:41.533 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T04:31:42.534 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T04:31:42.534 DEBUG:teuthology.orchestra.run.smithi114:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T04:31:42.644 INFO:teuthology.orchestra.run.smithi114.stdout:ttyS1 2024-09-16T04:31:42.670 DEBUG:teuthology.parallel:result is None 2024-09-16T04:31:46.749 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-09-16T04:31:58.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-09-16T04:31:58.751 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:32:01.055 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-09-16T04:32:01.493 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-09-16T04:32:01.493 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T04:32:01.493 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-09-16T04:32:01.508 INFO:teuthology.orchestra.run.smithi046.stdout:5.14.0-505.el9.x86_64 2024-09-16T04:32:01.508 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T04:32:01.508 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T04:32:01.508 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T04:32:02.510 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T04:32:02.510 DEBUG:teuthology.orchestra.run.smithi046:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T04:32:02.589 INFO:teuthology.orchestra.run.smithi046.stdout:ttyS1 2024-09-16T04:32:02.618 DEBUG:teuthology.parallel:result is None 2024-09-16T04:32:02.619 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T04:32:02.630 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T04:32:02.630 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T04:32:02.632 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T04:32:02.653 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T04:32:02.670 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T04:32:02.703 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T04:32:02.704 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T04:32:02.706 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T04:32:02.803 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T04:32:02.811 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T04:32:02.811 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T04:32:02.814 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T04:32:02.845 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:32:02.855 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:32:02.857 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:32:02.866 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:32:02.868 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T04:32:02.876 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T04:32:02.876 DEBUG:teuthology.orchestra.run.smithi046:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T04:32:02.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T04:32:02.938 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T04:32:02.953 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T04:32:02.953 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T04:32:02.977 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T04:32:02.994 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:32:03.076 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:32:03.126 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:32:03.211 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:32:03.250 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T04:32:03.251 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T04:32:03.337 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:32:03.381 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:32:03.443 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:32:03.490 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:32:03.527 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T04:32:03.528 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T04:32:03.593 DEBUG:teuthology.orchestra.run.smithi046:> sudo service rsyslog restart 2024-09-16T04:32:03.595 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2024-09-16T04:32:03.635 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T04:32:03.681 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T04:32:04.050 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T04:32:04.064 INFO:teuthology.task.internal:Starting timer... 2024-09-16T04:32:04.064 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T04:32:04.089 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T04:32:04.169 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-16T04:32:04.170 DEBUG:teuthology.orchestra.run.smithi046:> sudo service auditd rotate 2024-09-16T04:32:04.216 INFO:teuthology.orchestra.run.smithi046.stdout:Rotating logs: 2024-09-16T04:32:04.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2024-09-16T04:32:04.273 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2024-09-16T04:32:04.275 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T04:32:04.275 DEBUG:teuthology.orchestra.run.smithi046:> /usr/sbin/getenforce 2024-09-16T04:32:04.318 INFO:teuthology.orchestra.run.smithi046.stdout:Permissive 2024-09-16T04:32:04.319 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2024-09-16T04:32:04.350 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2024-09-16T04:32:04.350 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi046.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi114.front.sepia.ceph.com': 'permissive'} 2024-09-16T04:32:04.350 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T04:32:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:32:04.386 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T04:32:04.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:32:04.416 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T04:32:04.417 DEBUG:teuthology.orchestra.run.smithi046:> sudo /usr/sbin/setenforce permissive 2024-09-16T04:32:04.460 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2024-09-16T04:32:04.500 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T04:32:04.510 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T04:32:04.521 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T04:32:04.522 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi046.front.sepia.ceph.com,smithi114.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T04:35:54.200 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi046.front.sepia.ceph.com'), Remote(name='ubuntu@smithi114.front.sepia.ceph.com')] 2024-09-16T04:35:54.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-09-16T04:35:54.203 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:35:54.287 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-09-16T04:35:54.373 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-09-16T04:35:54.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-16T04:35:54.374 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:35:54.496 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-09-16T04:35:54.573 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-16T04:35:54.573 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T04:35:54.584 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T04:35:54.585 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T04:35:54.585 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:35:54.588 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T04:35:54.588 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:35:54.617 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T04:35:54.633 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T04:35:54.643 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T04:35:54.657 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T04:35:54.660 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: ntpd: command not found 2024-09-16T04:35:54.672 INFO:teuthology.orchestra.run.smithi046.stdout:506 Cannot talk to daemon 2024-09-16T04:35:54.682 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2024-09-16T04:35:54.687 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T04:35:54.693 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2024-09-16T04:35:54.702 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T04:35:54.707 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T04:35:54.721 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T04:35:54.757 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-09-16T04:35:54.760 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T04:35:54.760 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-09-16T04:35:54.761 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:35:54.761 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:35:54.761 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:35:54.761 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:35:54.778 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-09-16T04:35:54.782 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T04:35:54.782 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-09-16T04:35:54.782 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:35:54.783 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:35:54.783 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:35:54.783 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:35:54.784 INFO:teuthology.run_tasks:Running task install... 2024-09-16T04:35:54.792 DEBUG:teuthology.task.install:project ceph 2024-09-16T04:35:54.792 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}} 2024-09-16T04:35:54.792 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'} 2024-09-16T04:35:54.792 INFO:teuthology.task.install:Using flavor: default 2024-09-16T04:35:54.798 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T04:35:54.798 INFO:teuthology.task.install:extra packages: [] 2024-09-16T04:35:54.798 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T04:35:54.799 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:35:54.805 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T04:35:54.805 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:35:54.950 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T04:35:54.950 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T04:35:54.969 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T04:35:54.970 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T04:35:55.059 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T04:35:55.059 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T04:35:55.060 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T04:35:55.076 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T04:35:55.076 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T04:35:55.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T04:35:55.087 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T04:35:55.087 DEBUG:teuthology.orchestra.run.smithi046:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T04:35:55.103 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T04:35:55.103 DEBUG:teuthology.orchestra.run.smithi114:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T04:35:55.155 DEBUG:teuthology.orchestra.run.smithi046:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T04:35:55.169 DEBUG:teuthology.orchestra.run.smithi114:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T04:35:55.232 DEBUG:teuthology.orchestra.run.smithi046:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T04:35:55.243 DEBUG:teuthology.orchestra.run.smithi114:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T04:35:55.297 INFO:teuthology.orchestra.run.smithi046.stdout:check_obsoletes = 1 2024-09-16T04:35:55.299 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum clean all 2024-09-16T04:35:55.307 INFO:teuthology.orchestra.run.smithi114.stdout:check_obsoletes = 1 2024-09-16T04:35:55.309 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum clean all 2024-09-16T04:35:55.537 INFO:teuthology.orchestra.run.smithi114.stdout:45 files removed 2024-09-16T04:35:55.562 INFO:teuthology.orchestra.run.smithi046.stdout:45 files removed 2024-09-16T04:35:55.563 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-radosgw 2024-09-16T04:35:55.590 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-radosgw 2024-09-16T04:35:56.114 INFO:teuthology.orchestra.run.smithi114.stdout:ceph packages for x86_64 315 kB/s | 83 kB 00:00 2024-09-16T04:35:56.164 INFO:teuthology.orchestra.run.smithi046.stdout:ceph packages for x86_64 302 kB/s | 83 kB 00:00 2024-09-16T04:35:56.407 INFO:teuthology.orchestra.run.smithi114.stdout:ceph noarch packages 47 kB/s | 12 kB 00:00 2024-09-16T04:35:56.465 INFO:teuthology.orchestra.run.smithi046.stdout:ceph noarch packages 46 kB/s | 12 kB 00:00 2024-09-16T04:35:56.684 INFO:teuthology.orchestra.run.smithi114.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-16T04:35:56.741 INFO:teuthology.orchestra.run.smithi046.stdout:ceph source packages 8.0 kB/s | 1.9 kB 00:00 2024-09-16T04:35:57.599 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.2 MB 00:00 2024-09-16T04:35:58.366 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.2 MB 00:01 2024-09-16T04:36:01.458 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 7.5 MB/s | 20 MB 00:02 2024-09-16T04:36:02.559 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 6.6 MB/s | 20 MB 00:03 2024-09-16T04:36:05.959 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-16T04:36:07.552 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 19 kB 00:00 2024-09-16T04:36:08.301 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.5 MB 00:01 2024-09-16T04:36:08.658 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-16T04:36:09.894 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 75 kB/s | 19 kB 00:00 2024-09-16T04:36:10.956 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-16T04:36:14.314 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-16T04:36:16.453 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-09-16T04:36:16.861 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:36:16.885 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:36:16.885 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-09-16T04:36:16.885 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:36:16.885 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-09-16T04:36:16.885 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T04:36:16.885 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrading: 2024-09-16T04:36:16.885 INFO:teuthology.orchestra.run.smithi046.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T04:36:16.885 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T04:36:16.886 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-09-16T04:36:16.886 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T04:36:16.886 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T04:36:16.886 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T04:36:16.886 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T04:36:16.886 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T04:36:16.886 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T04:36:16.886 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T04:36:16.886 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T04:36:16.886 INFO:teuthology.orchestra.run.smithi046.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T04:36:16.886 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T04:36:16.886 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T04:36:16.886 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T04:36:16.887 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T04:36:16.887 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T04:36:16.887 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T04:36:16.887 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T04:36:16.887 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T04:36:16.887 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T04:36:16.887 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T04:36:16.887 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T04:36:16.887 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T04:36:16.887 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T04:36:16.887 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T04:36:16.887 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T04:36:16.887 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T04:36:16.888 INFO:teuthology.orchestra.run.smithi046.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T04:36:16.888 INFO:teuthology.orchestra.run.smithi046.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T04:36:16.888 INFO:teuthology.orchestra.run.smithi046.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T04:36:16.888 INFO:teuthology.orchestra.run.smithi046.stdout:Installing weak dependencies: 2024-09-16T04:36:16.888 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T04:36:16.888 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:36:16.888 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-09-16T04:36:16.888 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:36:16.888 INFO:teuthology.orchestra.run.smithi046.stdout:Install 30 Packages 2024-09-16T04:36:16.888 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrade 2 Packages 2024-09-16T04:36:16.888 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:36:16.889 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 60 M 2024-09-16T04:36:16.889 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-09-16T04:36:18.199 INFO:teuthology.orchestra.run.smithi046.stdout:(1/32): ceph-radosgw-19.1.1-326.g8921c8ef.el9.x 13 MB/s | 11 MB 00:00 2024-09-16T04:36:18.416 INFO:teuthology.orchestra.run.smithi046.stdout:(2/32): ceph-common-19.1.1-326.g8921c8ef.el9.x8 20 MB/s | 21 MB 00:01 2024-09-16T04:36:18.441 INFO:teuthology.orchestra.run.smithi046.stdout:(3/32): ceph-selinux-19.1.1-326.g8921c8ef.el9.x 104 kB/s | 25 kB 00:00 2024-09-16T04:36:18.474 INFO:teuthology.orchestra.run.smithi046.stdout:(4/32): libcephfs2-19.1.1-326.g8921c8ef.el9.x86 13 MB/s | 751 kB 00:00 2024-09-16T04:36:18.508 INFO:teuthology.orchestra.run.smithi046.stdout:(5/32): libradosstriper1-19.1.1-326.g8921c8ef.e 7.4 MB/s | 506 kB 00:00 2024-09-16T04:36:18.584 INFO:teuthology.orchestra.run.smithi046.stdout:(6/32): ceph-base-19.1.1-326.g8921c8ef.el9.x86_ 4.4 MB/s | 5.4 MB 00:01 2024-09-16T04:36:18.609 INFO:teuthology.orchestra.run.smithi046.stdout:(7/32): python3-ceph-argparse-19.1.1-326.g8921c 450 kB/s | 45 kB 00:00 2024-09-16T04:36:18.634 INFO:teuthology.orchestra.run.smithi046.stdout:(8/32): python3-ceph-common-19.1.1-326.g8921c8e 2.7 MB/s | 136 kB 00:00 2024-09-16T04:36:18.659 INFO:teuthology.orchestra.run.smithi046.stdout:(9/32): python3-cephfs-19.1.1-326.g8921c8ef.el9 3.1 MB/s | 161 kB 00:00 2024-09-16T04:36:18.685 INFO:teuthology.orchestra.run.smithi046.stdout:(10/32): python3-rados-19.1.1-326.g8921c8ef.el9 6.2 MB/s | 320 kB 00:00 2024-09-16T04:36:18.710 INFO:teuthology.orchestra.run.smithi046.stdout:(11/32): python3-rbd-19.1.1-326.g8921c8ef.el9.x 5.9 MB/s | 300 kB 00:00 2024-09-16T04:36:18.744 INFO:teuthology.orchestra.run.smithi046.stdout:(12/32): python3-rgw-19.1.1-326.g8921c8ef.el9.x 1.7 MB/s | 99 kB 00:00 2024-09-16T04:36:18.827 INFO:teuthology.orchestra.run.smithi046.stdout:(13/32): librgw2-19.1.1-326.g8921c8ef.el9.x86_6 15 MB/s | 5.3 MB 00:00 2024-09-16T04:36:18.829 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:36:18.838 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:36:18.838 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-16T04:36:18.838 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:36:18.838 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T04:36:18.838 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T04:36:18.839 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrading: 2024-09-16T04:36:18.839 INFO:teuthology.orchestra.run.smithi114.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T04:36:18.839 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T04:36:18.839 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-16T04:36:18.839 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T04:36:18.839 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T04:36:18.839 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T04:36:18.839 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T04:36:18.839 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T04:36:18.839 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T04:36:18.839 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T04:36:18.839 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T04:36:18.839 INFO:teuthology.orchestra.run.smithi114.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T04:36:18.840 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T04:36:18.840 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T04:36:18.840 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T04:36:18.840 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T04:36:18.840 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T04:36:18.840 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T04:36:18.840 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T04:36:18.840 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T04:36:18.840 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T04:36:18.840 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T04:36:18.840 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T04:36:18.840 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T04:36:18.840 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T04:36:18.841 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T04:36:18.841 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T04:36:18.841 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T04:36:18.841 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T04:36:18.841 INFO:teuthology.orchestra.run.smithi114.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T04:36:18.841 INFO:teuthology.orchestra.run.smithi114.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T04:36:18.841 INFO:teuthology.orchestra.run.smithi114.stdout:Installing weak dependencies: 2024-09-16T04:36:18.841 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T04:36:18.841 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:36:18.841 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T04:36:18.841 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:36:18.841 INFO:teuthology.orchestra.run.smithi114.stdout:Install 30 Packages 2024-09-16T04:36:18.841 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrade 2 Packages 2024-09-16T04:36:18.841 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:36:18.842 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 60 M 2024-09-16T04:36:18.842 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T04:36:18.869 INFO:teuthology.orchestra.run.smithi046.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 673 kB/s | 107 kB 00:00 2024-09-16T04:36:18.895 INFO:teuthology.orchestra.run.smithi046.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-16T04:36:18.936 INFO:teuthology.orchestra.run.smithi046.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 416 kB/s | 45 kB 00:00 2024-09-16T04:36:18.979 INFO:teuthology.orchestra.run.smithi046.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-16T04:36:19.012 INFO:teuthology.orchestra.run.smithi046.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 662 kB 00:00 2024-09-16T04:36:19.046 INFO:teuthology.orchestra.run.smithi046.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-09-16T04:36:19.071 INFO:teuthology.orchestra.run.smithi046.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.3 MB/s | 221 kB 00:00 2024-09-16T04:36:19.096 INFO:teuthology.orchestra.run.smithi046.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 717 kB/s | 36 kB 00:00 2024-09-16T04:36:19.180 INFO:teuthology.orchestra.run.smithi046.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 355 kB/s | 47 kB 00:00 2024-09-16T04:36:19.264 INFO:teuthology.orchestra.run.smithi046.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.6 MB/s | 308 kB 00:00 2024-09-16T04:36:19.297 INFO:teuthology.orchestra.run.smithi046.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 222 kB/s | 26 kB 00:00 2024-09-16T04:36:19.322 INFO:teuthology.orchestra.run.smithi046.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 831 kB/s | 49 kB 00:00 2024-09-16T04:36:19.356 INFO:teuthology.orchestra.run.smithi046.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-09-16T04:36:19.390 INFO:teuthology.orchestra.run.smithi046.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-16T04:36:19.423 INFO:teuthology.orchestra.run.smithi046.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.1 MB/s | 839 kB 00:00 2024-09-16T04:36:19.490 INFO:teuthology.orchestra.run.smithi046.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-16T04:36:19.566 INFO:teuthology.orchestra.run.smithi046.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-16T04:36:19.630 INFO:teuthology.orchestra.run.smithi114.stdout:(1/32): ceph-base-19.1.1-326.g8921c8ef.el9.x86_ 12 MB/s | 5.4 MB 00:00 2024-09-16T04:36:19.722 INFO:teuthology.orchestra.run.smithi114.stdout:(2/32): ceph-selinux-19.1.1-326.g8921c8ef.el9.x 275 kB/s | 25 kB 00:00 2024-09-16T04:36:19.783 INFO:teuthology.orchestra.run.smithi046.stdout:(31/32): librbd1-19.1.1-326.g8921c8ef.el9.x86_6 11 MB/s | 3.2 MB 00:00 2024-09-16T04:36:19.814 INFO:teuthology.orchestra.run.smithi114.stdout:(3/32): libcephfs2-19.1.1-326.g8921c8ef.el9.x86 8.0 MB/s | 751 kB 00:00 2024-09-16T04:36:19.906 INFO:teuthology.orchestra.run.smithi114.stdout:(4/32): libradosstriper1-19.1.1-326.g8921c8ef.e 5.4 MB/s | 506 kB 00:00 2024-09-16T04:36:20.170 INFO:teuthology.orchestra.run.smithi046.stdout:(32/32): librados2-19.1.1-326.g8921c8ef.el9.x86 4.6 MB/s | 3.4 MB 00:00 2024-09-16T04:36:20.175 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:36:20.176 INFO:teuthology.orchestra.run.smithi046.stdout:Total 18 MB/s | 60 MB 00:03 2024-09-16T04:36:20.206 INFO:teuthology.orchestra.run.smithi114.stdout:(5/32): ceph-radosgw-19.1.1-326.g8921c8ef.el9.x 10 MB/s | 11 MB 00:01 2024-09-16T04:36:20.288 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-09-16T04:36:20.306 INFO:teuthology.orchestra.run.smithi114.stdout:(6/32): librgw2-19.1.1-326.g8921c8ef.el9.x86_64 13 MB/s | 5.3 MB 00:00 2024-09-16T04:36:20.323 INFO:teuthology.orchestra.run.smithi114.stdout:(7/32): python3-ceph-argparse-19.1.1-326.g8921c 386 kB/s | 45 kB 00:00 2024-09-16T04:36:20.329 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-09-16T04:36:20.329 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-09-16T04:36:20.348 INFO:teuthology.orchestra.run.smithi114.stdout:(8/32): python3-ceph-common-19.1.1-326.g8921c8e 3.2 MB/s | 136 kB 00:00 2024-09-16T04:36:20.373 INFO:teuthology.orchestra.run.smithi114.stdout:(9/32): python3-cephfs-19.1.1-326.g8921c8ef.el9 3.2 MB/s | 161 kB 00:00 2024-09-16T04:36:20.407 INFO:teuthology.orchestra.run.smithi114.stdout:(10/32): python3-rados-19.1.1-326.g8921c8ef.el9 5.3 MB/s | 320 kB 00:00 2024-09-16T04:36:20.436 INFO:teuthology.orchestra.run.smithi114.stdout:(11/32): python3-rbd-19.1.1-326.g8921c8ef.el9.x 5.0 MB/s | 300 kB 00:00 2024-09-16T04:36:20.457 INFO:teuthology.orchestra.run.smithi114.stdout:(12/32): python3-rgw-19.1.1-326.g8921c8ef.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-16T04:36:20.675 INFO:teuthology.orchestra.run.smithi114.stdout:(13/32): ceph-common-19.1.1-326.g8921c8ef.el9.x 14 MB/s | 21 MB 00:01 2024-09-16T04:36:20.708 INFO:teuthology.orchestra.run.smithi114.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 388 kB/s | 107 kB 00:00 2024-09-16T04:36:20.775 INFO:teuthology.orchestra.run.smithi114.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 505 kB/s | 160 kB 00:00 2024-09-16T04:36:20.800 INFO:teuthology.orchestra.run.smithi114.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 362 kB/s | 45 kB 00:00 2024-09-16T04:36:20.826 INFO:teuthology.orchestra.run.smithi114.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-09-16T04:36:20.851 INFO:teuthology.orchestra.run.smithi114.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 838 kB/s | 42 kB 00:00 2024-09-16T04:36:20.876 INFO:teuthology.orchestra.run.smithi114.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-16T04:36:20.901 INFO:teuthology.orchestra.run.smithi114.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 716 kB/s | 36 kB 00:00 2024-09-16T04:36:20.927 INFO:teuthology.orchestra.run.smithi114.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 945 kB/s | 47 kB 00:00 2024-09-16T04:36:20.952 INFO:teuthology.orchestra.run.smithi114.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-09-16T04:36:21.019 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-09-16T04:36:21.019 INFO:teuthology.orchestra.run.smithi114.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 499 kB/s | 26 kB 00:00 2024-09-16T04:36:21.019 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-09-16T04:36:21.044 INFO:teuthology.orchestra.run.smithi114.stdout:(24/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.1 MB/s | 308 kB 00:00 2024-09-16T04:36:21.070 INFO:teuthology.orchestra.run.smithi114.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2024-09-16T04:36:21.086 INFO:teuthology.orchestra.run.smithi114.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-16T04:36:21.129 INFO:teuthology.orchestra.run.smithi114.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-16T04:36:21.203 INFO:teuthology.orchestra.run.smithi114.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.1 MB/s | 839 kB 00:00 2024-09-16T04:36:21.287 INFO:teuthology.orchestra.run.smithi114.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-09-16T04:36:21.404 INFO:teuthology.orchestra.run.smithi114.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 9.3 MB/s | 4.4 MB 00:00 2024-09-16T04:36:21.479 INFO:teuthology.orchestra.run.smithi114.stdout:(31/32): librados2-19.1.1-326.g8921c8ef.el9.x86 12 MB/s | 3.4 MB 00:00 2024-09-16T04:36:21.571 INFO:teuthology.orchestra.run.smithi114.stdout:(32/32): librbd1-19.1.1-326.g8921c8ef.el9.x86_6 11 MB/s | 3.2 MB 00:00 2024-09-16T04:36:21.576 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:36:21.576 INFO:teuthology.orchestra.run.smithi114.stdout:Total 22 MB/s | 60 MB 00:02 2024-09-16T04:36:21.677 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T04:36:21.712 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T04:36:21.712 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T04:36:22.394 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T04:36:22.394 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T04:36:22.561 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-09-16T04:36:22.659 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-16T04:36:22.696 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-16T04:36:22.729 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-16T04:36:22.752 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T04:36:22.776 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T04:36:22.922 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T04:36:22.984 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T04:36:23.401 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-16T04:36:23.440 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:36:23.532 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:36:23.640 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 8/34 2024-09-16T04:36:23.691 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T04:36:23.763 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T04:36:23.800 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-16T04:36:23.850 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 11/34 2024-09-16T04:36:23.862 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T04:36:23.923 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T04:36:23.938 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-16T04:36:23.952 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:36:23.968 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-16T04:36:23.993 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-16T04:36:24.016 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:36:24.016 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T04:36:24.043 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T04:36:24.051 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-16T04:36:24.171 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-16T04:36:24.201 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-16T04:36:24.242 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-16T04:36:24.261 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T04:36:24.284 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-16T04:36:24.313 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-16T04:36:24.321 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T04:36:24.346 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-16T04:36:24.652 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-16T04:36:24.674 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-16T04:36:24.716 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T04:36:24.762 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:36:24.808 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T04:36:24.838 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 23/34 2024-09-16T04:36:24.854 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:36:24.883 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-16T04:36:24.940 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 8/34 2024-09-16T04:36:24.972 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T04:36:25.026 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T04:36:25.055 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-16T04:36:25.098 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 11/34 2024-09-16T04:36:25.167 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T04:36:25.191 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:36:25.255 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:36:25.289 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-16T04:36:25.392 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-16T04:36:25.406 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-16T04:36:25.419 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T04:36:25.448 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-16T04:36:25.473 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T04:36:25.497 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-16T04:36:25.539 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-16T04:36:25.554 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 27/34 2024-09-16T04:36:25.568 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-16T04:36:25.588 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 28/34 2024-09-16T04:36:25.610 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-16T04:36:25.918 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-16T04:36:25.968 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T04:36:26.056 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T04:36:26.097 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 23/34 2024-09-16T04:36:26.139 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-16T04:36:26.617 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-16T04:36:26.643 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T04:36:26.697 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T04:36:26.774 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 27/34 2024-09-16T04:36:26.812 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 28/34 2024-09-16T04:36:27.989 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:36:28.018 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:36:28.642 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:36:28.680 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T04:36:28.777 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T04:36:28.778 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T04:36:28.778 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T04:36:28.778 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:36:28.806 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:36:29.028 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:36:29.067 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:36:29.646 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:36:29.687 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T04:36:29.782 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T04:36:29.782 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T04:36:29.782 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T04:36:29.782 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:36:29.830 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:36:39.179 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:36:39.179 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /sys 2024-09-16T04:36:39.179 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /proc 2024-09-16T04:36:39.179 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /mnt 2024-09-16T04:36:39.179 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /var/tmp 2024-09-16T04:36:39.179 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /home 2024-09-16T04:36:39.179 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /root 2024-09-16T04:36:39.179 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /tmp 2024-09-16T04:36:39.179 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:36:39.893 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:36:39.893 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /sys 2024-09-16T04:36:39.894 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /proc 2024-09-16T04:36:39.894 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /mnt 2024-09-16T04:36:39.894 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /var/tmp 2024-09-16T04:36:39.894 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /home 2024-09-16T04:36:39.894 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /root 2024-09-16T04:36:39.894 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /tmp 2024-09-16T04:36:39.894 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:36:40.174 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T04:36:40.214 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T04:36:40.214 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:36:40.214 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T04:36:40.214 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:36:40.214 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:36:40.215 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:36:40.216 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T04:36:40.300 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T04:36:40.301 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:36:40.914 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T04:36:40.939 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T04:36:40.939 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:36:40.939 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T04:36:40.939 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:36:40.939 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:36:40.940 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:36:40.941 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T04:36:41.033 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T04:36:41.033 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:36:41.623 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:36:41.624 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 1/34 2024-09-16T04:36:41.624 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2/34 2024-09-16T04:36:41.624 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 3/34 2024-09-16T04:36:41.624 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 4/34 2024-09-16T04:36:41.624 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 5/34 2024-09-16T04:36:41.624 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 6/34 2024-09-16T04:36:41.624 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:36:41.624 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 8/34 2024-09-16T04:36:41.624 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 9/34 2024-09-16T04:36:41.624 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 10/34 2024-09-16T04:36:41.624 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 11/34 2024-09-16T04:36:41.625 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T04:36:41.625 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:36:41.625 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-16T04:36:41.625 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-16T04:36:41.625 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-16T04:36:41.625 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-16T04:36:41.625 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-16T04:36:41.625 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-16T04:36:41.625 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-16T04:36:41.625 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-16T04:36:41.625 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-16T04:36:41.625 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-16T04:36:41.625 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-16T04:36:41.625 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-16T04:36:41.626 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-16T04:36:41.626 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-16T04:36:41.626 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-16T04:36:41.626 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-16T04:36:41.626 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-16T04:36:41.626 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:36:41.626 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-16T04:36:41.626 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 33/34 2024-09-16T04:36:42.236 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:36:42.236 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 1/34 2024-09-16T04:36:42.236 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2/34 2024-09-16T04:36:42.236 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 3/34 2024-09-16T04:36:42.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 4/34 2024-09-16T04:36:42.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 5/34 2024-09-16T04:36:42.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 6/34 2024-09-16T04:36:42.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:36:42.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 8/34 2024-09-16T04:36:42.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 9/34 2024-09-16T04:36:42.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 10/34 2024-09-16T04:36:42.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 11/34 2024-09-16T04:36:42.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T04:36:42.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:36:42.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-16T04:36:42.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-16T04:36:42.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-16T04:36:42.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-16T04:36:42.238 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-16T04:36:42.238 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-16T04:36:42.238 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-16T04:36:42.238 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-16T04:36:42.238 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-16T04:36:42.238 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-16T04:36:42.238 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-16T04:36:42.238 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-16T04:36:42.238 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-16T04:36:42.238 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-16T04:36:42.238 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-16T04:36:42.238 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-16T04:36:42.238 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-16T04:36:42.239 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:36:42.239 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-16T04:36:42.239 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 33/34 2024-09-16T04:36:42.691 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:36:42.691 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:36:42.691 INFO:teuthology.orchestra.run.smithi046.stdout:Upgraded: 2024-09-16T04:36:42.691 INFO:teuthology.orchestra.run.smithi046.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:42.691 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:42.691 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-09-16T04:36:42.691 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T04:36:42.691 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:42.691 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:42.691 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:42.692 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:42.692 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T04:36:42.692 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T04:36:42.692 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T04:36:42.692 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:42.692 INFO:teuthology.orchestra.run.smithi046.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T04:36:42.692 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T04:36:42.692 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T04:36:42.692 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:42.692 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T04:36:42.692 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:42.692 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T04:36:42.693 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T04:36:42.693 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T04:36:42.693 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:42.693 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:42.693 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:42.693 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T04:36:42.693 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:42.693 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:42.693 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:42.693 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T04:36:42.693 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T04:36:42.693 INFO:teuthology.orchestra.run.smithi046.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T04:36:42.693 INFO:teuthology.orchestra.run.smithi046.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T04:36:42.693 INFO:teuthology.orchestra.run.smithi046.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T04:36:42.694 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:36:42.694 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:36:43.035 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-test 2024-09-16T04:36:43.105 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:36:43.105 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:36:43.105 INFO:teuthology.orchestra.run.smithi114.stdout:Upgraded: 2024-09-16T04:36:43.105 INFO:teuthology.orchestra.run.smithi114.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:43.105 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:43.105 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T04:36:43.106 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T04:36:43.106 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:43.106 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:43.106 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:43.106 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:43.106 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T04:36:43.106 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T04:36:43.106 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T04:36:43.106 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:43.106 INFO:teuthology.orchestra.run.smithi114.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T04:36:43.106 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T04:36:43.106 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T04:36:43.106 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:43.106 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T04:36:43.107 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:43.107 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T04:36:43.107 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T04:36:43.107 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T04:36:43.107 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:43.107 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:43.107 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:43.107 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T04:36:43.107 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:43.107 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:43.107 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:43.107 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T04:36:43.107 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T04:36:43.107 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T04:36:43.108 INFO:teuthology.orchestra.run.smithi114.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T04:36:43.108 INFO:teuthology.orchestra.run.smithi114.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T04:36:43.108 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:36:43.108 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:36:43.433 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-test 2024-09-16T04:36:43.623 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:29 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:36:43.765 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:36:43.765 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:36:43.765 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-09-16T04:36:43.765 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:36:43.765 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-09-16T04:36:43.766 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T04:36:43.766 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-09-16T04:36:43.766 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T04:36:43.766 INFO:teuthology.orchestra.run.smithi046.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T04:36:43.766 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T04:36:43.766 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:36:43.766 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-09-16T04:36:43.766 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:36:43.766 INFO:teuthology.orchestra.run.smithi046.stdout:Install 4 Packages 2024-09-16T04:36:43.766 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:36:43.767 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 49 M 2024-09-16T04:36:43.767 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 209 M 2024-09-16T04:36:43.767 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-09-16T04:36:44.001 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:27 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:36:44.133 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:36:44.134 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:36:44.134 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-16T04:36:44.134 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:36:44.135 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T04:36:44.135 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T04:36:44.135 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-16T04:36:44.135 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T04:36:44.135 INFO:teuthology.orchestra.run.smithi114.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T04:36:44.135 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T04:36:44.136 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:36:44.136 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T04:36:44.136 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:36:44.136 INFO:teuthology.orchestra.run.smithi114.stdout:Install 4 Packages 2024-09-16T04:36:44.136 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:36:44.137 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 49 M 2024-09-16T04:36:44.138 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 209 M 2024-09-16T04:36:44.138 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T04:36:44.242 INFO:teuthology.orchestra.run.smithi046.stdout:(1/4): libcephsqlite-19.1.1-326.g8921c8ef.el9.x 748 kB/s | 163 kB 00:00 2024-09-16T04:36:44.284 INFO:teuthology.orchestra.run.smithi046.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.1 MB/s | 304 kB 00:00 2024-09-16T04:36:44.310 INFO:teuthology.orchestra.run.smithi046.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 952 kB/s | 64 kB 00:00 2024-09-16T04:36:44.571 INFO:teuthology.orchestra.run.smithi114.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.9 MB/s | 304 kB 00:00 2024-09-16T04:36:44.613 INFO:teuthology.orchestra.run.smithi114.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-16T04:36:44.855 INFO:teuthology.orchestra.run.smithi114.stdout:(3/4): libcephsqlite-19.1.1-326.g8921c8ef.el9.x 372 kB/s | 163 kB 00:00 2024-09-16T04:36:45.643 INFO:teuthology.orchestra.run.smithi046.stdout:(4/4): ceph-test-19.1.1-326.g8921c8ef.el9.x86_6 30 MB/s | 49 MB 00:01 2024-09-16T04:36:45.646 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:36:45.646 INFO:teuthology.orchestra.run.smithi046.stdout:Total 26 MB/s | 49 MB 00:01 2024-09-16T04:36:45.674 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-09-16T04:36:45.696 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-09-16T04:36:45.696 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-09-16T04:36:46.103 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-09-16T04:36:46.104 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-09-16T04:36:46.654 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-09-16T04:36:46.663 INFO:teuthology.orchestra.run.smithi114.stdout:(4/4): ceph-test-19.1.1-326.g8921c8ef.el9.x86_6 22 MB/s | 49 MB 00:02 2024-09-16T04:36:46.666 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:36:46.666 INFO:teuthology.orchestra.run.smithi114.stdout:Total 19 MB/s | 49 MB 00:02 2024-09-16T04:36:46.694 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T04:36:46.714 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T04:36:46.714 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T04:36:46.730 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T04:36:46.784 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T04:36:46.820 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T04:36:47.109 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T04:36:47.109 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T04:36:47.627 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T04:36:47.705 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T04:36:47.746 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T04:36:47.782 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T04:36:51.673 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T04:36:51.755 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T04:36:52.373 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T04:36:52.447 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T04:36:54.097 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T04:36:54.098 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 1/4 2024-09-16T04:36:54.098 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2/4 2024-09-16T04:36:54.098 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T04:36:54.417 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T04:36:54.418 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:36:54.418 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-09-16T04:36:54.418 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:54.418 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:54.418 INFO:teuthology.orchestra.run.smithi046.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T04:36:54.418 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T04:36:54.418 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:36:54.418 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:36:54.705 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph 2024-09-16T04:36:54.837 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T04:36:54.837 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 1/4 2024-09-16T04:36:54.837 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2/4 2024-09-16T04:36:54.837 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T04:36:55.169 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T04:36:55.169 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:36:55.169 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T04:36:55.169 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:55.169 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:36:55.169 INFO:teuthology.orchestra.run.smithi114.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T04:36:55.169 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T04:36:55.169 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:36:55.169 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:36:55.293 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:41 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:36:55.437 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:36:55.439 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:36:55.440 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-09-16T04:36:55.440 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:36:55.440 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-09-16T04:36:55.440 INFO:teuthology.orchestra.run.smithi046.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T04:36:55.440 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-09-16T04:36:55.441 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T04:36:55.441 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T04:36:55.441 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T04:36:55.441 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T04:36:55.441 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T04:36:55.441 INFO:teuthology.orchestra.run.smithi046.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T04:36:55.442 INFO:teuthology.orchestra.run.smithi046.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T04:36:55.442 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T04:36:55.442 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T04:36:55.442 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T04:36:55.442 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T04:36:55.442 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T04:36:55.442 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T04:36:55.442 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T04:36:55.442 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T04:36:55.442 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T04:36:55.442 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T04:36:55.443 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T04:36:55.443 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T04:36:55.443 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T04:36:55.443 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T04:36:55.443 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T04:36:55.443 INFO:teuthology.orchestra.run.smithi046.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T04:36:55.443 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T04:36:55.443 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T04:36:55.443 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T04:36:55.443 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T04:36:55.443 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T04:36:55.443 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T04:36:55.443 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T04:36:55.444 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T04:36:55.444 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T04:36:55.444 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T04:36:55.444 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T04:36:55.444 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T04:36:55.444 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T04:36:55.444 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:36:55.444 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-09-16T04:36:55.444 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:36:55.445 INFO:teuthology.orchestra.run.smithi046.stdout:Install 37 Packages 2024-09-16T04:36:55.445 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:36:55.445 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 30 M 2024-09-16T04:36:55.445 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 108 M 2024-09-16T04:36:55.446 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-09-16T04:36:55.480 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph 2024-09-16T04:36:56.049 INFO:teuthology.orchestra.run.smithi046.stdout:(1/37): ceph-19.1.1-326.g8921c8ef.el9.x86_64.rp 53 kB/s | 6.5 kB 00:00 2024-09-16T04:36:56.058 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:40 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:36:56.201 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:36:56.204 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:36:56.204 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-16T04:36:56.204 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:36:56.204 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T04:36:56.204 INFO:teuthology.orchestra.run.smithi114.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T04:36:56.204 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-16T04:36:56.204 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T04:36:56.204 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T04:36:56.204 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T04:36:56.204 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T04:36:56.204 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T04:36:56.205 INFO:teuthology.orchestra.run.smithi114.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T04:36:56.205 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T04:36:56.205 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T04:36:56.205 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T04:36:56.205 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T04:36:56.205 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T04:36:56.205 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T04:36:56.205 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T04:36:56.205 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T04:36:56.205 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T04:36:56.205 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T04:36:56.205 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T04:36:56.205 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T04:36:56.205 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T04:36:56.206 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T04:36:56.206 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T04:36:56.206 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T04:36:56.206 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T04:36:56.206 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T04:36:56.206 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T04:36:56.206 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T04:36:56.206 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T04:36:56.206 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T04:36:56.206 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T04:36:56.206 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T04:36:56.206 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T04:36:56.206 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T04:36:56.207 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T04:36:56.207 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T04:36:56.207 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T04:36:56.207 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T04:36:56.207 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:36:56.207 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T04:36:56.207 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:36:56.207 INFO:teuthology.orchestra.run.smithi114.stdout:Install 37 Packages 2024-09-16T04:36:56.207 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:36:56.208 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 30 M 2024-09-16T04:36:56.208 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 108 M 2024-09-16T04:36:56.208 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T04:36:56.232 INFO:teuthology.orchestra.run.smithi046.stdout:(2/37): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_6 4.9 MB/s | 1.5 MB 00:00 2024-09-16T04:36:56.283 INFO:teuthology.orchestra.run.smithi046.stdout:(3/37): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_6 6.7 MB/s | 2.4 MB 00:00 2024-09-16T04:36:56.383 INFO:teuthology.orchestra.run.smithi046.stdout:(4/37): ceph-mgr-modules-core-19.1.1-326.g8921c 2.4 MB/s | 249 kB 00:00 2024-09-16T04:36:56.450 INFO:teuthology.orchestra.run.smithi046.stdout:(5/37): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_6 12 MB/s | 4.7 MB 00:00 2024-09-16T04:36:56.617 INFO:teuthology.orchestra.run.smithi046.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-16T04:36:56.706 INFO:teuthology.orchestra.run.smithi114.stdout:(1/37): ceph-19.1.1-326.g8921c8ef.el9.x86_64.rp 39 kB/s | 6.5 kB 00:00 2024-09-16T04:36:56.817 INFO:teuthology.orchestra.run.smithi046.stdout:(7/37): ceph-osd-19.1.1-326.g8921c8ef.el9.x86_6 29 MB/s | 17 MB 00:00 2024-09-16T04:36:56.843 INFO:teuthology.orchestra.run.smithi046.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 473 kB/s | 106 kB 00:00 2024-09-16T04:36:56.865 INFO:teuthology.orchestra.run.smithi114.stdout:(2/37): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_6 4.6 MB/s | 1.5 MB 00:00 2024-09-16T04:36:56.893 INFO:teuthology.orchestra.run.smithi046.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-16T04:36:56.915 INFO:teuthology.orchestra.run.smithi114.stdout:(3/37): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_6 6.3 MB/s | 2.4 MB 00:00 2024-09-16T04:36:56.935 INFO:teuthology.orchestra.run.smithi046.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.5 MB/s | 1.2 MB 00:00 2024-09-16T04:36:56.960 INFO:teuthology.orchestra.run.smithi046.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-16T04:36:57.011 INFO:teuthology.orchestra.run.smithi046.stdout:(12/37): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-09-16T04:36:57.024 INFO:teuthology.orchestra.run.smithi114.stdout:(4/37): ceph-mgr-modules-core-19.1.1-326.g8921c 2.3 MB/s | 249 kB 00:00 2024-09-16T04:36:57.036 INFO:teuthology.orchestra.run.smithi046.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-09-16T04:36:57.061 INFO:teuthology.orchestra.run.smithi046.stdout:(14/37): python3-pycparser-2.20-6.el9.noarch.rp 556 kB/s | 135 kB 00:00 2024-09-16T04:36:57.086 INFO:teuthology.orchestra.run.smithi046.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 461 kB/s | 35 kB 00:00 2024-09-16T04:36:57.116 INFO:teuthology.orchestra.run.smithi114.stdout:(5/37): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_6 11 MB/s | 4.7 MB 00:00 2024-09-16T04:36:57.120 INFO:teuthology.orchestra.run.smithi046.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 916 kB/s | 77 kB 00:00 2024-09-16T04:36:57.145 INFO:teuthology.orchestra.run.smithi046.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 707 kB/s | 42 kB 00:00 2024-09-16T04:36:57.171 INFO:teuthology.orchestra.run.smithi046.stdout:(18/37): python3-pytz-2021.1-5.el9.noarch.rpm 467 kB/s | 51 kB 00:00 2024-09-16T04:36:57.196 INFO:teuthology.orchestra.run.smithi046.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 296 kB/s | 22 kB 00:00 2024-09-16T04:36:57.254 INFO:teuthology.orchestra.run.smithi046.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 521 kB/s | 43 kB 00:00 2024-09-16T04:36:57.280 INFO:teuthology.orchestra.run.smithi046.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-09-16T04:36:57.291 INFO:teuthology.orchestra.run.smithi114.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 948 kB/s | 253 kB 00:00 2024-09-16T04:36:57.297 INFO:teuthology.orchestra.run.smithi046.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 648 kB/s | 11 kB 00:00 2024-09-16T04:36:57.322 INFO:teuthology.orchestra.run.smithi046.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-09-16T04:36:57.339 INFO:teuthology.orchestra.run.smithi046.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 422 kB/s | 18 kB 00:00 2024-09-16T04:36:57.356 INFO:teuthology.orchestra.run.smithi046.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 690 kB/s | 23 kB 00:00 2024-09-16T04:36:57.373 INFO:teuthology.orchestra.run.smithi046.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 579 kB/s | 19 kB 00:00 2024-09-16T04:36:57.390 INFO:teuthology.orchestra.run.smithi046.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 591 kB/s | 20 kB 00:00 2024-09-16T04:36:57.408 INFO:teuthology.orchestra.run.smithi046.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.3 MB/s | 46 kB 00:00 2024-09-16T04:36:57.440 INFO:teuthology.orchestra.run.smithi046.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.9 MB/s | 358 kB 00:00 2024-09-16T04:36:57.450 INFO:teuthology.orchestra.run.smithi114.stdout:(7/37): ceph-osd-19.1.1-326.g8921c8ef.el9.x86_6 29 MB/s | 17 MB 00:00 2024-09-16T04:36:57.475 INFO:teuthology.orchestra.run.smithi114.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 581 kB/s | 106 kB 00:00 2024-09-16T04:36:57.491 INFO:teuthology.orchestra.run.smithi046.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 784 kB/s | 79 kB 00:00 2024-09-16T04:36:57.508 INFO:teuthology.orchestra.run.smithi046.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 245 kB/s | 16 kB 00:00 2024-09-16T04:36:57.525 INFO:teuthology.orchestra.run.smithi114.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-16T04:36:57.533 INFO:teuthology.orchestra.run.smithi046.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-16T04:36:57.550 INFO:teuthology.orchestra.run.smithi046.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 861 kB/s | 36 kB 00:00 2024-09-16T04:36:57.575 INFO:teuthology.orchestra.run.smithi046.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-09-16T04:36:57.584 INFO:teuthology.orchestra.run.smithi114.stdout:(10/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-16T04:36:57.600 INFO:teuthology.orchestra.run.smithi046.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-09-16T04:36:57.617 INFO:teuthology.orchestra.run.smithi046.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 479 kB/s | 20 kB 00:00 2024-09-16T04:36:57.626 INFO:teuthology.orchestra.run.smithi114.stdout:(11/37): python3-cryptography-36.0.1-4.el9.x86_ 2.4 MB/s | 1.2 MB 00:00 2024-09-16T04:36:57.651 INFO:teuthology.orchestra.run.smithi114.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 673 kB/s | 135 kB 00:00 2024-09-16T04:36:57.659 INFO:teuthology.orchestra.run.smithi046.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-16T04:36:57.664 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:36:57.664 INFO:teuthology.orchestra.run.smithi046.stdout:Total 14 MB/s | 30 MB 00:02 2024-09-16T04:36:57.718 INFO:teuthology.orchestra.run.smithi114.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 521 kB/s | 35 kB 00:00 2024-09-16T04:36:57.743 INFO:teuthology.orchestra.run.smithi114.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-16T04:36:57.769 INFO:teuthology.orchestra.run.smithi114.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 188 kB 00:00 2024-09-16T04:36:57.794 INFO:teuthology.orchestra.run.smithi114.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-16T04:36:57.819 INFO:teuthology.orchestra.run.smithi114.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 676 kB/s | 51 kB 00:00 2024-09-16T04:36:57.820 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-09-16T04:36:57.844 INFO:teuthology.orchestra.run.smithi114.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 553 kB/s | 42 kB 00:00 2024-09-16T04:36:57.865 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-09-16T04:36:57.865 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-09-16T04:36:57.886 INFO:teuthology.orchestra.run.smithi114.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-16T04:36:57.920 INFO:teuthology.orchestra.run.smithi114.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 5.3 MB/s | 172 kB 00:00 2024-09-16T04:36:57.945 INFO:teuthology.orchestra.run.smithi114.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-16T04:36:57.970 INFO:teuthology.orchestra.run.smithi114.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 429 kB/s | 11 kB 00:00 2024-09-16T04:36:57.996 INFO:teuthology.orchestra.run.smithi114.stdout:(23/37): python3-cherrypy-18.6.1-2.el9.noarch.r 4.6 MB/s | 358 kB 00:00 2024-09-16T04:36:58.021 INFO:teuthology.orchestra.run.smithi114.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-09-16T04:36:58.046 INFO:teuthology.orchestra.run.smithi114.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-09-16T04:36:58.071 INFO:teuthology.orchestra.run.smithi114.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 389 kB/s | 19 kB 00:00 2024-09-16T04:36:58.097 INFO:teuthology.orchestra.run.smithi114.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-16T04:36:58.127 INFO:teuthology.orchestra.run.smithi114.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 861 kB/s | 46 kB 00:00 2024-09-16T04:36:58.151 INFO:teuthology.orchestra.run.smithi114.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.6 MB/s | 79 kB 00:00 2024-09-16T04:36:58.176 INFO:teuthology.orchestra.run.smithi114.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 657 kB/s | 16 kB 00:00 2024-09-16T04:36:58.201 INFO:teuthology.orchestra.run.smithi114.stdout:(31/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.6 MB/s | 90 kB 00:00 2024-09-16T04:36:58.227 INFO:teuthology.orchestra.run.smithi114.stdout:(32/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-09-16T04:36:58.244 INFO:teuthology.orchestra.run.smithi114.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 851 kB/s | 36 kB 00:00 2024-09-16T04:36:58.277 INFO:teuthology.orchestra.run.smithi114.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-16T04:36:58.294 INFO:teuthology.orchestra.run.smithi114.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-16T04:36:58.319 INFO:teuthology.orchestra.run.smithi114.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-09-16T04:36:58.371 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-09-16T04:36:58.372 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-09-16T04:36:58.444 INFO:teuthology.orchestra.run.smithi114.stdout:(37/37): lua-devel-5.4.4-4.el9.x86_64.rpm 36 kB/s | 22 kB 00:00 2024-09-16T04:36:58.448 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:36:58.448 INFO:teuthology.orchestra.run.smithi114.stdout:Total 14 MB/s | 30 MB 00:02 2024-09-16T04:36:58.596 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T04:36:58.637 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T04:36:58.637 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T04:36:59.141 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T04:36:59.141 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T04:36:59.783 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-09-16T04:36:59.854 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T04:36:59.889 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T04:36:59.915 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T04:37:00.000 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T04:37:00.040 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T04:37:00.072 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T04:37:00.169 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T04:37:00.246 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T04:37:00.301 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T04:37:00.345 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T04:37:00.374 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T04:37:00.417 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T04:37:00.451 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T04:37:00.492 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T04:37:00.506 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T04:37:00.535 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T04:37:00.564 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T04:37:00.569 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T04:37:00.607 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T04:37:00.619 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T04:37:00.641 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T04:37:00.659 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T04:37:00.711 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T04:37:00.735 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T04:37:00.749 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T04:37:00.782 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T04:37:00.873 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T04:37:00.896 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T04:37:00.951 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T04:37:00.986 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T04:37:01.000 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T04:37:01.037 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T04:37:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T04:37:01.074 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T04:37:01.074 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T04:37:01.108 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T04:37:01.141 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T04:37:01.142 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T04:37:01.182 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T04:37:01.206 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T04:37:01.217 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T04:37:01.252 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T04:37:01.299 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T04:37:01.341 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T04:37:01.404 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T04:37:01.422 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T04:37:01.454 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T04:37:01.535 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T04:37:01.570 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T04:37:01.634 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T04:37:01.648 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T04:37:01.695 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T04:37:01.701 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T04:37:01.749 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T04:37:01.806 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T04:37:01.827 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T04:37:01.861 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T04:37:01.998 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 32/37 2024-09-16T04:37:02.029 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T04:37:02.045 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T04:37:02.056 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T04:37:02.056 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:37:02.056 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T04:37:02.056 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T04:37:02.056 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T04:37:02.056 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:02.103 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T04:37:02.189 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T04:37:02.302 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T04:37:02.355 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T04:37:02.478 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T04:37:02.651 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 32/37 2024-09-16T04:37:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T04:37:02.706 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T04:37:02.706 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:37:02.706 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T04:37:02.706 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T04:37:02.706 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T04:37:02.706 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:03.571 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T04:37:03.599 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T04:37:03.599 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:37:03.599 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T04:37:03.600 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T04:37:03.600 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T04:37:03.600 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:04.064 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T04:37:04.089 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T04:37:04.089 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:37:04.089 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T04:37:04.089 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T04:37:04.089 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T04:37:04.089 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:04.138 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T04:37:04.166 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T04:37:04.166 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:37:04.166 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T04:37:04.166 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T04:37:04.167 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T04:37:04.167 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:04.331 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T04:37:04.356 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T04:37:04.356 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:37:04.356 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T04:37:04.356 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T04:37:04.356 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T04:37:04.356 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:04.422 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T04:37:04.614 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T04:37:04.638 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T04:37:04.638 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:37:04.638 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T04:37:04.639 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T04:37:04.639 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T04:37:04.639 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:04.865 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T04:37:04.889 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T04:37:04.889 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:37:04.889 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T04:37:04.889 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T04:37:04.889 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T04:37:04.889 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:04.953 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T04:37:06.289 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T04:37:06.289 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/37 2024-09-16T04:37:06.289 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2/37 2024-09-16T04:37:06.289 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 3/37 2024-09-16T04:37:06.289 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 4/37 2024-09-16T04:37:06.290 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 5/37 2024-09-16T04:37:06.290 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 6/37 2024-09-16T04:37:06.290 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T04:37:06.290 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T04:37:06.290 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T04:37:06.290 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T04:37:06.290 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T04:37:06.290 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T04:37:06.290 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T04:37:06.290 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T04:37:06.290 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T04:37:06.290 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T04:37:06.290 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T04:37:06.291 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T04:37:06.291 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T04:37:06.291 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T04:37:06.291 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T04:37:06.291 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T04:37:06.291 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T04:37:06.291 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T04:37:06.291 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T04:37:06.292 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T04:37:06.293 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T04:37:06.293 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T04:37:06.293 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T04:37:06.293 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T04:37:06.293 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T04:37:06.293 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T04:37:06.293 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T04:37:06.293 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T04:37:06.293 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T04:37:06.293 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T04:37:06.832 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T04:37:06.833 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/37 2024-09-16T04:37:06.833 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2/37 2024-09-16T04:37:06.833 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 3/37 2024-09-16T04:37:06.833 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 4/37 2024-09-16T04:37:06.833 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 5/37 2024-09-16T04:37:06.833 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 6/37 2024-09-16T04:37:06.833 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T04:37:06.833 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T04:37:06.833 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T04:37:06.833 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T04:37:06.833 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T04:37:06.834 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T04:37:06.834 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T04:37:06.834 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T04:37:06.834 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T04:37:06.834 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T04:37:06.834 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T04:37:06.834 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T04:37:06.834 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T04:37:06.834 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T04:37:06.834 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T04:37:06.834 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T04:37:06.834 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T04:37:06.835 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T04:37:06.835 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T04:37:06.835 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T04:37:06.835 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T04:37:06.835 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T04:37:06.836 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T04:37:06.836 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T04:37:06.836 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T04:37:06.836 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T04:37:06.836 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T04:37:06.836 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T04:37:06.836 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T04:37:06.836 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T04:37:07.270 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T04:37:07.270 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:07.270 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-09-16T04:37:07.270 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:07.270 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:07.270 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:07.270 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:37:07.271 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:07.271 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:07.271 INFO:teuthology.orchestra.run.smithi046.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T04:37:07.271 INFO:teuthology.orchestra.run.smithi046.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T04:37:07.271 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T04:37:07.271 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T04:37:07.271 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T04:37:07.271 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T04:37:07.271 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T04:37:07.271 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T04:37:07.271 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T04:37:07.271 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T04:37:07.271 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T04:37:07.272 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T04:37:07.272 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T04:37:07.272 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T04:37:07.272 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T04:37:07.272 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T04:37:07.272 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T04:37:07.272 INFO:teuthology.orchestra.run.smithi046.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T04:37:07.272 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T04:37:07.272 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T04:37:07.272 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T04:37:07.272 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T04:37:07.272 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T04:37:07.272 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T04:37:07.273 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T04:37:07.273 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T04:37:07.273 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T04:37:07.273 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T04:37:07.273 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T04:37:07.273 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T04:37:07.273 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T04:37:07.273 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:07.273 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:37:07.651 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T04:37:07.652 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:07.652 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T04:37:07.652 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:07.652 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:07.652 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:07.652 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:37:07.652 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:07.652 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:07.652 INFO:teuthology.orchestra.run.smithi114.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T04:37:07.653 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T04:37:07.653 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T04:37:07.653 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T04:37:07.653 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T04:37:07.653 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T04:37:07.653 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T04:37:07.653 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T04:37:07.653 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T04:37:07.653 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T04:37:07.653 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T04:37:07.653 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T04:37:07.653 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T04:37:07.653 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T04:37:07.654 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T04:37:07.654 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T04:37:07.654 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T04:37:07.654 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T04:37:07.654 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T04:37:07.654 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T04:37:07.654 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T04:37:07.654 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T04:37:07.654 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T04:37:07.654 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T04:37:07.654 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T04:37:07.654 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T04:37:07.654 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T04:37:07.655 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T04:37:07.655 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T04:37:07.655 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T04:37:07.655 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T04:37:07.655 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:07.655 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:37:07.661 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-base 2024-09-16T04:37:08.023 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-base 2024-09-16T04:37:08.253 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:54 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:37:08.347 INFO:teuthology.orchestra.run.smithi046.stdout:Package ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:37:08.391 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:37:08.392 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-09-16T04:37:08.392 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:37:08.457 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install cephadm 2024-09-16T04:37:08.627 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:52 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:37:08.716 INFO:teuthology.orchestra.run.smithi114.stdout:Package ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:37:08.758 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:37:08.759 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-16T04:37:08.759 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:37:08.831 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install cephadm 2024-09-16T04:37:09.045 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:37:09.182 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:37:09.183 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:09.183 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-09-16T04:37:09.183 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:09.183 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-09-16T04:37:09.183 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T04:37:09.184 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:09.184 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-09-16T04:37:09.184 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:09.184 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-09-16T04:37:09.184 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:09.184 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 765 k 2024-09-16T04:37:09.184 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 771 k 2024-09-16T04:37:09.184 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-09-16T04:37:09.389 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:37:09.447 INFO:teuthology.orchestra.run.smithi046.stdout:cephadm-19.1.1-326.g8921c8ef.el9.noarch.rpm 2.8 MB/s | 765 kB 00:00 2024-09-16T04:37:09.447 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:09.448 INFO:teuthology.orchestra.run.smithi046.stdout:Total 2.8 MB/s | 765 kB 00:00 2024-09-16T04:37:09.448 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-09-16T04:37:09.454 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-09-16T04:37:09.454 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-09-16T04:37:09.460 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-09-16T04:37:09.460 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-09-16T04:37:09.520 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:37:09.521 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:09.521 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-16T04:37:09.521 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:09.521 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T04:37:09.522 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T04:37:09.522 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:09.522 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T04:37:09.522 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:09.522 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-16T04:37:09.522 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:09.522 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 765 k 2024-09-16T04:37:09.522 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 771 k 2024-09-16T04:37:09.522 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T04:37:09.532 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-09-16T04:37:09.765 INFO:teuthology.orchestra.run.smithi114.stdout:cephadm-19.1.1-326.g8921c8ef.el9.noarch.rpm 3.1 MB/s | 765 kB 00:00 2024-09-16T04:37:09.765 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:09.765 INFO:teuthology.orchestra.run.smithi114.stdout:Total 3.1 MB/s | 765 kB 00:00 2024-09-16T04:37:09.765 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T04:37:09.769 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T04:37:09.769 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T04:37:09.775 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T04:37:09.776 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T04:37:09.851 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T04:37:09.942 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:37:10.037 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:37:10.333 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:37:10.417 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:37:10.418 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:37:10.689 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:37:10.689 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:10.689 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-09-16T04:37:10.689 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:37:10.689 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:10.689 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:37:10.761 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:37:10.889 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T04:37:10.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:37:10.993 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:10.993 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T04:37:10.993 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:37:10.993 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:10.993 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:37:11.201 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T04:37:11.505 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:57 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:37:11.643 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:37:11.644 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:11.644 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-09-16T04:37:11.645 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:11.645 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-09-16T04:37:11.645 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T04:37:11.645 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:11.645 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-09-16T04:37:11.645 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:11.645 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-09-16T04:37:11.645 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:11.645 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 145 k 2024-09-16T04:37:11.645 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 435 k 2024-09-16T04:37:11.645 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-09-16T04:37:11.772 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:37:11.845 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-immutable-object-cache-19.1.1-326.g8921c8e 729 kB/s | 145 kB 00:00 2024-09-16T04:37:11.845 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:11.846 INFO:teuthology.orchestra.run.smithi046.stdout:Total 723 kB/s | 145 kB 00:00 2024-09-16T04:37:11.846 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-09-16T04:37:11.854 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-09-16T04:37:11.854 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-09-16T04:37:11.908 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:37:11.911 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:11.911 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-09-16T04:37:11.911 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:11.911 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T04:37:11.911 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T04:37:11.912 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:11.912 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T04:37:11.912 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:11.912 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-16T04:37:11.912 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:11.912 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 145 k 2024-09-16T04:37:11.912 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 435 k 2024-09-16T04:37:11.912 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T04:37:11.914 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-09-16T04:37:11.915 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-09-16T04:37:12.083 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-09-16T04:37:12.096 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-immutable-object-cache-19.1.1-326.g8921c8e 785 kB/s | 145 kB 00:00 2024-09-16T04:37:12.096 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:12.096 INFO:teuthology.orchestra.run.smithi114.stdout:Total 779 kB/s | 145 kB 00:00 2024-09-16T04:37:12.096 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T04:37:12.104 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T04:37:12.104 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T04:37:12.154 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:37:12.161 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T04:37:12.161 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T04:37:12.181 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:37:12.181 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:37:12.181 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T04:37:12.181 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:12.320 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T04:37:12.413 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:37:12.437 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:37:12.437 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:37:12.437 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T04:37:12.437 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:13.123 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:37:13.123 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:13.123 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-09-16T04:37:13.124 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:13.124 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:13.124 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:37:13.344 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:37:13.344 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:13.344 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T04:37:13.344 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:13.344 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:13.344 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:37:13.350 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr 2024-09-16T04:37:13.579 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr 2024-09-16T04:37:13.945 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:37:14.039 INFO:teuthology.orchestra.run.smithi046.stdout:Package ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:37:14.085 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:37:14.086 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-09-16T04:37:14.086 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:37:14.150 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T04:37:14.150 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:37:14.242 INFO:teuthology.orchestra.run.smithi114.stdout:Package ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:37:14.284 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:37:14.285 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-16T04:37:14.285 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:37:14.351 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T04:37:14.736 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:00 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:37:14.874 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:37:14.875 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:14.875 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-09-16T04:37:14.876 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:14.876 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-09-16T04:37:14.876 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T04:37:14.876 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-09-16T04:37:14.876 INFO:teuthology.orchestra.run.smithi046.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T04:37:14.876 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T04:37:14.876 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T04:37:14.876 INFO:teuthology.orchestra.run.smithi046.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T04:37:14.876 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T04:37:14.876 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T04:37:14.876 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T04:37:14.876 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T04:37:14.876 INFO:teuthology.orchestra.run.smithi046.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T04:37:14.876 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T04:37:14.877 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T04:37:14.877 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:14.877 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-09-16T04:37:14.877 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:14.877 INFO:teuthology.orchestra.run.smithi046.stdout:Install 12 Packages 2024-09-16T04:37:14.877 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:14.877 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 8.7 M 2024-09-16T04:37:14.878 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 104 M 2024-09-16T04:37:14.878 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-09-16T04:37:14.915 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:37:15.046 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:37:15.047 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:15.047 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-16T04:37:15.047 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:15.048 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T04:37:15.048 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T04:37:15.048 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-16T04:37:15.048 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T04:37:15.048 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T04:37:15.048 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T04:37:15.048 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T04:37:15.048 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T04:37:15.048 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T04:37:15.048 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T04:37:15.048 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T04:37:15.048 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T04:37:15.048 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T04:37:15.049 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T04:37:15.049 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:15.049 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T04:37:15.049 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:15.049 INFO:teuthology.orchestra.run.smithi114.stdout:Install 12 Packages 2024-09-16T04:37:15.049 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:15.049 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 8.7 M 2024-09-16T04:37:15.050 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 104 M 2024-09-16T04:37:15.050 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T04:37:15.485 INFO:teuthology.orchestra.run.smithi046.stdout:(1/12): ceph-grafana-dashboards-19.1.1-326.g892 219 kB/s | 32 kB 00:00 2024-09-16T04:37:15.510 INFO:teuthology.orchestra.run.smithi046.stdout:(2/12): ceph-prometheus-alerts-19.1.1-326.g8921 96 kB/s | 16 kB 00:00 2024-09-16T04:37:15.727 INFO:teuthology.orchestra.run.smithi046.stdout:(3/12): ceph-mgr-dashboard-19.1.1-326.g8921c8ef 9.2 MB/s | 3.6 MB 00:00 2024-09-16T04:37:15.886 INFO:teuthology.orchestra.run.smithi046.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.3 MB/s | 863 kB 00:00 2024-09-16T04:37:15.933 INFO:teuthology.orchestra.run.smithi114.stdout:(1/12): ceph-grafana-dashboards-19.1.1-326.g892 216 kB/s | 32 kB 00:00 2024-09-16T04:37:15.953 INFO:teuthology.orchestra.run.smithi046.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 8.1 MB/s | 548 kB 00:00 2024-09-16T04:37:15.957 INFO:teuthology.orchestra.run.smithi114.stdout:(2/12): ceph-prometheus-alerts-19.1.1-326.g8921 95 kB/s | 16 kB 00:00 2024-09-16T04:37:15.978 INFO:teuthology.orchestra.run.smithi046.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 776 kB/s | 19 kB 00:00 2024-09-16T04:37:16.095 INFO:teuthology.orchestra.run.smithi046.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 17 MB/s | 2.0 MB 00:00 2024-09-16T04:37:16.129 INFO:teuthology.orchestra.run.smithi046.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.2 MB/s | 144 kB 00:00 2024-09-16T04:37:16.162 INFO:teuthology.orchestra.run.smithi046.stdout:(9/12): python3-protobuf-3.14.0-13.el9.noarch.r 413 kB/s | 269 kB 00:00 2024-09-16T04:37:16.187 INFO:teuthology.orchestra.run.smithi046.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 526 kB/s | 31 kB 00:00 2024-09-16T04:37:16.202 INFO:teuthology.orchestra.run.smithi114.stdout:(3/12): ceph-mgr-dashboard-19.1.1-326.g8921c8ef 8.5 MB/s | 3.6 MB 00:00 2024-09-16T04:37:16.254 INFO:teuthology.orchestra.run.smithi046.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-09-16T04:37:16.358 INFO:teuthology.orchestra.run.smithi114.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.4 MB/s | 863 kB 00:00 2024-09-16T04:37:16.421 INFO:teuthology.orchestra.run.smithi046.stdout:(12/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.1 MB/s | 1.0 MB 00:00 2024-09-16T04:37:16.425 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:16.426 INFO:teuthology.orchestra.run.smithi046.stdout:Total 5.6 MB/s | 8.7 MB 00:01 2024-09-16T04:37:16.433 INFO:teuthology.orchestra.run.smithi114.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 7.2 MB/s | 548 kB 00:00 2024-09-16T04:37:16.458 INFO:teuthology.orchestra.run.smithi114.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 779 kB/s | 19 kB 00:00 2024-09-16T04:37:16.479 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-09-16T04:37:16.499 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-09-16T04:37:16.499 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-09-16T04:37:16.584 INFO:teuthology.orchestra.run.smithi114.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 16 MB/s | 2.0 MB 00:00 2024-09-16T04:37:16.609 INFO:teuthology.orchestra.run.smithi114.stdout:(8/12): python3-protobuf-3.14.0-13.el9.noarch.r 413 kB/s | 269 kB 00:00 2024-09-16T04:37:16.634 INFO:teuthology.orchestra.run.smithi114.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-09-16T04:37:16.676 INFO:teuthology.orchestra.run.smithi114.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 5.4 MB/s | 188 kB 00:00 2024-09-16T04:37:16.701 INFO:teuthology.orchestra.run.smithi114.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 334 kB/s | 31 kB 00:00 2024-09-16T04:37:16.716 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-09-16T04:37:16.716 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-09-16T04:37:16.885 INFO:teuthology.orchestra.run.smithi114.stdout:(12/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.1 MB/s | 1.0 MB 00:00 2024-09-16T04:37:16.889 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:16.889 INFO:teuthology.orchestra.run.smithi114.stdout:Total 4.7 MB/s | 8.7 MB 00:01 2024-09-16T04:37:16.941 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T04:37:16.959 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T04:37:16.960 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T04:37:17.169 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T04:37:17.169 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T04:37:17.258 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-09-16T04:37:17.347 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T04:37:17.399 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T04:37:17.455 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T04:37:17.484 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T04:37:17.591 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T04:37:17.700 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T04:37:17.725 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T04:37:17.786 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T04:37:17.818 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T04:37:17.820 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T04:37:17.868 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T04:37:17.873 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T04:37:17.906 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T04:37:17.909 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T04:37:17.941 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 10/12 2024-09-16T04:37:18.000 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T04:37:18.142 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T04:37:18.234 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T04:37:18.278 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T04:37:18.315 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T04:37:18.346 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 10/12 2024-09-16T04:37:19.054 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 11/12 2024-09-16T04:37:19.092 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T04:37:19.403 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 11/12 2024-09-16T04:37:19.438 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T04:37:20.438 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T04:37:20.439 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 1/12 2024-09-16T04:37:20.439 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 2/12 2024-09-16T04:37:20.439 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 3/12 2024-09-16T04:37:20.439 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T04:37:20.439 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T04:37:20.439 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T04:37:20.439 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T04:37:20.439 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T04:37:20.439 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T04:37:20.439 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T04:37:20.440 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T04:37:20.772 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T04:37:20.772 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:20.772 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-09-16T04:37:20.772 INFO:teuthology.orchestra.run.smithi046.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T04:37:20.772 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:37:20.772 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:37:20.772 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:37:20.772 INFO:teuthology.orchestra.run.smithi046.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T04:37:20.772 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T04:37:20.772 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T04:37:20.772 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T04:37:20.773 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T04:37:20.773 INFO:teuthology.orchestra.run.smithi046.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T04:37:20.773 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T04:37:20.773 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T04:37:20.773 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:20.773 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:37:20.848 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T04:37:20.849 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 1/12 2024-09-16T04:37:20.849 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 2/12 2024-09-16T04:37:20.849 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 3/12 2024-09-16T04:37:20.849 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T04:37:20.849 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T04:37:20.849 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T04:37:20.849 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T04:37:20.849 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T04:37:20.849 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T04:37:20.849 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T04:37:20.849 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T04:37:21.027 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T04:37:21.214 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T04:37:21.214 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:21.214 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T04:37:21.214 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T04:37:21.214 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:37:21.214 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:37:21.214 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:37:21.214 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T04:37:21.214 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T04:37:21.214 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T04:37:21.215 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T04:37:21.215 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T04:37:21.215 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T04:37:21.215 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T04:37:21.215 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T04:37:21.215 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:21.215 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:37:21.470 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T04:37:21.616 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:07 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:37:21.757 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:37:21.758 INFO:teuthology.orchestra.run.smithi046.stdout:====================================================================================== 2024-09-16T04:37:21.758 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-09-16T04:37:21.758 INFO:teuthology.orchestra.run.smithi046.stdout:====================================================================================== 2024-09-16T04:37:21.758 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-09-16T04:37:21.758 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T04:37:21.758 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-09-16T04:37:21.758 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T04:37:21.759 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T04:37:21.759 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T04:37:21.759 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T04:37:21.759 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T04:37:21.759 INFO:teuthology.orchestra.run.smithi046.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T04:37:21.759 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T04:37:21.759 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T04:37:21.759 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T04:37:21.759 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T04:37:21.759 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T04:37:21.759 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:21.759 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-09-16T04:37:21.759 INFO:teuthology.orchestra.run.smithi046.stdout:====================================================================================== 2024-09-16T04:37:21.759 INFO:teuthology.orchestra.run.smithi046.stdout:Install 12 Packages 2024-09-16T04:37:21.760 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:21.760 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 38 M 2024-09-16T04:37:21.760 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 212 M 2024-09-16T04:37:21.760 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-09-16T04:37:22.012 INFO:teuthology.orchestra.run.smithi046.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 901 kB/s | 191 kB 00:00 2024-09-16T04:37:22.033 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:06 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:37:22.063 INFO:teuthology.orchestra.run.smithi046.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.0 MB/s | 801 kB 00:00 2024-09-16T04:37:22.130 INFO:teuthology.orchestra.run.smithi046.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 279 kB/s | 33 kB 00:00 2024-09-16T04:37:22.167 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:37:22.168 INFO:teuthology.orchestra.run.smithi114.stdout:====================================================================================== 2024-09-16T04:37:22.168 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-16T04:37:22.168 INFO:teuthology.orchestra.run.smithi114.stdout:====================================================================================== 2024-09-16T04:37:22.169 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T04:37:22.169 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T04:37:22.169 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-16T04:37:22.169 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T04:37:22.169 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T04:37:22.169 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T04:37:22.169 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T04:37:22.169 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T04:37:22.169 INFO:teuthology.orchestra.run.smithi114.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T04:37:22.169 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T04:37:22.169 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T04:37:22.169 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T04:37:22.169 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T04:37:22.169 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T04:37:22.170 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:22.170 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T04:37:22.170 INFO:teuthology.orchestra.run.smithi114.stdout:====================================================================================== 2024-09-16T04:37:22.170 INFO:teuthology.orchestra.run.smithi114.stdout:Install 12 Packages 2024-09-16T04:37:22.170 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:22.170 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 38 M 2024-09-16T04:37:22.170 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 212 M 2024-09-16T04:37:22.171 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T04:37:22.171 INFO:teuthology.orchestra.run.smithi046.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 438 kB/s | 18 kB 00:00 2024-09-16T04:37:22.230 INFO:teuthology.orchestra.run.smithi046.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 662 kB/s | 38 kB 00:00 2024-09-16T04:37:22.322 INFO:teuthology.orchestra.run.smithi046.stdout:(6/12): ceph-mgr-diskprediction-local-19.1.1-32 14 MB/s | 7.4 MB 00:00 2024-09-16T04:37:22.381 INFO:teuthology.orchestra.run.smithi046.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.4 MB/s | 3.0 MB 00:00 2024-09-16T04:37:22.497 INFO:teuthology.orchestra.run.smithi046.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 44 MB/s | 5.1 MB 00:00 2024-09-16T04:37:22.540 INFO:teuthology.orchestra.run.smithi046.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.8 MB/s | 250 kB 00:00 2024-09-16T04:37:22.573 INFO:teuthology.orchestra.run.smithi046.stdout:(10/12): python3-devel-3.9.19-8.el9.x86_64.rpm 975 kB/s | 245 kB 00:00 2024-09-16T04:37:22.584 INFO:teuthology.orchestra.run.smithi114.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 880 kB/s | 191 kB 00:00 2024-09-16T04:37:22.649 INFO:teuthology.orchestra.run.smithi046.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 12 MB/s | 4.9 MB 00:00 2024-09-16T04:37:22.685 INFO:teuthology.orchestra.run.smithi114.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 327 kB/s | 33 kB 00:00 2024-09-16T04:37:22.710 INFO:teuthology.orchestra.run.smithi114.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.3 MB/s | 801 kB 00:00 2024-09-16T04:37:22.835 INFO:teuthology.orchestra.run.smithi114.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 145 kB/s | 18 kB 00:00 2024-09-16T04:37:22.882 INFO:teuthology.orchestra.run.smithi046.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-09-16T04:37:22.883 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:22.883 INFO:teuthology.orchestra.run.smithi046.stdout:Total 34 MB/s | 38 MB 00:01 2024-09-16T04:37:22.936 INFO:teuthology.orchestra.run.smithi114.stdout:(5/12): ceph-mgr-diskprediction-local-19.1.1-32 13 MB/s | 7.4 MB 00:00 2024-09-16T04:37:22.961 INFO:teuthology.orchestra.run.smithi114.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 308 kB/s | 38 kB 00:00 2024-09-16T04:37:23.028 INFO:teuthology.orchestra.run.smithi114.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.7 MB/s | 3.0 MB 00:00 2024-09-16T04:37:23.086 INFO:teuthology.orchestra.run.smithi114.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-09-16T04:37:23.162 INFO:teuthology.orchestra.run.smithi114.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 5.1 MB 00:00 2024-09-16T04:37:23.187 INFO:teuthology.orchestra.run.smithi114.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.4 MB/s | 250 kB 00:00 2024-09-16T04:37:23.253 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-09-16T04:37:23.280 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-09-16T04:37:23.281 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-09-16T04:37:23.288 INFO:teuthology.orchestra.run.smithi114.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 14 MB/s | 4.9 MB 00:00 2024-09-16T04:37:23.495 INFO:teuthology.orchestra.run.smithi114.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 49 MB/s | 16 MB 00:00 2024-09-16T04:37:23.496 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:23.497 INFO:teuthology.orchestra.run.smithi114.stdout:Total 29 MB/s | 38 MB 00:01 2024-09-16T04:37:23.740 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-09-16T04:37:23.741 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-09-16T04:37:23.850 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T04:37:23.876 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T04:37:23.876 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T04:37:24.311 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T04:37:24.311 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T04:37:24.664 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-09-16T04:37:24.725 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T04:37:24.781 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T04:37:24.887 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T04:37:24.925 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T04:37:25.064 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T04:37:25.117 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T04:37:25.143 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T04:37:25.147 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T04:37:25.191 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T04:37:25.235 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T04:37:25.295 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T04:37:25.337 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T04:37:25.520 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T04:37:25.554 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T04:37:25.639 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T04:37:25.949 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T04:37:26.075 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T04:37:26.311 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T04:37:26.417 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T04:37:27.409 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T04:37:27.647 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T04:37:28.574 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T04:37:28.605 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T04:37:28.680 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T04:37:28.720 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T04:37:31.252 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T04:37:31.252 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 1/12 2024-09-16T04:37:31.252 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T04:37:31.253 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T04:37:31.253 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T04:37:31.253 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T04:37:31.253 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T04:37:31.253 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T04:37:31.253 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T04:37:31.253 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T04:37:31.253 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T04:37:31.253 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T04:37:31.313 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T04:37:31.313 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 1/12 2024-09-16T04:37:31.313 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T04:37:31.313 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T04:37:31.313 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T04:37:31.313 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T04:37:31.313 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T04:37:31.313 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T04:37:31.313 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T04:37:31.313 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T04:37:31.313 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T04:37:31.314 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T04:37:31.696 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T04:37:31.697 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:31.697 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-09-16T04:37:31.697 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:37:31.697 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T04:37:31.697 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T04:37:31.697 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T04:37:31.697 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T04:37:31.697 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T04:37:31.697 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T04:37:31.697 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T04:37:31.697 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T04:37:31.697 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T04:37:31.698 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T04:37:31.698 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T04:37:31.698 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:31.698 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:37:31.722 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T04:37:31.722 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:31.722 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T04:37:31.722 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:37:31.722 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T04:37:31.722 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T04:37:31.722 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T04:37:31.722 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T04:37:31.722 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T04:37:31.722 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T04:37:31.722 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T04:37:31.723 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T04:37:31.723 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T04:37:31.723 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T04:37:31.723 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T04:37:31.723 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:31.723 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:37:32.096 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr-rook 2024-09-16T04:37:32.114 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-rook 2024-09-16T04:37:32.686 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:18 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:37:32.694 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:16 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:37:32.826 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:37:32.828 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:32.828 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-09-16T04:37:32.828 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:32.828 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-09-16T04:37:32.828 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T04:37:32.828 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-09-16T04:37:32.828 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T04:37:32.828 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T04:37:32.829 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T04:37:32.829 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T04:37:32.829 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T04:37:32.829 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T04:37:32.829 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T04:37:32.829 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T04:37:32.829 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T04:37:32.829 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T04:37:32.829 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T04:37:32.829 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T04:37:32.829 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:32.829 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-09-16T04:37:32.829 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:32.830 INFO:teuthology.orchestra.run.smithi046.stdout:Install 13 Packages 2024-09-16T04:37:32.830 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:32.830 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 2.2 M 2024-09-16T04:37:32.830 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 26 M 2024-09-16T04:37:32.830 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-09-16T04:37:32.830 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:37:32.830 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:32.831 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-16T04:37:32.831 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:32.831 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T04:37:32.831 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T04:37:32.831 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-16T04:37:32.831 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T04:37:32.831 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T04:37:32.831 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T04:37:32.831 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T04:37:32.831 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T04:37:32.831 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T04:37:32.831 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T04:37:32.831 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T04:37:32.832 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T04:37:32.832 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T04:37:32.832 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T04:37:32.832 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T04:37:32.832 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:32.832 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T04:37:32.832 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:32.832 INFO:teuthology.orchestra.run.smithi114.stdout:Install 13 Packages 2024-09-16T04:37:32.832 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:32.833 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 2.2 M 2024-09-16T04:37:32.833 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 26 M 2024-09-16T04:37:32.833 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T04:37:33.171 INFO:teuthology.orchestra.run.smithi114.stdout:(1/13): ceph-mgr-rook-19.1.1-326.g8921c8ef.el9. 288 kB/s | 49 kB 00:00 2024-09-16T04:37:33.205 INFO:teuthology.orchestra.run.smithi114.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 125 kB/s | 26 kB 00:00 2024-09-16T04:37:33.230 INFO:teuthology.orchestra.run.smithi114.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 83 kB/s | 19 kB 00:00 2024-09-16T04:37:33.424 INFO:teuthology.orchestra.run.smithi046.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 298 kB/s | 26 kB 00:00 2024-09-16T04:37:33.430 INFO:teuthology.orchestra.run.smithi114.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 858 kB/s | 222 kB 00:00 2024-09-16T04:37:33.449 INFO:teuthology.orchestra.run.smithi046.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 173 kB/s | 19 kB 00:00 2024-09-16T04:37:33.532 INFO:teuthology.orchestra.run.smithi046.stdout:(3/13): ceph-mgr-rook-19.1.1-326.g8921c8ef.el9. 253 kB/s | 49 kB 00:00 2024-09-16T04:37:33.555 INFO:teuthology.orchestra.run.smithi114.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 488 kB/s | 159 kB 00:00 2024-09-16T04:37:33.572 INFO:teuthology.orchestra.run.smithi114.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 378 kB/s | 54 kB 00:00 2024-09-16T04:37:33.591 INFO:teuthology.orchestra.run.smithi046.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-09-16T04:37:33.614 INFO:teuthology.orchestra.run.smithi114.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 550 kB/s | 32 kB 00:00 2024-09-16T04:37:33.616 INFO:teuthology.orchestra.run.smithi046.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.3 MB/s | 222 kB 00:00 2024-09-16T04:37:33.639 INFO:teuthology.orchestra.run.smithi114.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 681 kB/s | 279 kB 00:00 2024-09-16T04:37:33.650 INFO:teuthology.orchestra.run.smithi046.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 913 kB/s | 54 kB 00:00 2024-09-16T04:37:33.656 INFO:teuthology.orchestra.run.smithi114.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 168 kB/s | 14 kB 00:00 2024-09-16T04:37:33.684 INFO:teuthology.orchestra.run.smithi046.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 482 kB/s | 32 kB 00:00 2024-09-16T04:37:33.690 INFO:teuthology.orchestra.run.smithi114.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.2 MB/s | 59 kB 00:00 2024-09-16T04:37:33.709 INFO:teuthology.orchestra.run.smithi046.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 250 kB/s | 14 kB 00:00 2024-09-16T04:37:33.715 INFO:teuthology.orchestra.run.smithi114.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.2 MB/s | 220 kB 00:00 2024-09-16T04:37:33.734 INFO:teuthology.orchestra.run.smithi046.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.5 MB/s | 220 kB 00:00 2024-09-16T04:37:33.740 INFO:teuthology.orchestra.run.smithi114.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-16T04:37:33.760 INFO:teuthology.orchestra.run.smithi046.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-16T04:37:33.785 INFO:teuthology.orchestra.run.smithi046.stdout:(11/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.1 MB/s | 279 kB 00:00 2024-09-16T04:37:33.807 INFO:teuthology.orchestra.run.smithi114.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-09-16T04:37:33.810 INFO:teuthology.orchestra.run.smithi046.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-16T04:37:33.810 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:33.810 INFO:teuthology.orchestra.run.smithi114.stdout:Total 2.3 MB/s | 2.2 MB 00:00 2024-09-16T04:37:33.852 INFO:teuthology.orchestra.run.smithi046.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-16T04:37:33.854 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:33.854 INFO:teuthology.orchestra.run.smithi046.stdout:Total 2.2 MB/s | 2.2 MB 00:01 2024-09-16T04:37:33.876 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T04:37:33.891 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T04:37:33.891 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T04:37:33.922 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-09-16T04:37:33.938 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-09-16T04:37:33.939 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-09-16T04:37:33.980 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T04:37:33.980 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T04:37:34.036 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-09-16T04:37:34.036 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-09-16T04:37:34.478 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T04:37:34.645 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T04:37:34.674 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-09-16T04:37:34.701 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T04:37:34.773 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T04:37:34.818 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T04:37:34.849 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T04:37:34.859 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T04:37:34.905 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T04:37:34.970 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T04:37:34.980 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T04:37:35.021 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T04:37:35.063 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T04:37:35.071 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T04:37:35.120 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T04:37:35.178 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T04:37:35.287 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T04:37:35.341 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T04:37:35.837 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T04:37:35.892 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T04:37:35.925 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T04:37:35.973 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T04:37:35.995 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T04:37:36.097 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T04:37:36.162 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T04:37:36.195 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T04:37:36.245 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T04:37:36.273 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T04:37:36.734 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T04:37:36.735 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 1/13 2024-09-16T04:37:36.735 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T04:37:36.735 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T04:37:36.735 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T04:37:36.735 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T04:37:36.735 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T04:37:36.735 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T04:37:36.735 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T04:37:36.735 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T04:37:36.735 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T04:37:36.735 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T04:37:36.735 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T04:37:37.061 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T04:37:37.061 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:37.062 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T04:37:37.062 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:37:37.062 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T04:37:37.062 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T04:37:37.062 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T04:37:37.062 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T04:37:37.062 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T04:37:37.062 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T04:37:37.062 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T04:37:37.062 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T04:37:37.062 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T04:37:37.063 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T04:37:37.063 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T04:37:37.063 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T04:37:37.063 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:37.063 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:37:37.168 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T04:37:37.169 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 1/13 2024-09-16T04:37:37.169 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T04:37:37.169 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T04:37:37.169 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T04:37:37.169 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T04:37:37.169 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T04:37:37.169 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T04:37:37.169 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T04:37:37.169 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T04:37:37.169 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T04:37:37.169 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T04:37:37.169 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T04:37:37.323 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T04:37:37.512 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T04:37:37.512 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:37.512 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-09-16T04:37:37.513 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:37:37.513 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T04:37:37.513 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T04:37:37.513 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T04:37:37.513 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T04:37:37.513 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T04:37:37.513 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T04:37:37.513 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T04:37:37.513 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T04:37:37.513 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T04:37:37.513 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T04:37:37.513 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T04:37:37.513 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T04:37:37.514 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:37.514 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:37:37.891 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:21 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:37:37.893 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T04:37:38.025 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:37:38.026 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:38.026 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-16T04:37:38.026 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:38.026 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T04:37:38.026 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T04:37:38.026 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-16T04:37:38.026 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T04:37:38.026 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T04:37:38.026 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T04:37:38.026 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T04:37:38.027 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T04:37:38.027 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:38.027 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T04:37:38.027 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:38.027 INFO:teuthology.orchestra.run.smithi114.stdout:Install 6 Packages 2024-09-16T04:37:38.027 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:38.027 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 7.0 M 2024-09-16T04:37:38.028 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 33 M 2024-09-16T04:37:38.028 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T04:37:38.344 INFO:teuthology.orchestra.run.smithi114.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-09-16T04:37:38.419 INFO:teuthology.orchestra.run.smithi114.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-09-16T04:37:38.444 INFO:teuthology.orchestra.run.smithi114.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-16T04:37:38.470 INFO:teuthology.orchestra.run.smithi114.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-16T04:37:38.490 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:24 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:37:38.562 INFO:teuthology.orchestra.run.smithi114.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-09-16T04:37:38.629 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:37:38.630 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:38.630 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-09-16T04:37:38.630 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:38.630 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-09-16T04:37:38.630 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T04:37:38.630 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-09-16T04:37:38.630 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T04:37:38.630 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T04:37:38.630 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T04:37:38.630 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T04:37:38.631 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T04:37:38.631 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:38.631 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-09-16T04:37:38.631 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:38.631 INFO:teuthology.orchestra.run.smithi046.stdout:Install 6 Packages 2024-09-16T04:37:38.631 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:38.631 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 7.0 M 2024-09-16T04:37:38.632 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 33 M 2024-09-16T04:37:38.632 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-09-16T04:37:38.695 INFO:teuthology.orchestra.run.smithi114.stdout:(6/6): ceph-mgr-cephadm-19.1.1-326.g8921c8ef.el 298 kB/s | 146 kB 00:00 2024-09-16T04:37:38.697 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:38.697 INFO:teuthology.orchestra.run.smithi114.stdout:Total 10 MB/s | 7.0 MB 00:00 2024-09-16T04:37:38.769 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T04:37:38.780 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T04:37:38.780 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T04:37:38.879 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T04:37:38.880 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T04:37:38.938 INFO:teuthology.orchestra.run.smithi046.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-16T04:37:38.988 INFO:teuthology.orchestra.run.smithi046.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-16T04:37:39.013 INFO:teuthology.orchestra.run.smithi046.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-09-16T04:37:39.040 INFO:teuthology.orchestra.run.smithi046.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-16T04:37:39.155 INFO:teuthology.orchestra.run.smithi046.stdout:(5/6): ceph-mgr-cephadm-19.1.1-326.g8921c8ef.el 350 kB/s | 146 kB 00:00 2024-09-16T04:37:39.189 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T04:37:39.360 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T04:37:39.405 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T04:37:39.926 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T04:37:40.016 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T04:37:40.031 INFO:teuthology.orchestra.run.smithi046.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.6 MB/s | 6.0 MB 00:01 2024-09-16T04:37:40.032 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:40.033 INFO:teuthology.orchestra.run.smithi046.stdout:Total 5.0 MB/s | 7.0 MB 00:01 2024-09-16T04:37:40.088 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T04:37:40.106 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-09-16T04:37:40.118 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T04:37:40.118 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-09-16T04:37:40.118 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-09-16T04:37:40.225 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-09-16T04:37:40.225 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-09-16T04:37:40.548 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-09-16T04:37:40.738 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T04:37:40.801 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T04:37:40.917 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T04:37:40.917 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/6 2024-09-16T04:37:40.917 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T04:37:40.917 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T04:37:40.917 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T04:37:40.917 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T04:37:41.185 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T04:37:41.185 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:41.185 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T04:37:41.185 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:37:41.185 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T04:37:41.185 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T04:37:41.186 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T04:37:41.186 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T04:37:41.186 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T04:37:41.186 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:41.186 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:37:41.363 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T04:37:41.472 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T04:37:41.476 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-fuse 2024-09-16T04:37:41.543 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T04:37:41.603 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T04:37:42.047 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:26 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:37:42.181 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:37:42.182 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:42.182 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-16T04:37:42.182 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:42.182 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T04:37:42.182 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T04:37:42.183 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-16T04:37:42.183 INFO:teuthology.orchestra.run.smithi114.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T04:37:42.183 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:42.183 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T04:37:42.183 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:42.183 INFO:teuthology.orchestra.run.smithi114.stdout:Install 2 Packages 2024-09-16T04:37:42.183 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:42.184 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 976 k 2024-09-16T04:37:42.184 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 2.9 M 2024-09-16T04:37:42.184 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T04:37:42.347 INFO:teuthology.orchestra.run.smithi114.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 557 kB/s | 80 kB 00:00 2024-09-16T04:37:42.383 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T04:37:42.383 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/6 2024-09-16T04:37:42.383 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T04:37:42.383 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T04:37:42.383 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T04:37:42.383 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T04:37:42.514 INFO:teuthology.orchestra.run.smithi114.stdout:(2/2): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86_6 2.8 MB/s | 896 kB 00:00 2024-09-16T04:37:42.514 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:42.515 INFO:teuthology.orchestra.run.smithi114.stdout:Total 2.9 MB/s | 976 kB 00:00 2024-09-16T04:37:42.526 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T04:37:42.549 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T04:37:42.549 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T04:37:42.630 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T04:37:42.631 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T04:37:42.676 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T04:37:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:42.677 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-09-16T04:37:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:37:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T04:37:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T04:37:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T04:37:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T04:37:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T04:37:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:42.677 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:37:42.826 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T04:37:42.930 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-fuse 2024-09-16T04:37:42.958 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T04:37:43.039 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T04:37:43.523 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:29 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:37:43.663 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:37:43.664 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:43.664 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-09-16T04:37:43.664 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:43.664 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-09-16T04:37:43.664 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T04:37:43.665 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-09-16T04:37:43.665 INFO:teuthology.orchestra.run.smithi046.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T04:37:43.665 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:43.665 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-09-16T04:37:43.665 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:43.665 INFO:teuthology.orchestra.run.smithi046.stdout:Install 2 Packages 2024-09-16T04:37:43.665 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:43.666 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 976 k 2024-09-16T04:37:43.666 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 2.9 M 2024-09-16T04:37:43.666 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-09-16T04:37:43.735 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T04:37:43.736 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/2 2024-09-16T04:37:43.825 INFO:teuthology.orchestra.run.smithi046.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 705 kB/s | 80 kB 00:00 2024-09-16T04:37:44.000 INFO:teuthology.orchestra.run.smithi046.stdout:(2/2): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86_6 3.0 MB/s | 896 kB 00:00 2024-09-16T04:37:44.000 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:44.001 INFO:teuthology.orchestra.run.smithi046.stdout:Total 2.8 MB/s | 976 kB 00:00 2024-09-16T04:37:44.001 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T04:37:44.001 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:44.001 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T04:37:44.002 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T04:37:44.002 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:44.002 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:37:44.013 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-09-16T04:37:44.037 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-09-16T04:37:44.037 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-09-16T04:37:44.122 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-09-16T04:37:44.122 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-09-16T04:37:44.313 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-09-16T04:37:44.346 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-volume 2024-09-16T04:37:44.457 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T04:37:44.528 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T04:37:44.927 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:28 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:37:45.066 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:37:45.066 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:45.066 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-16T04:37:45.067 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:45.067 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T04:37:45.067 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T04:37:45.067 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:45.067 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T04:37:45.067 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:45.067 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-16T04:37:45.067 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:45.067 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 285 k 2024-09-16T04:37:45.067 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 1.3 M 2024-09-16T04:37:45.067 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T04:37:45.302 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-volume-19.1.1-326.g8921c8ef.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-16T04:37:45.302 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:45.303 INFO:teuthology.orchestra.run.smithi114.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-16T04:37:45.303 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T04:37:45.311 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T04:37:45.311 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T04:37:45.345 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T04:37:45.345 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T04:37:45.386 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T04:37:45.386 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/2 2024-09-16T04:37:45.573 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T04:37:45.646 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:37:45.662 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:37:45.662 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:37:45.662 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T04:37:45.662 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:45.668 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T04:37:45.668 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:45.668 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-09-16T04:37:45.668 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T04:37:45.668 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:45.668 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:37:45.960 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-volume 2024-09-16T04:37:46.563 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:32 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:37:46.703 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:37:46.703 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:46.704 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-09-16T04:37:46.704 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:46.704 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-09-16T04:37:46.704 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T04:37:46.704 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:46.704 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-09-16T04:37:46.704 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:46.704 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-09-16T04:37:46.704 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:46.704 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 285 k 2024-09-16T04:37:46.704 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 1.3 M 2024-09-16T04:37:46.704 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-09-16T04:37:46.729 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:37:46.729 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:46.729 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T04:37:46.729 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:37:46.729 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:46.729 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:37:47.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librados-devel 2024-09-16T04:37:47.131 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-volume-19.1.1-326.g8921c8ef.el9.noarch.rpm 670 kB/s | 285 kB 00:00 2024-09-16T04:37:47.132 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:47.132 INFO:teuthology.orchestra.run.smithi046.stdout:Total 667 kB/s | 285 kB 00:00 2024-09-16T04:37:47.132 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-09-16T04:37:47.141 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-09-16T04:37:47.141 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-09-16T04:37:47.173 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-09-16T04:37:47.173 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-09-16T04:37:47.404 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-09-16T04:37:47.474 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:37:47.492 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:37:47.492 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:37:47.492 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T04:37:47.492 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:47.591 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:31 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:37:47.728 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:37:47.729 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:47.729 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-09-16T04:37:47.730 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:47.730 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T04:37:47.730 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T04:37:47.730 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:47.730 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T04:37:47.730 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:47.730 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-16T04:37:47.730 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:47.730 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 127 k 2024-09-16T04:37:47.730 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 456 k 2024-09-16T04:37:47.730 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T04:37:48.090 INFO:teuthology.orchestra.run.smithi114.stdout:librados-devel-19.1.1-326.g8921c8ef.el9.x86_64. 352 kB/s | 127 kB 00:00 2024-09-16T04:37:48.091 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:48.091 INFO:teuthology.orchestra.run.smithi114.stdout:Total 351 kB/s | 127 kB 00:00 2024-09-16T04:37:48.091 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T04:37:48.097 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T04:37:48.097 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T04:37:48.167 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T04:37:48.167 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T04:37:48.336 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T04:37:48.437 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:37:48.519 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:37:48.519 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:48.519 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-09-16T04:37:48.519 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:37:48.519 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:48.519 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:37:48.824 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install librados-devel 2024-09-16T04:37:48.896 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:37:49.147 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:37:49.147 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:49.148 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T04:37:49.148 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:49.148 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:49.148 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:37:49.384 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install libcephfs2 2024-09-16T04:37:49.417 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:35 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:37:49.556 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:37:49.557 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:49.557 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-09-16T04:37:49.557 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:49.557 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-09-16T04:37:49.557 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T04:37:49.557 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:49.558 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-09-16T04:37:49.558 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:49.558 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-09-16T04:37:49.558 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:49.558 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 127 k 2024-09-16T04:37:49.558 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 456 k 2024-09-16T04:37:49.558 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-09-16T04:37:49.936 INFO:teuthology.orchestra.run.smithi046.stdout:librados-devel-19.1.1-326.g8921c8ef.el9.x86_64. 335 kB/s | 127 kB 00:00 2024-09-16T04:37:49.936 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:49.937 INFO:teuthology.orchestra.run.smithi046.stdout:Total 334 kB/s | 127 kB 00:00 2024-09-16T04:37:49.937 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-09-16T04:37:49.943 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-09-16T04:37:49.943 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-09-16T04:37:49.960 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:33 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:37:50.017 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-09-16T04:37:50.017 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-09-16T04:37:50.053 INFO:teuthology.orchestra.run.smithi114.stdout:Package libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:37:50.098 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:37:50.099 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-16T04:37:50.099 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:37:50.172 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install libcephfs-devel 2024-09-16T04:37:50.192 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-09-16T04:37:50.296 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:37:50.737 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:37:50.739 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:34 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:37:50.873 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:37:50.874 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:50.874 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-09-16T04:37:50.874 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:50.875 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T04:37:50.875 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T04:37:50.875 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:50.875 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T04:37:50.875 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:50.875 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-16T04:37:50.875 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:50.875 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 32 k 2024-09-16T04:37:50.875 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 142 k 2024-09-16T04:37:50.875 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T04:37:51.016 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:37:51.017 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:51.017 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-09-16T04:37:51.017 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:51.017 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:51.017 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:37:51.035 INFO:teuthology.orchestra.run.smithi114.stdout:libcephfs-devel-19.1.1-326.g8921c8ef.el9.x86_64 197 kB/s | 32 kB 00:00 2024-09-16T04:37:51.035 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:51.036 INFO:teuthology.orchestra.run.smithi114.stdout:Total 196 kB/s | 32 kB 00:00 2024-09-16T04:37:51.036 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T04:37:51.039 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T04:37:51.040 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T04:37:51.059 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T04:37:51.059 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T04:37:51.148 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T04:37:51.247 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install libcephfs2 2024-09-16T04:37:51.290 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:37:51.653 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:37:51.843 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:37:51.893 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:37:51.893 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:51.893 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T04:37:51.893 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:51.893 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:51.893 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:37:51.937 INFO:teuthology.orchestra.run.smithi046.stdout:Package libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:37:51.983 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:37:51.984 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-09-16T04:37:51.985 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:37:52.039 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install libcephfs-devel 2024-09-16T04:37:52.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librados2 2024-09-16T04:37:52.631 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:37:52.641 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:36 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:37:52.731 INFO:teuthology.orchestra.run.smithi114.stdout:Package librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:37:52.771 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:37:52.772 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:52.772 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-09-16T04:37:52.772 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:52.772 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-09-16T04:37:52.772 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T04:37:52.772 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:52.772 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-09-16T04:37:52.773 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:52.773 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-09-16T04:37:52.773 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:52.773 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 32 k 2024-09-16T04:37:52.773 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 142 k 2024-09-16T04:37:52.773 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-09-16T04:37:52.774 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:37:52.775 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-16T04:37:52.775 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:37:52.851 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librbd1 2024-09-16T04:37:52.923 INFO:teuthology.orchestra.run.smithi046.stdout:libcephfs-devel-19.1.1-326.g8921c8ef.el9.x86_64 211 kB/s | 32 kB 00:00 2024-09-16T04:37:52.924 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:52.924 INFO:teuthology.orchestra.run.smithi046.stdout:Total 209 kB/s | 32 kB 00:00 2024-09-16T04:37:52.924 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-09-16T04:37:52.928 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-09-16T04:37:52.928 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-09-16T04:37:52.946 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-09-16T04:37:52.946 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-09-16T04:37:53.038 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-09-16T04:37:53.157 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:37:53.424 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:37:53.515 INFO:teuthology.orchestra.run.smithi114.stdout:Package librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:37:53.516 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:37:53.559 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:37:53.560 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-16T04:37:53.560 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:37:53.631 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rados 2024-09-16T04:37:53.774 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:37:53.774 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:53.774 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-09-16T04:37:53.774 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:53.774 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:53.774 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:37:53.939 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install librados2 2024-09-16T04:37:54.203 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:37:54.294 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:37:54.336 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:37:54.337 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-16T04:37:54.337 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:37:54.415 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rgw 2024-09-16T04:37:54.533 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:37:54.627 INFO:teuthology.orchestra.run.smithi046.stdout:Package librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:37:54.673 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:37:54.674 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-09-16T04:37:54.674 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:37:54.731 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install librbd1 2024-09-16T04:37:54.986 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:37:55.079 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:37:55.123 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:37:55.124 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-16T04:37:55.124 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:37:55.196 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-cephfs 2024-09-16T04:37:55.324 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:37:55.418 INFO:teuthology.orchestra.run.smithi046.stdout:Package librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:37:55.464 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:37:55.465 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-09-16T04:37:55.465 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:37:55.523 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-rados 2024-09-16T04:37:55.762 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:37:55.854 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:37:55.897 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:37:55.898 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-16T04:37:55.898 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:37:55.973 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rbd 2024-09-16T04:37:56.120 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:37:56.215 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:37:56.261 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:37:56.262 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-09-16T04:37:56.262 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:37:56.327 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-rgw 2024-09-16T04:37:56.546 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:37:56.638 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:37:56.682 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:37:56.683 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-16T04:37:56.683 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:37:56.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-fuse 2024-09-16T04:37:56.919 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:37:57.013 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:37:57.059 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:37:57.060 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-09-16T04:37:57.060 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:37:57.126 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-cephfs 2024-09-16T04:37:57.324 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:37:57.459 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:37:57.460 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:57.460 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-09-16T04:37:57.460 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:57.460 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T04:37:57.460 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T04:37:57.461 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:57.461 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T04:37:57.461 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:57.461 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-16T04:37:57.461 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:57.461 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 85 k 2024-09-16T04:37:57.461 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 231 k 2024-09-16T04:37:57.461 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T04:37:57.722 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:37:57.815 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:37:57.838 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-fuse-19.1.1-326.g8921c8ef.el9.x86_64.rpm 226 kB/s | 85 kB 00:00 2024-09-16T04:37:57.838 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:57.839 INFO:teuthology.orchestra.run.smithi114.stdout:Total 225 kB/s | 85 kB 00:00 2024-09-16T04:37:57.839 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T04:37:57.846 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T04:37:57.846 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T04:37:57.861 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:37:57.862 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-09-16T04:37:57.862 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:37:57.910 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T04:37:57.910 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T04:37:57.934 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-rbd 2024-09-16T04:37:58.060 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T04:37:58.160 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:37:58.539 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:37:58.562 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:37:58.637 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:37:58.684 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:37:58.685 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-09-16T04:37:58.685 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:37:58.751 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install rbd-fuse 2024-09-16T04:37:58.809 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:37:58.809 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:58.809 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T04:37:58.809 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:37:58.810 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:58.810 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:37:59.093 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-mirror 2024-09-16T04:37:59.350 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:37:59.492 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:37:59.493 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:59.493 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-09-16T04:37:59.493 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:59.493 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-09-16T04:37:59.493 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T04:37:59.493 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:59.493 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-09-16T04:37:59.494 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:37:59.494 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-09-16T04:37:59.494 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:37:59.494 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 85 k 2024-09-16T04:37:59.494 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 231 k 2024-09-16T04:37:59.494 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-09-16T04:37:59.688 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:37:59.834 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:37:59.835 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:59.835 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-09-16T04:37:59.835 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:59.835 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T04:37:59.835 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T04:37:59.835 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:59.835 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T04:37:59.835 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:37:59.835 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-16T04:37:59.835 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:37:59.836 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 3.1 M 2024-09-16T04:37:59.836 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 13 M 2024-09-16T04:37:59.836 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T04:37:59.877 INFO:teuthology.orchestra.run.smithi046.stdout:rbd-fuse-19.1.1-326.g8921c8ef.el9.x86_64.rpm 223 kB/s | 85 kB 00:00 2024-09-16T04:37:59.877 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:37:59.877 INFO:teuthology.orchestra.run.smithi046.stdout:Total 222 kB/s | 85 kB 00:00 2024-09-16T04:37:59.878 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-09-16T04:37:59.886 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-09-16T04:37:59.886 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-09-16T04:37:59.954 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-09-16T04:37:59.954 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-09-16T04:38:00.221 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-09-16T04:38:00.328 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:38:00.853 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:38:00.867 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-mirror-19.1.1-326.g8921c8ef.el9.x86_64.rpm 3.0 MB/s | 3.1 MB 00:01 2024-09-16T04:38:00.867 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:38:00.868 INFO:teuthology.orchestra.run.smithi114.stdout:Total 3.0 MB/s | 3.1 MB 00:01 2024-09-16T04:38:00.868 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T04:38:00.877 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T04:38:00.877 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T04:38:00.958 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T04:38:00.958 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T04:38:01.115 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:38:01.116 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:01.116 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-09-16T04:38:01.116 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:38:01.116 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:01.116 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:38:01.398 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install rbd-mirror 2024-09-16T04:38:01.402 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T04:38:01.468 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:38:01.494 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:38:01.494 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:38:01.494 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T04:38:01.494 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T04:38:01.495 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T04:38:01.495 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:02.000 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:38:02.141 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:38:02.142 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:38:02.142 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-09-16T04:38:02.142 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:38:02.142 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-09-16T04:38:02.142 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T04:38:02.142 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:02.142 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-09-16T04:38:02.143 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:38:02.143 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-09-16T04:38:02.143 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:02.143 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 3.1 M 2024-09-16T04:38:02.143 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 13 M 2024-09-16T04:38:02.143 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-09-16T04:38:02.638 INFO:teuthology.orchestra.run.smithi046.stdout:rbd-mirror-19.1.1-326.g8921c8ef.el9.x86_64.rpm 6.3 MB/s | 3.1 MB 00:00 2024-09-16T04:38:02.639 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:38:02.639 INFO:teuthology.orchestra.run.smithi046.stdout:Total 6.2 MB/s | 3.1 MB 00:00 2024-09-16T04:38:02.639 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-09-16T04:38:02.649 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-09-16T04:38:02.650 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-09-16T04:38:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:38:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:02.677 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T04:38:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:38:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:02.677 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:38:02.736 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-09-16T04:38:02.736 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-09-16T04:38:02.953 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-nbd 2024-09-16T04:38:03.200 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-09-16T04:38:03.264 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:38:03.292 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:38:03.292 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:38:03.292 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T04:38:03.292 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T04:38:03.292 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T04:38:03.292 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:03.534 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 04:36:16 AM UTC. 2024-09-16T04:38:03.667 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T04:38:03.668 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:38:03.668 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-09-16T04:38:03.668 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:38:03.668 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T04:38:03.668 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T04:38:03.668 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:03.668 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T04:38:03.669 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T04:38:03.669 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-16T04:38:03.669 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:03.669 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 170 k 2024-09-16T04:38:03.669 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 494 k 2024-09-16T04:38:03.669 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T04:38:03.878 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_64.rpm 814 kB/s | 170 kB 00:00 2024-09-16T04:38:03.879 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:38:03.879 INFO:teuthology.orchestra.run.smithi114.stdout:Total 809 kB/s | 170 kB 00:00 2024-09-16T04:38:03.879 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T04:38:03.887 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T04:38:03.887 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T04:38:03.952 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T04:38:03.952 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T04:38:04.102 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T04:38:04.208 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:38:04.366 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:38:04.366 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:04.366 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-09-16T04:38:04.366 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:38:04.366 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:04.366 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:38:04.612 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install rbd-nbd 2024-09-16T04:38:04.685 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:38:04.953 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:38:04.953 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:04.953 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T04:38:04.953 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:38:04.953 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:04.954 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T04:38:05.211 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:51 ago on Mon 16 Sep 2024 04:36:14 AM UTC. 2024-09-16T04:38:05.292 DEBUG:teuthology.parallel:result is None 2024-09-16T04:38:05.353 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-09-16T04:38:05.353 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:38:05.353 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-09-16T04:38:05.354 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:38:05.354 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-09-16T04:38:05.354 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T04:38:05.354 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:05.354 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-09-16T04:38:05.354 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-09-16T04:38:05.354 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-09-16T04:38:05.354 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:05.354 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 170 k 2024-09-16T04:38:05.354 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 494 k 2024-09-16T04:38:05.354 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-09-16T04:38:05.726 INFO:teuthology.orchestra.run.smithi046.stdout:rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_64.rpm 459 kB/s | 170 kB 00:00 2024-09-16T04:38:05.727 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:38:05.727 INFO:teuthology.orchestra.run.smithi046.stdout:Total 457 kB/s | 170 kB 00:00 2024-09-16T04:38:05.727 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-09-16T04:38:05.736 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-09-16T04:38:05.736 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-09-16T04:38:05.805 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-09-16T04:38:05.805 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-09-16T04:38:05.965 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-09-16T04:38:06.076 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:38:06.519 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:38:06.807 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:38:06.807 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:06.807 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-09-16T04:38:06.807 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:38:06.808 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:06.808 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-09-16T04:38:07.090 DEBUG:teuthology.parallel:result is None 2024-09-16T04:38:07.090 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:38:07.236 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T04:38:07.259 INFO:teuthology.orchestra.run.smithi046.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T04:38:07.260 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T04:38:07.260 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T04:38:07.262 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:38:07.408 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T04:38:07.433 INFO:teuthology.orchestra.run.smithi114.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T04:38:07.433 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T04:38:07.433 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T04:38:07.435 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T04:38:07.435 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T04:38:07.435 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T04:38:07.464 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T04:38:07.464 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T04:38:07.503 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T04:38:07.504 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T04:38:07.504 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T04:38:07.532 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T04:38:07.599 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T04:38:07.599 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T04:38:07.629 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T04:38:07.699 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T04:38:07.699 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T04:38:07.699 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T04:38:07.728 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T04:38:07.796 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T04:38:07.796 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T04:38:07.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T04:38:07.893 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T04:38:07.893 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T04:38:07.893 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T04:38:07.921 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T04:38:07.989 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T04:38:07.989 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T04:38:08.018 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T04:38:08.086 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T04:38:08.169 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T04:38:08.169 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 777 /var/log/ceph 2024-09-16T04:38:08.172 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /var/log/ceph 2024-09-16T04:38:08.199 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T04:38:08.199 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T04:38:08.241 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T04:38:08.266 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T04:38:08.267 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T04:38:08.309 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T04:38:08.343 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T04:38:08.344 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'slow request', 'overall HEALTH_'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T04:38:08.344 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906031', 'branch': 'squid', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/many workloads/pool-create-delete}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7906031', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'slow request', 'overall HEALTH_'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 700, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 5616, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'sleep_before_teardown': 0, 'subset': '15927/100000', 'suite': 'rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4', 'targets': {'smithi046.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNMLqXwawDOhk5uLrKgnesDWCzWHSdcfDOfa/3p9Posol0v5qa0BAvoS1qWIDkZRfdk/yXM9lBElVBQSehJO1jQ=', 'smithi114.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE+GRyLy3+p2Go8uZ+tF/JqOLdZIsCS79Rnej/qSyoEotA1p9KvZcCEfdhMrhw62EGFDhE8bwRlwvsBmk5AOsEc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'freeze_mon_duration': 20, 'freeze_mon_probability': 10, 'revive_delay': 20, 'thrash_delay': 1, 'thrash_many': True}}, {'exec': {'client.0': ['ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_21:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T04:38:08.344 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T04:38:08.376 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T04:38:08.392 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T04:38:08.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T04:38:08.459 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T04:38:08.459 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T04:38:08.516 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T04:38:08.516 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_1 2024-09-16T04:38:08.574 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T04:38:08.575 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:38:08.575 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-16T04:38:08.575 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:38:08.575 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:38:08.576 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-09-16 04:38:03.546767484 +0000 2024-09-16T04:38:08.576 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-09-16 04:35:34.593882555 +0000 2024-09-16T04:38:08.576 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-09-16 04:35:34.593882555 +0000 2024-09-16T04:38:08.576 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-09-16 04:35:34.593882555 +0000 2024-09-16T04:38:08.577 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T04:38:08.640 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-09-16T04:38:08.640 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-09-16T04:38:08.640 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000101883 s, 5.0 MB/s 2024-09-16T04:38:08.641 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T04:38:08.700 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_2 2024-09-16T04:38:08.757 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T04:38:08.757 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:38:08.757 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-16T04:38:08.757 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:38:08.757 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:38:08.757 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-09-16 04:38:03.546767484 +0000 2024-09-16T04:38:08.757 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-09-16 04:35:34.921879321 +0000 2024-09-16T04:38:08.757 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-09-16 04:35:34.921879321 +0000 2024-09-16T04:38:08.757 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-09-16 04:35:34.921879321 +0000 2024-09-16T04:38:08.757 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T04:38:08.825 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-09-16T04:38:08.826 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-09-16T04:38:08.826 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000108654 s, 4.7 MB/s 2024-09-16T04:38:08.827 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T04:38:08.885 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_3 2024-09-16T04:38:08.943 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T04:38:08.943 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:38:08.943 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-16T04:38:08.944 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:38:08.944 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:38:08.944 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-09-16 04:38:03.547767472 +0000 2024-09-16T04:38:08.944 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-09-16 04:35:35.255876027 +0000 2024-09-16T04:38:08.944 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-09-16 04:35:35.255876027 +0000 2024-09-16T04:38:08.944 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-09-16 04:35:35.255876027 +0000 2024-09-16T04:38:08.944 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T04:38:09.009 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-09-16T04:38:09.009 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-09-16T04:38:09.010 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000103371 s, 5.0 MB/s 2024-09-16T04:38:09.011 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T04:38:09.068 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_4 2024-09-16T04:38:09.124 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T04:38:09.124 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:38:09.124 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-16T04:38:09.125 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:38:09.125 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:38:09.125 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-09-16 04:38:03.547767472 +0000 2024-09-16T04:38:09.125 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-09-16 04:35:35.585872773 +0000 2024-09-16T04:38:09.125 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-09-16 04:35:35.585872773 +0000 2024-09-16T04:38:09.125 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-09-16 04:35:35.585872773 +0000 2024-09-16T04:38:09.125 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T04:38:09.189 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-09-16T04:38:09.189 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-09-16T04:38:09.190 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000121645 s, 4.2 MB/s 2024-09-16T04:38:09.191 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T04:38:09.248 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T04:38:09.248 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T04:38:09.248 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T04:38:09.266 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T04:38:09.267 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2024-09-16T04:38:09.323 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T04:38:09.324 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:38:09.324 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-16T04:38:09.324 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:38:09.324 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:38:09.324 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-16 04:38:01.781944946 +0000 2024-09-16T04:38:09.324 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-16 04:35:27.462018638 +0000 2024-09-16T04:38:09.324 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-16 04:35:27.462018638 +0000 2024-09-16T04:38:09.324 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-16 04:35:27.462018638 +0000 2024-09-16T04:38:09.324 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T04:38:09.389 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-16T04:38:09.389 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-16T04:38:09.389 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000203377 s, 2.5 MB/s 2024-09-16T04:38:09.390 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T04:38:09.448 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2024-09-16T04:38:09.505 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T04:38:09.505 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:38:09.505 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-16T04:38:09.505 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:38:09.505 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:38:09.506 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-16 04:38:01.782944925 +0000 2024-09-16T04:38:09.506 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-16 04:35:27.802011442 +0000 2024-09-16T04:38:09.506 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-16 04:35:27.802011442 +0000 2024-09-16T04:38:09.506 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-16 04:35:27.802011442 +0000 2024-09-16T04:38:09.506 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T04:38:09.570 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-16T04:38:09.570 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-16T04:38:09.570 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000184325 s, 2.8 MB/s 2024-09-16T04:38:09.571 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T04:38:09.628 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2024-09-16T04:38:09.684 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T04:38:09.684 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:38:09.684 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-16T04:38:09.684 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:38:09.685 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:38:09.685 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-16 04:38:01.782944925 +0000 2024-09-16T04:38:09.685 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-16 04:35:28.114004838 +0000 2024-09-16T04:38:09.685 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-16 04:35:28.114004838 +0000 2024-09-16T04:38:09.685 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-16 04:35:28.114004838 +0000 2024-09-16T04:38:09.685 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T04:38:09.748 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-16T04:38:09.748 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-16T04:38:09.748 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000171621 s, 3.0 MB/s 2024-09-16T04:38:09.750 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T04:38:09.807 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2024-09-16T04:38:09.864 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T04:38:09.864 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:38:09.864 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-16T04:38:09.864 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:38:09.864 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:38:09.864 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-16 04:38:01.782944925 +0000 2024-09-16T04:38:09.864 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-16 04:35:28.446997790 +0000 2024-09-16T04:38:09.864 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-16 04:35:28.446997790 +0000 2024-09-16T04:38:09.865 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-16 04:35:28.446997790 +0000 2024-09-16T04:38:09.865 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T04:38:09.928 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-16T04:38:09.928 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-16T04:38:09.928 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000214981 s, 2.4 MB/s 2024-09-16T04:38:09.930 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T04:38:09.985 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T04:38:09.986 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi046.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi114.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T04:38:09.986 INFO:tasks.ceph:Generating config... 2024-09-16T04:38:09.987 INFO:tasks.ceph:[client] debug monc = 20 2024-09-16T04:38:09.987 INFO:tasks.ceph:[client] debug ms = 1 2024-09-16T04:38:09.987 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T04:38:09.987 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-16T04:38:09.987 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T04:38:09.988 INFO:tasks.ceph:[global] ms type = async 2024-09-16T04:38:09.988 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T04:38:09.988 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T04:38:09.988 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T04:38:09.988 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T04:38:09.988 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T04:38:09.988 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-16T04:38:09.988 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-16T04:38:09.988 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-16T04:38:09.988 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-16T04:38:09.988 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-16T04:38:09.989 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-16T04:38:09.989 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-16T04:38:09.989 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-16T04:38:09.989 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-09-16T04:38:09.991 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-09-16T04:38:09.991 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T04:38:09.991 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T04:38:09.991 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-16T04:38:09.992 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T04:38:09.992 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T04:38:09.992 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T04:38:09.992 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T04:38:09.992 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T04:38:09.992 INFO:tasks.ceph:[osd] mon client ping interval = 4 2024-09-16T04:38:09.992 INFO:tasks.ceph:[osd] mon client ping timeout = 12 2024-09-16T04:38:09.992 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T04:38:09.993 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T04:38:09.993 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T04:38:09.993 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-16T04:38:09.993 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-16T04:38:09.993 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T04:38:09.993 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-16T04:38:09.993 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-16T04:38:09.993 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T04:38:09.993 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-16T04:38:09.993 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-16T04:38:09.993 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T04:38:09.994 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T04:38:10.050 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T04:38:10.053 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T04:38:10.099 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T04:38:10.126 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.46'), ('mon.c', '[v2:172.21.15.46:3301,v1:172.21.15.46:6790]'), ('mon.b', '172.21.15.114')] 2024-09-16T04:38:10.126 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.46,[v2:172.21.15.46:3301,v1:172.21.15.46:6790],172.21.15.114', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-16T04:38:10.127 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T04:38:10.127 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T04:38:10.182 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.46 --addv c '[v2:172.21.15.46:3301,v1:172.21.15.46:6790]' --add b 172.21.15.114 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:38:10.270 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:38:10.270 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: generated fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:10.270 INFO:teuthology.orchestra.run.smithi046.stdout:setting min_mon_release = quincy 2024-09-16T04:38:10.270 INFO:teuthology.orchestra.run.smithi046.stdout:epoch 0 2024-09-16T04:38:10.271 INFO:teuthology.orchestra.run.smithi046.stdout:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:10.271 INFO:teuthology.orchestra.run.smithi046.stdout:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:10.271 INFO:teuthology.orchestra.run.smithi046.stdout:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:10.271 INFO:teuthology.orchestra.run.smithi046.stdout:min_mon_release 17 (quincy) 2024-09-16T04:38:10.271 INFO:teuthology.orchestra.run.smithi046.stdout:election_strategy: 1 2024-09-16T04:38:10.271 INFO:teuthology.orchestra.run.smithi046.stdout:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:10.271 INFO:teuthology.orchestra.run.smithi046.stdout:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:10.271 INFO:teuthology.orchestra.run.smithi046.stdout:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:10.271 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T04:38:10.273 DEBUG:teuthology.orchestra.run.smithi046:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T04:38:10.289 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 01e423f6-9e74-4519-981b-340ad8faba4f... 2024-09-16T04:38:10.290 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T04:38:10.331 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T04:38:10.373 INFO:teuthology.orchestra.run.smithi046.stdout:[global] 2024-09-16T04:38:10.374 INFO:teuthology.orchestra.run.smithi046.stdout: chdir = "" 2024-09-16T04:38:10.374 INFO:teuthology.orchestra.run.smithi046.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T04:38:10.374 INFO:teuthology.orchestra.run.smithi046.stdout: auth supported = cephx 2024-09-16T04:38:10.374 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.374 INFO:teuthology.orchestra.run.smithi046.stdout: filestore xattr use omap = true 2024-09-16T04:38:10.374 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.374 INFO:teuthology.orchestra.run.smithi046.stdout: mon clock drift allowed = 1.000 2024-09-16T04:38:10.374 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.374 INFO:teuthology.orchestra.run.smithi046.stdout: osd crush chooseleaf type = 0 2024-09-16T04:38:10.374 INFO:teuthology.orchestra.run.smithi046.stdout: auth debug = true 2024-09-16T04:38:10.374 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.374 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on old message = true 2024-09-16T04:38:10.375 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on bug = true 2024-09-16T04:38:10.375 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.375 INFO:teuthology.orchestra.run.smithi046.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T04:38:10.375 INFO:teuthology.orchestra.run.smithi046.stdout: mon pg warn max object skew = 0 2024-09-16T04:38:10.375 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.375 INFO:teuthology.orchestra.run.smithi046.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T04:38:10.375 INFO:teuthology.orchestra.run.smithi046.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T04:38:10.375 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.375 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default size = 2 2024-09-16T04:38:10.375 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.375 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow primary affinity = true 2024-09-16T04:38:10.375 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow pg remap = true 2024-09-16T04:38:10.375 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on legacy crush tunables = false 2024-09-16T04:38:10.375 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on crush straw calc version zero = false 2024-09-16T04:38:10.376 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on no sortbitwise = false 2024-09-16T04:38:10.376 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on osd down out interval zero = false 2024-09-16T04:38:10.376 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on too few osds = false 2024-09-16T04:38:10.376 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T04:38:10.376 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T04:38:10.376 INFO:teuthology.orchestra.run.smithi046.stdout: mon_allow_pool_size_one = true 2024-09-16T04:38:10.376 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.376 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T04:38:10.376 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.376 INFO:teuthology.orchestra.run.smithi046.stdout: osd default data pool replay window = 5 2024-09-16T04:38:10.376 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.376 INFO:teuthology.orchestra.run.smithi046.stdout: mon allow pool delete = true 2024-09-16T04:38:10.376 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.377 INFO:teuthology.orchestra.run.smithi046.stdout: mon cluster log file level = debug 2024-09-16T04:38:10.377 INFO:teuthology.orchestra.run.smithi046.stdout: debug asserts on shutdown = true 2024-09-16T04:38:10.377 INFO:teuthology.orchestra.run.smithi046.stdout: mon health detail to clog = false 2024-09-16T04:38:10.377 INFO:teuthology.orchestra.run.smithi046.stdout: mon host = "172.21.15.46,[v2:172.21.15.46:3301,v1:172.21.15.46:6790],172.21.15.114" 2024-09-16T04:38:10.377 INFO:teuthology.orchestra.run.smithi046.stdout: mon client directed command retry = 5 2024-09-16T04:38:10.377 INFO:teuthology.orchestra.run.smithi046.stdout: mon election default strategy = 1 2024-09-16T04:38:10.377 INFO:teuthology.orchestra.run.smithi046.stdout: ms inject socket failures = 5000 2024-09-16T04:38:10.377 INFO:teuthology.orchestra.run.smithi046.stdout: ms type = async 2024-09-16T04:38:10.377 INFO:teuthology.orchestra.run.smithi046.stdout: fsid = 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:10.377 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.377 INFO:teuthology.orchestra.run.smithi046.stdout:[osd] 2024-09-16T04:38:10.377 INFO:teuthology.orchestra.run.smithi046.stdout: osd journal size = 100 2024-09-16T04:38:10.377 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.377 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub load threshold = 5.0 2024-09-16T04:38:10.378 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub max interval = 600 2024-09-16T04:38:10.378 INFO:teuthology.orchestra.run.smithi046.stdout: osd mclock profile = high_recovery_ops 2024-09-16T04:38:10.378 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.378 INFO:teuthology.orchestra.run.smithi046.stdout: osd recover clone overlap = true 2024-09-16T04:38:10.378 INFO:teuthology.orchestra.run.smithi046.stdout: osd recovery max chunk = 1048576 2024-09-16T04:38:10.378 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.378 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug shutdown = true 2024-09-16T04:38:10.378 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug op order = true 2024-09-16T04:38:10.378 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug verify stray on activate = true 2024-09-16T04:38:10.378 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug trim objects = true 2024-09-16T04:38:10.378 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.378 INFO:teuthology.orchestra.run.smithi046.stdout: osd open classes on start = true 2024-09-16T04:38:10.378 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug pg log writeout = true 2024-09-16T04:38:10.378 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.379 INFO:teuthology.orchestra.run.smithi046.stdout: osd deep scrub update digest min age = 30 2024-09-16T04:38:10.379 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.379 INFO:teuthology.orchestra.run.smithi046.stdout: osd map max advance = 10 2024-09-16T04:38:10.379 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.379 INFO:teuthology.orchestra.run.smithi046.stdout: journal zero on create = true 2024-09-16T04:38:10.379 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.379 INFO:teuthology.orchestra.run.smithi046.stdout: filestore ondisk finisher threads = 3 2024-09-16T04:38:10.379 INFO:teuthology.orchestra.run.smithi046.stdout: filestore apply finisher threads = 3 2024-09-16T04:38:10.379 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.379 INFO:teuthology.orchestra.run.smithi046.stdout: bdev debug aio = true 2024-09-16T04:38:10.379 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug misdirected ops = true 2024-09-16T04:38:10.379 INFO:teuthology.orchestra.run.smithi046.stdout: bluefs allocator = hybrid 2024-09-16T04:38:10.379 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore allocator = hybrid 2024-09-16T04:38:10.379 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore block size = 96636764160 2024-09-16T04:38:10.380 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore fsck on mount = True 2024-09-16T04:38:10.380 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore zero block detection = True 2024-09-16T04:38:10.380 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluefs = 20 2024-09-16T04:38:10.380 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluestore = 20 2024-09-16T04:38:10.380 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-09-16T04:38:10.380 INFO:teuthology.orchestra.run.smithi046.stdout: debug osd = 20 2024-09-16T04:38:10.380 INFO:teuthology.orchestra.run.smithi046.stdout: debug rocksdb = 10 2024-09-16T04:38:10.380 INFO:teuthology.orchestra.run.smithi046.stdout: mon client ping interval = 4 2024-09-16T04:38:10.380 INFO:teuthology.orchestra.run.smithi046.stdout: mon client ping timeout = 12 2024-09-16T04:38:10.380 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T04:38:10.380 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd full ratio = 0.9 2024-09-16T04:38:10.380 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd nearfull ratio = 0.8 2024-09-16T04:38:10.380 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug verify cached snaps = True 2024-09-16T04:38:10.381 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug verify missing on start = True 2024-09-16T04:38:10.381 INFO:teuthology.orchestra.run.smithi046.stdout: osd failsafe full ratio = 0.95 2024-09-16T04:38:10.381 INFO:teuthology.orchestra.run.smithi046.stdout: osd mclock override recovery settings = True 2024-09-16T04:38:10.381 INFO:teuthology.orchestra.run.smithi046.stdout: osd objectstore = bluestore 2024-09-16T04:38:10.381 INFO:teuthology.orchestra.run.smithi046.stdout: osd op queue = debug_random 2024-09-16T04:38:10.381 INFO:teuthology.orchestra.run.smithi046.stdout: osd op queue cut off = debug_random 2024-09-16T04:38:10.381 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.381 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr] 2024-09-16T04:38:10.381 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-09-16T04:38:10.381 INFO:teuthology.orchestra.run.smithi046.stdout: debug mgr = 20 2024-09-16T04:38:10.381 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 20 2024-09-16T04:38:10.382 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2024-09-16T04:38:10.382 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:38:10.382 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:38:10.382 INFO:teuthology.orchestra.run.smithi046.stdout: mgr/telemetry/nag = false 2024-09-16T04:38:10.382 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.382 INFO:teuthology.orchestra.run.smithi046.stdout:[mon] 2024-09-16T04:38:10.382 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-09-16T04:38:10.382 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 20 2024-09-16T04:38:10.382 INFO:teuthology.orchestra.run.smithi046.stdout: debug paxos = 20 2024-09-16T04:38:10.382 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2024-09-16T04:38:10.382 INFO:teuthology.orchestra.run.smithi046.stdout: mon data avail warn = 5 2024-09-16T04:38:10.382 INFO:teuthology.orchestra.run.smithi046.stdout: mon mgr mkfs grace = 240 2024-09-16T04:38:10.382 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:38:10.382 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd reporter subtree level = osd 2024-09-16T04:38:10.383 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd prime pg temp = true 2024-09-16T04:38:10.383 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:38:10.383 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.383 INFO:teuthology.orchestra.run.smithi046.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T04:38:10.383 INFO:teuthology.orchestra.run.smithi046.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T04:38:10.383 INFO:teuthology.orchestra.run.smithi046.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T04:38:10.383 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.383 INFO:teuthology.orchestra.run.smithi046.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T04:38:10.383 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T04:38:10.383 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T04:38:10.383 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.383 INFO:teuthology.orchestra.run.smithi046.stdout: # 1m isn't quite enough 2024-09-16T04:38:10.383 INFO:teuthology.orchestra.run.smithi046.stdout: mon_down_mkfs_grace = 2m 2024-09-16T04:38:10.383 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.384 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_filestore_osds = false 2024-09-16T04:38:10.384 INFO:teuthology.orchestra.run.smithi046.stdout: mon min osdmap epochs = 25 2024-09-16T04:38:10.384 INFO:teuthology.orchestra.run.smithi046.stdout: mon osdmap full prune interval = 2 2024-09-16T04:38:10.384 INFO:teuthology.orchestra.run.smithi046.stdout: mon osdmap full prune min = 15 2024-09-16T04:38:10.384 INFO:teuthology.orchestra.run.smithi046.stdout: mon osdmap full prune txsize = 2 2024-09-16T04:38:10.384 INFO:teuthology.orchestra.run.smithi046.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T04:38:10.384 INFO:teuthology.orchestra.run.smithi046.stdout: mon scrub inject missing keys = 0.05 2024-09-16T04:38:10.384 INFO:teuthology.orchestra.run.smithi046.stdout: mon scrub interval = 300 2024-09-16T04:38:10.384 INFO:teuthology.orchestra.run.smithi046.stdout: paxos service trim min = 5 2024-09-16T04:38:10.384 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:10.384 INFO:teuthology.orchestra.run.smithi046.stdout:[client] 2024-09-16T04:38:10.384 INFO:teuthology.orchestra.run.smithi046.stdout: rgw cache enabled = true 2024-09-16T04:38:10.384 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable ops log = true 2024-09-16T04:38:10.385 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable usage log = true 2024-09-16T04:38:10.385 INFO:teuthology.orchestra.run.smithi046.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T04:38:10.385 INFO:teuthology.orchestra.run.smithi046.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T04:38:10.385 INFO:teuthology.orchestra.run.smithi046.stdout: debug monc = 20 2024-09-16T04:38:10.385 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-09-16T04:38:10.385 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.a] 2024-09-16T04:38:10.385 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.c] 2024-09-16T04:38:10.385 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.b] 2024-09-16T04:38:10.387 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2024-09-16T04:38:10.387 INFO:teuthology.orchestra.run.smithi114.stdout: chdir = "" 2024-09-16T04:38:10.387 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T04:38:10.387 INFO:teuthology.orchestra.run.smithi114.stdout: auth supported = cephx 2024-09-16T04:38:10.387 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.387 INFO:teuthology.orchestra.run.smithi114.stdout: filestore xattr use omap = true 2024-09-16T04:38:10.387 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.387 INFO:teuthology.orchestra.run.smithi114.stdout: mon clock drift allowed = 1.000 2024-09-16T04:38:10.387 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.387 INFO:teuthology.orchestra.run.smithi114.stdout: osd crush chooseleaf type = 0 2024-09-16T04:38:10.387 INFO:teuthology.orchestra.run.smithi114.stdout: auth debug = true 2024-09-16T04:38:10.387 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.388 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on old message = true 2024-09-16T04:38:10.388 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on bug = true 2024-09-16T04:38:10.388 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.388 INFO:teuthology.orchestra.run.smithi114.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T04:38:10.388 INFO:teuthology.orchestra.run.smithi114.stdout: mon pg warn max object skew = 0 2024-09-16T04:38:10.388 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.388 INFO:teuthology.orchestra.run.smithi114.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T04:38:10.388 INFO:teuthology.orchestra.run.smithi114.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T04:38:10.388 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.388 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default size = 2 2024-09-16T04:38:10.388 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.388 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow primary affinity = true 2024-09-16T04:38:10.388 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow pg remap = true 2024-09-16T04:38:10.389 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on legacy crush tunables = false 2024-09-16T04:38:10.389 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on crush straw calc version zero = false 2024-09-16T04:38:10.389 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on no sortbitwise = false 2024-09-16T04:38:10.389 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on osd down out interval zero = false 2024-09-16T04:38:10.389 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on too few osds = false 2024-09-16T04:38:10.389 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T04:38:10.389 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T04:38:10.389 INFO:teuthology.orchestra.run.smithi114.stdout: mon_allow_pool_size_one = true 2024-09-16T04:38:10.389 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.389 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T04:38:10.389 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.389 INFO:teuthology.orchestra.run.smithi114.stdout: osd default data pool replay window = 5 2024-09-16T04:38:10.389 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.390 INFO:teuthology.orchestra.run.smithi114.stdout: mon allow pool delete = true 2024-09-16T04:38:10.390 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.390 INFO:teuthology.orchestra.run.smithi114.stdout: mon cluster log file level = debug 2024-09-16T04:38:10.390 INFO:teuthology.orchestra.run.smithi114.stdout: debug asserts on shutdown = true 2024-09-16T04:38:10.390 INFO:teuthology.orchestra.run.smithi114.stdout: mon health detail to clog = false 2024-09-16T04:38:10.390 INFO:teuthology.orchestra.run.smithi114.stdout: mon host = "172.21.15.46,[v2:172.21.15.46:3301,v1:172.21.15.46:6790],172.21.15.114" 2024-09-16T04:38:10.390 INFO:teuthology.orchestra.run.smithi114.stdout: mon client directed command retry = 5 2024-09-16T04:38:10.390 INFO:teuthology.orchestra.run.smithi114.stdout: mon election default strategy = 1 2024-09-16T04:38:10.390 INFO:teuthology.orchestra.run.smithi114.stdout: ms inject socket failures = 5000 2024-09-16T04:38:10.390 INFO:teuthology.orchestra.run.smithi114.stdout: ms type = async 2024-09-16T04:38:10.390 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:10.390 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.390 INFO:teuthology.orchestra.run.smithi114.stdout:[osd] 2024-09-16T04:38:10.390 INFO:teuthology.orchestra.run.smithi114.stdout: osd journal size = 100 2024-09-16T04:38:10.391 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.391 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub load threshold = 5.0 2024-09-16T04:38:10.391 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub max interval = 600 2024-09-16T04:38:10.391 INFO:teuthology.orchestra.run.smithi114.stdout: osd mclock profile = high_recovery_ops 2024-09-16T04:38:10.391 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.391 INFO:teuthology.orchestra.run.smithi114.stdout: osd recover clone overlap = true 2024-09-16T04:38:10.391 INFO:teuthology.orchestra.run.smithi114.stdout: osd recovery max chunk = 1048576 2024-09-16T04:38:10.391 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.391 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug shutdown = true 2024-09-16T04:38:10.391 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug op order = true 2024-09-16T04:38:10.391 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify stray on activate = true 2024-09-16T04:38:10.391 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug trim objects = true 2024-09-16T04:38:10.392 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.392 INFO:teuthology.orchestra.run.smithi114.stdout: osd open classes on start = true 2024-09-16T04:38:10.392 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug pg log writeout = true 2024-09-16T04:38:10.392 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.392 INFO:teuthology.orchestra.run.smithi114.stdout: osd deep scrub update digest min age = 30 2024-09-16T04:38:10.392 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.392 INFO:teuthology.orchestra.run.smithi114.stdout: osd map max advance = 10 2024-09-16T04:38:10.392 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.392 INFO:teuthology.orchestra.run.smithi114.stdout: journal zero on create = true 2024-09-16T04:38:10.392 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.392 INFO:teuthology.orchestra.run.smithi114.stdout: filestore ondisk finisher threads = 3 2024-09-16T04:38:10.392 INFO:teuthology.orchestra.run.smithi114.stdout: filestore apply finisher threads = 3 2024-09-16T04:38:10.392 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.393 INFO:teuthology.orchestra.run.smithi114.stdout: bdev debug aio = true 2024-09-16T04:38:10.393 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug misdirected ops = true 2024-09-16T04:38:10.393 INFO:teuthology.orchestra.run.smithi114.stdout: bluefs allocator = hybrid 2024-09-16T04:38:10.393 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore allocator = hybrid 2024-09-16T04:38:10.393 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore block size = 96636764160 2024-09-16T04:38:10.393 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore fsck on mount = True 2024-09-16T04:38:10.393 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore zero block detection = True 2024-09-16T04:38:10.393 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluefs = 20 2024-09-16T04:38:10.393 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluestore = 20 2024-09-16T04:38:10.393 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-09-16T04:38:10.393 INFO:teuthology.orchestra.run.smithi114.stdout: debug osd = 20 2024-09-16T04:38:10.393 INFO:teuthology.orchestra.run.smithi114.stdout: debug rocksdb = 10 2024-09-16T04:38:10.393 INFO:teuthology.orchestra.run.smithi114.stdout: mon client ping interval = 4 2024-09-16T04:38:10.394 INFO:teuthology.orchestra.run.smithi114.stdout: mon client ping timeout = 12 2024-09-16T04:38:10.394 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T04:38:10.394 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd full ratio = 0.9 2024-09-16T04:38:10.394 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd nearfull ratio = 0.8 2024-09-16T04:38:10.394 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify cached snaps = True 2024-09-16T04:38:10.394 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify missing on start = True 2024-09-16T04:38:10.394 INFO:teuthology.orchestra.run.smithi114.stdout: osd failsafe full ratio = 0.95 2024-09-16T04:38:10.394 INFO:teuthology.orchestra.run.smithi114.stdout: osd mclock override recovery settings = True 2024-09-16T04:38:10.394 INFO:teuthology.orchestra.run.smithi114.stdout: osd objectstore = bluestore 2024-09-16T04:38:10.394 INFO:teuthology.orchestra.run.smithi114.stdout: osd op queue = debug_random 2024-09-16T04:38:10.394 INFO:teuthology.orchestra.run.smithi114.stdout: osd op queue cut off = debug_random 2024-09-16T04:38:10.394 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.394 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr] 2024-09-16T04:38:10.394 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-09-16T04:38:10.395 INFO:teuthology.orchestra.run.smithi114.stdout: debug mgr = 20 2024-09-16T04:38:10.395 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-09-16T04:38:10.395 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-09-16T04:38:10.395 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:38:10.395 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:38:10.395 INFO:teuthology.orchestra.run.smithi114.stdout: mgr/telemetry/nag = false 2024-09-16T04:38:10.395 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.395 INFO:teuthology.orchestra.run.smithi114.stdout:[mon] 2024-09-16T04:38:10.395 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-09-16T04:38:10.395 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-09-16T04:38:10.395 INFO:teuthology.orchestra.run.smithi114.stdout: debug paxos = 20 2024-09-16T04:38:10.395 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-09-16T04:38:10.395 INFO:teuthology.orchestra.run.smithi114.stdout: mon data avail warn = 5 2024-09-16T04:38:10.395 INFO:teuthology.orchestra.run.smithi114.stdout: mon mgr mkfs grace = 240 2024-09-16T04:38:10.396 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:38:10.396 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd reporter subtree level = osd 2024-09-16T04:38:10.396 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd prime pg temp = true 2024-09-16T04:38:10.396 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:38:10.396 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.396 INFO:teuthology.orchestra.run.smithi114.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T04:38:10.396 INFO:teuthology.orchestra.run.smithi114.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T04:38:10.396 INFO:teuthology.orchestra.run.smithi114.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T04:38:10.396 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.396 INFO:teuthology.orchestra.run.smithi114.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T04:38:10.396 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T04:38:10.396 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T04:38:10.396 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.397 INFO:teuthology.orchestra.run.smithi114.stdout: # 1m isn't quite enough 2024-09-16T04:38:10.397 INFO:teuthology.orchestra.run.smithi114.stdout: mon_down_mkfs_grace = 2m 2024-09-16T04:38:10.397 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.397 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_filestore_osds = false 2024-09-16T04:38:10.397 INFO:teuthology.orchestra.run.smithi114.stdout: mon min osdmap epochs = 25 2024-09-16T04:38:10.397 INFO:teuthology.orchestra.run.smithi114.stdout: mon osdmap full prune interval = 2 2024-09-16T04:38:10.397 INFO:teuthology.orchestra.run.smithi114.stdout: mon osdmap full prune min = 15 2024-09-16T04:38:10.397 INFO:teuthology.orchestra.run.smithi114.stdout: mon osdmap full prune txsize = 2 2024-09-16T04:38:10.397 INFO:teuthology.orchestra.run.smithi114.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T04:38:10.397 INFO:teuthology.orchestra.run.smithi114.stdout: mon scrub inject missing keys = 0.05 2024-09-16T04:38:10.397 INFO:teuthology.orchestra.run.smithi114.stdout: mon scrub interval = 300 2024-09-16T04:38:10.397 INFO:teuthology.orchestra.run.smithi114.stdout: paxos service trim min = 5 2024-09-16T04:38:10.397 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T04:38:10.397 INFO:teuthology.orchestra.run.smithi114.stdout:[client] 2024-09-16T04:38:10.398 INFO:teuthology.orchestra.run.smithi114.stdout: rgw cache enabled = true 2024-09-16T04:38:10.398 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable ops log = true 2024-09-16T04:38:10.398 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable usage log = true 2024-09-16T04:38:10.398 INFO:teuthology.orchestra.run.smithi114.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T04:38:10.398 INFO:teuthology.orchestra.run.smithi114.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T04:38:10.398 INFO:teuthology.orchestra.run.smithi114.stdout: debug monc = 20 2024-09-16T04:38:10.398 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-09-16T04:38:10.398 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.a] 2024-09-16T04:38:10.398 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.c] 2024-09-16T04:38:10.398 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.b] 2024-09-16T04:38:10.398 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T04:38:10.399 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T04:38:10.483 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T04:38:10.483 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T04:38:10.483 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T04:38:10.499 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T04:38:10.499 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T04:38:10.555 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi046.front.sepia.ceph.com 2024-09-16T04:38:10.555 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T04:38:10.556 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T04:38:10.556 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T04:38:10.631 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T04:38:10.632 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:38:10.687 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T04:38:10.688 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T04:38:10.688 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T04:38:10.688 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T04:38:10.722 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T04:38:10.723 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:38:10.777 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T04:38:10.777 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T04:38:10.778 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T04:38:10.879 INFO:teuthology.orchestra.run.smithi114.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T04:38:10.882 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T04:38:10.882 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T04:38:10.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T04:38:10.924 INFO:teuthology.orchestra.run.smithi114.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T04:38:10.936 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T04:38:10.936 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi046.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi114.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T04:38:10.936 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T04:38:10.963 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T04:38:10.963 INFO:tasks.ceph:role: osd.0 2024-09-16T04:38:10.963 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi046.front.sepia.ceph.com 2024-09-16T04:38:10.963 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T04:38:11.031 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:38:11.032 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:38:11.032 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:38:11.032 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:38:11.032 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:38:11.032 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:38:11.032 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:38:11.032 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:38:11.032 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:38:11.032 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:38:11.061 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-09-16T04:38:11.063 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-09-16T04:38:11.063 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T04:38:11.132 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T04:38:11.204 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T04:38:11.270 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T04:38:11.270 INFO:tasks.ceph:role: osd.1 2024-09-16T04:38:11.270 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi046.front.sepia.ceph.com 2024-09-16T04:38:11.270 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T04:38:11.336 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:38:11.336 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:38:11.336 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:38:11.336 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:38:11.336 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:38:11.336 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:38:11.337 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:38:11.337 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:38:11.337 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:38:11.337 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:38:11.369 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-09-16T04:38:11.372 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-09-16T04:38:11.372 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T04:38:11.442 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T04:38:11.512 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T04:38:11.579 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T04:38:11.579 INFO:tasks.ceph:role: osd.2 2024-09-16T04:38:11.579 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi046.front.sepia.ceph.com 2024-09-16T04:38:11.579 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T04:38:11.644 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:38:11.644 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:38:11.644 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:38:11.644 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:38:11.644 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:38:11.644 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:38:11.644 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:38:11.644 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:38:11.644 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:38:11.644 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:38:11.676 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-09-16T04:38:11.678 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-09-16T04:38:11.678 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T04:38:11.754 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T04:38:11.824 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:38:11.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:11.916+0000 7f8c95c13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T04:38:11.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:11.916+0000 7f8c95c13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T04:38:11.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:11.916+0000 7f8c95c13640 -1 bdev(0x55ce38f2b800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-16T04:38:11.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:11.917+0000 7f8c95c13640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T04:38:14.830 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T04:38:14.861 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:38:14.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:14.954+0000 7f5233aef640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T04:38:14.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:14.954+0000 7f5233aef640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T04:38:14.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:14.955+0000 7f5233aef640 -1 bdev(0x564a98609800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-16T04:38:14.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:14.955+0000 7f5233aef640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T04:38:17.873 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T04:38:17.905 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:38:17.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:17.995+0000 7fe92a60b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T04:38:17.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:17.995+0000 7fe92a60b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T04:38:17.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:17.995+0000 7fe92a60b640 -1 bdev(0x560a661d9800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-16T04:38:17.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:17.995+0000 7fe92a60b640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T04:38:20.916 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T04:38:20.948 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T04:38:20.978 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T04:38:20.978 INFO:tasks.ceph:role: osd.3 2024-09-16T04:38:20.978 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T04:38:20.978 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T04:38:21.056 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:38:21.056 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:38:21.056 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:38:21.056 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:38:21.056 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:38:21.056 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:38:21.056 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:38:21.056 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:38:21.057 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:38:21.057 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:38:21.557 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-09-16T04:38:21.559 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-09-16T04:38:21.560 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-16T04:38:21.702 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T04:38:21.736 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T04:38:21.804 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T04:38:21.804 INFO:tasks.ceph:role: osd.4 2024-09-16T04:38:21.804 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T04:38:21.804 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T04:38:21.878 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:38:21.878 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:38:21.878 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:38:21.878 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:38:21.878 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:38:21.879 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:38:21.879 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:38:21.879 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:38:21.879 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:38:21.879 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:38:22.326 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-09-16T04:38:22.328 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-09-16T04:38:22.328 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-16T04:38:22.422 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T04:38:22.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T04:38:22.515 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T04:38:22.515 INFO:tasks.ceph:role: osd.5 2024-09-16T04:38:22.515 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T04:38:22.515 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T04:38:22.588 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:38:22.604 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:38:22.604 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:38:22.605 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:38:22.605 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:38:22.605 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:38:22.605 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:38:22.605 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:38:22.605 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:38:22.605 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:38:23.071 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-09-16T04:38:23.076 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-09-16T04:38:23.076 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-16T04:38:23.226 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T04:38:23.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:38:23.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:23.408+0000 7f57834f1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T04:38:23.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:23.412+0000 7f57834f1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T04:38:23.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:23.412+0000 7f57834f1640 -1 bdev(0x5565364b3800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-16T04:38:23.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:23.412+0000 7f57834f1640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T04:38:26.333 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T04:38:26.359 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:38:26.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:26.445+0000 7f8b72419640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T04:38:26.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:26.445+0000 7f8b72419640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T04:38:26.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:26.445+0000 7f8b72419640 -1 bdev(0x5581bacb3800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-16T04:38:26.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:26.446+0000 7f8b72419640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T04:38:29.372 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T04:38:29.402 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:38:29.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:29.491+0000 7fbecee16640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T04:38:29.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:29.491+0000 7fbecee16640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T04:38:29.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:29.491+0000 7fbecee16640 -1 bdev(0x5572fa579800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-16T04:38:29.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:29.492+0000 7fbecee16640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T04:38:32.422 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T04:38:32.452 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T04:38:32.453 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T04:38:32.453 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T04:38:32.484 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T04:38:32.484 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T04:38:32.553 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T04:38:32.553 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T04:38:32.622 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T04:38:32.622 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T04:38:32.650 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T04:38:32.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T04:38:32.717 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T04:38:32.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T04:38:32.783 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T04:38:32.783 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T04:38:32.848 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T04:38:32.849 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T04:38:32.904 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T04:38:32.904 DEBUG:teuthology.orchestra.run.smithi046:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T04:38:32.907 DEBUG:teuthology.orchestra.run.smithi114:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T04:38:32.947 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.0] 2024-09-16T04:38:32.948 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAztudmLsm6NhAA8URNq8Ox+P5rSKFdsDRt7g== 2024-09-16T04:38:32.948 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.1] 2024-09-16T04:38:32.948 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2tudmAX3+OBAAAhtq4OSD99a9zPjl7RWqWQ== 2024-09-16T04:38:32.948 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.2] 2024-09-16T04:38:32.948 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA5tudmTcRtOxAAznnxF2zY1Xsz/EPdQPrKng== 2024-09-16T04:38:32.948 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr.x] 2024-09-16T04:38:32.948 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAytudmhnxrNBAAPBlQPqWExHYhBY7d3RDTxg== 2024-09-16T04:38:32.948 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.3] 2024-09-16T04:38:32.948 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA/tudmEw1tGBAAcUQvtihFvH+A41SYVso/dw== 2024-09-16T04:38:32.948 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.4] 2024-09-16T04:38:32.948 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBCtudmA2qdGhAAAz1YASlzVuR7ZGWB0FuqUA== 2024-09-16T04:38:32.948 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.5] 2024-09-16T04:38:32.948 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBFtudmnRVbHRAAGKj7W7gMwjLeRMITOc/lOg== 2024-09-16T04:38:32.949 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2024-09-16T04:38:32.949 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAytudmr04INxAADRxKq9wyHZz7r3U84EyfKQ== 2024-09-16T04:38:32.968 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.0] 2024-09-16T04:38:32.968 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAztudmLsm6NhAA8URNq8Ox+P5rSKFdsDRt7g== 2024-09-16T04:38:32.968 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.1] 2024-09-16T04:38:32.968 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQA2tudmAX3+OBAAAhtq4OSD99a9zPjl7RWqWQ== 2024-09-16T04:38:32.968 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.2] 2024-09-16T04:38:32.968 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQA5tudmTcRtOxAAznnxF2zY1Xsz/EPdQPrKng== 2024-09-16T04:38:32.968 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr.x] 2024-09-16T04:38:32.969 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAytudmhnxrNBAAPBlQPqWExHYhBY7d3RDTxg== 2024-09-16T04:38:32.969 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.3] 2024-09-16T04:38:32.969 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQA/tudmEw1tGBAAcUQvtihFvH+A41SYVso/dw== 2024-09-16T04:38:32.969 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.4] 2024-09-16T04:38:32.969 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQBCtudmA2qdGhAAAz1YASlzVuR7ZGWB0FuqUA== 2024-09-16T04:38:32.969 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.5] 2024-09-16T04:38:32.969 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQBFtudmnRVbHRAAGKj7W7gMwjLeRMITOc/lOg== 2024-09-16T04:38:32.969 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-09-16T04:38:32.969 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAytudmr04INxAADRxKq9wyHZz7r3U84EyfKQ== 2024-09-16T04:38:32.970 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:38:32.993 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:38:33.061 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:38:33.064 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:38:33.117 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:38:33.119 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:38:33.171 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T04:38:33.173 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T04:38:33.225 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:38:33.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:38:33.278 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:38:33.280 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:38:33.331 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:38:33.333 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:38:33.383 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T04:38:33.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T04:38:33.439 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T04:38:33.439 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T04:38:33.465 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T04:38:33.562 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T04:38:33.588 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T04:38:33.654 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T04:38:33.751 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T04:38:33.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T04:38:33.804 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T04:38:33.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T04:38:33.923 DEBUG:teuthology.orchestra.run.smithi046:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:38:33.926 DEBUG:teuthology.orchestra.run.smithi114:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:38:33.979 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T04:38:33.979 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T04:38:33.979 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T04:38:33.982 INFO:tasks.ceph.mon.a:Started 2024-09-16T04:38:33.982 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:38:33.982 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:38:33.984 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:38:33.984 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T04:38:33.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T04:38:34.022 INFO:tasks.ceph.mon.b:Started 2024-09-16T04:38:34.022 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T04:38:34.022 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T04:38:34.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T04:38:34.024 INFO:tasks.ceph.mgr.x:Started 2024-09-16T04:38:34.024 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T04:38:34.025 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T04:38:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.125+0000 7f811396c640 10 monclient: get_monmap_and_config 2024-09-16T04:38:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.125+0000 7f811396c640 10 monclient: build_initial_monmap 2024-09-16T04:38:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.125+0000 7f811396c640 10 monclient: monmap: 2024-09-16T04:38:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:38:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:34.125888+0000 2024-09-16T04:38:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:34.125888+0000 2024-09-16T04:38:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:38:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:38:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:38:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:38:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:38:34.128 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f811396c640 1 Processor -- start 2024-09-16T04:38:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f811396c640 1 -- start start 2024-09-16T04:38:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f811396c640 10 monclient: init 2024-09-16T04:38:34.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f811396c640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:34.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f811396c640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-16T04:38:34.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f811396c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f810c0a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:38:34.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f811396c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 0x7f810c0a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:34.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f811396c640 10 monclient(hunting): picked mon.noname-a con 0x7f810c0a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:38:34.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f811396c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f810c13be90 addr v1:172.21.15.114:6789/0 2024-09-16T04:38:34.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f811396c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f810c09b940 con 0x7f810c0a3c10 2024-09-16T04:38:34.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f811396c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f810c09ab00 con 0x7f810c13be90 2024-09-16T04:38:34.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f811396c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f811396c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f810c09b030 con 0x7f810c0a3ff0 2024-09-16T04:38:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f811396c640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f811396c640 10 monclient(hunting): authenticate will time out at 712.541726s 2024-09-16T04:38:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f8110f07640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 0x7f810c0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f8111708640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f810c0a3c10 0x7f810c09bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58626/0 (socket says 172.21.15.46:58626) 2024-09-16T04:38:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f8111708640 1 -- 172.21.15.46:0/200911504 learned_addr learned my addr 172.21.15.46:0/200911504 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:38:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f8110f07640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 0x7f810c0a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51660/0 (socket says 172.21.15.46:51660) 2024-09-16T04:38:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f8110f07640 10 monclient(hunting): get_auth_request con 0x7f810c0a3ff0 auth_method 0 2024-09-16T04:38:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f8110f07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f8110f07640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f8110f07640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f8110f07640 1 -- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 msgr2=0x7f810c0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:38:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f8110f07640 1 -- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 msgr2=0x7f810c0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:38:34.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.126+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 0x7f810c0a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:38:34.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.127+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 0x7f810c0a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:38:34.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.328+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 0x7f810c0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:34.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.328+0000 7f8110f07640 10 monclient(hunting): get_auth_request con 0x7f810c0a3ff0 auth_method 0 2024-09-16T04:38:34.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.328+0000 7f8110f07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:34.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.328+0000 7f8110f07640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:34.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.328+0000 7f8110f07640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:38:34.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.328+0000 7f8110f07640 1 -- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 msgr2=0x7f810c0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:38:34.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.328+0000 7f8110f07640 1 -- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 msgr2=0x7f810c0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:38:34.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.328+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 0x7f810c0a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:38:34.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.328+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 0x7f810c0a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:38:34.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.729+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 0x7f810c0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:34.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.729+0000 7f8110f07640 10 monclient(hunting): get_auth_request con 0x7f810c0a3ff0 auth_method 0 2024-09-16T04:38:34.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.729+0000 7f8110f07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:34.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.729+0000 7f8110f07640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:34.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.729+0000 7f8110f07640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:38:34.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.729+0000 7f8110f07640 1 -- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 msgr2=0x7f810c0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:38:34.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.729+0000 7f8110f07640 1 -- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 msgr2=0x7f810c0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:38:34.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.729+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 0x7f810c0a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:38:34.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:34.729+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 0x7f810c0a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:38:35.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:35.530+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 0x7f810c0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:35.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:35.530+0000 7f8110f07640 10 monclient(hunting): get_auth_request con 0x7f810c0a3ff0 auth_method 0 2024-09-16T04:38:35.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:35.530+0000 7f8110f07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:35.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:35.530+0000 7f8110f07640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:35.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:35.530+0000 7f8110f07640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:38:35.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:35.530+0000 7f8110f07640 1 -- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 msgr2=0x7f810c0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:38:35.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:35.530+0000 7f8110f07640 1 -- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 msgr2=0x7f810c0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:38:35.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:35.530+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 0x7f810c0a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:38:35.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:35.530+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 0x7f810c0a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:38:37.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.126+0000 7f80feffd640 10 monclient(hunting): tick 2024-09-16T04:38:37.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.126+0000 7f80feffd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:38:37.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.126+0000 7f80feffd640 1 monclient(hunting): continuing hunt 2024-09-16T04:38:37.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.126+0000 7f80feffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:38:37.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.126+0000 7f80feffd640 1 -- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 msgr2=0x7f810c0a43d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:38:37.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.126+0000 7f80feffd640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 0x7f810c0a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:37.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.126+0000 7f80feffd640 1 -- 172.21.15.46:0/200911504 >> v1:172.21.15.114:6789/0 conn(0x7f810c13be90 legacy=0x7f810c13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:37.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.126+0000 7f80feffd640 1 -- 172.21.15.46:0/200911504 >> v1:172.21.15.46:6789/0 conn(0x7f810c0a3c10 legacy=0x7f810c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:37.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.126+0000 7f80feffd640 10 monclient: _add_conns ranks=[2,4] 2024-09-16T04:38:37.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.126+0000 7f80feffd640 1 --2- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 0x7f80f0005b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:37.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f80feffd640 10 monclient(hunting): picked mon.noname-b con 0x7f80f0003710 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:37.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f80feffd640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 0x7f80f0008670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:37.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f80feffd640 10 monclient(hunting): picked mon.noname-c con 0x7f80f00061d0 addr v2:172.21.15.114:3300/0 2024-09-16T04:38:37.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f80feffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:37.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f80feffd640 1 -- 172.21.15.46:0/200911504 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f810c09b030 con 0x7f80f00061d0 2024-09-16T04:38:37.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f80feffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:37.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f80feffd640 1 -- 172.21.15.46:0/200911504 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f80f0008d30 con 0x7f80f0003710 2024-09-16T04:38:37.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f80feffd640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:37.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f8111708640 1 --2- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 0x7f80f0005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:37.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f8111708640 10 monclient(hunting): get_auth_request con 0x7f80f0003710 auth_method 0 2024-09-16T04:38:37.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f8111708640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:37.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f8111708640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:37.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f8111708640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:37.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f8111708640 1 -- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 msgr2=0x7f80f0005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:38:37.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f8111708640 1 -- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 msgr2=0x7f80f0005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:38:37.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f8111708640 1 --2- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 0x7f80f0005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:38:37.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f8111708640 1 --2- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 0x7f80f0005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:38:37.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 0x7f80f0008670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:37.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f8110f07640 10 monclient(hunting): get_auth_request con 0x7f80f00061d0 auth_method 0 2024-09-16T04:38:37.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f8110f07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:37.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f8110f07640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:37.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f8110f07640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:37.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f8110f07640 1 -- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 msgr2=0x7f80f0008670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:38:37.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f8110f07640 1 -- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 msgr2=0x7f80f0008670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:38:37.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 0x7f80f0008670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:38:37.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.127+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 0x7f80f0008670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:38:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.328+0000 7f8111708640 1 --2- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 0x7f80f0005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.328+0000 7f8111708640 10 monclient(hunting): get_auth_request con 0x7f80f0003710 auth_method 0 2024-09-16T04:38:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.328+0000 7f8111708640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.328+0000 7f8111708640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.328+0000 7f8111708640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:38:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.328+0000 7f8111708640 1 -- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 msgr2=0x7f80f0005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:38:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.328+0000 7f8111708640 1 -- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 msgr2=0x7f80f0005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:38:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.328+0000 7f8111708640 1 --2- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 0x7f80f0005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:38:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.328+0000 7f8111708640 1 --2- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 0x7f80f0005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:38:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.328+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 0x7f80f0008670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:37.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.328+0000 7f8110f07640 10 monclient(hunting): get_auth_request con 0x7f80f00061d0 auth_method 0 2024-09-16T04:38:37.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.328+0000 7f8110f07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:37.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.328+0000 7f8110f07640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:37.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.328+0000 7f8110f07640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:38:37.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.328+0000 7f8110f07640 1 -- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 msgr2=0x7f80f0008670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:38:37.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.328+0000 7f8110f07640 1 -- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 msgr2=0x7f80f0008670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:38:37.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.328+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 0x7f80f0008670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:38:37.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.328+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 0x7f80f0008670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:38:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.729+0000 7f8111708640 1 --2- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 0x7f80f0005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.729+0000 7f8111708640 10 monclient(hunting): get_auth_request con 0x7f80f0003710 auth_method 0 2024-09-16T04:38:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.729+0000 7f8111708640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.729+0000 7f8111708640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.729+0000 7f8111708640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:38:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.729+0000 7f8111708640 1 -- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 msgr2=0x7f80f0005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:38:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.729+0000 7f8111708640 1 -- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 msgr2=0x7f80f0005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:38:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.729+0000 7f8111708640 1 --2- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 0x7f80f0005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:38:37.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.729+0000 7f8111708640 1 --2- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 0x7f80f0005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:38:37.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.729+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 0x7f80f0008670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:37.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.729+0000 7f8110f07640 10 monclient(hunting): get_auth_request con 0x7f80f00061d0 auth_method 0 2024-09-16T04:38:37.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.729+0000 7f8110f07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:37.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.729+0000 7f8110f07640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:37.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.729+0000 7f8110f07640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:38:37.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.729+0000 7f8110f07640 1 -- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 msgr2=0x7f80f0008670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:38:37.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.729+0000 7f8110f07640 1 -- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 msgr2=0x7f80f0008670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:38:37.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.729+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 0x7f80f0008670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:38:37.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:37.729+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 0x7f80f0008670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:38:38.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:38.530+0000 7f8111708640 1 --2- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 0x7f80f0005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:38.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:38.530+0000 7f8111708640 10 monclient(hunting): get_auth_request con 0x7f80f0003710 auth_method 0 2024-09-16T04:38:38.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:38.530+0000 7f8111708640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:38.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:38.530+0000 7f8111708640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:38.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:38.530+0000 7f8111708640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:38:38.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:38.530+0000 7f8111708640 1 -- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 msgr2=0x7f80f0005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:38:38.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:38.530+0000 7f8111708640 1 -- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 msgr2=0x7f80f0005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:38:38.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:38.530+0000 7f8111708640 1 --2- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 0x7f80f0005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:38:38.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:38.530+0000 7f8111708640 1 --2- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 0x7f80f0005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:38:38.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:38.530+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 0x7f80f0008670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:38.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:38.530+0000 7f8110f07640 10 monclient(hunting): get_auth_request con 0x7f80f00061d0 auth_method 0 2024-09-16T04:38:38.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:38.530+0000 7f8110f07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:38.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:38.530+0000 7f8110f07640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:38.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:38.530+0000 7f8110f07640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:38:38.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:38.531+0000 7f8110f07640 1 -- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 msgr2=0x7f80f0008670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:38:38.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:38.531+0000 7f8110f07640 1 -- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 msgr2=0x7f80f0008670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:38:38.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:38.531+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 0x7f80f0008670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:38:38.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:38.531+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 0x7f80f0008670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:38:39.410 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:39.409+0000 7faefb42d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T04:38:39.451 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:39.450+0000 7faefb42d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T04:38:39.587 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:39.586+0000 7faefb42d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T04:38:39.637 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:39.635+0000 7faefb42d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T04:38:39.786 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:39.784+0000 7faefb42d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T04:38:39.895 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:39.893+0000 7faefb42d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T04:38:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f80feffd640 10 monclient(hunting): tick 2024-09-16T04:38:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f80feffd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:38:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f80feffd640 1 monclient(hunting): continuing hunt 2024-09-16T04:38:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f80feffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:38:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f80feffd640 1 -- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 msgr2=0x7f80f0005b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:38:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f80feffd640 1 --2- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003710 0x7f80f0005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f80feffd640 1 -- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 msgr2=0x7f80f0008670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:38:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f80feffd640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f00061d0 0x7f80f0008670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f80feffd640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-16T04:38:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f8111f09640 1 -- 172.21.15.46:0/200911504 reap_dead start 2024-09-16T04:38:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f80feffd640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 0x7f80f000b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f80feffd640 10 monclient(hunting): picked mon.noname-a con 0x7f810c0a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:38:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f80feffd640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f0003b50 0x7f80f00065b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f80feffd640 10 monclient(hunting): picked mon.noname-c con 0x7f80f0003b50 addr v2:172.21.15.114:3300/0 2024-09-16T04:38:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f80feffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f80f0006af0 addr v1:172.21.15.114:6789/0 2024-09-16T04:38:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f80feffd640 1 -- 172.21.15.46:0/200911504 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f810c09b310 con 0x7f80f0006af0 2024-09-16T04:38:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f80feffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f80feffd640 1 -- 172.21.15.46:0/200911504 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f810c09b030 con 0x7f810c0a3ff0 2024-09-16T04:38:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f80feffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f80feffd640 1 -- 172.21.15.46:0/200911504 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f80f0008d30 con 0x7f80f0003b50 2024-09-16T04:38:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f80feffd640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f8111708640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.46:3300/0 conn(0x7f810c0a3ff0 0x7f80f000b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f8111708640 10 monclient(hunting): get_auth_request con 0x7f810c0a3ff0 auth_method 0 2024-09-16T04:38:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f8111708640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f8111708640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f8111708640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f8111708640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f8110f07640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f0003b50 0x7f80f00065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f8111708640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f8111708640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f8110f07640 10 monclient(hunting): get_auth_request con 0x7f80f0003b50 auth_method 0 2024-09-16T04:38:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f8110f07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f8110f07640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f8110f07640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f8110f07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f8110f07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.127+0000 7f8110f07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.128+0000 7f8111708640 10 monclient(hunting): handle_auth_done global_id 4104 payload 274 2024-09-16T04:38:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.128+0000 7f8111708640 1 -- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f0003b50 msgr2=0x7f80f00065b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:38:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.128+0000 7f8111708640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f0003b50 0x7f80f00065b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.128+0000 7f8111708640 1 -- 172.21.15.46:0/200911504 >> v1:172.21.15.114:6789/0 conn(0x7f80f0006af0 legacy=0x7f80f0006f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.128+0000 7f8111708640 10 monclient: _finish_hunting 0 2024-09-16T04:38:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.128+0000 7f8111708640 1 monclient: found mon.noname-a 2024-09-16T04:38:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.128+0000 7f8111708640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.128+0000 7f8111708640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:38:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.128+0000 7f80fffff640 1 -- 172.21.15.46:0/200911504 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 405764912 0 0) 0x7f810c09b310 con 0x7f80f0006af0 2024-09-16T04:38:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.128+0000 7f8111708640 1 -- 172.21.15.46:0/200911504 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f00088f0 con 0x7f810c0a3ff0 2024-09-16T04:38:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.128+0000 7f80fffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:38:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.129+0000 7f8111708640 1 --2- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f810c0a3ff0 0x7f80f000b180 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f810c06cb60 tx=0x7f810002ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd84446a77421c77 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.129+0000 7f80fffff640 1 -- 172.21.15.46:0/200911504 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8100031070 con 0x7f810c0a3ff0 2024-09-16T04:38:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.129+0000 7f80fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.129+0000 7f80fffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:38:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.129+0000 7f80fffff640 10 monclient: dump: 2024-09-16T04:38:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:38:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:40.137 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.129+0000 7f80fffff640 10 monclient: _finish_auth 0 2024-09-16T04:38:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.129+0000 7f80fffff640 10 monclient: _check_auth_tickets 2024-09-16T04:38:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.129+0000 7f80fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.129+0000 7f80fffff640 1 -- 172.21.15.46:0/200911504 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f810002b810 con 0x7f810c0a3ff0 2024-09-16T04:38:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.129+0000 7f811396c640 5 monclient: authenticate success, global_id 4104 2024-09-16T04:38:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.129+0000 7f811396c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:38:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.129+0000 7f80fffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.129+0000 7f811396c640 10 monclient: get_monmap_and_config success 2024-09-16T04:38:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.129+0000 7f811396c640 4 set_mon_vals no callback set 2024-09-16T04:38:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.129+0000 7f80fffff640 1 -- 172.21.15.46:0/200911504 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810002bc00 con 0x7f810c0a3ff0 2024-09-16T04:38:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.129+0000 7f80fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.129+0000 7f80fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:38:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.129+0000 7f80fffff640 10 monclient: dump: 2024-09-16T04:38:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:38:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:40.139 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.130+0000 7f811396c640 10 monclient: shutdown 2024-09-16T04:38:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.130+0000 7f811396c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.130+0000 7f811396c640 1 -- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f810c0a3ff0 msgr2=0x7f80f000b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.130+0000 7f811396c640 1 --2- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f810c0a3ff0 0x7f80f000b180 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f810c06cb60 tx=0x7f810002ad10 comp rx=0 tx=0).stop 2024-09-16T04:38:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.130+0000 7f811396c640 1 -- 172.21.15.46:0/200911504 shutdown_connections 2024-09-16T04:38:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.130+0000 7f811396c640 1 --2- 172.21.15.46:0/200911504 >> v2:172.21.15.114:3300/0 conn(0x7f80f0003b50 0x7f80f00065b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.130+0000 7f811396c640 1 --2- 172.21.15.46:0/200911504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f810c0a3ff0 0x7f80f000b180 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.130+0000 7f811396c640 1 -- 172.21.15.46:0/200911504 >> 172.21.15.46:0/200911504 conn(0x7f810c0a6340 msgr2=0x7f810c06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.130+0000 7f811396c640 1 -- 172.21.15.46:0/200911504 shutdown_connections 2024-09-16T04:38:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.130+0000 7f811396c640 1 -- 172.21.15.46:0/200911504 wait complete. 2024-09-16T04:38:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.131+0000 7f811396c640 10 monclient: build_initial_monmap 2024-09-16T04:38:40.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.131+0000 7f811396c640 10 monclient: monmap: 2024-09-16T04:38:40.140 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:38:40.140 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:38:40.140 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:38:40.140 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:38:40.140 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:40.140 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:40.140 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:38:40.140 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:38:40.140 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:38:40.140 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:40.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.131+0000 7f811396c640 1 Processor -- start 2024-09-16T04:38:40.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.131+0000 7f811396c640 1 -- start start 2024-09-16T04:38:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.131+0000 7f811396c640 10 monclient: init 2024-09-16T04:38:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f811396c640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f811396c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:38:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f811396c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003b50 0x7f810c1374a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f811396c640 10 monclient(hunting): picked mon.noname-c con 0x7f80f0003b50 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f811396c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80f0006af0 0x7f810c11fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f811396c640 10 monclient(hunting): picked mon.noname-a con 0x7f80f0006af0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:38:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f811396c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f810c0a3ff0 0x7f810c120250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f811396c640 10 monclient(hunting): picked mon.noname-b con 0x7f810c0a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:38:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f811396c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f811396c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f810c09be70 con 0x7f80f0006af0 2024-09-16T04:38:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f811396c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f811396c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f810c09bcf0 con 0x7f810c0a3ff0 2024-09-16T04:38:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f811396c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f811396c640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f810c09bff0 con 0x7f80f0003b50 2024-09-16T04:38:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f811396c640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f811396c640 10 monclient(hunting): authenticate will time out at 718.547458s 2024-09-16T04:38:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8111708640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003b50 0x7f810c1374a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8110f07640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80f0006af0 0x7f810c11fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8111708640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003b50 0x7f810c1374a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:43988/0 (socket says 172.21.15.46:43988) 2024-09-16T04:38:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8110f07640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80f0006af0 0x7f810c11fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40962/0 (socket says 172.21.15.46:40962) 2024-09-16T04:38:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8111708640 1 -- 172.21.15.46:0/1910701068 learned_addr learned my addr 172.21.15.46:0/1910701068 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:38:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8111708640 10 monclient(hunting): get_auth_request con 0x7f80f0003b50 auth_method 0 2024-09-16T04:38:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8111708640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8111708640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8111708640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:40.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8110f07640 10 monclient(hunting): get_auth_request con 0x7f80f0006af0 auth_method 0 2024-09-16T04:38:40.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8110f07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:40.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8110f07640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:40.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8110f07640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:40.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8111f09640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f810c0a3ff0 0x7f810c120250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:40.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8111f09640 10 monclient(hunting): get_auth_request con 0x7f810c0a3ff0 auth_method 0 2024-09-16T04:38:40.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8111f09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:40.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8111f09640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:40.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8111f09640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:40.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8111708640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:40.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8111708640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:40.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8111708640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:40.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8110f07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:40.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8110f07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8110f07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8111f09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8111f09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8111f09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.132+0000 7f8110f07640 10 monclient(hunting): handle_auth_done global_id 4107 payload 824 2024-09-16T04:38:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f8110f07640 1 -- 172.21.15.46:0/1910701068 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003b50 msgr2=0x7f810c1374a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f8110f07640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003b50 0x7f810c1374a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f8110f07640 1 -- 172.21.15.46:0/1910701068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f810c0a3ff0 msgr2=0x7f810c120250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f8110f07640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f810c0a3ff0 0x7f810c120250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f8110f07640 10 monclient: _finish_hunting 0 2024-09-16T04:38:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f8110f07640 1 monclient: found mon.noname-a 2024-09-16T04:38:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f8110f07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f8110f07640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:38:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f8110f07640 1 -- 172.21.15.46:0/1910701068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f810c13aea0 con 0x7f80f0006af0 2024-09-16T04:38:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f8111708640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003b50 0x7f810c1374a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:38:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f8110f07640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80f0006af0 0x7f810c11fd10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f81080189d0 tx=0x7f8108018e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f80fdffb640 1 -- 172.21.15.46:0/1910701068 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8108016ce0 con 0x7f80f0006af0 2024-09-16T04:38:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f80fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f80fdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:38:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f80fdffb640 10 monclient: dump: 2024-09-16T04:38:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:38:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:40.156 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f80fdffb640 10 monclient: _finish_auth 0 2024-09-16T04:38:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f80fdffb640 10 monclient: _check_auth_tickets 2024-09-16T04:38:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f80fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f80fdffb640 1 -- 172.21.15.46:0/1910701068 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8108028070 con 0x7f80f0006af0 2024-09-16T04:38:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f811396c640 5 monclient: authenticate success, global_id 4107 2024-09-16T04:38:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f811396c640 10 monclient: _renew_subs 2024-09-16T04:38:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f811396c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:38:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f811396c640 1 -- 172.21.15.46:0/1910701068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f810c0a0ac0 con 0x7f80f0006af0 2024-09-16T04:38:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f80fdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f80fdffb640 1 -- 172.21.15.46:0/1910701068 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8108023620 con 0x7f80f0006af0 2024-09-16T04:38:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f80fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f80fdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:38:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f80fdffb640 10 monclient: dump: 2024-09-16T04:38:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:38:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:40.157 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f811396c640 10 monclient: _renew_subs 2024-09-16T04:38:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f811396c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:38:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f80ff7fe640 4 set_mon_vals no callback set 2024-09-16T04:38:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.133+0000 7f811396c640 1 -- 172.21.15.46:0/1910701068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f810c1235d0 con 0x7f80f0006af0 2024-09-16T04:38:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f8110f07640 1 -- 172.21.15.46:0/1910701068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80f0006af0 msgr2=0x7f810c11fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:38:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f8110f07640 1 -- 172.21.15.46:0/1910701068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80f0006af0 msgr2=0x7f810c11fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:38:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f8110f07640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80f0006af0 0x7f810c11fd10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f81080189d0 tx=0x7f8108018e90 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:38:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f8110f07640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80f0006af0 0x7f810c11fd10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f81080189d0 tx=0x7f8108018e90 comp rx=0 tx=0).stop 2024-09-16T04:38:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f80fdffb640 10 monclient: ms_handle_reset current mon [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:38:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f80fdffb640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f80fdffb640 1 -- 172.21.15.46:0/1910701068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80f0006af0 msgr2=0x7f810c11fd10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:38:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f80fdffb640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80f0006af0 0x7f810c11fd10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f80fdffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:38:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f80fdffb640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f810c0a3ff0 0x7f80f00065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f80fdffb640 10 monclient(hunting): picked mon.b con 0x7f810c0a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:38:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f80fdffb640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003b50 0x7f80f0002f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f80fdffb640 10 monclient(hunting): picked mon.c con 0x7f80f0003b50 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f80fdffb640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80f0005560 0x7f80f00059e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f80fdffb640 10 monclient(hunting): picked mon.a con 0x7f80f0005560 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:38:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f80fdffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f80fdffb640 1 -- 172.21.15.46:0/1910701068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f80f000d660 con 0x7f80f0005560 2024-09-16T04:38:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f80fdffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f80fdffb640 1 -- 172.21.15.46:0/1910701068 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f80f000d7e0 con 0x7f810c0a3ff0 2024-09-16T04:38:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f80fdffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f80fdffb640 1 -- 172.21.15.46:0/1910701068 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f80f000d960 con 0x7f80f0003b50 2024-09-16T04:38:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f80fdffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f8111f09640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80f0005560 0x7f80f00059e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f8110f07640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003b50 0x7f80f0002f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f8111f09640 10 monclient(hunting): get_auth_request con 0x7f80f0005560 auth_method 0 2024-09-16T04:38:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f8111f09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f8111f09640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f8111f09640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:38:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.134+0000 7f8110f07640 10 monclient(hunting): get_auth_request con 0x7f80f0003b50 auth_method 0 2024-09-16T04:38:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8110f07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8110f07640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111708640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f810c0a3ff0 0x7f80f00065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8110f07640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:38:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111708640 10 monclient(hunting): get_auth_request con 0x7f810c0a3ff0 auth_method 0 2024-09-16T04:38:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111708640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111708640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111708640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:38:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:38:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111708640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111708640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111708640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:38:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8110f07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8110f07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8110f07640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:38:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f811396c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f811396c640 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 10 monclient(hunting): handle_auth_done global_id 4107 payload 293 2024-09-16T04:38:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 1 -- 172.21.15.46:0/1910701068 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003b50 msgr2=0x7f80f0002f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003b50 0x7f80f0002f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 1 -- 172.21.15.46:0/1910701068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f810c0a3ff0 msgr2=0x7f80f00065b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f810c0a3ff0 0x7f80f00065b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 10 monclient: _finish_hunting 0 2024-09-16T04:38:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 1 monclient: found mon.a 2024-09-16T04:38:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:38:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 1 -- 172.21.15.46:0/1910701068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f80f000db40 con 0x7f80f0005560 2024-09-16T04:38:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:38:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 1 -- 172.21.15.46:0/1910701068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f810c0a43d0 con 0x7f80f0005560 2024-09-16T04:38:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:38:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 1 -- 172.21.15.46:0/1910701068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f810c09ab00 con 0x7f80f0005560 2024-09-16T04:38:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 10 monclient: _finish_auth 0 2024-09-16T04:38:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 10 monclient: _check_auth_tickets 2024-09-16T04:38:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111708640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f810c0a3ff0 0x7f80f00065b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:38:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f8111f09640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80f0005560 0x7f80f00059e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8104002090 tx=0x7f8108008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f80fdffb640 1 -- 172.21.15.46:0/1910701068 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8104001900 con 0x7f80f0005560 2024-09-16T04:38:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f80fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f80fdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:38:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f80fdffb640 10 monclient: dump: 2024-09-16T04:38:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:38:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:40.164 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:40.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f80fdffb640 1 -- 172.21.15.46:0/1910701068 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8104018070 con 0x7f80f0005560 2024-09-16T04:38:40.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f80fdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:40.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f80fdffb640 1 -- 172.21.15.46:0/1910701068 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8104005b40 con 0x7f80f0005560 2024-09-16T04:38:40.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.135+0000 7f80feffd640 4 set_mon_vals no callback set 2024-09-16T04:38:40.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.136+0000 7f80fdffb640 1 -- 172.21.15.46:0/1910701068 <== mon.0 v2:172.21.15.46:3300/0 4 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8104010040 con 0x7f80f0005560 2024-09-16T04:38:40.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.138+0000 7f80fdffb640 1 -- 172.21.15.46:0/1910701068 <== mon.0 v2:172.21.15.46:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f810401c8f0 con 0x7f80f0005560 2024-09-16T04:38:40.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.138+0000 7f80fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:40.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.138+0000 7f80fdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:38:40.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.140+0000 7f80fdffb640 1 -- 172.21.15.46:0/1910701068 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f810401cb40 con 0x7f80f0005560 2024-09-16T04:38:40.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.140+0000 7f80fdffb640 10 monclient: handle_mon_command_ack 1 not found 2024-09-16T04:38:40.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.205+0000 7f811396c640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T04:38:40.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.205+0000 7f811396c640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T04:38:40.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.205+0000 7f811396c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:38:40.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.205+0000 7f811396c640 1 -- 172.21.15.46:0/1910701068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f810c122180 con 0x7f80f0005560 2024-09-16T04:38:40.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.206+0000 7f80fdffb640 1 -- 172.21.15.46:0/1910701068 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f810401cdb0 con 0x7f80f0005560 2024-09-16T04:38:40.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.207+0000 7f80fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T04:38:40.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.207+0000 7f80fdffb640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-16T04:38:40.208 INFO:teuthology.orchestra.run.smithi046.stderr:adjusted tunables profile to default 2024-09-16T04:38:40.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.207+0000 7f811396c640 10 monclient: shutdown 2024-09-16T04:38:40.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.207+0000 7f811396c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:40.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.207+0000 7f811396c640 1 -- 172.21.15.46:0/1910701068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80f0005560 msgr2=0x7f80f00059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:40.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.207+0000 7f811396c640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80f0005560 0x7f80f00059e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8104002090 tx=0x7f8108008040 comp rx=0 tx=0).stop 2024-09-16T04:38:40.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.208+0000 7f811396c640 1 -- 172.21.15.46:0/1910701068 shutdown_connections 2024-09-16T04:38:40.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.208+0000 7f811396c640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80f0005560 0x7f80f00059e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:40.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.208+0000 7f811396c640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f80f0003b50 0x7f80f0002f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:40.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.208+0000 7f811396c640 1 --2- 172.21.15.46:0/1910701068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f810c0a3ff0 0x7f80f00065b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:40.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.208+0000 7f811396c640 1 -- 172.21.15.46:0/1910701068 >> 172.21.15.46:0/1910701068 conn(0x7f810c0a6340 msgr2=0x7f810c0a1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:40.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.208+0000 7f811396c640 1 -- 172.21.15.46:0/1910701068 shutdown_connections 2024-09-16T04:38:40.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:40.208+0000 7f811396c640 1 -- 172.21.15.46:0/1910701068 wait complete. 2024-09-16T04:38:40.221 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T04:38:40.221 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T04:38:40.222 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T04:38:40.222 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T04:38:40.227 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:40.226+0000 7faefb42d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T04:38:40.249 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T04:38:40.249 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T04:38:40.277 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:40.275+0000 7faefb42d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T04:38:40.316 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-16T04:38:40.316 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T04:38:40.383 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T04:38:40.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T04:38:40.408 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T04:38:40.408 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T04:38:40.430 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:40.429+0000 7faefb42d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T04:38:40.455 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T04:38:40.455 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T04:38:40.520 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new 7ff88780-0630-4bb1-9f5a-5331d7724d17 0 2024-09-16T04:38:40.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.658+0000 7fe8e0d8d640 10 monclient: get_monmap_and_config 2024-09-16T04:38:40.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.658+0000 7fe8e0d8d640 10 monclient: build_initial_monmap 2024-09-16T04:38:40.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.658+0000 7fe8e0d8d640 10 monclient: monmap: 2024-09-16T04:38:40.660 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T04:38:40.660 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:40.660 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:40.659040+0000 2024-09-16T04:38:40.660 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:40.659040+0000 2024-09-16T04:38:40.660 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:40.660 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:40.660 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:38:40.660 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:38:40.660 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:38:40.660 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:38:40.660 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:38:40.660 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:40.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.658+0000 7fe8e0d8d640 1 Processor -- start 2024-09-16T04:38:40.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.658+0000 7fe8e0d8d640 1 -- start start 2024-09-16T04:38:40.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.658+0000 7fe8e0d8d640 10 monclient: init 2024-09-16T04:38:40.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8e0d8d640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:40.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8e0d8d640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-16T04:38:40.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8e0d8d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe8dc0a3c10 0x7fe8dc09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:40.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8e0d8d640 10 monclient(hunting): picked mon.noname-c con 0x7fe8dc0a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:38:40.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8e0d8d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe8dc0a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:38:40.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8e0d8d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe8dc13be90 addr v1:172.21.15.114:6789/0 2024-09-16T04:38:40.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8e0d8d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8dc09ab70 con 0x7fe8dc0a3ff0 2024-09-16T04:38:40.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8e0d8d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8dc0a4d60 con 0x7fe8dc13be90 2024-09-16T04:38:40.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8e0d8d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:40.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8e0d8d640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fe8dc09b030 con 0x7fe8dc0a3c10 2024-09-16T04:38:40.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8e0d8d640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:40.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8e0d8d640 10 monclient(hunting): authenticate will time out at 738.135425s 2024-09-16T04:38:40.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8da59c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe8dc0a3c10 0x7fe8dc09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:40.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8dad9d640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe8dc13be90 0x7fe8dc13e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36550/0 (socket says 172.21.15.114:36550) 2024-09-16T04:38:40.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8da59c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe8dc0a3c10 0x7fe8dc09bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48816/0 (socket says 172.21.15.114:48816) 2024-09-16T04:38:40.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8dad9d640 1 -- 172.21.15.114:0/566536086 learned_addr learned my addr 172.21.15.114:0/566536086 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T04:38:40.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8da59c640 10 monclient(hunting): get_auth_request con 0x7fe8dc0a3c10 auth_method 0 2024-09-16T04:38:40.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8da59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8da59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8da59c640 10 monclient(hunting): handle_auth_done global_id 4111 payload 274 2024-09-16T04:38:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8da59c640 1 -- 172.21.15.114:0/566536086 >> v1:172.21.15.114:6789/0 conn(0x7fe8dc13be90 legacy=0x7fe8dc13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8da59c640 1 -- 172.21.15.114:0/566536086 >> v1:172.21.15.46:6789/0 conn(0x7fe8dc0a3ff0 legacy=0x7fe8dc0a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:38:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8da59c640 10 monclient: _finish_hunting 0 2024-09-16T04:38:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8da59c640 1 monclient: found mon.noname-c 2024-09-16T04:38:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8da59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8da59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:38:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.659+0000 7fe8da59c640 1 -- 172.21.15.114:0/566536086 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8dc09b310 con 0x7fe8dc0a3c10 2024-09-16T04:38:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.660+0000 7fe8da59c640 1 -- 172.21.15.114:0/566536086 >> v2:172.21.15.114:3300/0 conn(0x7fe8dc0a3c10 msgr2=0x7fe8dc09bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:38:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.660+0000 7fe8da59c640 1 -- 172.21.15.114:0/566536086 >> v2:172.21.15.114:3300/0 conn(0x7fe8dc0a3c10 msgr2=0x7fe8dc09bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:38:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.660+0000 7fe8da59c640 1 --2- 172.21.15.114:0/566536086 >> v2:172.21.15.114:3300/0 conn(0x7fe8dc0a3c10 0x7fe8dc09bcf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7fe8cc0097a0 tx=0x7fe8cc02a0b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:38:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.660+0000 7fe8da59c640 1 --2- 172.21.15.114:0/566536086 >> v2:172.21.15.114:3300/0 conn(0x7fe8dc0a3c10 0x7fe8dc09bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:38:40.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8da59c640 1 --2- 172.21.15.114:0/566536086 >> v2:172.21.15.114:3300/0 conn(0x7fe8dc0a3c10 0x7fe8dc09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:40.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8da59c640 10 monclient: get_auth_request con 0x7fe8dc0a3c10 auth_method 0 2024-09-16T04:38:40.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8da59c640 0 --2- 172.21.15.114:0/566536086 >> v2:172.21.15.114:3300/0 conn(0x7fe8dc0a3c10 0x7fe8dc09bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T04:38:40.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8da59c640 1 --2- 172.21.15.114:0/566536086 >> v2:172.21.15.114:3300/0 conn(0x7fe8dc0a3c10 0x7fe8dc09bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:40.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d959a640 10 monclient: ms_handle_reset current mon v2:172.21.15.114:3300/0 2024-09-16T04:38:40.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d959a640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:40.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d959a640 1 -- 172.21.15.114:0/566536086 >> v2:172.21.15.114:3300/0 conn(0x7fe8dc0a3c10 msgr2=0x7fe8dc09bcf0 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-16T04:38:40.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d959a640 1 --2- 172.21.15.114:0/566536086 >> v2:172.21.15.114:3300/0 conn(0x7fe8dc0a3c10 0x7fe8dc09bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:40.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d959a640 10 monclient: _add_conns ranks=[2,3] 2024-09-16T04:38:40.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d959a640 1 --2- 172.21.15.114:0/566536086 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe8c8003500 0x7fe8c8005950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:40.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d959a640 10 monclient(hunting): picked mon.noname-b con 0x7fe8c8003500 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:40.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d959a640 1 --2- 172.21.15.114:0/566536086 >> v2:172.21.15.46:3300/0 conn(0x7fe8c8006270 0x7fe8c8008730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:40.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d959a640 10 monclient(hunting): picked mon.noname-a con 0x7fe8c8006270 addr v2:172.21.15.46:3300/0 2024-09-16T04:38:40.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d959a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:40.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d959a640 1 -- 172.21.15.114:0/566536086 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe8c8009000 con 0x7fe8c8006270 2024-09-16T04:38:40.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d959a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:40.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d959a640 1 -- 172.21.15.114:0/566536086 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe8c80091b0 con 0x7fe8c8003500 2024-09-16T04:38:40.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d959a640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:40.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d959a640 10 monclient(hunting): ms_handle_reset stray mon v2:172.21.15.114:3300/0 2024-09-16T04:38:40.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d9d9b640 1 --2- 172.21.15.114:0/566536086 >> v2:172.21.15.46:3300/0 conn(0x7fe8c8006270 0x7fe8c8008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:40.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8da59c640 1 --2- 172.21.15.114:0/566536086 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe8c8003500 0x7fe8c8005950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:40.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8da59c640 10 monclient(hunting): get_auth_request con 0x7fe8c8003500 auth_method 0 2024-09-16T04:38:40.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:40.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8da59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:40.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8da59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:38:40.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d9d9b640 10 monclient(hunting): get_auth_request con 0x7fe8c8006270 auth_method 0 2024-09-16T04:38:40.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:40.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:40.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:38:40.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:40.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:40.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8da59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:38:40.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:40.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:40.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.861+0000 7fe8d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:38:40.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.862+0000 7fe8da59c640 10 monclient(hunting): handle_auth_done global_id 4111 payload 293 2024-09-16T04:38:40.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.862+0000 7fe8da59c640 1 -- 172.21.15.114:0/566536086 >> v2:172.21.15.46:3300/0 conn(0x7fe8c8006270 msgr2=0x7fe8c8008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:38:40.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.862+0000 7fe8da59c640 1 --2- 172.21.15.114:0/566536086 >> v2:172.21.15.46:3300/0 conn(0x7fe8c8006270 0x7fe8c8008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:40.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.862+0000 7fe8da59c640 10 monclient: _finish_hunting 0 2024-09-16T04:38:40.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.862+0000 7fe8da59c640 1 monclient: found mon.noname-b 2024-09-16T04:38:40.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.862+0000 7fe8da59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:40.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.862+0000 7fe8da59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:38:40.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.862+0000 7fe8da59c640 1 -- 172.21.15.114:0/566536086 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8c8009470 con 0x7fe8c8003500 2024-09-16T04:38:40.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.862+0000 7fe8da59c640 1 -- 172.21.15.114:0/566536086 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe8c8003500 msgr2=0x7fe8c8005950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:38:40.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.862+0000 7fe8da59c640 1 -- 172.21.15.114:0/566536086 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe8c8003500 msgr2=0x7fe8c8005950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:38:40.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.862+0000 7fe8da59c640 1 --2- 172.21.15.114:0/566536086 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe8c8003500 0x7fe8c8005950 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7fe8dc06ef00 tx=0x7fe8cc009190 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:38:40.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:40.862+0000 7fe8da59c640 1 --2- 172.21.15.114:0/566536086 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe8c8003500 0x7fe8c8005950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:38:41.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8da59c640 1 --2- 172.21.15.114:0/566536086 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe8c8003500 0x7fe8c8005950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8da59c640 10 monclient: get_auth_request con 0x7fe8c8003500 auth_method 0 2024-09-16T04:38:41.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8da59c640 0 --2- 172.21.15.114:0/566536086 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe8c8003500 0x7fe8c8005950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T04:38:41.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8da59c640 1 --2- 172.21.15.114:0/566536086 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe8c8003500 0x7fe8c8005950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8d959a640 10 monclient: ms_handle_reset current mon [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:41.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8dad9d640 1 -- 172.21.15.114:0/566536086 reap_dead start 2024-09-16T04:38:41.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8d959a640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:41.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8d959a640 1 -- 172.21.15.114:0/566536086 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe8c8003500 msgr2=0x7fe8c8005950 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-16T04:38:41.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8d959a640 1 --2- 172.21.15.114:0/566536086 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe8c8003500 0x7fe8c8005950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8d959a640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-16T04:38:41.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8d959a640 1 --2- 172.21.15.114:0/566536086 >> v2:172.21.15.46:3300/0 conn(0x7fe8c8006720 0x7fe8c8009700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8d959a640 10 monclient(hunting): picked mon.noname-a con 0x7fe8c8006720 addr v2:172.21.15.46:3300/0 2024-09-16T04:38:41.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8d959a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe8c8009db0 addr v1:172.21.15.114:6789/0 2024-09-16T04:38:41.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8d959a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe8c800dcd0 addr v1:172.21.15.46:6789/0 2024-09-16T04:38:41.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8d959a640 1 -- 172.21.15.114:0/566536086 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8c8011c80 con 0x7fe8c800dcd0 2024-09-16T04:38:41.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8d959a640 1 -- 172.21.15.114:0/566536086 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8c8012ee0 con 0x7fe8c8009db0 2024-09-16T04:38:41.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8d959a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8d959a640 1 -- 172.21.15.114:0/566536086 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe8c8014120 con 0x7fe8c8006720 2024-09-16T04:38:41.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8d959a640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:41.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8d959a640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:41.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8da59c640 1 --2- 172.21.15.114:0/566536086 >> v2:172.21.15.46:3300/0 conn(0x7fe8c8006720 0x7fe8c8009700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8da59c640 10 monclient(hunting): get_auth_request con 0x7fe8c8006720 auth_method 0 2024-09-16T04:38:41.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:41.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8da59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8da59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:38:41.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:41.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:41.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.063+0000 7fe8da59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:38:41.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8d959a640 1 -- 172.21.15.114:0/566536086 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4214828825 0 0) 0x7fe8c8012ee0 con 0x7fe8c8009db0 2024-09-16T04:38:41.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8d959a640 10 monclient(hunting): _negotiate 2024-09-16T04:38:41.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8d959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8d959a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:38:41.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8d959a640 1 -- 172.21.15.114:0/566536086 --> v1:172.21.15.114:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fe8c8011e50 con 0x7fe8c8009db0 2024-09-16T04:38:41.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8da59c640 10 monclient(hunting): handle_auth_done global_id 4111 payload 293 2024-09-16T04:38:41.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8da59c640 1 -- 172.21.15.114:0/566536086 >> v1:172.21.15.114:6789/0 conn(0x7fe8c8009db0 legacy=0x7fe8c800c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8da59c640 1 -- 172.21.15.114:0/566536086 >> v1:172.21.15.46:6789/0 conn(0x7fe8c800dcd0 legacy=0x7fe8c8010190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8da59c640 10 monclient: _finish_hunting 0 2024-09-16T04:38:41.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8da59c640 1 monclient: found mon.noname-a 2024-09-16T04:38:41.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8da59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:41.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8da59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:38:41.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8da59c640 1 -- 172.21.15.114:0/566536086 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8c80142f0 con 0x7fe8c8006720 2024-09-16T04:38:41.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8da59c640 1 --2- 172.21.15.114:0/566536086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8c8006720 0x7fe8c8009700 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc06ef00 tx=0x7fe8cc02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f54169065daa7d16 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:41.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8d959a640 1 -- 172.21.15.114:0/566536086 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc044070 con 0x7fe8c8006720 2024-09-16T04:38:41.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:41.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8d959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:38:41.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8d959a640 10 monclient: dump: 2024-09-16T04:38:41.078 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:41.078 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.078 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.078 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.078 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:41.078 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.078 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:41.078 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:41.078 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:41.078 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8d959a640 10 monclient: _finish_auth 0 2024-09-16T04:38:41.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8d959a640 10 monclient: _check_auth_tickets 2024-09-16T04:38:41.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:41.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8d959a640 1 -- 172.21.15.114:0/566536086 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8cc0338b0 con 0x7fe8c8006720 2024-09-16T04:38:41.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8e0d8d640 5 monclient: authenticate success, global_id 4111 2024-09-16T04:38:41.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8e0d8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:38:41.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8d959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:41.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8e0d8d640 10 monclient: get_monmap_and_config success 2024-09-16T04:38:41.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8e0d8d640 4 set_mon_vals no callback set 2024-09-16T04:38:41.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8d959a640 1 -- 172.21.15.114:0/566536086 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc033430 con 0x7fe8c8006720 2024-09-16T04:38:41.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:41.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8d959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:38:41.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.064+0000 7fe8d959a640 10 monclient: dump: 2024-09-16T04:38:41.080 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:41.080 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.080 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.080 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.080 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:41.080 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.080 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:41.080 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:41.080 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:41.080 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.065+0000 7fe8e0d8d640 10 monclient: shutdown 2024-09-16T04:38:41.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.065+0000 7fe8e0d8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:41.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.065+0000 7fe8e0d8d640 1 -- 172.21.15.114:0/566536086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8c8006720 msgr2=0x7fe8c8009700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.065+0000 7fe8e0d8d640 1 --2- 172.21.15.114:0/566536086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8c8006720 0x7fe8c8009700 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc06ef00 tx=0x7fe8cc02e000 comp rx=0 tx=0).stop 2024-09-16T04:38:41.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.065+0000 7fe8e0d8d640 1 -- 172.21.15.114:0/566536086 shutdown_connections 2024-09-16T04:38:41.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.065+0000 7fe8e0d8d640 1 --2- 172.21.15.114:0/566536086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8c8006720 0x7fe8c8009700 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.065+0000 7fe8e0d8d640 1 -- 172.21.15.114:0/566536086 >> 172.21.15.114:0/566536086 conn(0x7fe8dc0a5d10 msgr2=0x7fe8dc0a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:41.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.065+0000 7fe8e0d8d640 1 -- 172.21.15.114:0/566536086 shutdown_connections 2024-09-16T04:38:41.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.065+0000 7fe8e0d8d640 1 -- 172.21.15.114:0/566536086 wait complete. 2024-09-16T04:38:41.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.066+0000 7fe8e0d8d640 10 monclient: build_initial_monmap 2024-09-16T04:38:41.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.066+0000 7fe8e0d8d640 10 monclient: monmap: 2024-09-16T04:38:41.081 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T04:38:41.081 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:38:41.081 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T04:38:41.081 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T04:38:41.082 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:41.082 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.082 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:38:41.082 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:38:41.082 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:38:41.082 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.066+0000 7fe8e0d8d640 1 Processor -- start 2024-09-16T04:38:41.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.066+0000 7fe8e0d8d640 1 -- start start 2024-09-16T04:38:41.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.066+0000 7fe8e0d8d640 10 monclient: init 2024-09-16T04:38:41.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.066+0000 7fe8e0d8d640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:41.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.066+0000 7fe8e0d8d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:38:41.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.066+0000 7fe8e0d8d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8c8006720 0x7fe8dc1cc030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.066+0000 7fe8e0d8d640 10 monclient(hunting): picked mon.noname-a con 0x7fe8c8006720 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:38:41.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8e0d8d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe8c8009db0 0x7fe8dc1cc570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8e0d8d640 10 monclient(hunting): picked mon.noname-c con 0x7fe8c8009db0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:41.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8e0d8d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8c800dcd0 0x7fe8dc1ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8e0d8d640 10 monclient(hunting): picked mon.noname-b con 0x7fe8c800dcd0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:38:41.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8e0d8d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8e0d8d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe8dc0a4580 con 0x7fe8c8006720 2024-09-16T04:38:41.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8e0d8d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8e0d8d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe8dc0a4400 con 0x7fe8c800dcd0 2024-09-16T04:38:41.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8e0d8d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8e0d8d640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe8dc0a4700 con 0x7fe8c8009db0 2024-09-16T04:38:41.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8e0d8d640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:41.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8e0d8d640 10 monclient(hunting): authenticate will time out at 738.543049s 2024-09-16T04:38:41.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8dad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8c800dcd0 0x7fe8dc1ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8dad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8c800dcd0 0x7fe8dc1ccab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48836/0 (socket says 172.21.15.114:48836) 2024-09-16T04:38:41.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8dad9d640 1 -- 172.21.15.114:0/517736472 learned_addr learned my addr 172.21.15.114:0/517736472 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T04:38:41.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8dad9d640 10 monclient(hunting): get_auth_request con 0x7fe8c800dcd0 auth_method 0 2024-09-16T04:38:41.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:41.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8dad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8dad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:41.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d9d9b640 1 --2- 172.21.15.114:0/517736472 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe8c8009db0 0x7fe8dc1cc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8da59c640 1 --2- 172.21.15.114:0/517736472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8c8006720 0x7fe8dc1cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8dad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:41.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8dad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:41.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8dad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:41.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d9d9b640 10 monclient(hunting): get_auth_request con 0x7fe8c8009db0 auth_method 0 2024-09-16T04:38:41.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:41.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:41.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8da59c640 10 monclient(hunting): get_auth_request con 0x7fe8c8006720 auth_method 0 2024-09-16T04:38:41.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:41.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8da59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8da59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:41.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:41.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:41.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:41.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:41.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:41.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:41.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8dad9d640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2024-09-16T04:38:41.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8dad9d640 1 -- 172.21.15.114:0/517736472 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe8c8009db0 msgr2=0x7fe8dc1cc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8dad9d640 1 --2- 172.21.15.114:0/517736472 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe8c8009db0 0x7fe8dc1cc570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8dad9d640 1 -- 172.21.15.114:0/517736472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8c8006720 msgr2=0x7fe8dc1cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8dad9d640 1 --2- 172.21.15.114:0/517736472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8c8006720 0x7fe8dc1cc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8dad9d640 10 monclient: _finish_hunting 0 2024-09-16T04:38:41.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8dad9d640 1 monclient: found mon.noname-b 2024-09-16T04:38:41.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8dad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:41.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8dad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8dad9d640 1 -- 172.21.15.114:0/517736472 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8dc0a3650 con 0x7fe8c800dcd0 2024-09-16T04:38:41.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8dad9d640 1 --2- 172.21.15.114:0/517736472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8c800dcd0 0x7fe8dc1ccab0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe8d400c790 tx=0x7fe8d400cc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:41.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d2ffd640 1 -- 172.21.15.114:0/517736472 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d4017070 con 0x7fe8c800dcd0 2024-09-16T04:38:41.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:41.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:38:41.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d2ffd640 10 monclient: dump: 2024-09-16T04:38:41.087 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:41.087 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.087 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.087 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.087 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:41.087 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.087 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:41.087 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:41.087 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:41.087 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:38:41.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:38:41.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:41.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d2ffd640 1 -- 172.21.15.114:0/517736472 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8d400aa50 con 0x7fe8c800dcd0 2024-09-16T04:38:41.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8e0d8d640 5 monclient: authenticate success, global_id 4117 2024-09-16T04:38:41.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8e0d8d640 10 monclient: _renew_subs 2024-09-16T04:38:41.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8e0d8d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8e0d8d640 1 -- 172.21.15.114:0/517736472 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8dc0a3100 con 0x7fe8c800dcd0 2024-09-16T04:38:41.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:41.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d2ffd640 1 -- 172.21.15.114:0/517736472 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d400cea0 con 0x7fe8c800dcd0 2024-09-16T04:38:41.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:41.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:38:41.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d2ffd640 10 monclient: dump: 2024-09-16T04:38:41.088 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:41.088 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.089 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.089 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.089 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:41.089 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.089 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:41.089 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:41.089 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:41.089 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8d8d99640 4 set_mon_vals no callback set 2024-09-16T04:38:41.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8e0d8d640 10 monclient: _renew_subs 2024-09-16T04:38:41.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8e0d8d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.067+0000 7fe8e0d8d640 1 -- 172.21.15.114:0/517736472 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8dc123740 con 0x7fe8c800dcd0 2024-09-16T04:38:41.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.068+0000 7fe8d2ffd640 1 -- 172.21.15.114:0/517736472 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe8d401a070 con 0x7fe8c800dcd0 2024-09-16T04:38:41.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.068+0000 7fe8d2ffd640 1 -- 172.21.15.114:0/517736472 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fe8d401c810 con 0x7fe8c800dcd0 2024-09-16T04:38:41.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.069+0000 7fe8e0d8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:41.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.069+0000 7fe8e0d8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:41.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.069+0000 7fe8e0d8d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.069+0000 7fe8e0d8d640 1 -- 172.21.15.114:0/517736472 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8dc09bcf0 con 0x7fe8c800dcd0 2024-09-16T04:38:41.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.071+0000 7fe8d2ffd640 1 -- 172.21.15.114:0/517736472 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fe8d4021020 con 0x7fe8c800dcd0 2024-09-16T04:38:41.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.071+0000 7fe8d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:41.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.071+0000 7fe8d2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:38:41.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.139+0000 7fe8e0d8d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7ff88780-0630-4bb1-9f5a-5331d7724d17", "id": 0}] 2024-09-16T04:38:41.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.139+0000 7fe8e0d8d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7ff88780-0630-4bb1-9f5a-5331d7724d17", "id": 0}] 2024-09-16T04:38:41.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.139+0000 7fe8e0d8d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.139+0000 7fe8e0d8d640 1 -- 172.21.15.114:0/517736472 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7ff88780-0630-4bb1-9f5a-5331d7724d17", "id": 0} v 0) -- 0x7fe8dc069150 con 0x7fe8c800dcd0 2024-09-16T04:38:41.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.145+0000 7fe8d2ffd640 1 -- 172.21.15.114:0/517736472 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7ff88780-0630-4bb1-9f5a-5331d7724d17", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fe8dc0a3650 con 0x7fe8c800dcd0 2024-09-16T04:38:41.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.145+0000 7fe8d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7ff88780-0630-4bb1-9f5a-5331d7724d17", "id": 0}] 2024-09-16T04:38:41.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.145+0000 7fe8d2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:38:41.147 INFO:teuthology.orchestra.run.smithi114.stdout:0 2024-09-16T04:38:41.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.146+0000 7fe8e0d8d640 10 monclient: shutdown 2024-09-16T04:38:41.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.146+0000 7fe8e0d8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:41.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.146+0000 7fe8e0d8d640 1 -- 172.21.15.114:0/517736472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8c800dcd0 msgr2=0x7fe8dc1ccab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.146+0000 7fe8e0d8d640 1 --2- 172.21.15.114:0/517736472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8c800dcd0 0x7fe8dc1ccab0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe8d400c790 tx=0x7fe8d400cc50 comp rx=0 tx=0).stop 2024-09-16T04:38:41.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.146+0000 7fe8e0d8d640 1 -- 172.21.15.114:0/517736472 shutdown_connections 2024-09-16T04:38:41.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.146+0000 7fe8e0d8d640 1 --2- 172.21.15.114:0/517736472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8c800dcd0 0x7fe8dc1ccab0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.146+0000 7fe8e0d8d640 1 --2- 172.21.15.114:0/517736472 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe8c8009db0 0x7fe8dc1cc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.146+0000 7fe8e0d8d640 1 --2- 172.21.15.114:0/517736472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8c8006720 0x7fe8dc1cc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.146+0000 7fe8e0d8d640 1 -- 172.21.15.114:0/517736472 >> 172.21.15.114:0/517736472 conn(0x7fe8dc0a5d10 msgr2=0x7fe8dc0a7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:41.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.146+0000 7fe8e0d8d640 1 -- 172.21.15.114:0/517736472 shutdown_connections 2024-09-16T04:38:41.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.146+0000 7fe8e0d8d640 1 -- 172.21.15.114:0/517736472 wait complete. 2024-09-16T04:38:41.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new 37be2e88-32d0-4a30-8979-899b34f3be17 1 2024-09-16T04:38:41.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.257+0000 7fea2c653640 10 monclient: get_monmap_and_config 2024-09-16T04:38:41.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.257+0000 7fea2c653640 10 monclient: build_initial_monmap 2024-09-16T04:38:41.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.257+0000 7fea2c653640 10 monclient: monmap: 2024-09-16T04:38:41.259 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T04:38:41.259 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.259 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:41.258318+0000 2024-09-16T04:38:41.259 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:41.258318+0000 2024-09-16T04:38:41.259 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:41.259 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.259 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:38:41.259 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:38:41.259 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:38:41.260 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:38:41.260 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:38:41.260 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea2c653640 1 Processor -- start 2024-09-16T04:38:41.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea2c653640 1 -- start start 2024-09-16T04:38:41.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea2c653640 10 monclient: init 2024-09-16T04:38:41.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea2c653640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:41.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea2c653640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-16T04:38:41.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea2c653640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fea240a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:38:41.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea2c653640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fea240a3ff0 0x7fea240a4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea2c653640 10 monclient(hunting): picked mon.noname-c con 0x7fea240a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:38:41.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea2c653640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fea2413be90 0x7fea2413e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea2c653640 10 monclient(hunting): picked mon.noname-b con 0x7fea2413be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea2c653640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea2409ab70 con 0x7fea240a3c10 2024-09-16T04:38:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea2c653640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea2c653640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fea240a4d60 con 0x7fea240a3ff0 2024-09-16T04:38:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea2c653640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea2c653640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fea2409b030 con 0x7fea2413be90 2024-09-16T04:38:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea2c653640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea2c653640 10 monclient(hunting): authenticate will time out at 738.734790s 2024-09-16T04:38:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea29bee640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fea240a3ff0 0x7fea240a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea29bee640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fea240a3ff0 0x7fea240a4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48840/0 (socket says 172.21.15.114:48840) 2024-09-16T04:38:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea29bee640 1 -- 172.21.15.114:0/2975710504 learned_addr learned my addr 172.21.15.114:0/2975710504 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T04:38:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea29bee640 10 monclient(hunting): get_auth_request con 0x7fea240a3ff0 auth_method 0 2024-09-16T04:38:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea29bee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:41.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea29bee640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.258+0000 7fea29bee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:41.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.259+0000 7fea29bee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:41.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.259+0000 7fea29bee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:41.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.259+0000 7fea29bee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:41.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.259+0000 7fea2abf0640 1 --2- 172.21.15.114:0/2975710504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fea2413be90 0x7fea2413e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.259+0000 7fea29bee640 10 monclient(hunting): handle_auth_done global_id 4120 payload 274 2024-09-16T04:38:41.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.259+0000 7fea29bee640 1 -- 172.21.15.114:0/2975710504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fea2413be90 msgr2=0x7fea2413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:38:41.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.259+0000 7fea29bee640 1 --2- 172.21.15.114:0/2975710504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fea2413be90 0x7fea2413e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.259+0000 7fea29bee640 1 -- 172.21.15.114:0/2975710504 >> v1:172.21.15.46:6789/0 conn(0x7fea240a3c10 legacy=0x7fea2409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:38:41.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.259+0000 7fea29bee640 10 monclient: _finish_hunting 0 2024-09-16T04:38:41.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.259+0000 7fea29bee640 1 monclient: found mon.noname-c 2024-09-16T04:38:41.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.259+0000 7fea29bee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:41.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.259+0000 7fea29bee640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.259+0000 7fea29bee640 1 -- 172.21.15.114:0/2975710504 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea2409b310 con 0x7fea240a3ff0 2024-09-16T04:38:41.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.259+0000 7fea29bee640 1 --2- 172.21.15.114:0/2975710504 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea240a3ff0 0x7fea240a4400 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fea14009870 tx=0x7fea1402a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cc529f595791d67 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:41.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea293ed640 1 -- 172.21.15.114:0/2975710504 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea14037070 con 0x7fea240a3ff0 2024-09-16T04:38:41.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea293ed640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:41.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea293ed640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:38:41.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea293ed640 10 monclient: dump: 2024-09-16T04:38:41.271 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:41.271 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.271 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.271 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.271 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:41.271 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.271 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:41.271 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:41.271 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:41.272 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea293ed640 10 monclient: _finish_auth 0 2024-09-16T04:38:41.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea293ed640 10 monclient: _check_auth_tickets 2024-09-16T04:38:41.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea293ed640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:41.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea293ed640 1 -- 172.21.15.114:0/2975710504 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea1402ae10 con 0x7fea240a3ff0 2024-09-16T04:38:41.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea2c653640 5 monclient: authenticate success, global_id 4120 2024-09-16T04:38:41.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea2c653640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:38:41.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea293ed640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:41.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea2c653640 10 monclient: get_monmap_and_config success 2024-09-16T04:38:41.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea293ed640 1 -- 172.21.15.114:0/2975710504 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea14032660 con 0x7fea240a3ff0 2024-09-16T04:38:41.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea293ed640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:41.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea2c653640 4 set_mon_vals no callback set 2024-09-16T04:38:41.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea293ed640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:38:41.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea293ed640 10 monclient: dump: 2024-09-16T04:38:41.272 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:41.273 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.273 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.273 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.273 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:41.273 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.273 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:41.273 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:41.273 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:41.273 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea2c653640 10 monclient: shutdown 2024-09-16T04:38:41.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea2c653640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:41.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea2c653640 1 -- 172.21.15.114:0/2975710504 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea240a3ff0 msgr2=0x7fea240a4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea2c653640 1 --2- 172.21.15.114:0/2975710504 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea240a3ff0 0x7fea240a4400 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fea14009870 tx=0x7fea1402a100 comp rx=0 tx=0).stop 2024-09-16T04:38:41.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea2c653640 1 -- 172.21.15.114:0/2975710504 shutdown_connections 2024-09-16T04:38:41.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea2c653640 1 --2- 172.21.15.114:0/2975710504 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fea2413be90 0x7fea2413e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea2c653640 1 --2- 172.21.15.114:0/2975710504 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea240a3ff0 0x7fea240a4400 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea2c653640 1 -- 172.21.15.114:0/2975710504 >> 172.21.15.114:0/2975710504 conn(0x7fea240a5d10 msgr2=0x7fea240a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:41.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea2c653640 1 -- 172.21.15.114:0/2975710504 shutdown_connections 2024-09-16T04:38:41.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.260+0000 7fea2c653640 1 -- 172.21.15.114:0/2975710504 wait complete. 2024-09-16T04:38:41.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.261+0000 7fea2c653640 10 monclient: build_initial_monmap 2024-09-16T04:38:41.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.261+0000 7fea2c653640 10 monclient: monmap: 2024-09-16T04:38:41.275 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T04:38:41.275 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:38:41.275 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T04:38:41.275 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T04:38:41.275 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:41.275 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.275 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:38:41.276 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:38:41.276 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:38:41.276 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.261+0000 7fea2c653640 1 Processor -- start 2024-09-16T04:38:41.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.261+0000 7fea2c653640 1 -- start start 2024-09-16T04:38:41.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.261+0000 7fea2c653640 10 monclient: init 2024-09-16T04:38:41.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2c653640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:41.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2c653640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:38:41.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2c653640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea240a3c10 0x7fea2409fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2c653640 10 monclient(hunting): picked mon.noname-b con 0x7fea240a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:38:41.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2c653640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fea240a3ff0 0x7fea24130d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2c653640 10 monclient(hunting): picked mon.noname-c con 0x7fea240a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:41.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2c653640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea2413be90 0x7fea24131290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2c653640 10 monclient(hunting): picked mon.noname-a con 0x7fea2413be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:38:41.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2c653640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2c653640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fea2406e550 con 0x7fea2413be90 2024-09-16T04:38:41.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2c653640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2c653640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fea2406e3d0 con 0x7fea240a3c10 2024-09-16T04:38:41.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2c653640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2c653640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fea2413e7c0 con 0x7fea240a3ff0 2024-09-16T04:38:41.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2c653640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:41.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2c653640 10 monclient(hunting): authenticate will time out at 738.738238s 2024-09-16T04:38:41.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2a3ef640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea240a3c10 0x7fea2409fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2a3ef640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea240a3c10 0x7fea2409fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48854/0 (socket says 172.21.15.114:48854) 2024-09-16T04:38:41.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2a3ef640 1 -- 172.21.15.114:0/487009249 learned_addr learned my addr 172.21.15.114:0/487009249 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T04:38:41.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2a3ef640 10 monclient(hunting): get_auth_request con 0x7fea240a3c10 auth_method 0 2024-09-16T04:38:41.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2a3ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:41.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2a3ef640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2a3ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:41.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2a3ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:41.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2a3ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:41.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2a3ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:41.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2abf0640 1 --2- 172.21.15.114:0/487009249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea2413be90 0x7fea24131290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea29bee640 1 --2- 172.21.15.114:0/487009249 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fea240a3ff0 0x7fea24130d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2abf0640 10 monclient(hunting): get_auth_request con 0x7fea2413be90 auth_method 0 2024-09-16T04:38:41.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2abf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:41.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2abf0640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2abf0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:41.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea29bee640 10 monclient(hunting): get_auth_request con 0x7fea240a3ff0 auth_method 0 2024-09-16T04:38:41.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea29bee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:41.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea29bee640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea29bee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:41.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2abf0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:41.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2abf0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:41.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2abf0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:41.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2a3ef640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2024-09-16T04:38:41.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2a3ef640 1 -- 172.21.15.114:0/487009249 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fea240a3ff0 msgr2=0x7fea24130d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2a3ef640 1 --2- 172.21.15.114:0/487009249 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fea240a3ff0 0x7fea24130d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2a3ef640 1 -- 172.21.15.114:0/487009249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea2413be90 msgr2=0x7fea24131290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2a3ef640 1 --2- 172.21.15.114:0/487009249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea2413be90 0x7fea24131290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2a3ef640 10 monclient: _finish_hunting 0 2024-09-16T04:38:41.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2a3ef640 1 monclient: found mon.noname-b 2024-09-16T04:38:41.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2a3ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:41.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2a3ef640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2a3ef640 1 -- 172.21.15.114:0/487009249 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea2409f1f0 con 0x7fea240a3c10 2024-09-16T04:38:41.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea29bee640 1 --2- 172.21.15.114:0/487009249 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fea240a3ff0 0x7fea24130d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:38:41.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2a3ef640 1 --2- 172.21.15.114:0/487009249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea240a3c10 0x7fea2409fc60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fea2000c490 tx=0x7fea2000c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:41.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea1affd640 1 -- 172.21.15.114:0/487009249 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea20016020 con 0x7fea240a3c10 2024-09-16T04:38:41.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:41.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea1affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:38:41.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea1affd640 10 monclient: dump: 2024-09-16T04:38:41.281 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:41.281 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.281 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.281 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.281 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:41.281 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.281 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:41.281 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:41.281 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:41.281 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea1affd640 10 monclient: _finish_auth 0 2024-09-16T04:38:41.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea1affd640 10 monclient: _check_auth_tickets 2024-09-16T04:38:41.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:41.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea1affd640 1 -- 172.21.15.114:0/487009249 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea20012830 con 0x7fea240a3c10 2024-09-16T04:38:41.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2c653640 5 monclient: authenticate success, global_id 4123 2024-09-16T04:38:41.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2c653640 10 monclient: _renew_subs 2024-09-16T04:38:41.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2c653640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea2c653640 1 -- 172.21.15.114:0/487009249 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea2409f4d0 con 0x7fea240a3c10 2024-09-16T04:38:41.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea1affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:41.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea1affd640 1 -- 172.21.15.114:0/487009249 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea20012c20 con 0x7fea240a3c10 2024-09-16T04:38:41.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:41.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea1affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:38:41.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea1affd640 10 monclient: dump: 2024-09-16T04:38:41.282 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:41.282 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.282 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.282 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.283 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:41.283 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.283 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:41.283 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:41.283 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:41.283 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.262+0000 7fea28bec640 4 set_mon_vals no callback set 2024-09-16T04:38:41.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.263+0000 7fea2c653640 10 monclient: _renew_subs 2024-09-16T04:38:41.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.263+0000 7fea2c653640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.263+0000 7fea2c653640 1 -- 172.21.15.114:0/487009249 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea2409ef10 con 0x7fea240a3c10 2024-09-16T04:38:41.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.263+0000 7fea1affd640 1 -- 172.21.15.114:0/487009249 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fea20019020 con 0x7fea240a3c10 2024-09-16T04:38:41.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.263+0000 7fea1affd640 1 -- 172.21.15.114:0/487009249 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fea2001a7e0 con 0x7fea240a3c10 2024-09-16T04:38:41.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.263+0000 7fea2c653640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:41.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.263+0000 7fea2c653640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:41.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.263+0000 7fea2c653640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.263+0000 7fea2c653640 1 -- 172.21.15.114:0/487009249 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea24131830 con 0x7fea240a3c10 2024-09-16T04:38:41.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.265+0000 7fea1affd640 1 -- 172.21.15.114:0/487009249 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fea20020030 con 0x7fea240a3c10 2024-09-16T04:38:41.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.266+0000 7fea1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:41.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.266+0000 7fea1affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:38:41.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.336+0000 7fea2c653640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "37be2e88-32d0-4a30-8979-899b34f3be17", "id": 1}] 2024-09-16T04:38:41.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.336+0000 7fea2c653640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "37be2e88-32d0-4a30-8979-899b34f3be17", "id": 1}] 2024-09-16T04:38:41.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.336+0000 7fea2c653640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.336+0000 7fea2c653640 1 -- 172.21.15.114:0/487009249 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd new", "uuid": "37be2e88-32d0-4a30-8979-899b34f3be17", "id": 1} v 0) -- 0x7fea24067e40 con 0x7fea240a3c10 2024-09-16T04:38:41.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.344+0000 7fea1affd640 1 -- 172.21.15.114:0/487009249 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "37be2e88-32d0-4a30-8979-899b34f3be17", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fea20012dc0 con 0x7fea240a3c10 2024-09-16T04:38:41.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.344+0000 7fea1affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "37be2e88-32d0-4a30-8979-899b34f3be17", "id": 1}] 2024-09-16T04:38:41.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.344+0000 7fea1affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:38:41.346 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-09-16T04:38:41.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.345+0000 7fea2c653640 10 monclient: shutdown 2024-09-16T04:38:41.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.345+0000 7fea2c653640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:41.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.345+0000 7fea2c653640 1 -- 172.21.15.114:0/487009249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea240a3c10 msgr2=0x7fea2409fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.345+0000 7fea2c653640 1 --2- 172.21.15.114:0/487009249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea240a3c10 0x7fea2409fc60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fea2000c490 tx=0x7fea2000c950 comp rx=0 tx=0).stop 2024-09-16T04:38:41.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.345+0000 7fea2c653640 1 -- 172.21.15.114:0/487009249 shutdown_connections 2024-09-16T04:38:41.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.345+0000 7fea2c653640 1 --2- 172.21.15.114:0/487009249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea2413be90 0x7fea24131290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.345+0000 7fea2c653640 1 --2- 172.21.15.114:0/487009249 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fea240a3ff0 0x7fea24130d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.345+0000 7fea2c653640 1 --2- 172.21.15.114:0/487009249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea240a3c10 0x7fea2409fc60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.345+0000 7fea2c653640 1 -- 172.21.15.114:0/487009249 >> 172.21.15.114:0/487009249 conn(0x7fea240a5d10 msgr2=0x7fea2413b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:41.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.345+0000 7fea2c653640 1 -- 172.21.15.114:0/487009249 shutdown_connections 2024-09-16T04:38:41.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.345+0000 7fea2c653640 1 -- 172.21.15.114:0/487009249 wait complete. 2024-09-16T04:38:41.357 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new 73b9a838-944e-48cd-afab-5260608d1c6e 2 2024-09-16T04:38:41.403 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:41.401+0000 7faefb42d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T04:38:41.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.452+0000 7fd4a1933640 10 monclient: get_monmap_and_config 2024-09-16T04:38:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.452+0000 7fd4a1933640 10 monclient: build_initial_monmap 2024-09-16T04:38:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.453+0000 7fd4a1933640 10 monclient: monmap: 2024-09-16T04:38:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T04:38:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:41.453882+0000 2024-09-16T04:38:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:41.453882+0000 2024-09-16T04:38:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:38:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:38:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:38:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:38:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:38:41.455 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.453+0000 7fd4a1933640 1 Processor -- start 2024-09-16T04:38:41.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.453+0000 7fd4a1933640 1 -- start start 2024-09-16T04:38:41.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.453+0000 7fd4a1933640 10 monclient: init 2024-09-16T04:38:41.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd4a1933640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:41.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd4a1933640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T04:38:41.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd4a1933640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd49c0a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:38:41.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd4a1933640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd49c0a3ff0 0x7fd49c0a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd4a1933640 10 monclient(hunting): picked mon.noname-b con 0x7fd49c0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:41.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd4a1933640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd49c13be90 0x7fd49c13e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd4a1933640 10 monclient(hunting): picked mon.noname-c con 0x7fd49c13be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:38:41.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd4a1933640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd49c09ab70 con 0x7fd49c0a3c10 2024-09-16T04:38:41.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd4a1933640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd4a1933640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fd49c0a4d60 con 0x7fd49c13be90 2024-09-16T04:38:41.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd4a1933640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd4a1933640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd49c09b030 con 0x7fd49c0a3ff0 2024-09-16T04:38:41.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd4a1933640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:41.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd4a1933640 10 monclient(hunting): authenticate will time out at 738.930148s 2024-09-16T04:38:41.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49b7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd49c13be90 0x7fd49c13e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49b7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd49c13be90 0x7fd49c13e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48864/0 (socket says 172.21.15.114:48864) 2024-09-16T04:38:41.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49b7fe640 1 -- 172.21.15.114:0/1766558528 learned_addr learned my addr 172.21.15.114:0/1766558528 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T04:38:41.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49b7fe640 10 monclient(hunting): get_auth_request con 0x7fd49c13be90 auth_method 0 2024-09-16T04:38:41.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:41.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:41.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49a7fc640 1 --2- 172.21.15.114:0/1766558528 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd49c0a3ff0 0x7fd49c0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49a7fc640 10 monclient(hunting): get_auth_request con 0x7fd49c0a3ff0 auth_method 0 2024-09-16T04:38:41.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:41.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:41.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:41.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:41.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:41.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:41.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:41.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:41.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49a7fc640 1 -- 172.21.15.114:0/1766558528 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd49c0a3ff0 msgr2=0x7fd49c0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:38:41.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49a7fc640 1 -- 172.21.15.114:0/1766558528 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd49c0a3ff0 msgr2=0x7fd49c0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:38:41.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49a7fc640 1 --2- 172.21.15.114:0/1766558528 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd49c0a3ff0 0x7fd49c0a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:38:41.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49a7fc640 1 --2- 172.21.15.114:0/1766558528 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd49c0a3ff0 0x7fd49c0a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:38:41.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49b7fe640 10 monclient(hunting): handle_auth_done global_id 4126 payload 274 2024-09-16T04:38:41.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49b7fe640 1 -- 172.21.15.114:0/1766558528 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd49c0a3ff0 msgr2=0x7fd49c0a43d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:38:41.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49b7fe640 1 --2- 172.21.15.114:0/1766558528 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd49c0a3ff0 0x7fd49c0a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49b7fe640 1 -- 172.21.15.114:0/1766558528 >> v1:172.21.15.46:6789/0 conn(0x7fd49c0a3c10 legacy=0x7fd49c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:38:41.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49b7fe640 1 monclient: found mon.noname-c 2024-09-16T04:38:41.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:41.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.454+0000 7fd49b7fe640 1 -- 172.21.15.114:0/1766558528 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd49c09b310 con 0x7fd49c13be90 2024-09-16T04:38:41.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.455+0000 7fd49b7fe640 1 --2- 172.21.15.114:0/1766558528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd49c13be90 0x7fd49c13e280 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd494009e50 tx=0x7fd49402a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18c3a871c7c16202 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:41.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.455+0000 7fd499ffb640 1 -- 172.21.15.114:0/1766558528 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd494036070 con 0x7fd49c13be90 2024-09-16T04:38:41.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.455+0000 7fd499ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:41.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.455+0000 7fd499ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:38:41.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.455+0000 7fd499ffb640 10 monclient: dump: 2024-09-16T04:38:41.466 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:41.466 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.466 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.466 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.466 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:41.466 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.466 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:41.466 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:41.466 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:41.466 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.455+0000 7fd499ffb640 10 monclient: _finish_auth 0 2024-09-16T04:38:41.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.455+0000 7fd499ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:38:41.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.455+0000 7fd499ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:41.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.455+0000 7fd4a1933640 5 monclient: authenticate success, global_id 4126 2024-09-16T04:38:41.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.455+0000 7fd499ffb640 1 -- 172.21.15.114:0/1766558528 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd494031600 con 0x7fd49c13be90 2024-09-16T04:38:41.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.455+0000 7fd4a1933640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:38:41.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.455+0000 7fd499ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:41.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.455+0000 7fd4a1933640 10 monclient: get_monmap_and_config success 2024-09-16T04:38:41.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.455+0000 7fd4a1933640 4 set_mon_vals no callback set 2024-09-16T04:38:41.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.455+0000 7fd499ffb640 1 -- 172.21.15.114:0/1766558528 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4940319d0 con 0x7fd49c13be90 2024-09-16T04:38:41.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.455+0000 7fd499ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:41.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.455+0000 7fd499ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:38:41.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.455+0000 7fd499ffb640 10 monclient: dump: 2024-09-16T04:38:41.468 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:41.468 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.468 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.468 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.468 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:41.468 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.468 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:41.468 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:41.468 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:41.468 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.456+0000 7fd4a1933640 10 monclient: shutdown 2024-09-16T04:38:41.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.456+0000 7fd4a1933640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:41.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.456+0000 7fd4a1933640 1 -- 172.21.15.114:0/1766558528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd49c13be90 msgr2=0x7fd49c13e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.456+0000 7fd4a1933640 1 --2- 172.21.15.114:0/1766558528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd49c13be90 0x7fd49c13e280 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd494009e50 tx=0x7fd49402a5f0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.456+0000 7fd4a1933640 1 -- 172.21.15.114:0/1766558528 shutdown_connections 2024-09-16T04:38:41.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.456+0000 7fd4a1933640 1 --2- 172.21.15.114:0/1766558528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd49c13be90 0x7fd49c13e280 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.456+0000 7fd4a1933640 1 --2- 172.21.15.114:0/1766558528 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd49c0a3ff0 0x7fd49c0a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.456+0000 7fd4a1933640 1 -- 172.21.15.114:0/1766558528 >> 172.21.15.114:0/1766558528 conn(0x7fd49c0a5d10 msgr2=0x7fd49c0a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:41.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.456+0000 7fd4a1933640 1 -- 172.21.15.114:0/1766558528 shutdown_connections 2024-09-16T04:38:41.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.456+0000 7fd4a1933640 1 -- 172.21.15.114:0/1766558528 wait complete. 2024-09-16T04:38:41.469 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:41.456+0000 7faefb42d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T04:38:41.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.456+0000 7fd4a1933640 10 monclient: build_initial_monmap 2024-09-16T04:38:41.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.456+0000 7fd4a1933640 10 monclient: monmap: 2024-09-16T04:38:41.476 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T04:38:41.476 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:38:41.477 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T04:38:41.477 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T04:38:41.477 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:41.477 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.477 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:38:41.477 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:38:41.477 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:38:41.478 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd4a1933640 1 Processor -- start 2024-09-16T04:38:41.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd4a1933640 1 -- start start 2024-09-16T04:38:41.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd4a1933640 10 monclient: init 2024-09-16T04:38:41.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd4a1933640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:41.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd4a1933640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:38:41.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd4a1933640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd49c0a3c10 0x7fd49c1cbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd4a1933640 10 monclient(hunting): picked mon.noname-c con 0x7fd49c0a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:41.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd4a1933640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd49c0a3ff0 0x7fd49c13a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd4a1933640 10 monclient(hunting): picked mon.noname-b con 0x7fd49c0a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:38:41.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd4a1933640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd49c13be90 0x7fd49c13ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd4a1933640 10 monclient(hunting): picked mon.noname-a con 0x7fd49c13be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:38:41.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd4a1933640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd4a1933640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd49c06e550 con 0x7fd49c13be90 2024-09-16T04:38:41.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd4a1933640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd4a1933640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd49c06e3d0 con 0x7fd49c0a3ff0 2024-09-16T04:38:41.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd4a1933640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd4a1933640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd49c13e7c0 con 0x7fd49c0a3c10 2024-09-16T04:38:41.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd4a1933640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:41.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd4a1933640 10 monclient(hunting): authenticate will time out at 738.933693s 2024-09-16T04:38:41.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49a7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd49c0a3ff0 0x7fd49c13a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49a7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd49c0a3ff0 0x7fd49c13a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48870/0 (socket says 172.21.15.114:48870) 2024-09-16T04:38:41.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49a7fc640 1 -- 172.21.15.114:0/2924434243 learned_addr learned my addr 172.21.15.114:0/2924434243 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T04:38:41.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49a7fc640 10 monclient(hunting): get_auth_request con 0x7fd49c0a3ff0 auth_method 0 2024-09-16T04:38:41.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:41.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49b7fe640 1 --2- 172.21.15.114:0/2924434243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd49c13be90 0x7fd49c13ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:41.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49affd640 1 --2- 172.21.15.114:0/2924434243 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd49c0a3c10 0x7fd49c1cbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49b7fe640 10 monclient(hunting): get_auth_request con 0x7fd49c13be90 auth_method 0 2024-09-16T04:38:41.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:41.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:41.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49affd640 10 monclient(hunting): get_auth_request con 0x7fd49c0a3c10 auth_method 0 2024-09-16T04:38:41.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:41.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:41.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:41.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:41.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:41.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:41.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:41.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.457+0000 7fd49b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:41.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd49affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:41.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd49affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:41.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd49affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:41.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd49a7fc640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2024-09-16T04:38:41.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd49a7fc640 1 -- 172.21.15.114:0/2924434243 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd49c0a3c10 msgr2=0x7fd49c1cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd49a7fc640 1 --2- 172.21.15.114:0/2924434243 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd49c0a3c10 0x7fd49c1cbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd49a7fc640 1 -- 172.21.15.114:0/2924434243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd49c13be90 msgr2=0x7fd49c13ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd49a7fc640 1 --2- 172.21.15.114:0/2924434243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd49c13be90 0x7fd49c13ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd49a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:38:41.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd49a7fc640 1 monclient: found mon.noname-b 2024-09-16T04:38:41.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd49a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:41.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd49a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd49a7fc640 1 -- 172.21.15.114:0/2924434243 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd49c09f090 con 0x7fd49c0a3ff0 2024-09-16T04:38:41.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd49b7fe640 1 --2- 172.21.15.114:0/2924434243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd49c13be90 0x7fd49c13ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:38:41.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd49a7fc640 1 --2- 172.21.15.114:0/2924434243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd49c0a3ff0 0x7fd49c13a710 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd49000e820 tx=0x7fd49000ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:41.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd483fff640 1 -- 172.21.15.114:0/2924434243 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd490017070 con 0x7fd49c0a3ff0 2024-09-16T04:38:41.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd483fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:41.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd483fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:38:41.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd483fff640 10 monclient: dump: 2024-09-16T04:38:41.488 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:41.488 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.488 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.488 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.488 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:41.489 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.489 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:41.489 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:41.489 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:41.489 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd483fff640 10 monclient: _finish_auth 0 2024-09-16T04:38:41.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd483fff640 10 monclient: _check_auth_tickets 2024-09-16T04:38:41.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd483fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:41.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd483fff640 1 -- 172.21.15.114:0/2924434243 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd49000c940 con 0x7fd49c0a3ff0 2024-09-16T04:38:41.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd4a1933640 5 monclient: authenticate success, global_id 4129 2024-09-16T04:38:41.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd4a1933640 10 monclient: _renew_subs 2024-09-16T04:38:41.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd4a1933640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd4a1933640 1 -- 172.21.15.114:0/2924434243 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd49c0a0400 con 0x7fd49c0a3ff0 2024-09-16T04:38:41.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd483fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:41.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd483fff640 1 -- 172.21.15.114:0/2924434243 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49000cd50 con 0x7fd49c0a3ff0 2024-09-16T04:38:41.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd483fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:41.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd483fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:38:41.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd483fff640 10 monclient: dump: 2024-09-16T04:38:41.490 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:41.490 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.490 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.490 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.490 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:41.490 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.490 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:41.491 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:41.491 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:41.491 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd4a1933640 10 monclient: _renew_subs 2024-09-16T04:38:41.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd4a1933640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd4a1933640 1 -- 172.21.15.114:0/2924434243 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd49c09ff10 con 0x7fd49c0a3ff0 2024-09-16T04:38:41.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd4997fa640 4 set_mon_vals no callback set 2024-09-16T04:38:41.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd483fff640 1 -- 172.21.15.114:0/2924434243 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd49001a070 con 0x7fd49c0a3ff0 2024-09-16T04:38:41.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.458+0000 7fd483fff640 1 -- 172.21.15.114:0/2924434243 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fd490006640 con 0x7fd49c0a3ff0 2024-09-16T04:38:41.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.459+0000 7fd4a1933640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:41.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.459+0000 7fd4a1933640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:41.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.459+0000 7fd4a1933640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.459+0000 7fd4a1933640 1 -- 172.21.15.114:0/2924434243 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd49c0a43d0 con 0x7fd49c0a3ff0 2024-09-16T04:38:41.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.463+0000 7fd483fff640 1 -- 172.21.15.114:0/2924434243 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fd490006020 con 0x7fd49c0a3ff0 2024-09-16T04:38:41.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.463+0000 7fd483fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:41.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.463+0000 7fd483fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:38:41.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.531+0000 7fd4a1933640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "73b9a838-944e-48cd-afab-5260608d1c6e", "id": 2}] 2024-09-16T04:38:41.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.531+0000 7fd4a1933640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "73b9a838-944e-48cd-afab-5260608d1c6e", "id": 2}] 2024-09-16T04:38:41.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.531+0000 7fd4a1933640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.531+0000 7fd4a1933640 1 -- 172.21.15.114:0/2924434243 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd new", "uuid": "73b9a838-944e-48cd-afab-5260608d1c6e", "id": 2} v 0) -- 0x7fd49c069150 con 0x7fd49c0a3ff0 2024-09-16T04:38:41.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.540+0000 7fd483fff640 1 -- 172.21.15.114:0/2924434243 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "73b9a838-944e-48cd-afab-5260608d1c6e", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fd490005990 con 0x7fd49c0a3ff0 2024-09-16T04:38:41.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.540+0000 7fd483fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "73b9a838-944e-48cd-afab-5260608d1c6e", "id": 2}] 2024-09-16T04:38:41.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.540+0000 7fd483fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:38:41.542 INFO:teuthology.orchestra.run.smithi114.stdout:2 2024-09-16T04:38:41.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.541+0000 7fd4a1933640 10 monclient: shutdown 2024-09-16T04:38:41.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.541+0000 7fd4a1933640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:41.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.541+0000 7fd4a1933640 1 -- 172.21.15.114:0/2924434243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd49c0a3ff0 msgr2=0x7fd49c13a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.541+0000 7fd4a1933640 1 --2- 172.21.15.114:0/2924434243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd49c0a3ff0 0x7fd49c13a710 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd49000e820 tx=0x7fd49000ece0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.541+0000 7fd4a1933640 1 -- 172.21.15.114:0/2924434243 shutdown_connections 2024-09-16T04:38:41.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.541+0000 7fd4a1933640 1 --2- 172.21.15.114:0/2924434243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd49c13be90 0x7fd49c13ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.541+0000 7fd4a1933640 1 --2- 172.21.15.114:0/2924434243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd49c0a3ff0 0x7fd49c13a710 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.541+0000 7fd4a1933640 1 --2- 172.21.15.114:0/2924434243 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd49c0a3c10 0x7fd49c1cbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.541+0000 7fd4a1933640 1 -- 172.21.15.114:0/2924434243 >> 172.21.15.114:0/2924434243 conn(0x7fd49c0a5d10 msgr2=0x7fd49c13de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:41.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.541+0000 7fd4a1933640 1 -- 172.21.15.114:0/2924434243 shutdown_connections 2024-09-16T04:38:41.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.541+0000 7fd4a1933640 1 -- 172.21.15.114:0/2924434243 wait complete. 2024-09-16T04:38:41.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new d0c22455-5685-4ade-b540-3d5428a7cb46 3 2024-09-16T04:38:41.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.648+0000 7f87a7f03640 10 monclient: get_monmap_and_config 2024-09-16T04:38:41.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.648+0000 7f87a7f03640 10 monclient: build_initial_monmap 2024-09-16T04:38:41.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.649+0000 7f87a7f03640 10 monclient: monmap: 2024-09-16T04:38:41.651 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T04:38:41.651 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.651 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:41.649903+0000 2024-09-16T04:38:41.651 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:41.649903+0000 2024-09-16T04:38:41.651 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:41.651 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.651 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:38:41.651 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:38:41.652 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:38:41.652 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:38:41.652 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:38:41.652 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.649+0000 7f87a7f03640 1 Processor -- start 2024-09-16T04:38:41.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.649+0000 7f87a7f03640 1 -- start start 2024-09-16T04:38:41.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.649+0000 7f87a7f03640 10 monclient: init 2024-09-16T04:38:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a7f03640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a7f03640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-16T04:38:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a7f03640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f87a00a3c10 0x7f87a009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a7f03640 10 monclient(hunting): picked mon.noname-b con 0x7f87a00a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a7f03640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f87a00a3ff0 0x7f87a00a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a7f03640 10 monclient(hunting): picked mon.noname-c con 0x7f87a00a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:38:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a7f03640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f87a01369d0 addr v1:172.21.15.46:6789/0 2024-09-16T04:38:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a7f03640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87a009ab70 con 0x7f87a01369d0 2024-09-16T04:38:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a7f03640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a7f03640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f87a00a4d60 con 0x7f87a00a3ff0 2024-09-16T04:38:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a7f03640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a7f03640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f87a009b030 con 0x7f87a00a3c10 2024-09-16T04:38:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a7f03640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a7f03640 10 monclient(hunting): authenticate will time out at 739.126353s 2024-09-16T04:38:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a549e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f87a00a3ff0 0x7f87a00a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a549e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f87a00a3ff0 0x7f87a00a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48886/0 (socket says 172.21.15.114:48886) 2024-09-16T04:38:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a549e640 1 -- 172.21.15.114:0/1349236282 learned_addr learned my addr 172.21.15.114:0/1349236282 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T04:38:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a549e640 10 monclient(hunting): get_auth_request con 0x7f87a00a3ff0 auth_method 0 2024-09-16T04:38:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a549e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a549e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a549e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a549e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a549e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a549e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a5c9f640 1 --2- 172.21.15.114:0/1349236282 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f87a00a3c10 0x7f87a009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a5c9f640 10 monclient(hunting): get_auth_request con 0x7f87a00a3c10 auth_method 0 2024-09-16T04:38:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a5c9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a5c9f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a5c9f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a5c9f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a5c9f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a5c9f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a549e640 10 monclient(hunting): handle_auth_done global_id 4132 payload 274 2024-09-16T04:38:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a549e640 1 -- 172.21.15.114:0/1349236282 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f87a00a3c10 msgr2=0x7f87a009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:38:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a549e640 1 --2- 172.21.15.114:0/1349236282 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f87a00a3c10 0x7f87a009bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a549e640 1 -- 172.21.15.114:0/1349236282 >> v1:172.21.15.46:6789/0 conn(0x7f87a01369d0 legacy=0x7f87a0136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a549e640 10 monclient: _finish_hunting 0 2024-09-16T04:38:41.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a549e640 1 monclient: found mon.noname-c 2024-09-16T04:38:41.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a549e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:41.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a549e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.650+0000 7f87a549e640 1 -- 172.21.15.114:0/1349236282 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a009b310 con 0x7f87a00a3ff0 2024-09-16T04:38:41.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.651+0000 7f87a549e640 1 --2- 172.21.15.114:0/1349236282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87a00a3ff0 0x7f87a00a43d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f879c009870 tx=0x7f879c02a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37c8238a8a0ae20d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:41.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.651+0000 7f87a4c9d640 1 -- 172.21.15.114:0/1349236282 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879c037070 con 0x7f87a00a3ff0 2024-09-16T04:38:41.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.651+0000 7f87a4c9d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:41.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.651+0000 7f87a4c9d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:38:41.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.651+0000 7f87a4c9d640 10 monclient: dump: 2024-09-16T04:38:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:41.663 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.651+0000 7f87a4c9d640 10 monclient: _finish_auth 0 2024-09-16T04:38:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.651+0000 7f87a4c9d640 10 monclient: _check_auth_tickets 2024-09-16T04:38:41.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.651+0000 7f87a4c9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:41.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.651+0000 7f87a4c9d640 1 -- 172.21.15.114:0/1349236282 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f879c032440 con 0x7f87a00a3ff0 2024-09-16T04:38:41.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.651+0000 7f87a7f03640 5 monclient: authenticate success, global_id 4132 2024-09-16T04:38:41.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.651+0000 7f87a7f03640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:38:41.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.651+0000 7f87a4c9d640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:41.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.651+0000 7f87a7f03640 10 monclient: get_monmap_and_config success 2024-09-16T04:38:41.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.651+0000 7f87a4c9d640 1 -- 172.21.15.114:0/1349236282 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879c032810 con 0x7f87a00a3ff0 2024-09-16T04:38:41.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.651+0000 7f87a7f03640 4 set_mon_vals no callback set 2024-09-16T04:38:41.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.651+0000 7f87a4c9d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:41.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.651+0000 7f87a4c9d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:38:41.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.651+0000 7f87a4c9d640 10 monclient: dump: 2024-09-16T04:38:41.664 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:41.664 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.665 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.665 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.665 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:41.665 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.665 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:41.665 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:41.665 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:41.665 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.652+0000 7f87a7f03640 10 monclient: shutdown 2024-09-16T04:38:41.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.652+0000 7f87a7f03640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:41.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.652+0000 7f87a7f03640 1 -- 172.21.15.114:0/1349236282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87a00a3ff0 msgr2=0x7f87a00a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.652+0000 7f87a7f03640 1 --2- 172.21.15.114:0/1349236282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87a00a3ff0 0x7f87a00a43d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f879c009870 tx=0x7f879c02a1c0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.652+0000 7f87a7f03640 1 -- 172.21.15.114:0/1349236282 shutdown_connections 2024-09-16T04:38:41.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.652+0000 7f87a7f03640 1 --2- 172.21.15.114:0/1349236282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87a00a3ff0 0x7f87a00a43d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.652+0000 7f87a7f03640 1 --2- 172.21.15.114:0/1349236282 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f87a00a3c10 0x7f87a009bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.652+0000 7f87a7f03640 1 -- 172.21.15.114:0/1349236282 >> 172.21.15.114:0/1349236282 conn(0x7f87a00a5d10 msgr2=0x7f87a00a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:41.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.652+0000 7f87a7f03640 1 -- 172.21.15.114:0/1349236282 shutdown_connections 2024-09-16T04:38:41.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.652+0000 7f87a7f03640 1 -- 172.21.15.114:0/1349236282 wait complete. 2024-09-16T04:38:41.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.652+0000 7f87a7f03640 10 monclient: build_initial_monmap 2024-09-16T04:38:41.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.652+0000 7f87a7f03640 10 monclient: monmap: 2024-09-16T04:38:41.666 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T04:38:41.666 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:38:41.666 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T04:38:41.666 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T04:38:41.667 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:41.667 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.667 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:38:41.667 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:38:41.667 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:38:41.667 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a7f03640 1 Processor -- start 2024-09-16T04:38:41.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a7f03640 1 -- start start 2024-09-16T04:38:41.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a7f03640 10 monclient: init 2024-09-16T04:38:41.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a7f03640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:41.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a7f03640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:38:41.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a7f03640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f87a00a3c10 0x7f87a01cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a7f03640 10 monclient(hunting): picked mon.noname-c con 0x7f87a00a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:41.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a7f03640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87a00a3ff0 0x7f87a01cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a7f03640 10 monclient(hunting): picked mon.noname-b con 0x7f87a00a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:38:41.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a7f03640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87a01369d0 0x7f87a0123210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a7f03640 10 monclient(hunting): picked mon.noname-a con 0x7f87a01369d0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:38:41.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a7f03640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a7f03640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f87a00a4910 con 0x7f87a01369d0 2024-09-16T04:38:41.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a7f03640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a7f03640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f87a006e3d0 con 0x7f87a00a3ff0 2024-09-16T04:38:41.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a7f03640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a7f03640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f87a0126580 con 0x7f87a00a3c10 2024-09-16T04:38:41.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a7f03640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:41.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a7f03640 10 monclient(hunting): authenticate will time out at 739.129803s 2024-09-16T04:38:41.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a549e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87a00a3ff0 0x7f87a01cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a549e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87a00a3ff0 0x7f87a01cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48898/0 (socket says 172.21.15.114:48898) 2024-09-16T04:38:41.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a549e640 1 -- 172.21.15.114:0/2402315176 learned_addr learned my addr 172.21.15.114:0/2402315176 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T04:38:41.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a549e640 10 monclient(hunting): get_auth_request con 0x7f87a00a3ff0 auth_method 0 2024-09-16T04:38:41.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a549e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:41.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a549e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a549e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:41.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a5c9f640 1 --2- 172.21.15.114:0/2402315176 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f87a00a3c10 0x7f87a01cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a549e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:41.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a549e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:41.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a64a0640 1 --2- 172.21.15.114:0/2402315176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87a01369d0 0x7f87a0123210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a549e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:41.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a5c9f640 10 monclient(hunting): get_auth_request con 0x7f87a00a3c10 auth_method 0 2024-09-16T04:38:41.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a5c9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:41.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a5c9f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a5c9f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:41.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a64a0640 10 monclient(hunting): get_auth_request con 0x7f87a01369d0 auth_method 0 2024-09-16T04:38:41.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.653+0000 7f87a64a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:41.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a64a0640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a64a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:41.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a5c9f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:41.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a5c9f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:41.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a5c9f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:41.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a64a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:41.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a64a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:41.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a64a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:41.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a549e640 10 monclient(hunting): handle_auth_done global_id 4135 payload 824 2024-09-16T04:38:41.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a549e640 1 -- 172.21.15.114:0/2402315176 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f87a00a3c10 msgr2=0x7f87a01cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a549e640 1 --2- 172.21.15.114:0/2402315176 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f87a00a3c10 0x7f87a01cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a549e640 1 -- 172.21.15.114:0/2402315176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87a01369d0 msgr2=0x7f87a0123210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a549e640 1 --2- 172.21.15.114:0/2402315176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87a01369d0 0x7f87a0123210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a549e640 10 monclient: _finish_hunting 0 2024-09-16T04:38:41.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a549e640 1 monclient: found mon.noname-b 2024-09-16T04:38:41.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a549e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:41.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a549e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a549e640 1 -- 172.21.15.114:0/2402315176 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a01259c0 con 0x7f87a00a3ff0 2024-09-16T04:38:41.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a549e640 1 --2- 172.21.15.114:0/2402315176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87a00a3ff0 0x7f87a01cc420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f879c02a200 tx=0x7f879c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:41.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87967fc640 1 -- 172.21.15.114:0/2402315176 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879c037040 con 0x7f87a00a3ff0 2024-09-16T04:38:41.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:41.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:38:41.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87967fc640 10 monclient: dump: 2024-09-16T04:38:41.672 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:41.672 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.672 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.672 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.672 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:41.672 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.672 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:41.672 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:41.672 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:41.672 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87967fc640 10 monclient: _finish_auth 0 2024-09-16T04:38:41.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87967fc640 10 monclient: _check_auth_tickets 2024-09-16T04:38:41.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:41.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87967fc640 1 -- 172.21.15.114:0/2402315176 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f879c042070 con 0x7f87a00a3ff0 2024-09-16T04:38:41.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a7f03640 5 monclient: authenticate success, global_id 4135 2024-09-16T04:38:41.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a7f03640 10 monclient: _renew_subs 2024-09-16T04:38:41.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a7f03640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a7f03640 1 -- 172.21.15.114:0/2402315176 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87a0127150 con 0x7f87a00a3ff0 2024-09-16T04:38:41.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87967fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:41.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87967fc640 1 -- 172.21.15.114:0/2402315176 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879c030880 con 0x7f87a00a3ff0 2024-09-16T04:38:41.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:41.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:38:41.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87967fc640 10 monclient: dump: 2024-09-16T04:38:41.674 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:41.674 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.674 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.674 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.674 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:41.674 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.674 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:41.674 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:41.674 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:41.674 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f8797fff640 4 set_mon_vals no callback set 2024-09-16T04:38:41.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a7f03640 10 monclient: _renew_subs 2024-09-16T04:38:41.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a7f03640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87a7f03640 1 -- 172.21.15.114:0/2402315176 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87a006def0 con 0x7f87a00a3ff0 2024-09-16T04:38:41.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87967fc640 1 -- 172.21.15.114:0/2402315176 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f879c02ad10 con 0x7f87a00a3ff0 2024-09-16T04:38:41.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.654+0000 7f87967fc640 1 -- 172.21.15.114:0/2402315176 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f879c047cb0 con 0x7f87a00a3ff0 2024-09-16T04:38:41.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.655+0000 7f87a7f03640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:41.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.655+0000 7f87a7f03640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:41.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.655+0000 7f87a7f03640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.655+0000 7f87a7f03640 1 -- 172.21.15.114:0/2402315176 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87a0137580 con 0x7f87a00a3ff0 2024-09-16T04:38:41.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.659+0000 7f87967fc640 1 -- 172.21.15.114:0/2402315176 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f879c047820 con 0x7f87a00a3ff0 2024-09-16T04:38:41.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.659+0000 7f87967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:41.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.659+0000 7f87967fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:38:41.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.728+0000 7f87a7f03640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d0c22455-5685-4ade-b540-3d5428a7cb46", "id": 3}] 2024-09-16T04:38:41.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.728+0000 7f87a7f03640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d0c22455-5685-4ade-b540-3d5428a7cb46", "id": 3}] 2024-09-16T04:38:41.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.728+0000 7f87a7f03640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.728+0000 7f87a7f03640 1 -- 172.21.15.114:0/2402315176 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d0c22455-5685-4ade-b540-3d5428a7cb46", "id": 3} v 0) -- 0x7f87a0069150 con 0x7f87a00a3ff0 2024-09-16T04:38:41.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.734+0000 7f87967fc640 1 -- 172.21.15.114:0/2402315176 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d0c22455-5685-4ade-b540-3d5428a7cb46", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f879c047aa0 con 0x7f87a00a3ff0 2024-09-16T04:38:41.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.734+0000 7f87967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d0c22455-5685-4ade-b540-3d5428a7cb46", "id": 3}] 2024-09-16T04:38:41.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.734+0000 7f87967fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:38:41.736 INFO:teuthology.orchestra.run.smithi114.stdout:3 2024-09-16T04:38:41.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.735+0000 7f87a7f03640 10 monclient: shutdown 2024-09-16T04:38:41.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.735+0000 7f87a7f03640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:41.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.735+0000 7f87a7f03640 1 -- 172.21.15.114:0/2402315176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87a00a3ff0 msgr2=0x7f87a01cc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.735+0000 7f87a7f03640 1 --2- 172.21.15.114:0/2402315176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87a00a3ff0 0x7f87a01cc420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f879c02a200 tx=0x7f879c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.735+0000 7f87a7f03640 1 -- 172.21.15.114:0/2402315176 shutdown_connections 2024-09-16T04:38:41.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.735+0000 7f87a7f03640 1 --2- 172.21.15.114:0/2402315176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87a01369d0 0x7f87a0123210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.735+0000 7f87a7f03640 1 --2- 172.21.15.114:0/2402315176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87a00a3ff0 0x7f87a01cc420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.735+0000 7f87a7f03640 1 --2- 172.21.15.114:0/2402315176 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f87a00a3c10 0x7f87a01cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.735+0000 7f87a7f03640 1 -- 172.21.15.114:0/2402315176 >> 172.21.15.114:0/2402315176 conn(0x7f87a00a5d10 msgr2=0x7f87a009e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:41.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.735+0000 7f87a7f03640 1 -- 172.21.15.114:0/2402315176 shutdown_connections 2024-09-16T04:38:41.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.735+0000 7f87a7f03640 1 -- 172.21.15.114:0/2402315176 wait complete. 2024-09-16T04:38:41.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new f37724e4-2796-43c8-975b-4458986c4ea6 4 2024-09-16T04:38:41.843 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:41.842+0000 7faefb42d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T04:38:41.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.843+0000 7ff64b3c2640 10 monclient: get_monmap_and_config 2024-09-16T04:38:41.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.843+0000 7ff64b3c2640 10 monclient: build_initial_monmap 2024-09-16T04:38:41.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.843+0000 7ff64b3c2640 10 monclient: monmap: 2024-09-16T04:38:41.845 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T04:38:41.845 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.845 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:41.844547+0000 2024-09-16T04:38:41.845 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:41.844547+0000 2024-09-16T04:38:41.845 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:41.845 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.845 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:38:41.845 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:38:41.845 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:38:41.845 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:38:41.846 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:38:41.846 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.844+0000 7ff64b3c2640 1 Processor -- start 2024-09-16T04:38:41.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.844+0000 7ff64b3c2640 1 -- start start 2024-09-16T04:38:41.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.844+0000 7ff64b3c2640 10 monclient: init 2024-09-16T04:38:41.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.844+0000 7ff64b3c2640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:41.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.844+0000 7ff64b3c2640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-16T04:38:41.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.844+0000 7ff64b3c2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff64409fa20 addr v1:172.21.15.46:6789/0 2024-09-16T04:38:41.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.844+0000 7ff64b3c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff644137b00 addr v1:172.21.15.114:6789/0 2024-09-16T04:38:41.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.844+0000 7ff64b3c2640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff64413b610 0x7ff64413da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.844+0000 7ff64b3c2640 10 monclient(hunting): picked mon.noname-c con 0x7ff64413b610 addr v2:172.21.15.114:3300/0 2024-09-16T04:38:41.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.844+0000 7ff64b3c2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff64409d420 con 0x7ff64409fa20 2024-09-16T04:38:41.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.844+0000 7ff64b3c2640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff64406c1e0 con 0x7ff644137b00 2024-09-16T04:38:41.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.844+0000 7ff64b3c2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.844+0000 7ff64b3c2640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ff64409dc50 con 0x7ff64413b610 2024-09-16T04:38:41.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.844+0000 7ff64b3c2640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:41.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.844+0000 7ff64b3c2640 10 monclient(hunting): authenticate will time out at 739.320904s 2024-09-16T04:38:41.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.845+0000 7ff64995f640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff64413b610 0x7ff64413da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.845+0000 7ff64895d640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff644137b00 0x7ff644139ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36570/0 (socket says 172.21.15.114:36570) 2024-09-16T04:38:41.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.845+0000 7ff64895d640 1 -- 172.21.15.114:0/2373710514 learned_addr learned my addr 172.21.15.114:0/2373710514 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T04:38:41.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.845+0000 7ff64995f640 10 monclient(hunting): get_auth_request con 0x7ff64413b610 auth_method 0 2024-09-16T04:38:41.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.845+0000 7ff64995f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:41.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.845+0000 7ff64995f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.845+0000 7ff64995f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:41.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.845+0000 7ff64995f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:41.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.845+0000 7ff64995f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:41.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.845+0000 7ff64995f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:41.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.845+0000 7ff64995f640 10 monclient(hunting): handle_auth_done global_id 4138 payload 274 2024-09-16T04:38:41.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.845+0000 7ff64995f640 1 -- 172.21.15.114:0/2373710514 >> v1:172.21.15.114:6789/0 conn(0x7ff644137b00 legacy=0x7ff644139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.845+0000 7ff64995f640 1 -- 172.21.15.114:0/2373710514 >> v1:172.21.15.46:6789/0 conn(0x7ff64409fa20 legacy=0x7ff64409fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.845+0000 7ff64995f640 10 monclient: _finish_hunting 0 2024-09-16T04:38:41.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.845+0000 7ff64995f640 1 monclient: found mon.noname-c 2024-09-16T04:38:41.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.845+0000 7ff64995f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:41.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.845+0000 7ff64995f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.845+0000 7ff64995f640 1 -- 172.21.15.114:0/2373710514 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff64406bd50 con 0x7ff64413b610 2024-09-16T04:38:41.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.845+0000 7ff63bfff640 1 -- 172.21.15.114:0/2373710514 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1316238813 0 0) 0x7ff64406c1e0 con 0x7ff644137b00 2024-09-16T04:38:41.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.845+0000 7ff63bfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:38:41.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff64995f640 1 --2- 172.21.15.114:0/2373710514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64413b610 0x7ff64413da00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff63c009d90 tx=0x7ff63c02a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1afde1dbb8306a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:41.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff63bfff640 1 -- 172.21.15.114:0/2373710514 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63c031600 con 0x7ff64413b610 2024-09-16T04:38:41.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff63bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:41.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff63bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:38:41.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff63bfff640 10 monclient: dump: 2024-09-16T04:38:41.855 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:41.855 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.855 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.855 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.856 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:41.856 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.856 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:41.856 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:41.856 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:41.856 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff63bfff640 10 monclient: _finish_auth 0 2024-09-16T04:38:41.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff63bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:38:41.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff63bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:41.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff64b3c2640 5 monclient: authenticate success, global_id 4138 2024-09-16T04:38:41.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff63bfff640 1 -- 172.21.15.114:0/2373710514 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff63c0317a0 con 0x7ff64413b610 2024-09-16T04:38:41.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff64b3c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:38:41.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff63bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:41.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff64b3c2640 10 monclient: get_monmap_and_config success 2024-09-16T04:38:41.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff64b3c2640 4 set_mon_vals no callback set 2024-09-16T04:38:41.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff63bfff640 1 -- 172.21.15.114:0/2373710514 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63c031b70 con 0x7ff64413b610 2024-09-16T04:38:41.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff63bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:41.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff63bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:38:41.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff63bfff640 10 monclient: dump: 2024-09-16T04:38:41.858 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:41.858 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.858 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.858 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.858 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:41.858 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.858 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:41.858 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:41.858 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:41.858 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff64b3c2640 10 monclient: shutdown 2024-09-16T04:38:41.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff64b3c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:41.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff64b3c2640 1 -- 172.21.15.114:0/2373710514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64413b610 msgr2=0x7ff64413da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff64b3c2640 1 --2- 172.21.15.114:0/2373710514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64413b610 0x7ff64413da00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff63c009d90 tx=0x7ff63c02a5d0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff64b3c2640 1 -- 172.21.15.114:0/2373710514 shutdown_connections 2024-09-16T04:38:41.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff64b3c2640 1 --2- 172.21.15.114:0/2373710514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64413b610 0x7ff64413da00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff64b3c2640 1 -- 172.21.15.114:0/2373710514 >> 172.21.15.114:0/2373710514 conn(0x7ff6440a6570 msgr2=0x7ff6440a29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:41.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff64b3c2640 1 -- 172.21.15.114:0/2373710514 shutdown_connections 2024-09-16T04:38:41.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.846+0000 7ff64b3c2640 1 -- 172.21.15.114:0/2373710514 wait complete. 2024-09-16T04:38:41.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.847+0000 7ff64b3c2640 10 monclient: build_initial_monmap 2024-09-16T04:38:41.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.847+0000 7ff64b3c2640 10 monclient: monmap: 2024-09-16T04:38:41.860 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T04:38:41.860 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:38:41.860 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T04:38:41.860 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T04:38:41.860 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:41.860 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.860 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:38:41.860 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:38:41.860 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:38:41.861 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.847+0000 7ff64b3c2640 1 Processor -- start 2024-09-16T04:38:41.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.847+0000 7ff64b3c2640 1 -- start start 2024-09-16T04:38:41.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.847+0000 7ff64b3c2640 10 monclient: init 2024-09-16T04:38:41.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64b3c2640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:41.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64b3c2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:38:41.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64b3c2640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff64409fa20 0x7ff64411c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64b3c2640 10 monclient(hunting): picked mon.noname-c con 0x7ff64409fa20 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:41.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64b3c2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff644137b00 0x7ff64411ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64b3c2640 10 monclient(hunting): picked mon.noname-b con 0x7ff644137b00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:38:41.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64b3c2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff64413b610 0x7ff64411cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:41.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64b3c2640 10 monclient(hunting): picked mon.noname-a con 0x7ff64413b610 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:38:41.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64b3c2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64b3c2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff64409d640 con 0x7ff64413b610 2024-09-16T04:38:41.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64b3c2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64b3c2640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff64406c460 con 0x7ff644137b00 2024-09-16T04:38:41.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64b3c2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:41.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64b3c2640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7ff64411ed60 con 0x7ff64409fa20 2024-09-16T04:38:41.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64b3c2640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:41.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64b3c2640 10 monclient(hunting): authenticate will time out at 739.324294s 2024-09-16T04:38:41.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64895d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff644137b00 0x7ff64411ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64895d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff644137b00 0x7ff64411ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48908/0 (socket says 172.21.15.114:48908) 2024-09-16T04:38:41.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64895d640 1 -- 172.21.15.114:0/1372290974 learned_addr learned my addr 172.21.15.114:0/1372290974 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T04:38:41.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64895d640 10 monclient(hunting): get_auth_request con 0x7ff644137b00 auth_method 0 2024-09-16T04:38:41.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64895d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:41.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64895d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64895d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:41.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64895d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:41.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64895d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:41.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64895d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:41.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64995f640 1 --2- 172.21.15.114:0/1372290974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff64413b610 0x7ff64411cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64915e640 1 --2- 172.21.15.114:0/1372290974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff64409fa20 0x7ff64411c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:41.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64995f640 10 monclient(hunting): get_auth_request con 0x7ff64413b610 auth_method 0 2024-09-16T04:38:41.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64995f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:41.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64995f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:41.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64995f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:41.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64895d640 10 monclient(hunting): handle_auth_done global_id 4144 payload 824 2024-09-16T04:38:41.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64895d640 1 -- 172.21.15.114:0/1372290974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff64409fa20 msgr2=0x7ff64411c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64895d640 1 --2- 172.21.15.114:0/1372290974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff64409fa20 0x7ff64411c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64895d640 1 -- 172.21.15.114:0/1372290974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff64413b610 msgr2=0x7ff64411cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64895d640 1 --2- 172.21.15.114:0/1372290974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff64413b610 0x7ff64411cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64895d640 10 monclient: _finish_hunting 0 2024-09-16T04:38:41.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64895d640 1 monclient: found mon.noname-b 2024-09-16T04:38:41.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64895d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:41.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64895d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64895d640 1 -- 172.21.15.114:0/1372290974 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6440a4320 con 0x7ff644137b00 2024-09-16T04:38:41.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64915e640 10 monclient: get_auth_request con 0x7ff64409fa20 auth_method 0 2024-09-16T04:38:41.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64915e640 1 --2- 172.21.15.114:0/1372290974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff64409fa20 0x7ff64411c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:38:41.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64895d640 1 --2- 172.21.15.114:0/1372290974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff644137b00 0x7ff64411ca40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff63400c500 tx=0x7ff63400c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:41.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff639ffb640 1 -- 172.21.15.114:0/1372290974 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff634018070 con 0x7ff644137b00 2024-09-16T04:38:41.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff639ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:41.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff639ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:38:41.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff639ffb640 10 monclient: dump: 2024-09-16T04:38:41.865 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:41.865 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.865 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.865 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.865 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:41.865 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.865 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:41.866 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:41.866 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:41.866 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff639ffb640 10 monclient: _finish_auth 0 2024-09-16T04:38:41.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff639ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:38:41.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff639ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:41.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff639ffb640 1 -- 172.21.15.114:0/1372290974 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6340074e0 con 0x7ff644137b00 2024-09-16T04:38:41.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64b3c2640 5 monclient: authenticate success, global_id 4144 2024-09-16T04:38:41.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff639ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:41.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff639ffb640 1 -- 172.21.15.114:0/1372290974 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff634004a20 con 0x7ff644137b00 2024-09-16T04:38:41.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64b3c2640 10 monclient: _renew_subs 2024-09-16T04:38:41.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64b3c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff64b3c2640 1 -- 172.21.15.114:0/1372290974 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff64409d110 con 0x7ff644137b00 2024-09-16T04:38:41.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.849+0000 7ff639ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:41.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.849+0000 7ff639ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:38:41.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.849+0000 7ff639ffb640 10 monclient: dump: 2024-09-16T04:38:41.867 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:41.867 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:41.867 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.867 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:41.867 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:41.867 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:41.867 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:41.867 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:41.867 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:41.867 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:41.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.848+0000 7ff63b7fe640 4 set_mon_vals no callback set 2024-09-16T04:38:41.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.849+0000 7ff639ffb640 1 -- 172.21.15.114:0/1372290974 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff63401b070 con 0x7ff644137b00 2024-09-16T04:38:41.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.849+0000 7ff64b3c2640 10 monclient: _renew_subs 2024-09-16T04:38:41.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.849+0000 7ff64b3c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.849+0000 7ff64b3c2640 1 -- 172.21.15.114:0/1372290974 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff64411f040 con 0x7ff644137b00 2024-09-16T04:38:41.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.849+0000 7ff639ffb640 1 -- 172.21.15.114:0/1372290974 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7ff63401ccc0 con 0x7ff644137b00 2024-09-16T04:38:41.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.850+0000 7ff64b3c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:41.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.850+0000 7ff64b3c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:41.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.850+0000 7ff64b3c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.850+0000 7ff64b3c2640 1 -- 172.21.15.114:0/1372290974 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff64409fe00 con 0x7ff644137b00 2024-09-16T04:38:41.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.852+0000 7ff639ffb640 1 -- 172.21.15.114:0/1372290974 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7ff634022020 con 0x7ff644137b00 2024-09-16T04:38:41.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.852+0000 7ff639ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:41.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.852+0000 7ff639ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:38:41.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.920+0000 7ff64b3c2640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f37724e4-2796-43c8-975b-4458986c4ea6", "id": 4}] 2024-09-16T04:38:41.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.921+0000 7ff64b3c2640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f37724e4-2796-43c8-975b-4458986c4ea6", "id": 4}] 2024-09-16T04:38:41.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.921+0000 7ff64b3c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:41.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.921+0000 7ff64b3c2640 1 -- 172.21.15.114:0/1372290974 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f37724e4-2796-43c8-975b-4458986c4ea6", "id": 4} v 0) -- 0x7ff64411db90 con 0x7ff644137b00 2024-09-16T04:38:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.928+0000 7ff639ffb640 1 -- 172.21.15.114:0/1372290974 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f37724e4-2796-43c8-975b-4458986c4ea6", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7ff634004c10 con 0x7ff644137b00 2024-09-16T04:38:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.928+0000 7ff639ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f37724e4-2796-43c8-975b-4458986c4ea6", "id": 4}] 2024-09-16T04:38:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.928+0000 7ff639ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:38:41.930 INFO:teuthology.orchestra.run.smithi114.stdout:4 2024-09-16T04:38:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.929+0000 7ff64b3c2640 10 monclient: shutdown 2024-09-16T04:38:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.929+0000 7ff64b3c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.929+0000 7ff64b3c2640 1 -- 172.21.15.114:0/1372290974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff644137b00 msgr2=0x7ff64411ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.929+0000 7ff64b3c2640 1 --2- 172.21.15.114:0/1372290974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff644137b00 0x7ff64411ca40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff63400c500 tx=0x7ff63400c9c0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.929+0000 7ff64b3c2640 1 -- 172.21.15.114:0/1372290974 shutdown_connections 2024-09-16T04:38:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.929+0000 7ff64b3c2640 1 --2- 172.21.15.114:0/1372290974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff64413b610 0x7ff64411cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.929+0000 7ff64b3c2640 1 --2- 172.21.15.114:0/1372290974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff644137b00 0x7ff64411ca40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.929+0000 7ff64b3c2640 1 --2- 172.21.15.114:0/1372290974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff64409fa20 0x7ff64411c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.929+0000 7ff64b3c2640 1 -- 172.21.15.114:0/1372290974 >> 172.21.15.114:0/1372290974 conn(0x7ff6440a6570 msgr2=0x7ff6440a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.929+0000 7ff64b3c2640 1 -- 172.21.15.114:0/1372290974 shutdown_connections 2024-09-16T04:38:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:41.929+0000 7ff64b3c2640 1 -- 172.21.15.114:0/1372290974 wait complete. 2024-09-16T04:38:41.932 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:41.930+0000 7faefb42d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T04:38:41.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new 979362db-e173-47a0-93d1-187742938a85 5 2024-09-16T04:38:42.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.036+0000 7fa34b267640 10 monclient: get_monmap_and_config 2024-09-16T04:38:42.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.036+0000 7fa34b267640 10 monclient: build_initial_monmap 2024-09-16T04:38:42.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.036+0000 7fa34b267640 10 monclient: monmap: 2024-09-16T04:38:42.038 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T04:38:42.038 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:42.038 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:42.037433+0000 2024-09-16T04:38:42.038 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:42.037433+0000 2024-09-16T04:38:42.038 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:42.038 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:42.038 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:38:42.038 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:38:42.039 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:38:42.039 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:38:42.039 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:38:42.039 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:42.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.037+0000 7fa34b267640 1 Processor -- start 2024-09-16T04:38:42.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.037+0000 7fa34b267640 1 -- start start 2024-09-16T04:38:42.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.037+0000 7fa34b267640 10 monclient: init 2024-09-16T04:38:42.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.037+0000 7fa34b267640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:42.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.037+0000 7fa34b267640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-16T04:38:42.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.037+0000 7fa34b267640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa3440a3c10 0x7fa34409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:42.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.037+0000 7fa34b267640 10 monclient(hunting): picked mon.noname-c con 0x7fa3440a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:38:42.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.037+0000 7fa34b267640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3440a3ff0 0x7fa3440a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:42.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.037+0000 7fa34b267640 10 monclient(hunting): picked mon.noname-b con 0x7fa3440a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:42.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.037+0000 7fa34b267640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa3441369d0 addr v1:172.21.15.114:6789/0 2024-09-16T04:38:42.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.037+0000 7fa34b267640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa34409ab70 con 0x7fa3441369d0 2024-09-16T04:38:42.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.037+0000 7fa34b267640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:42.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.037+0000 7fa34b267640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fa3440a4d60 con 0x7fa3440a3c10 2024-09-16T04:38:42.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.037+0000 7fa34b267640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:42.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.037+0000 7fa34b267640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fa34409b030 con 0x7fa3440a3ff0 2024-09-16T04:38:42.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.037+0000 7fa34b267640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:42.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.037+0000 7fa34b267640 10 monclient(hunting): authenticate will time out at 739.513749s 2024-09-16T04:38:42.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.037+0000 7fa349804640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa3441369d0 0x7fa344136e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36574/0 (socket says 172.21.15.114:36574) 2024-09-16T04:38:42.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.037+0000 7fa349804640 1 -- 172.21.15.114:0/2229800009 learned_addr learned my addr 172.21.15.114:0/2229800009 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T04:38:42.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa349003640 1 --2- 172.21.15.114:0/2229800009 >> v2:172.21.15.114:3300/0 conn(0x7fa3440a3c10 0x7fa34409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:42.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa348802640 1 --2- 172.21.15.114:0/2229800009 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3440a3ff0 0x7fa3440a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:42.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa349003640 10 monclient(hunting): get_auth_request con 0x7fa3440a3c10 auth_method 0 2024-09-16T04:38:42.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa349003640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:42.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa349003640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:42.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa349003640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa348802640 10 monclient(hunting): get_auth_request con 0x7fa3440a3ff0 auth_method 0 2024-09-16T04:38:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa348802640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa348802640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa348802640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa349003640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa349003640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa349003640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa348802640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa348802640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa33bfff640 1 -- 172.21.15.114:0/2229800009 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1188824844 0 0) 0x7fa34409ab70 con 0x7fa3441369d0 2024-09-16T04:38:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa348802640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa33bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:38:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa33bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:42.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa33bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:42.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa33bfff640 10 monclient(hunting): my global_id is 4150 2024-09-16T04:38:42.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa33bfff640 1 -- 172.21.15.114:0/2229800009 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa330003520 con 0x7fa3441369d0 2024-09-16T04:38:42.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa349003640 10 monclient(hunting): handle_auth_done global_id 4147 payload 274 2024-09-16T04:38:42.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa349003640 1 -- 172.21.15.114:0/2229800009 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3440a3ff0 msgr2=0x7fa3440a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:38:42.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa349003640 1 --2- 172.21.15.114:0/2229800009 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3440a3ff0 0x7fa3440a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:42.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa349003640 1 -- 172.21.15.114:0/2229800009 >> v1:172.21.15.114:6789/0 conn(0x7fa3441369d0 legacy=0x7fa344136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:42.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa349003640 10 monclient: _finish_hunting 0 2024-09-16T04:38:42.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa349003640 1 monclient: found mon.noname-c 2024-09-16T04:38:42.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa349003640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:42.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa349003640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:38:42.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.038+0000 7fa349003640 1 -- 172.21.15.114:0/2229800009 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa34409b310 con 0x7fa3440a3c10 2024-09-16T04:38:42.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.039+0000 7fa349003640 1 --2- 172.21.15.114:0/2229800009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3440a3c10 0x7fa34409bcf0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa3400097a0 tx=0x7fa34002a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df18eefd3f84431c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:42.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.039+0000 7fa33bfff640 1 -- 172.21.15.114:0/2229800009 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa340037070 con 0x7fa3440a3c10 2024-09-16T04:38:42.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.039+0000 7fa33bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:42.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.039+0000 7fa33bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:38:42.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.039+0000 7fa33bfff640 10 monclient: dump: 2024-09-16T04:38:42.053 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:42.053 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:42.053 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:42.053 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:42.053 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:42.053 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:42.053 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:42.053 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:42.053 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:42.053 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:42.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.039+0000 7fa33bfff640 10 monclient: _finish_auth 0 2024-09-16T04:38:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.039+0000 7fa33bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:38:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.039+0000 7fa33bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.039+0000 7fa33bfff640 1 -- 172.21.15.114:0/2229800009 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa34002ae50 con 0x7fa3440a3c10 2024-09-16T04:38:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.039+0000 7fa34b267640 5 monclient: authenticate success, global_id 4147 2024-09-16T04:38:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.039+0000 7fa34b267640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:38:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.039+0000 7fa33bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.039+0000 7fa34b267640 10 monclient: get_monmap_and_config success 2024-09-16T04:38:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.039+0000 7fa33bfff640 1 -- 172.21.15.114:0/2229800009 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa340032660 con 0x7fa3440a3c10 2024-09-16T04:38:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.039+0000 7fa34b267640 4 set_mon_vals no callback set 2024-09-16T04:38:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.039+0000 7fa33bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.039+0000 7fa33bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:38:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.039+0000 7fa33bfff640 10 monclient: dump: 2024-09-16T04:38:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:42.055 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:42.055 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:42.055 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:42.055 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:42.055 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:42.055 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:42.055 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:42.055 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:42.055 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:42.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.040+0000 7fa34b267640 10 monclient: shutdown 2024-09-16T04:38:42.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.040+0000 7fa34b267640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:42.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.040+0000 7fa34b267640 1 -- 172.21.15.114:0/2229800009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3440a3c10 msgr2=0x7fa34409bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.040+0000 7fa34b267640 1 --2- 172.21.15.114:0/2229800009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3440a3c10 0x7fa34409bcf0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa3400097a0 tx=0x7fa34002a100 comp rx=0 tx=0).stop 2024-09-16T04:38:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.040+0000 7fa34b267640 1 -- 172.21.15.114:0/2229800009 shutdown_connections 2024-09-16T04:38:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.040+0000 7fa34b267640 1 --2- 172.21.15.114:0/2229800009 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3440a3ff0 0x7fa3440a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.040+0000 7fa34b267640 1 --2- 172.21.15.114:0/2229800009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3440a3c10 0x7fa34409bcf0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.040+0000 7fa34b267640 1 -- 172.21.15.114:0/2229800009 >> 172.21.15.114:0/2229800009 conn(0x7fa3440a5d10 msgr2=0x7fa3440a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.040+0000 7fa34b267640 1 -- 172.21.15.114:0/2229800009 shutdown_connections 2024-09-16T04:38:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.040+0000 7fa34b267640 1 -- 172.21.15.114:0/2229800009 wait complete. 2024-09-16T04:38:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.040+0000 7fa34b267640 10 monclient: build_initial_monmap 2024-09-16T04:38:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.040+0000 7fa34b267640 10 monclient: monmap: 2024-09-16T04:38:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T04:38:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:38:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T04:38:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T04:38:42.057 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:42.057 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:42.057 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:38:42.058 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:38:42.058 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:38:42.058 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:42.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa34b267640 1 Processor -- start 2024-09-16T04:38:42.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa34b267640 1 -- start start 2024-09-16T04:38:42.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa34b267640 10 monclient: init 2024-09-16T04:38:42.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa34b267640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:42.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa34b267640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:38:42.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa34b267640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3440a3c10 0x7fa3441cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:42.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa34b267640 10 monclient(hunting): picked mon.noname-b con 0x7fa3440a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:38:42.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa34b267640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3440a3ff0 0x7fa3441cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:42.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa34b267640 10 monclient(hunting): picked mon.noname-a con 0x7fa3440a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:38:42.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa34b267640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3441369d0 0x7fa344123060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:42.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa34b267640 10 monclient(hunting): picked mon.noname-c con 0x7fa3441369d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:42.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa34b267640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:42.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa34b267640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa3440a4910 con 0x7fa3440a3ff0 2024-09-16T04:38:42.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa34b267640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:42.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa34b267640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa34406e3d0 con 0x7fa3440a3c10 2024-09-16T04:38:42.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa34b267640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:42.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa34b267640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fa344126f70 con 0x7fa3441369d0 2024-09-16T04:38:42.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa34b267640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:42.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa34b267640 10 monclient(hunting): authenticate will time out at 739.517737s 2024-09-16T04:38:42.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa349003640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3440a3c10 0x7fa3441cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:42.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa349003640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3440a3c10 0x7fa3441cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48930/0 (socket says 172.21.15.114:48930) 2024-09-16T04:38:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa349003640 1 -- 172.21.15.114:0/2248822129 learned_addr learned my addr 172.21.15.114:0/2248822129 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T04:38:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa349003640 10 monclient(hunting): get_auth_request con 0x7fa3440a3c10 auth_method 0 2024-09-16T04:38:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa349003640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa349003640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa349003640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa349003640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa349003640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa349003640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa348802640 1 --2- 172.21.15.114:0/2248822129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3440a3ff0 0x7fa3441cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa348802640 10 monclient(hunting): get_auth_request con 0x7fa3440a3ff0 auth_method 0 2024-09-16T04:38:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa348802640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa348802640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa348802640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa349804640 1 --2- 172.21.15.114:0/2248822129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3441369d0 0x7fa344123060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa349804640 10 monclient(hunting): get_auth_request con 0x7fa3441369d0 auth_method 0 2024-09-16T04:38:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa349804640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa349804640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.041+0000 7fa349804640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa348802640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa348802640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa348802640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa349003640 10 monclient(hunting): handle_auth_done global_id 4153 payload 824 2024-09-16T04:38:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa349003640 1 -- 172.21.15.114:0/2248822129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3441369d0 msgr2=0x7fa344123060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:42.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa349003640 1 --2- 172.21.15.114:0/2248822129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3441369d0 0x7fa344123060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:42.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa349003640 1 -- 172.21.15.114:0/2248822129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3440a3ff0 msgr2=0x7fa3441cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:42.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa349003640 1 --2- 172.21.15.114:0/2248822129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3440a3ff0 0x7fa3441cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:42.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa349003640 10 monclient: _finish_hunting 0 2024-09-16T04:38:42.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa349003640 1 monclient: found mon.noname-b 2024-09-16T04:38:42.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa349003640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:42.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa349003640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3300/0 2024-09-16T04:38:42.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa349003640 1 -- 172.21.15.114:0/2248822129 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3441269b0 con 0x7fa3440a3c10 2024-09-16T04:38:42.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa349804640 1 --2- 172.21.15.114:0/2248822129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3441369d0 0x7fa344123060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:38:42.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa349003640 1 --2- 172.21.15.114:0/2248822129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3440a3c10 0x7fa3441cbee0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa340035040 tx=0x7fa3400046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:42.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa339ffb640 1 -- 172.21.15.114:0/2248822129 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa340041070 con 0x7fa3440a3c10 2024-09-16T04:38:42.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa339ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:42.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa339ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:38:42.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa339ffb640 10 monclient: dump: 2024-09-16T04:38:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:42.063 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa339ffb640 10 monclient: _finish_auth 0 2024-09-16T04:38:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa339ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:38:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa339ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa339ffb640 1 -- 172.21.15.114:0/2248822129 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa34003c460 con 0x7fa3440a3c10 2024-09-16T04:38:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa34b267640 5 monclient: authenticate success, global_id 4153 2024-09-16T04:38:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa339ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa339ffb640 1 -- 172.21.15.114:0/2248822129 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa340037040 con 0x7fa3440a3c10 2024-09-16T04:38:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa34b267640 10 monclient: _renew_subs 2024-09-16T04:38:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa34b267640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa34b267640 1 -- 172.21.15.114:0/2248822129 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa344125120 con 0x7fa3440a3c10 2024-09-16T04:38:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa339ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa339ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:38:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa339ffb640 10 monclient: dump: 2024-09-16T04:38:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T04:38:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T04:38:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:42.065 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:38:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa33b7fe640 4 set_mon_vals no callback set 2024-09-16T04:38:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa34b267640 10 monclient: _renew_subs 2024-09-16T04:38:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa34b267640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa34b267640 1 -- 172.21.15.114:0/2248822129 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3440a2b90 con 0x7fa3440a3c10 2024-09-16T04:38:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa339ffb640 1 -- 172.21.15.114:0/2248822129 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa34003a070 con 0x7fa3440a3c10 2024-09-16T04:38:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.042+0000 7fa339ffb640 1 -- 172.21.15.114:0/2248822129 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fa340045c00 con 0x7fa3440a3c10 2024-09-16T04:38:42.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.043+0000 7fa34b267640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:42.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.043+0000 7fa34b267640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:42.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.043+0000 7fa34b267640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:42.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.043+0000 7fa34b267640 1 -- 172.21.15.114:0/2248822129 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa34406da10 con 0x7fa3440a3c10 2024-09-16T04:38:42.066 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:42.044+0000 7faefb42d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T04:38:42.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.045+0000 7fa339ffb640 1 -- 172.21.15.114:0/2248822129 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fa340045680 con 0x7fa3440a3c10 2024-09-16T04:38:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.045+0000 7fa339ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.045+0000 7fa339ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:38:42.093 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:42.092+0000 7faefb42d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T04:38:42.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.115+0000 7fa34b267640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "979362db-e173-47a0-93d1-187742938a85", "id": 5}] 2024-09-16T04:38:42.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.115+0000 7fa34b267640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "979362db-e173-47a0-93d1-187742938a85", "id": 5}] 2024-09-16T04:38:42.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.115+0000 7fa34b267640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:38:42.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.115+0000 7fa34b267640 1 -- 172.21.15.114:0/2248822129 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd new", "uuid": "979362db-e173-47a0-93d1-187742938a85", "id": 5} v 0) -- 0x7fa344124c80 con 0x7fa3440a3c10 2024-09-16T04:38:42.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.123+0000 7fa339ffb640 1 -- 172.21.15.114:0/2248822129 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "979362db-e173-47a0-93d1-187742938a85", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fa340045860 con 0x7fa3440a3c10 2024-09-16T04:38:42.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.123+0000 7fa339ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "979362db-e173-47a0-93d1-187742938a85", "id": 5}] 2024-09-16T04:38:42.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.123+0000 7fa339ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:38:42.126 INFO:teuthology.orchestra.run.smithi114.stdout:5 2024-09-16T04:38:42.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.124+0000 7fa34b267640 10 monclient: shutdown 2024-09-16T04:38:42.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.124+0000 7fa34b267640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:42.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.124+0000 7fa34b267640 1 -- 172.21.15.114:0/2248822129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3440a3c10 msgr2=0x7fa3441cbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:42.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.124+0000 7fa34b267640 1 --2- 172.21.15.114:0/2248822129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3440a3c10 0x7fa3441cbee0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa340035040 tx=0x7fa3400046c0 comp rx=0 tx=0).stop 2024-09-16T04:38:42.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.124+0000 7fa34b267640 1 -- 172.21.15.114:0/2248822129 shutdown_connections 2024-09-16T04:38:42.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.124+0000 7fa34b267640 1 --2- 172.21.15.114:0/2248822129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3441369d0 0x7fa344123060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:42.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.124+0000 7fa34b267640 1 --2- 172.21.15.114:0/2248822129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3440a3ff0 0x7fa3441cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:42.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.124+0000 7fa34b267640 1 --2- 172.21.15.114:0/2248822129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3440a3c10 0x7fa3441cbee0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:42.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.124+0000 7fa34b267640 1 -- 172.21.15.114:0/2248822129 >> 172.21.15.114:0/2248822129 conn(0x7fa3440a5d10 msgr2=0x7fa3440a12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:42.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.124+0000 7fa34b267640 1 -- 172.21.15.114:0/2248822129 shutdown_connections 2024-09-16T04:38:42.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T04:38:42.124+0000 7fa34b267640 1 -- 172.21.15.114:0/2248822129 wait complete. 2024-09-16T04:38:42.136 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T04:38:42.137 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T04:38:42.139 INFO:tasks.ceph.osd.0:Started 2024-09-16T04:38:42.139 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T04:38:42.139 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T04:38:42.142 INFO:tasks.ceph.osd.1:Started 2024-09-16T04:38:42.142 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T04:38:42.142 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T04:38:42.146 INFO:tasks.ceph.osd.2:Started 2024-09-16T04:38:42.146 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T04:38:42.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T04:38:42.148 INFO:tasks.ceph.osd.3:Started 2024-09-16T04:38:42.148 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T04:38:42.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T04:38:42.150 INFO:tasks.ceph.osd.4:Started 2024-09-16T04:38:42.150 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T04:38:42.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T04:38:42.153 INFO:tasks.ceph.osd.5:Started 2024-09-16T04:38:42.154 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:38:42.215 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:42.213+0000 7faefb42d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T04:38:42.271 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:42.270+0000 7faefb42d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T04:38:42.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.275+0000 7f49a344c640 10 monclient: get_monmap_and_config 2024-09-16T04:38:42.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.275+0000 7f49a344c640 10 monclient: build_initial_monmap 2024-09-16T04:38:42.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.275+0000 7f49a344c640 10 monclient: monmap: 2024-09-16T04:38:42.276 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:38:42.277 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:42.277 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:42.275932+0000 2024-09-16T04:38:42.277 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:42.275932+0000 2024-09-16T04:38:42.277 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:42.277 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:42.277 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:38:42.277 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:38:42.277 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:38:42.277 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:38:42.277 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:38:42.277 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:42.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.276+0000 7f49a344c640 1 Processor -- start 2024-09-16T04:38:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.276+0000 7f49a344c640 1 -- start start 2024-09-16T04:38:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.276+0000 7f49a344c640 10 monclient: init 2024-09-16T04:38:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.276+0000 7f49a344c640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.276+0000 7f49a344c640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-16T04:38:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.276+0000 7f49a344c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f499c0a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:38:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.276+0000 7f49a344c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c0a3ff0 0x7f499c0a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.276+0000 7f49a344c640 10 monclient(hunting): picked mon.noname-b con 0x7f499c0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.276+0000 7f49a344c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f499c13be90 addr v1:172.21.15.46:6789/0 2024-09-16T04:38:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.276+0000 7f49a344c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f499c09ab70 con 0x7f499c13be90 2024-09-16T04:38:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.276+0000 7f49a344c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f499c0a4d60 con 0x7f499c0a3c10 2024-09-16T04:38:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.276+0000 7f49a344c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.276+0000 7f49a344c640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f499c09b030 con 0x7f499c0a3ff0 2024-09-16T04:38:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.276+0000 7f49a344c640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.276+0000 7f49a344c640 10 monclient(hunting): authenticate will time out at 720.691988s 2024-09-16T04:38:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f49a09e7640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c0a3ff0 0x7f499c0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f49a19e9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f499c13be90 0x7f499c13e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40226/0 (socket says 172.21.15.46:40226) 2024-09-16T04:38:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f49a19e9640 1 -- 172.21.15.46:0/2251549893 learned_addr learned my addr 172.21.15.46:0/2251549893 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:38:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f49a09e7640 10 monclient(hunting): get_auth_request con 0x7f499c0a3ff0 auth_method 0 2024-09-16T04:38:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f49a09e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f49a09e7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f49a09e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f49a09e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f49a09e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f49a09e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f4993fff640 1 -- 172.21.15.46:0/2251549893 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3803969601 0 0) 0x7f499c09ab70 con 0x7f499c13be90 2024-09-16T04:38:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f4993fff640 10 monclient(hunting): _negotiate 2024-09-16T04:38:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f4993fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f4993fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f4993fff640 10 monclient(hunting): my global_id is 4158 2024-09-16T04:38:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f4993fff640 1 -- 172.21.15.46:0/2251549893 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4988003520 con 0x7f499c13be90 2024-09-16T04:38:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f49a09e7640 10 monclient(hunting): handle_auth_done global_id 4145 payload 274 2024-09-16T04:38:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f49a09e7640 1 -- 172.21.15.46:0/2251549893 >> v1:172.21.15.114:6789/0 conn(0x7f499c0a3c10 legacy=0x7f499c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f4993fff640 1 -- 172.21.15.46:0/2251549893 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1431820097 0 0) 0x7f499c0a4d60 con 0x7f499c0a3c10 2024-09-16T04:38:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f49a09e7640 1 -- 172.21.15.46:0/2251549893 >> v1:172.21.15.46:6789/0 conn(0x7f499c13be90 legacy=0x7f499c13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f49a09e7640 10 monclient: _finish_hunting 0 2024-09-16T04:38:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f49a09e7640 1 monclient: found mon.noname-b 2024-09-16T04:38:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f49a09e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f49a09e7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:38:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f49a09e7640 1 -- 172.21.15.46:0/2251549893 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f499c09b310 con 0x7f499c0a3ff0 2024-09-16T04:38:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.277+0000 7f4993fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:38:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.278+0000 7f49a09e7640 1 --2- 172.21.15.46:0/2251549893 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c0a3ff0 0x7f499c0a43d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f49980097d0 tx=0x7f499802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0c323e453c8bdff server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.278+0000 7f4993fff640 1 -- 172.21.15.46:0/2251549893 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4998037070 con 0x7f499c0a3ff0 2024-09-16T04:38:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.278+0000 7f4993fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.278+0000 7f4993fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:38:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.278+0000 7f4993fff640 10 monclient: dump: 2024-09-16T04:38:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:38:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:42.288 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.278+0000 7f4993fff640 10 monclient: _finish_auth 0 2024-09-16T04:38:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.278+0000 7f4993fff640 10 monclient: _check_auth_tickets 2024-09-16T04:38:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.278+0000 7f4993fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.278+0000 7f4993fff640 1 -- 172.21.15.46:0/2251549893 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f499802ae50 con 0x7f499c0a3ff0 2024-09-16T04:38:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.278+0000 7f49a344c640 5 monclient: authenticate success, global_id 4145 2024-09-16T04:38:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.278+0000 7f49a344c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:38:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.278+0000 7f4993fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.278+0000 7f49a344c640 10 monclient: get_monmap_and_config success 2024-09-16T04:38:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.278+0000 7f49a344c640 4 set_mon_vals no callback set 2024-09-16T04:38:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.278+0000 7f4993fff640 1 -- 172.21.15.46:0/2251549893 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4998032640 con 0x7f499c0a3ff0 2024-09-16T04:38:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.278+0000 7f4993fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.278+0000 7f4993fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:38:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.278+0000 7f4993fff640 10 monclient: dump: 2024-09-16T04:38:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:38:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:42.290 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.278+0000 7f49a344c640 10 monclient: shutdown 2024-09-16T04:38:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.279+0000 7f49a344c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.279+0000 7f49a344c640 1 -- 172.21.15.46:0/2251549893 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c0a3ff0 msgr2=0x7f499c0a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.279+0000 7f49a344c640 1 --2- 172.21.15.46:0/2251549893 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c0a3ff0 0x7f499c0a43d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f49980097d0 tx=0x7f499802a100 comp rx=0 tx=0).stop 2024-09-16T04:38:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.279+0000 7f49a344c640 1 -- 172.21.15.46:0/2251549893 shutdown_connections 2024-09-16T04:38:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.279+0000 7f49a344c640 1 --2- 172.21.15.46:0/2251549893 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c0a3ff0 0x7f499c0a43d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.279+0000 7f49a344c640 1 -- 172.21.15.46:0/2251549893 >> 172.21.15.46:0/2251549893 conn(0x7f499c0a5d10 msgr2=0x7f499c0a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.279+0000 7f49a344c640 1 -- 172.21.15.46:0/2251549893 shutdown_connections 2024-09-16T04:38:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.279+0000 7f49a344c640 1 -- 172.21.15.46:0/2251549893 wait complete. 2024-09-16T04:38:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.279+0000 7f49a344c640 10 monclient: build_initial_monmap 2024-09-16T04:38:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.279+0000 7f49a344c640 10 monclient: monmap: 2024-09-16T04:38:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:38:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:38:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:38:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:38:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:38:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:38:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:38:42.292 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a344c640 1 Processor -- start 2024-09-16T04:38:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a344c640 1 -- start start 2024-09-16T04:38:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a344c640 10 monclient: init 2024-09-16T04:38:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a344c640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a344c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:38:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a344c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f499c0a3c10 0x7f499c1cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a344c640 10 monclient(hunting): picked mon.noname-b con 0x7f499c0a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:38:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a344c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f499c0a3ff0 0x7f499c1cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a344c640 10 monclient(hunting): picked mon.noname-a con 0x7f499c0a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:38:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a344c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c13be90 0x7f499c1388c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a344c640 10 monclient(hunting): picked mon.noname-c con 0x7f499c13be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a344c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a344c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f499c06e550 con 0x7f499c0a3ff0 2024-09-16T04:38:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a344c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a344c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f499c06e3d0 con 0x7f499c0a3c10 2024-09-16T04:38:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a344c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a344c640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f499c13f9a0 con 0x7f499c13be90 2024-09-16T04:38:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a344c640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a344c640 10 monclient(hunting): authenticate will time out at 720.695707s 2024-09-16T04:38:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a09e7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f499c0a3ff0 0x7f499c1cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a09e7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f499c0a3ff0 0x7f499c1cc0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41018/0 (socket says 172.21.15.46:41018) 2024-09-16T04:38:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a19e9640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c13be90 0x7f499c1388c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a09e7640 1 -- 172.21.15.46:0/1698602644 learned_addr learned my addr 172.21.15.46:0/1698602644 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:38:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a09e7640 10 monclient(hunting): get_auth_request con 0x7f499c0a3ff0 auth_method 0 2024-09-16T04:38:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a09e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a09e7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a09e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a11e8640 1 --2- 172.21.15.46:0/1698602644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f499c0a3c10 0x7f499c1cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a19e9640 10 monclient(hunting): get_auth_request con 0x7f499c13be90 auth_method 0 2024-09-16T04:38:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a19e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a19e9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a19e9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a11e8640 10 monclient(hunting): get_auth_request con 0x7f499c0a3c10 auth_method 0 2024-09-16T04:38:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a11e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a11e8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a11e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a09e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a09e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a09e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a19e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a19e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a19e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a11e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a11e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.280+0000 7f49a11e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f49a09e7640 10 monclient(hunting): handle_auth_done global_id 4161 payload 824 2024-09-16T04:38:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f49a09e7640 1 -- 172.21.15.46:0/1698602644 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c13be90 msgr2=0x7f499c1388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f49a09e7640 1 --2- 172.21.15.46:0/1698602644 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c13be90 0x7f499c1388c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f49a09e7640 1 -- 172.21.15.46:0/1698602644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f499c0a3c10 msgr2=0x7f499c1cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f49a09e7640 1 --2- 172.21.15.46:0/1698602644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f499c0a3c10 0x7f499c1cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f49a09e7640 10 monclient: _finish_hunting 0 2024-09-16T04:38:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f49a09e7640 1 monclient: found mon.noname-a 2024-09-16T04:38:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f49a09e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f49a09e7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:38:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f49a09e7640 1 -- 172.21.15.46:0/1698602644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f499c120230 con 0x7f499c0a3ff0 2024-09-16T04:38:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f49a19e9640 1 --2- 172.21.15.46:0/1698602644 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c13be90 0x7f499c1388c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:38:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f49a11e8640 1 --2- 172.21.15.46:0/1698602644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f499c0a3c10 0x7f499c1cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:38:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f49a09e7640 1 --2- 172.21.15.46:0/1698602644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f499c0a3ff0 0x7f499c1cc0e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4998000c00 tx=0x7f49980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f4991ffb640 1 -- 172.21.15.46:0/1698602644 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4998041070 con 0x7f499c0a3ff0 2024-09-16T04:38:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f4991ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f4991ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:38:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f4991ffb640 10 monclient: dump: 2024-09-16T04:38:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:38:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:42.297 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f4991ffb640 10 monclient: _finish_auth 0 2024-09-16T04:38:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f4991ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:38:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f4991ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f4991ffb640 1 -- 172.21.15.46:0/1698602644 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f499803c460 con 0x7f499c0a3ff0 2024-09-16T04:38:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f49a344c640 5 monclient: authenticate success, global_id 4161 2024-09-16T04:38:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f49a344c640 10 monclient: _renew_subs 2024-09-16T04:38:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f49a344c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:38:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f49a344c640 1 -- 172.21.15.46:0/1698602644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f499c13f330 con 0x7f499c0a3ff0 2024-09-16T04:38:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f4991ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f4991ffb640 1 -- 172.21.15.46:0/1698602644 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4998037040 con 0x7f499c0a3ff0 2024-09-16T04:38:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f4991ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f4991ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:38:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f4991ffb640 10 monclient: dump: 2024-09-16T04:38:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:38:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:42.299 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f49a344c640 10 monclient: _renew_subs 2024-09-16T04:38:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f49937fe640 4 set_mon_vals no callback set 2024-09-16T04:38:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f49a344c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:38:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.281+0000 7f49a344c640 1 -- 172.21.15.46:0/1698602644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f499c13ae40 con 0x7f499c0a3ff0 2024-09-16T04:38:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.282+0000 7f4991ffb640 1 -- 172.21.15.46:0/1698602644 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4998044070 con 0x7f499c0a3ff0 2024-09-16T04:38:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.282+0000 7f4991ffb640 1 -- 172.21.15.46:0/1698602644 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f4998046e00 con 0x7f499c0a3ff0 2024-09-16T04:38:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.282+0000 7f49a344c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.282+0000 7f49a344c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.282+0000 7f49a344c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:38:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.282+0000 7f49a344c640 1 -- 172.21.15.46:0/1698602644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f499c0a43d0 con 0x7f499c0a3ff0 2024-09-16T04:38:42.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.284+0000 7f4991ffb640 1 -- 172.21.15.46:0/1698602644 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f4998046a20 con 0x7f499c0a3ff0 2024-09-16T04:38:42.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.284+0000 7f4991ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:42.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.284+0000 7f4991ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:38:42.317 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:42.316+0000 7faefb42d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T04:38:42.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.345+0000 7f49a344c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:38:42.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.345+0000 7f49a344c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:38:42.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.345+0000 7f49a344c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:38:42.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.345+0000 7f49a344c640 1 -- 172.21.15.46:0/1698602644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f499c067e40 con 0x7f499c0a3ff0 2024-09-16T04:38:42.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.346+0000 7f4991ffb640 1 -- 172.21.15.46:0/1698602644 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f4998046c00 con 0x7f499c0a3ff0 2024-09-16T04:38:42.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.346+0000 7f4991ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:38:42.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.346+0000 7f4991ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:38:42.350 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:38:42.350 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":8,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","created":"2024-09-16T04:38:39.286708+0000","modified":"2024-09-16T04:38:42.117316+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T04:38:42.117316+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7ff88780-0630-4bb1-9f5a-5331d7724d17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"37be2e88-32d0-4a30-8979-899b34f3be17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"73b9a838-944e-48cd-afab-5260608d1c6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d0c22455-5685-4ade-b540-3d5428a7cb46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f37724e4-2796-43c8-975b-4458986c4ea6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"979362db-e173-47a0-93d1-187742938a85","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:38:42.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.347+0000 7f49a344c640 10 monclient: shutdown 2024-09-16T04:38:42.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.347+0000 7f49a344c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:42.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.347+0000 7f49a344c640 1 -- 172.21.15.46:0/1698602644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f499c0a3ff0 msgr2=0x7f499c1cc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:42.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.347+0000 7f49a344c640 1 --2- 172.21.15.46:0/1698602644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f499c0a3ff0 0x7f499c1cc0e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4998000c00 tx=0x7f49980046c0 comp rx=0 tx=0).stop 2024-09-16T04:38:42.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.347+0000 7f49a344c640 1 -- 172.21.15.46:0/1698602644 shutdown_connections 2024-09-16T04:38:42.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.347+0000 7f49a344c640 1 --2- 172.21.15.46:0/1698602644 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c13be90 0x7f499c1388c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:42.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.347+0000 7f49a344c640 1 --2- 172.21.15.46:0/1698602644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f499c0a3ff0 0x7f499c1cc0e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:42.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.347+0000 7f49a344c640 1 --2- 172.21.15.46:0/1698602644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f499c0a3c10 0x7f499c1cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:42.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.347+0000 7f49a344c640 1 -- 172.21.15.46:0/1698602644 >> 172.21.15.46:0/1698602644 conn(0x7f499c0a5d10 msgr2=0x7f499c0a23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:42.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.347+0000 7f49a344c640 1 -- 172.21.15.46:0/1698602644 shutdown_connections 2024-09-16T04:38:42.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:42.347+0000 7f49a344c640 1 -- 172.21.15.46:0/1698602644 wait complete. 2024-09-16T04:38:42.359 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T04:38:42.359 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T04:38:42.429 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:42.428+0000 7faefb42d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T04:38:42.478 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:42.477+0000 7faefb42d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T04:38:42.531 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:42.529+0000 7faefb42d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T04:38:42.601 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:42.599+0000 7faefb42d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T04:38:42.645 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:42.643+0000 7faefb42d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T04:38:42.689 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:42.687+0000 7faefb42d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T04:38:42.964 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T04:38:43.068 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.067+0000 7f49a1f2a640 10 monclient: get_monmap_and_config 2024-09-16T04:38:43.102 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.067+0000 7f49a1f2a640 10 monclient: build_initial_monmap 2024-09-16T04:38:43.102 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.067+0000 7f49a1f2a640 10 monclient: monmap: 2024-09-16T04:38:43.102 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-09-16T04:38:43.102 INFO:teuthology.misc.health.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:43.102 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-09-16T04:38:43.067609+0000 2024-09-16T04:38:43.102 INFO:teuthology.misc.health.smithi046.stderr:created 2024-09-16T04:38:43.067609+0000 2024-09-16T04:38:43.102 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:43.102 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:43.102 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:38:43.102 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:38:43.102 INFO:teuthology.misc.health.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:38:43.102 INFO:teuthology.misc.health.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:38:43.103 INFO:teuthology.misc.health.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:38:43.103 INFO:teuthology.misc.health.smithi046.stderr: 2024-09-16T04:38:43.110 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.067+0000 7f49a1f2a640 1 Processor -- start 2024-09-16T04:38:43.110 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.067+0000 7f49a1f2a640 1 -- start start 2024-09-16T04:38:43.111 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.067+0000 7f49a1f2a640 10 monclient: init 2024-09-16T04:38:43.111 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f49a1f2a640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:43.111 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f49a1f2a640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-16T04:38:43.111 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f49a1f2a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f499c0a2cc0 0x7f499c0a30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:43.111 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f49a1f2a640 10 monclient(hunting): picked mon.noname-a con 0x7f499c0a2cc0 addr v2:172.21.15.46:3300/0 2024-09-16T04:38:43.111 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f49a1f2a640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f499c0a06a0 0x7f499c0a3b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:43.111 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f49a1f2a640 10 monclient(hunting): picked mon.noname-c con 0x7f499c0a06a0 addr v2:172.21.15.114:3300/0 2024-09-16T04:38:43.111 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f49a1f2a640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c0a4050 0x7f499c0a4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:43.111 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f49a1f2a640 10 monclient(hunting): picked mon.noname-b con 0x7f499c0a4050 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:43.111 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f49a1f2a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:43.111 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f49a1f2a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f499c0a35e0 con 0x7f499c0a2cc0 2024-09-16T04:38:43.111 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f49a1f2a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:43.111 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f49a1f2a640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f499c0a52b0 con 0x7f499c0a06a0 2024-09-16T04:38:43.112 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f49a1f2a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:43.112 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f49a1f2a640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f499c0a4fa0 con 0x7f499c0a4050 2024-09-16T04:38:43.112 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f49a1f2a640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:43.112 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f49a1f2a640 10 monclient(hunting): authenticate will time out at 721.483520s 2024-09-16T04:38:43.112 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f499c0a2cc0 0x7f499c0a30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:43.112 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f499c0a2cc0 0x7f499c0a30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41034/0 (socket says 172.21.15.46:41034) 2024-09-16T04:38:43.112 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499b7fe640 1 -- 172.21.15.46:0/1668639695 learned_addr learned my addr 172.21.15.46:0/1668639695 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:38:43.112 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499b7fe640 10 monclient(hunting): get_auth_request con 0x7f499c0a2cc0 auth_method 0 2024-09-16T04:38:43.112 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:43.112 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:43.112 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:43.112 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499bfff640 1 --2- 172.21.15.46:0/1668639695 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c0a4050 0x7f499c0a4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:43.112 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499bfff640 10 monclient(hunting): get_auth_request con 0x7f499c0a4050 auth_method 0 2024-09-16T04:38:43.113 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:43.113 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:43.113 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:43.113 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:43.113 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:43.113 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499affd640 1 --2- 172.21.15.46:0/1668639695 >> v2:172.21.15.114:3300/0 conn(0x7f499c0a06a0 0x7f499c0a3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:43.113 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:43.113 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499affd640 10 monclient(hunting): get_auth_request con 0x7f499c0a06a0 auth_method 0 2024-09-16T04:38:43.114 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:43.114 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:43.114 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:43.114 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:43.114 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:43.114 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:43.114 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:43.114 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:43.114 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:43.114 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499b7fe640 10 monclient(hunting): handle_auth_done global_id 4164 payload 274 2024-09-16T04:38:43.114 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499b7fe640 1 -- 172.21.15.46:0/1668639695 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c0a4050 msgr2=0x7f499c0a4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:38:43.114 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499b7fe640 1 --2- 172.21.15.46:0/1668639695 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c0a4050 0x7f499c0a4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:43.114 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499b7fe640 1 -- 172.21.15.46:0/1668639695 >> v2:172.21.15.114:3300/0 conn(0x7f499c0a06a0 msgr2=0x7f499c0a3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:38:43.114 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499b7fe640 1 --2- 172.21.15.46:0/1668639695 >> v2:172.21.15.114:3300/0 conn(0x7f499c0a06a0 0x7f499c0a3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:43.115 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:38:43.115 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.068+0000 7f499b7fe640 1 monclient: found mon.noname-a 2024-09-16T04:38:43.115 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.069+0000 7f499b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:43.115 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.069+0000 7f499b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:38:43.115 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.069+0000 7f499b7fe640 1 -- 172.21.15.46:0/1668639695 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f499c0a0cc0 con 0x7f499c0a2cc0 2024-09-16T04:38:43.115 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.069+0000 7f499b7fe640 1 --2- 172.21.15.46:0/1668639695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f499c0a2cc0 0x7f499c0a30a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f499c073a00 tx=0x7f498c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439b5db78bdf497d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:43.115 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.069+0000 7f499a7fc640 1 -- 172.21.15.46:0/1668639695 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498c037070 con 0x7f499c0a2cc0 2024-09-16T04:38:43.115 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.069+0000 7f499a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:43.115 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.069+0000 7f499a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:38:43.115 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.069+0000 7f499a7fc640 10 monclient: dump: 2024-09-16T04:38:43.115 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-09-16T04:38:43.115 INFO:teuthology.misc.health.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:43.115 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:43.115 INFO:teuthology.misc.health.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:43.116 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:43.116 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:43.116 INFO:teuthology.misc.health.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:43.116 INFO:teuthology.misc.health.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:43.116 INFO:teuthology.misc.health.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:43.116 INFO:teuthology.misc.health.smithi046.stderr: 2024-09-16T04:38:43.116 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.069+0000 7f499a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:38:43.116 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.069+0000 7f499a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:38:43.116 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.069+0000 7f499a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:43.116 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.069+0000 7f499a7fc640 1 -- 172.21.15.46:0/1668639695 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f498c032440 con 0x7f499c0a2cc0 2024-09-16T04:38:43.116 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.069+0000 7f49a1f2a640 5 monclient: authenticate success, global_id 4164 2024-09-16T04:38:43.116 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.069+0000 7f49a1f2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:38:43.116 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.069+0000 7f499a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:43.117 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.069+0000 7f49a1f2a640 10 monclient: get_monmap_and_config success 2024-09-16T04:38:43.117 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.069+0000 7f499a7fc640 1 -- 172.21.15.46:0/1668639695 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498c032810 con 0x7f499c0a2cc0 2024-09-16T04:38:43.117 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.069+0000 7f49a1f2a640 4 set_mon_vals no callback set 2024-09-16T04:38:43.117 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.069+0000 7f499a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:43.117 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.069+0000 7f499a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:38:43.117 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.069+0000 7f499a7fc640 10 monclient: dump: 2024-09-16T04:38:43.117 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-09-16T04:38:43.117 INFO:teuthology.misc.health.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:43.117 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:43.117 INFO:teuthology.misc.health.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:43.117 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:43.117 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:43.117 INFO:teuthology.misc.health.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:43.117 INFO:teuthology.misc.health.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:43.117 INFO:teuthology.misc.health.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:43.118 INFO:teuthology.misc.health.smithi046.stderr: 2024-09-16T04:38:43.118 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.070+0000 7f49a1f2a640 10 monclient: shutdown 2024-09-16T04:38:43.118 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.070+0000 7f49a1f2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:43.118 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.070+0000 7f49a1f2a640 1 -- 172.21.15.46:0/1668639695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f499c0a2cc0 msgr2=0x7f499c0a30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:43.118 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.070+0000 7f49a1f2a640 1 --2- 172.21.15.46:0/1668639695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f499c0a2cc0 0x7f499c0a30a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f499c073a00 tx=0x7f498c02a140 comp rx=0 tx=0).stop 2024-09-16T04:38:43.118 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.070+0000 7f49a1f2a640 1 -- 172.21.15.46:0/1668639695 shutdown_connections 2024-09-16T04:38:43.118 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.070+0000 7f49a1f2a640 1 --2- 172.21.15.46:0/1668639695 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c0a4050 0x7f499c0a4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:43.118 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.070+0000 7f49a1f2a640 1 --2- 172.21.15.46:0/1668639695 >> v2:172.21.15.114:3300/0 conn(0x7f499c0a06a0 0x7f499c0a3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:43.118 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.070+0000 7f49a1f2a640 1 --2- 172.21.15.46:0/1668639695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f499c0a2cc0 0x7f499c0a30a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:43.118 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.070+0000 7f49a1f2a640 1 -- 172.21.15.46:0/1668639695 >> 172.21.15.46:0/1668639695 conn(0x7f499c0a5c10 msgr2=0x7f499c0a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:43.118 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.070+0000 7f49a1f2a640 1 -- 172.21.15.46:0/1668639695 shutdown_connections 2024-09-16T04:38:43.118 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.070+0000 7f49a1f2a640 1 -- 172.21.15.46:0/1668639695 wait complete. 2024-09-16T04:38:43.118 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.071+0000 7f49a1f2a640 10 monclient: build_initial_monmap 2024-09-16T04:38:43.118 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.071+0000 7f49a1f2a640 10 monclient: monmap: 2024-09-16T04:38:43.119 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-09-16T04:38:43.119 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:38:43.119 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-09-16T04:38:43.119 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-09-16T04:38:43.119 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:43.119 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:43.119 INFO:teuthology.misc.health.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:38:43.119 INFO:teuthology.misc.health.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:38:43.119 INFO:teuthology.misc.health.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:38:43.119 INFO:teuthology.misc.health.smithi046.stderr: 2024-09-16T04:38:43.119 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.071+0000 7f49a1f2a640 1 Processor -- start 2024-09-16T04:38:43.119 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.071+0000 7f49a1f2a640 1 -- start start 2024-09-16T04:38:43.119 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.071+0000 7f49a1f2a640 10 monclient: init 2024-09-16T04:38:43.119 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f49a1f2a640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:43.120 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f49a1f2a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:38:43.120 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f49a1f2a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f499c0a06a0 0x7f499c1cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:43.120 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f49a1f2a640 10 monclient(hunting): picked mon.noname-b con 0x7f499c0a06a0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:38:43.120 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f49a1f2a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f499c0a2cc0 0x7f499c135ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:43.120 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f49a1f2a640 10 monclient(hunting): picked mon.noname-a con 0x7f499c0a2cc0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:38:43.120 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f49a1f2a640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c0a4050 0x7f499c06eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:43.120 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f49a1f2a640 10 monclient(hunting): picked mon.noname-c con 0x7f499c0a4050 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:43.120 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f49a1f2a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:43.120 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f49a1f2a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f499c136e60 con 0x7f499c0a2cc0 2024-09-16T04:38:43.120 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f49a1f2a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:43.120 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f49a1f2a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f499c0a1280 con 0x7f499c0a06a0 2024-09-16T04:38:43.120 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f49a1f2a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:43.120 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f49a1f2a640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f499c136fe0 con 0x7f499c0a4050 2024-09-16T04:38:43.120 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f49a1f2a640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:43.121 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f49a1f2a640 10 monclient(hunting): authenticate will time out at 721.487222s 2024-09-16T04:38:43.121 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f499c0a2cc0 0x7f499c135ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:43.121 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499bfff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c0a4050 0x7f499c06eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:43.121 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499bfff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c0a4050 0x7f499c06eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:44068/0 (socket says 172.21.15.46:44068) 2024-09-16T04:38:43.121 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499bfff640 1 -- 172.21.15.46:0/1743421185 learned_addr learned my addr 172.21.15.46:0/1743421185 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:38:43.121 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499bfff640 10 monclient(hunting): get_auth_request con 0x7f499c0a4050 auth_method 0 2024-09-16T04:38:43.121 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:43.121 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:43.121 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:43.121 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499b7fe640 1 --2- 172.21.15.46:0/1743421185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f499c0a06a0 0x7f499c1cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:43.122 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499b7fe640 10 monclient(hunting): get_auth_request con 0x7f499c0a06a0 auth_method 0 2024-09-16T04:38:43.122 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f499c0a2cc0 0x7f499c135ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41046/0 (socket says 172.21.15.46:41046) 2024-09-16T04:38:43.122 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:43.122 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:43.122 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:43.122 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499affd640 10 monclient(hunting): get_auth_request con 0x7f499c0a2cc0 auth_method 0 2024-09-16T04:38:43.122 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:43.122 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:43.122 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:43.122 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:43.122 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:43.122 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:43.122 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:43.122 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:43.123 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:43.123 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:43.123 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:43.123 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:43.123 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499bfff640 10 monclient(hunting): handle_auth_done global_id 4154 payload 824 2024-09-16T04:38:43.123 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499bfff640 1 -- 172.21.15.46:0/1743421185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f499c0a06a0 msgr2=0x7f499c1cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:43.123 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499bfff640 1 --2- 172.21.15.46:0/1743421185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f499c0a06a0 0x7f499c1cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:43.123 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499bfff640 1 -- 172.21.15.46:0/1743421185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f499c0a2cc0 msgr2=0x7f499c135ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:43.123 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499bfff640 1 --2- 172.21.15.46:0/1743421185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f499c0a2cc0 0x7f499c135ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:43.123 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:38:43.123 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499bfff640 1 monclient: found mon.noname-c 2024-09-16T04:38:43.123 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:43.123 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:38:43.123 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499bfff640 1 -- 172.21.15.46:0/1743421185 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f499c0a0300 con 0x7f499c0a4050 2024-09-16T04:38:43.124 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499affd640 1 --2- 172.21.15.46:0/1743421185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f499c0a2cc0 0x7f499c135ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:38:43.124 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499b7fe640 1 --2- 172.21.15.46:0/1743421185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f499c0a06a0 0x7f499c1cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:38:43.124 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.072+0000 7f499bfff640 1 --2- 172.21.15.46:0/1743421185 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c0a4050 0x7f499c06eea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f499400efc0 tx=0x7f499400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:43.124 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f4983fff640 1 -- 172.21.15.46:0/1743421185 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4994018020 con 0x7f499c0a4050 2024-09-16T04:38:43.124 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f4983fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:43.124 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f4983fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:38:43.124 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f4983fff640 10 monclient: dump: 2024-09-16T04:38:43.124 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-09-16T04:38:43.124 INFO:teuthology.misc.health.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:43.124 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:43.124 INFO:teuthology.misc.health.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:43.124 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:43.124 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:43.124 INFO:teuthology.misc.health.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:43.125 INFO:teuthology.misc.health.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:43.125 INFO:teuthology.misc.health.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:43.125 INFO:teuthology.misc.health.smithi046.stderr: 2024-09-16T04:38:43.125 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f4983fff640 10 monclient: _finish_auth 0 2024-09-16T04:38:43.125 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f4983fff640 10 monclient: _check_auth_tickets 2024-09-16T04:38:43.125 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f4983fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:43.125 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f4983fff640 1 -- 172.21.15.46:0/1743421185 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4994014480 con 0x7f499c0a4050 2024-09-16T04:38:43.125 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f49a1f2a640 5 monclient: authenticate success, global_id 4154 2024-09-16T04:38:43.125 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f49a1f2a640 10 monclient: _renew_subs 2024-09-16T04:38:43.125 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f49a1f2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:38:43.125 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f49a1f2a640 1 -- 172.21.15.46:0/1743421185 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f499c09f340 con 0x7f499c0a4050 2024-09-16T04:38:43.125 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f4983fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:43.125 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f4983fff640 1 -- 172.21.15.46:0/1743421185 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4994014870 con 0x7f499c0a4050 2024-09-16T04:38:43.125 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f4983fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:43.126 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f4983fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:38:43.126 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f4983fff640 10 monclient: dump: 2024-09-16T04:38:43.126 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-09-16T04:38:43.126 INFO:teuthology.misc.health.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:43.126 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:43.126 INFO:teuthology.misc.health.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:43.126 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:43.126 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:43.126 INFO:teuthology.misc.health.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:43.126 INFO:teuthology.misc.health.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:43.126 INFO:teuthology.misc.health.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:43.126 INFO:teuthology.misc.health.smithi046.stderr: 2024-09-16T04:38:43.127 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f49a1f2a640 10 monclient: _renew_subs 2024-09-16T04:38:43.127 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f49a1f2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:38:43.127 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f49a1f2a640 1 -- 172.21.15.46:0/1743421185 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f499c072b30 con 0x7f499c0a4050 2024-09-16T04:38:43.127 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f4999ffb640 4 set_mon_vals no callback set 2024-09-16T04:38:43.127 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f4983fff640 1 -- 172.21.15.46:0/1743421185 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f499401b020 con 0x7f499c0a4050 2024-09-16T04:38:43.127 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.073+0000 7f4983fff640 1 -- 172.21.15.46:0/1743421185 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f49940139d0 con 0x7f499c0a4050 2024-09-16T04:38:43.127 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.074+0000 7f49a1f2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:43.127 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.074+0000 7f49a1f2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:43.127 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.074+0000 7f49a1f2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:38:43.127 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.074+0000 7f49a1f2a640 1 -- 172.21.15.46:0/1743421185 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f499c06e360 con 0x7f499c0a4050 2024-09-16T04:38:43.127 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.076+0000 7f4983fff640 1 -- 172.21.15.46:0/1743421185 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f4994022020 con 0x7f499c0a4050 2024-09-16T04:38:43.127 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.076+0000 7f4983fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:43.128 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.076+0000 7f4983fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:38:43.137 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.136+0000 7f49a1f2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:38:43.138 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.136+0000 7f49a1f2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:38:43.138 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.136+0000 7f49a1f2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:38:43.138 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.136+0000 7f49a1f2a640 1 -- 172.21.15.46:0/1743421185 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f499c1d02f0 con 0x7f499c0a4050 2024-09-16T04:38:43.139 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.137+0000 7f4983fff640 1 -- 172.21.15.46:0/1743421185 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f4994010040 con 0x7f499c0a4050 2024-09-16T04:38:43.139 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.137+0000 7f4983fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:38:43.139 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.137+0000 7f4983fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:38:43.139 INFO:teuthology.misc.health.smithi046.stdout: 2024-09-16T04:38:43.140 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":8,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","created":"2024-09-16T04:38:39.286708+0000","modified":"2024-09-16T04:38:42.117316+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T04:38:42.117316+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7ff88780-0630-4bb1-9f5a-5331d7724d17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"37be2e88-32d0-4a30-8979-899b34f3be17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"73b9a838-944e-48cd-afab-5260608d1c6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d0c22455-5685-4ade-b540-3d5428a7cb46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f37724e4-2796-43c8-975b-4458986c4ea6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"979362db-e173-47a0-93d1-187742938a85","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:38:43.141 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.137+0000 7f49a1f2a640 10 monclient: shutdown 2024-09-16T04:38:43.141 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.137+0000 7f49a1f2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:43.141 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.137+0000 7f49a1f2a640 1 -- 172.21.15.46:0/1743421185 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c0a4050 msgr2=0x7f499c06eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:43.141 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.137+0000 7f49a1f2a640 1 --2- 172.21.15.46:0/1743421185 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c0a4050 0x7f499c06eea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f499400efc0 tx=0x7f499400c490 comp rx=0 tx=0).stop 2024-09-16T04:38:43.141 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.137+0000 7f49a1f2a640 1 -- 172.21.15.46:0/1743421185 shutdown_connections 2024-09-16T04:38:43.142 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.137+0000 7f49a1f2a640 1 --2- 172.21.15.46:0/1743421185 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f499c0a4050 0x7f499c06eea0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:43.142 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.137+0000 7f49a1f2a640 1 --2- 172.21.15.46:0/1743421185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f499c0a2cc0 0x7f499c135ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:43.142 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.137+0000 7f49a1f2a640 1 --2- 172.21.15.46:0/1743421185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f499c0a06a0 0x7f499c1cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:43.142 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.137+0000 7f49a1f2a640 1 -- 172.21.15.46:0/1743421185 >> 172.21.15.46:0/1743421185 conn(0x7f499c0a5c10 msgr2=0x7f499c0a7830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:43.142 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.137+0000 7f49a1f2a640 1 -- 172.21.15.46:0/1743421185 shutdown_connections 2024-09-16T04:38:43.142 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:43.137+0000 7f49a1f2a640 1 -- 172.21.15.46:0/1743421185 wait complete. 2024-09-16T04:38:43.148 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-16T04:38:43.338 INFO:tasks.ceph.osd.2.smithi046.stderr:2024-09-16T04:38:43.337+0000 7f5ec5758640 -1 Falling back to public interface 2024-09-16T04:38:43.340 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:43.339+0000 7faefb42d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T04:38:43.358 INFO:tasks.ceph.osd.0.smithi046.stderr:2024-09-16T04:38:43.357+0000 7f5de240d640 -1 Falling back to public interface 2024-09-16T04:38:43.606 INFO:tasks.ceph.osd.1.smithi046.stderr:2024-09-16T04:38:43.605+0000 7fe1084b4640 -1 Falling back to public interface 2024-09-16T04:38:43.615 INFO:tasks.ceph.osd.4.smithi114.stderr:2024-09-16T04:38:43.613+0000 7ff8b4e13640 -1 Falling back to public interface 2024-09-16T04:38:43.618 INFO:tasks.ceph.osd.3.smithi114.stderr:2024-09-16T04:38:43.617+0000 7fa375e0d640 -1 Falling back to public interface 2024-09-16T04:38:43.624 INFO:tasks.ceph.osd.5.smithi114.stderr:2024-09-16T04:38:43.622+0000 7f71b234d640 -1 Falling back to public interface 2024-09-16T04:38:44.353 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:44.351+0000 7fae9ac27640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T04:38:44.353 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-09-16T04:38:44.352+0000 7fae9ac27640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T04:38:47.415 INFO:tasks.ceph.osd.2.smithi046.stderr:2024-09-16T04:38:47.414+0000 7f5ec5758640 -1 osd.2 0 log_to_monitors true 2024-09-16T04:38:47.423 INFO:tasks.ceph.osd.1.smithi046.stderr:2024-09-16T04:38:47.422+0000 7fe1084b4640 -1 osd.1 0 log_to_monitors true 2024-09-16T04:38:47.446 INFO:tasks.ceph.osd.5.smithi114.stderr:2024-09-16T04:38:47.443+0000 7f71b234d640 -1 osd.5 0 log_to_monitors true 2024-09-16T04:38:47.653 INFO:tasks.ceph.osd.3.smithi114.stderr:2024-09-16T04:38:47.651+0000 7fa375e0d640 -1 osd.3 0 log_to_monitors true 2024-09-16T04:38:47.681 INFO:tasks.ceph.osd.4.smithi114.stderr:2024-09-16T04:38:47.678+0000 7ff8b4e13640 -1 osd.4 0 log_to_monitors true 2024-09-16T04:38:47.694 INFO:tasks.ceph.osd.0.smithi046.stderr:2024-09-16T04:38:47.693+0000 7f5de240d640 -1 osd.0 0 log_to_monitors true 2024-09-16T04:38:48.482 INFO:tasks.ceph.osd.2.smithi046.stderr:2024-09-16T04:38:48.480+0000 7f5ec16e7640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T04:38:48.484 INFO:tasks.ceph.osd.5.smithi114.stderr:2024-09-16T04:38:48.482+0000 7f71ae2dc640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T04:38:48.495 INFO:tasks.ceph.osd.2.smithi046.stderr:2024-09-16T04:38:48.494+0000 7f5ebd50f640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:38:48.502 INFO:tasks.ceph.osd.5.smithi114.stderr:2024-09-16T04:38:48.500+0000 7f71aa104640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:38:49.080 INFO:tasks.ceph.osd.1.smithi046.stderr:2024-09-16T04:38:49.079+0000 7fe104443640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T04:38:49.098 INFO:tasks.ceph.osd.1.smithi046.stderr:2024-09-16T04:38:49.097+0000 7fe0fc263640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:38:49.493 INFO:tasks.ceph.osd.3.smithi114.stderr:2024-09-16T04:38:49.490+0000 7fa37172f640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T04:38:49.518 INFO:tasks.ceph.osd.3.smithi114.stderr:2024-09-16T04:38:49.516+0000 7fa36d557640 -1 osd.3 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:38:49.753 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T04:38:49.855 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.854+0000 7f7d61801640 10 monclient: get_monmap_and_config 2024-09-16T04:38:49.855 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.854+0000 7f7d61801640 10 monclient: build_initial_monmap 2024-09-16T04:38:49.855 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.854+0000 7f7d61801640 10 monclient: monmap: 2024-09-16T04:38:49.855 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-09-16T04:38:49.855 INFO:teuthology.misc.health.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:49.855 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-09-16T04:38:49.854618+0000 2024-09-16T04:38:49.856 INFO:teuthology.misc.health.smithi046.stderr:created 2024-09-16T04:38:49.854618+0000 2024-09-16T04:38:49.856 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:49.856 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:49.856 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:38:49.856 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:38:49.856 INFO:teuthology.misc.health.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:38:49.856 INFO:teuthology.misc.health.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:38:49.856 INFO:teuthology.misc.health.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:38:49.856 INFO:teuthology.misc.health.smithi046.stderr: 2024-09-16T04:38:49.859 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.854+0000 7f7d61801640 1 Processor -- start 2024-09-16T04:38:49.859 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.854+0000 7f7d61801640 1 -- start start 2024-09-16T04:38:49.859 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.854+0000 7f7d61801640 10 monclient: init 2024-09-16T04:38:49.859 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d61801640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:49.859 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d61801640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-16T04:38:49.859 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d61801640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d5c136ab0 addr v1:172.21.15.46:6789/0 2024-09-16T04:38:49.860 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d61801640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7d5c0a1bb0 0x7f7d5c13b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:49.860 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d61801640 10 monclient(hunting): picked mon.noname-a con 0x7f7d5c0a1bb0 addr v2:172.21.15.46:3300/0 2024-09-16T04:38:49.860 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d61801640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7d5c13bf00 addr v1:172.21.15.114:6789/0 2024-09-16T04:38:49.860 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d61801640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d5c0a0f60 con 0x7f7d5c136ab0 2024-09-16T04:38:49.860 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d61801640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d5c1375a0 con 0x7f7d5c13bf00 2024-09-16T04:38:49.860 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d61801640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:49.860 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d61801640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f7d5c0a11d0 con 0x7f7d5c0a1bb0 2024-09-16T04:38:49.860 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d61801640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:49.860 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d61801640 10 monclient(hunting): authenticate will time out at 728.270473s 2024-09-16T04:38:49.860 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d527fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7d5c0a1bb0 0x7f7d5c13b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:49.860 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d527fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7d5c0a1bb0 0x7f7d5c13b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40674/0 (socket says 172.21.15.46:40674) 2024-09-16T04:38:49.860 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d527fc640 1 -- 172.21.15.46:0/2840967704 learned_addr learned my addr 172.21.15.46:0/2840967704 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:38:49.861 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d527fc640 10 monclient(hunting): get_auth_request con 0x7f7d5c0a1bb0 auth_method 0 2024-09-16T04:38:49.861 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:49.861 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:49.861 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:49.861 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d527fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:49.861 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d527fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:49.861 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d527fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:49.861 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.855+0000 7f7d527fc640 10 monclient(hunting): handle_auth_done global_id 4206 payload 274 2024-09-16T04:38:49.861 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.856+0000 7f7d5a7fc640 1 -- 172.21.15.46:0/2840967704 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4136508650 0 0) 0x7f7d5c1375a0 con 0x7f7d5c13bf00 2024-09-16T04:38:49.861 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.856+0000 7f7d527fc640 1 -- 172.21.15.46:0/2840967704 >> v1:172.21.15.114:6789/0 conn(0x7f7d5c13bf00 legacy=0x7f7d5c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:49.861 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.856+0000 7f7d527fc640 1 -- 172.21.15.46:0/2840967704 >> v1:172.21.15.46:6789/0 conn(0x7f7d5c136ab0 legacy=0x7f7d5c136e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:49.862 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.856+0000 7f7d527fc640 10 monclient: _finish_hunting 0 2024-09-16T04:38:49.862 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.856+0000 7f7d527fc640 1 monclient: found mon.noname-a 2024-09-16T04:38:49.862 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.856+0000 7f7d527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:49.862 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.856+0000 7f7d527fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:38:49.862 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.856+0000 7f7d527fc640 1 -- 172.21.15.46:0/2840967704 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d5c06ea20 con 0x7f7d5c0a1bb0 2024-09-16T04:38:49.862 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.856+0000 7f7d5a7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:38:49.862 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.857+0000 7f7d527fc640 1 --2- 172.21.15.46:0/2840967704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d5c0a1bb0 0x7f7d5c13b9c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009870 tx=0x7f7d4402a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a197e9bb728d9ee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:49.862 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.857+0000 7f7d5a7fc640 1 -- 172.21.15.46:0/2840967704 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d44037070 con 0x7f7d5c0a1bb0 2024-09-16T04:38:49.862 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.857+0000 7f7d5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:49.862 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.857+0000 7f7d5a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:38:49.862 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.857+0000 7f7d5a7fc640 10 monclient: dump: 2024-09-16T04:38:49.862 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-09-16T04:38:49.862 INFO:teuthology.misc.health.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:49.862 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:49.863 INFO:teuthology.misc.health.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:49.863 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:49.863 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:49.863 INFO:teuthology.misc.health.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:49.863 INFO:teuthology.misc.health.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:49.863 INFO:teuthology.misc.health.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:49.863 INFO:teuthology.misc.health.smithi046.stderr: 2024-09-16T04:38:49.863 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.857+0000 7f7d5a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:38:49.863 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.857+0000 7f7d5a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:38:49.863 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.857+0000 7f7d5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:49.863 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.857+0000 7f7d61801640 5 monclient: authenticate success, global_id 4206 2024-09-16T04:38:49.863 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.857+0000 7f7d61801640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:38:49.863 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.857+0000 7f7d5a7fc640 1 -- 172.21.15.46:0/2840967704 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d4402ad60 con 0x7f7d5c0a1bb0 2024-09-16T04:38:49.864 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.857+0000 7f7d5a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:49.864 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.857+0000 7f7d5a7fc640 1 -- 172.21.15.46:0/2840967704 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d44032660 con 0x7f7d5c0a1bb0 2024-09-16T04:38:49.864 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.857+0000 7f7d5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:49.864 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.857+0000 7f7d5a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:38:49.864 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.857+0000 7f7d5a7fc640 10 monclient: dump: 2024-09-16T04:38:49.864 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-09-16T04:38:49.864 INFO:teuthology.misc.health.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:49.864 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:49.864 INFO:teuthology.misc.health.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:49.864 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:49.864 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:49.864 INFO:teuthology.misc.health.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:49.864 INFO:teuthology.misc.health.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:49.864 INFO:teuthology.misc.health.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:49.865 INFO:teuthology.misc.health.smithi046.stderr: 2024-09-16T04:38:49.865 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.857+0000 7f7d61801640 10 monclient: get_monmap_and_config success 2024-09-16T04:38:49.865 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.857+0000 7f7d61801640 4 set_mon_vals no callback set 2024-09-16T04:38:49.865 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.858+0000 7f7d61801640 10 monclient: shutdown 2024-09-16T04:38:49.865 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.858+0000 7f7d61801640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:49.865 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.858+0000 7f7d61801640 1 -- 172.21.15.46:0/2840967704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d5c0a1bb0 msgr2=0x7f7d5c13b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:49.865 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.858+0000 7f7d61801640 1 --2- 172.21.15.46:0/2840967704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d5c0a1bb0 0x7f7d5c13b9c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009870 tx=0x7f7d4402a0d0 comp rx=0 tx=0).stop 2024-09-16T04:38:49.865 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.858+0000 7f7d61801640 1 -- 172.21.15.46:0/2840967704 shutdown_connections 2024-09-16T04:38:49.865 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.858+0000 7f7d61801640 1 --2- 172.21.15.46:0/2840967704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d5c0a1bb0 0x7f7d5c13b9c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:49.865 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.858+0000 7f7d61801640 1 -- 172.21.15.46:0/2840967704 >> 172.21.15.46:0/2840967704 conn(0x7f7d5c0a9bc0 msgr2=0x7f7d5c0a5ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:49.865 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.858+0000 7f7d61801640 1 -- 172.21.15.46:0/2840967704 shutdown_connections 2024-09-16T04:38:49.865 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.858+0000 7f7d61801640 1 -- 172.21.15.46:0/2840967704 wait complete. 2024-09-16T04:38:49.869 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.859+0000 7f7d61801640 10 monclient: build_initial_monmap 2024-09-16T04:38:49.869 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.859+0000 7f7d61801640 10 monclient: monmap: 2024-09-16T04:38:49.869 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-09-16T04:38:49.869 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:38:49.869 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-09-16T04:38:49.869 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-09-16T04:38:49.870 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:49.870 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:49.870 INFO:teuthology.misc.health.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:38:49.870 INFO:teuthology.misc.health.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:38:49.870 INFO:teuthology.misc.health.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:38:49.870 INFO:teuthology.misc.health.smithi046.stderr: 2024-09-16T04:38:49.870 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.859+0000 7f7d61801640 1 Processor -- start 2024-09-16T04:38:49.870 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.859+0000 7f7d61801640 1 -- start start 2024-09-16T04:38:49.870 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.859+0000 7f7d61801640 10 monclient: init 2024-09-16T04:38:49.870 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d61801640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:49.870 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d61801640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:38:49.870 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d61801640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d5c0a1bb0 0x7f7d5c1cb920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:49.870 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d61801640 10 monclient(hunting): picked mon.noname-a con 0x7f7d5c0a1bb0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:38:49.870 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d61801640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7d5c136ab0 0x7f7d5c135920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:49.870 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d61801640 10 monclient(hunting): picked mon.noname-c con 0x7f7d5c136ab0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:49.871 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d61801640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d5c13bf00 0x7f7d5c09b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:49.871 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d61801640 10 monclient(hunting): picked mon.noname-b con 0x7f7d5c13bf00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:38:49.871 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d61801640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:49.871 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d61801640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5c06e0d0 con 0x7f7d5c0a1bb0 2024-09-16T04:38:49.871 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d61801640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:49.871 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d61801640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5c0a4bd0 con 0x7f7d5c13bf00 2024-09-16T04:38:49.871 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d61801640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:49.871 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d61801640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f7d5c06e250 con 0x7f7d5c136ab0 2024-09-16T04:38:49.871 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d61801640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:49.871 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d61801640 10 monclient(hunting): authenticate will time out at 728.275232s 2024-09-16T04:38:49.871 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d5affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d5c0a1bb0 0x7f7d5c1cb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:49.871 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d527fc640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7d5c136ab0 0x7f7d5c135920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:49.871 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d5affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d5c0a1bb0 0x7f7d5c1cb920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40688/0 (socket says 172.21.15.46:40688) 2024-09-16T04:38:49.872 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d527fc640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7d5c136ab0 0x7f7d5c135920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:47964/0 (socket says 172.21.15.46:47964) 2024-09-16T04:38:49.872 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d5affd640 1 -- 172.21.15.46:0/2179578319 learned_addr learned my addr 172.21.15.46:0/2179578319 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:38:49.872 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d5affd640 10 monclient(hunting): get_auth_request con 0x7f7d5c0a1bb0 auth_method 0 2024-09-16T04:38:49.872 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:49.872 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d5affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:49.872 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d5affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:49.872 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d527fc640 10 monclient(hunting): get_auth_request con 0x7f7d5c136ab0 auth_method 0 2024-09-16T04:38:49.872 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:49.872 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:49.872 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:49.872 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d5b7fe640 1 --2- 172.21.15.46:0/2179578319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d5c13bf00 0x7f7d5c09b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:49.872 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d5b7fe640 10 monclient(hunting): get_auth_request con 0x7f7d5c13bf00 auth_method 0 2024-09-16T04:38:49.872 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:49.873 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:49.873 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:49.873 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d527fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:49.873 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d527fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:49.873 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d527fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:49.873 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:49.873 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:49.873 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d5affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:49.873 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:49.873 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:49.873 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:49.873 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d527fc640 10 monclient(hunting): handle_auth_done global_id 4193 payload 824 2024-09-16T04:38:49.873 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d527fc640 1 -- 172.21.15.46:0/2179578319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d5c13bf00 msgr2=0x7f7d5c09b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:49.874 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d527fc640 1 --2- 172.21.15.46:0/2179578319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d5c13bf00 0x7f7d5c09b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:49.874 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d527fc640 1 -- 172.21.15.46:0/2179578319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d5c0a1bb0 msgr2=0x7f7d5c1cb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:49.874 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d527fc640 1 --2- 172.21.15.46:0/2179578319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d5c0a1bb0 0x7f7d5c1cb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:49.874 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d527fc640 10 monclient: _finish_hunting 0 2024-09-16T04:38:49.874 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d527fc640 1 monclient: found mon.noname-c 2024-09-16T04:38:49.874 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:49.874 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d527fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:38:49.874 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.860+0000 7f7d527fc640 1 -- 172.21.15.46:0/2179578319 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d5c09f310 con 0x7f7d5c136ab0 2024-09-16T04:38:49.874 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d527fc640 1 --2- 172.21.15.46:0/2179578319 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7d5c136ab0 0x7f7d5c135920 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7d44031000 tx=0x7f7d440046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:49.874 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d53fff640 1 -- 172.21.15.46:0/2179578319 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d44037040 con 0x7f7d5c136ab0 2024-09-16T04:38:49.874 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:49.874 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d53fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:38:49.874 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d53fff640 10 monclient: dump: 2024-09-16T04:38:49.874 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-09-16T04:38:49.875 INFO:teuthology.misc.health.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:49.875 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:49.875 INFO:teuthology.misc.health.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:49.875 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:49.875 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:49.875 INFO:teuthology.misc.health.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:49.875 INFO:teuthology.misc.health.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:49.875 INFO:teuthology.misc.health.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:49.875 INFO:teuthology.misc.health.smithi046.stderr: 2024-09-16T04:38:49.875 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d53fff640 10 monclient: _finish_auth 0 2024-09-16T04:38:49.875 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d53fff640 10 monclient: _check_auth_tickets 2024-09-16T04:38:49.875 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:49.875 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d61801640 5 monclient: authenticate success, global_id 4193 2024-09-16T04:38:49.876 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d53fff640 1 -- 172.21.15.46:0/2179578319 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d44004030 con 0x7f7d5c136ab0 2024-09-16T04:38:49.876 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d53fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:49.876 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d53fff640 1 -- 172.21.15.46:0/2179578319 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d44004440 con 0x7f7d5c136ab0 2024-09-16T04:38:49.876 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:49.876 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d53fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:38:49.876 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d53fff640 10 monclient: dump: 2024-09-16T04:38:49.876 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-09-16T04:38:49.876 INFO:teuthology.misc.health.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:49.876 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:49.876 INFO:teuthology.misc.health.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:49.876 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:49.876 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:49.876 INFO:teuthology.misc.health.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:49.876 INFO:teuthology.misc.health.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:49.877 INFO:teuthology.misc.health.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:49.877 INFO:teuthology.misc.health.smithi046.stderr: 2024-09-16T04:38:49.877 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d61801640 10 monclient: _renew_subs 2024-09-16T04:38:49.877 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d61801640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:38:49.877 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d61801640 1 -- 172.21.15.46:0/2179578319 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d5c09e140 con 0x7f7d5c136ab0 2024-09-16T04:38:49.877 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d59ffb640 4 set_mon_vals no callback set 2024-09-16T04:38:49.877 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d61801640 10 monclient: _renew_subs 2024-09-16T04:38:49.877 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d61801640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:38:49.877 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.861+0000 7f7d61801640 1 -- 172.21.15.46:0/2179578319 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d5c09e360 con 0x7f7d5c136ab0 2024-09-16T04:38:49.877 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.862+0000 7f7d53fff640 1 -- 172.21.15.46:0/2179578319 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 4) ==== 45681+0+0 (secure 0 0 0) 0x7f7d4403a070 con 0x7f7d5c136ab0 2024-09-16T04:38:49.877 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.862+0000 7f7d53fff640 1 --2- 172.21.15.46:0/2179578319 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7d3c038320 0x7f7d3c03a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:49.877 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.862+0000 7f7d53fff640 1 -- 172.21.15.46:0/2179578319 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(11..11 src has 1..11) ==== 2767+0+0 (secure 0 0 0) 0x7f7d4406b650 con 0x7f7d5c136ab0 2024-09-16T04:38:49.877 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.862+0000 7f7d5affd640 1 --2- 172.21.15.46:0/2179578319 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7d3c038320 0x7f7d3c03a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:49.877 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.862+0000 7f7d5affd640 10 monclient: get_auth_request con 0x7f7d3c038320 auth_method 0 2024-09-16T04:38:49.878 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.862+0000 7f7d61801640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:49.878 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.862+0000 7f7d61801640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:49.878 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.862+0000 7f7d61801640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:38:49.878 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.862+0000 7f7d61801640 1 -- 172.21.15.46:0/2179578319 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d5c13edb0 con 0x7f7d5c136ab0 2024-09-16T04:38:49.878 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.863+0000 7f7d5affd640 1 --2- 172.21.15.46:0/2179578319 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7d3c038320 0x7f7d3c03a7e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c001a20 tx=0x7f7d4c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:49.878 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.867+0000 7f7d53fff640 1 -- 172.21.15.46:0/2179578319 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7d5c13edb0 con 0x7f7d5c136ab0 2024-09-16T04:38:49.878 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.868+0000 7f7d53fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:49.878 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:49.868+0000 7f7d53fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:38:50.080 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:50.079+0000 7f7d61801640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:38:50.080 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:50.079+0000 7f7d61801640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:38:50.080 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:50.079+0000 7f7d61801640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:38:50.081 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:50.079+0000 7f7d61801640 1 -- 172.21.15.46:0/2179578319 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7d5c136e90 con 0x7f7d5c136ab0 2024-09-16T04:38:50.081 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:50.080+0000 7f7d53fff640 1 -- 172.21.15.46:0/2179578319 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) ==== 74+0+7051 (secure 0 0 0) 0x7f7d44041020 con 0x7f7d5c136ab0 2024-09-16T04:38:50.081 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:50.080+0000 7f7d53fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:38:50.081 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:50.080+0000 7f7d53fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:38:50.081 INFO:teuthology.misc.health.smithi046.stdout: 2024-09-16T04:38:50.082 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":11,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","created":"2024-09-16T04:38:39.286708+0000","modified":"2024-09-16T04:38:49.476889+0000","last_up_change":"2024-09-16T04:38:49.476889+0000","last_in_change":"2024-09-16T04:38:42.117316+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7ff88780-0630-4bb1-9f5a-5331d7724d17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"37be2e88-32d0-4a30-8979-899b34f3be17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6817","nonce":4217009477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6819","nonce":4217009477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6823","nonce":4217009477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6821","nonce":4217009477}]},"public_addr":"172.21.15.46:6817/4217009477","cluster_addr":"172.21.15.46:6819/4217009477","heartbeat_back_addr":"172.21.15.46:6823/4217009477","heartbeat_front_addr":"172.21.15.46:6821/4217009477","state":["exists","up"]},{"osd":2,"uuid":"73b9a838-944e-48cd-afab-5260608d1c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6801","nonce":3101080311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6803","nonce":3101080311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6807","nonce":3101080311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6805","nonce":3101080311}]},"public_addr":"172.21.15.46:6801/3101080311","cluster_addr":"172.21.15.46:6803/3101080311","heartbeat_back_addr":"172.21.15.46:6807/3101080311","heartbeat_front_addr":"172.21.15.46:6805/3101080311","state":["exists","up"]},{"osd":3,"uuid":"d0c22455-5685-4ade-b540-3d5428a7cb46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f37724e4-2796-43c8-975b-4458986c4ea6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"979362db-e173-47a0-93d1-187742938a85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6819","nonce":938868100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6821","nonce":938868100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6825","nonce":938868100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6823","nonce":938868100}]},"public_addr":"172.21.15.114:6819/938868100","cluster_addr":"172.21.15.114:6821/938868100","heartbeat_back_addr":"172.21.15.114:6825/938868100","heartbeat_front_addr":"172.21.15.114:6823/938868100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:38:50.084 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:50.082+0000 7f7d61801640 1 -- 172.21.15.46:0/2179578319 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7d3c038320 msgr2=0x7f7d3c03a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:50.084 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:50.082+0000 7f7d61801640 1 --2- 172.21.15.46:0/2179578319 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7d3c038320 0x7f7d3c03a7e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c001a20 tx=0x7f7d4c009040 comp rx=0 tx=0).stop 2024-09-16T04:38:50.085 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:50.082+0000 7f7d61801640 10 monclient: shutdown 2024-09-16T04:38:50.085 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:50.082+0000 7f7d61801640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:50.085 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:50.082+0000 7f7d61801640 1 -- 172.21.15.46:0/2179578319 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7d5c136ab0 msgr2=0x7f7d5c135920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:50.085 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:50.082+0000 7f7d61801640 1 --2- 172.21.15.46:0/2179578319 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7d5c136ab0 0x7f7d5c135920 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7d44031000 tx=0x7f7d440046c0 comp rx=0 tx=0).stop 2024-09-16T04:38:50.086 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:50.082+0000 7f7d61801640 1 -- 172.21.15.46:0/2179578319 shutdown_connections 2024-09-16T04:38:50.086 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:50.082+0000 7f7d61801640 1 --2- 172.21.15.46:0/2179578319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d5c13bf00 0x7f7d5c09b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:50.086 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:50.082+0000 7f7d61801640 1 --2- 172.21.15.46:0/2179578319 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7d5c136ab0 0x7f7d5c135920 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:50.086 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:50.082+0000 7f7d61801640 1 --2- 172.21.15.46:0/2179578319 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7d3c038320 0x7f7d3c03a7e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:50.086 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:50.082+0000 7f7d61801640 1 --2- 172.21.15.46:0/2179578319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d5c0a1bb0 0x7f7d5c1cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:50.087 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:50.082+0000 7f7d61801640 1 -- 172.21.15.46:0/2179578319 >> 172.21.15.46:0/2179578319 conn(0x7f7d5c0a9bc0 msgr2=0x7f7d5c13e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:50.087 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:50.082+0000 7f7d61801640 1 -- 172.21.15.46:0/2179578319 shutdown_connections 2024-09-16T04:38:50.087 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:50.082+0000 7f7d61801640 1 -- 172.21.15.46:0/2179578319 wait complete. 2024-09-16T04:38:50.093 DEBUG:teuthology.misc:3 of 6 OSDs are up 2024-09-16T04:38:50.098 INFO:tasks.ceph.osd.0.smithi046.stderr:2024-09-16T04:38:50.097+0000 7f5dddc97640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T04:38:50.122 INFO:tasks.ceph.osd.0.smithi046.stderr:2024-09-16T04:38:50.121+0000 7f5dd5ab7640 -1 osd.0 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:38:50.268 INFO:tasks.ceph.osd.4.smithi114.stderr:2024-09-16T04:38:50.265+0000 7ff8b06f5640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T04:38:50.284 INFO:tasks.ceph.osd.4.smithi114.stderr:2024-09-16T04:38:50.281+0000 7ff8a8515640 -1 osd.4 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:38:56.699 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T04:38:56.803 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.802+0000 7f3372450640 10 monclient: get_monmap_and_config 2024-09-16T04:38:56.804 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.802+0000 7f3372450640 10 monclient: build_initial_monmap 2024-09-16T04:38:56.804 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.802+0000 7f3372450640 10 monclient: monmap: 2024-09-16T04:38:56.804 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-09-16T04:38:56.804 INFO:teuthology.misc.health.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:56.804 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-09-16T04:38:56.803205+0000 2024-09-16T04:38:56.804 INFO:teuthology.misc.health.smithi046.stderr:created 2024-09-16T04:38:56.803205+0000 2024-09-16T04:38:56.804 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:56.804 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:56.804 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:38:56.804 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:38:56.804 INFO:teuthology.misc.health.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:38:56.804 INFO:teuthology.misc.health.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:38:56.804 INFO:teuthology.misc.health.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:38:56.805 INFO:teuthology.misc.health.smithi046.stderr: 2024-09-16T04:38:56.812 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.803+0000 7f3372450640 1 Processor -- start 2024-09-16T04:38:56.813 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.803+0000 7f3372450640 1 -- start start 2024-09-16T04:38:56.813 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.803+0000 7f3372450640 10 monclient: init 2024-09-16T04:38:56.813 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f3372450640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:56.813 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f3372450640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-16T04:38:56.813 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f3372450640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336c09fed0 0x7f336c09f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:56.813 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f3372450640 10 monclient(hunting): picked mon.noname-b con 0x7f336c09fed0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:56.813 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f3372450640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f336c0a02b0 addr v1:172.21.15.114:6789/0 2024-09-16T04:38:56.813 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f3372450640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f336c0a06b0 0x7f336c134dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:56.813 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f3372450640 10 monclient(hunting): picked mon.noname-a con 0x7f336c0a06b0 addr v2:172.21.15.46:3300/0 2024-09-16T04:38:56.813 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f3372450640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f336c09e450 con 0x7f336c0a02b0 2024-09-16T04:38:56.813 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f3372450640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:56.813 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f3372450640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f336c09efb0 con 0x7f336c0a06b0 2024-09-16T04:38:56.813 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f3372450640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:56.814 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f3372450640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f336c09e870 con 0x7f336c09fed0 2024-09-16T04:38:56.814 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f3372450640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:56.814 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f3372450640 10 monclient(hunting): authenticate will time out at 735.219298s 2024-09-16T04:38:56.814 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f336bfff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336c09fed0 0x7f336c09f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:56.814 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f33709ed640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f336c0a06b0 0x7f336c134dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:56.814 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f336bfff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336c09fed0 0x7f336c09f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:60792/0 (socket says 172.21.15.46:60792) 2024-09-16T04:38:56.814 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f33709ed640 10 monclient(hunting): get_auth_request con 0x7f336c0a06b0 auth_method 0 2024-09-16T04:38:56.814 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f33709ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:56.814 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f33709ed640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:56.814 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f33709ed640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:56.814 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f336bfff640 1 -- 172.21.15.46:0/2684040834 learned_addr learned my addr 172.21.15.46:0/2684040834 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:38:56.814 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f336bfff640 10 monclient(hunting): get_auth_request con 0x7f336c09fed0 auth_method 0 2024-09-16T04:38:56.814 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f336bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:56.814 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f336bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:56.814 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f336bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:56.815 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f336bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:56.815 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f336bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:56.815 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f336bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:56.815 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f33709ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:56.815 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f33709ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:56.815 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f33709ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:56.815 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f336bfff640 10 monclient(hunting): handle_auth_done global_id 4202 payload 274 2024-09-16T04:38:56.815 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f336bfff640 1 -- 172.21.15.46:0/2684040834 >> v2:172.21.15.46:3300/0 conn(0x7f336c0a06b0 msgr2=0x7f336c134dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:38:56.815 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f336bfff640 1 --2- 172.21.15.46:0/2684040834 >> v2:172.21.15.46:3300/0 conn(0x7f336c0a06b0 0x7f336c134dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:56.815 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f336bfff640 1 -- 172.21.15.46:0/2684040834 >> v1:172.21.15.114:6789/0 conn(0x7f336c0a02b0 legacy=0x7f336c1346c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:38:56.815 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f336bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:38:56.815 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f336bfff640 1 monclient: found mon.noname-b 2024-09-16T04:38:56.815 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f336bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:56.815 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f336bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:38:56.816 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f336bfff640 1 -- 172.21.15.46:0/2684040834 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f336c0a0ed0 con 0x7f336c09fed0 2024-09-16T04:38:56.816 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.804+0000 7f33709ed640 1 --2- 172.21.15.46:0/2684040834 >> v2:172.21.15.46:3300/0 conn(0x7f336c0a06b0 0x7f336c134dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:38:56.816 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.805+0000 7f336bfff640 1 --2- 172.21.15.46:0/2684040834 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336c09fed0 0x7f336c09f700 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f335c009870 tx=0x7f335c02a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c420192f1e412b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:56.816 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.805+0000 7f336affd640 1 -- 172.21.15.46:0/2684040834 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335c037070 con 0x7f336c09fed0 2024-09-16T04:38:56.816 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.805+0000 7f336affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:56.816 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.805+0000 7f336affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:38:56.816 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.805+0000 7f336affd640 10 monclient: dump: 2024-09-16T04:38:56.816 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-09-16T04:38:56.816 INFO:teuthology.misc.health.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:56.816 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:56.816 INFO:teuthology.misc.health.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:56.816 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:56.816 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:56.816 INFO:teuthology.misc.health.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:56.816 INFO:teuthology.misc.health.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:56.817 INFO:teuthology.misc.health.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:56.817 INFO:teuthology.misc.health.smithi046.stderr: 2024-09-16T04:38:56.817 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.805+0000 7f336affd640 10 monclient: _finish_auth 0 2024-09-16T04:38:56.817 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.805+0000 7f336affd640 10 monclient: _check_auth_tickets 2024-09-16T04:38:56.817 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.805+0000 7f336affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:56.817 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.805+0000 7f336affd640 1 -- 172.21.15.46:0/2684040834 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f335c032440 con 0x7f336c09fed0 2024-09-16T04:38:56.817 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.805+0000 7f3372450640 5 monclient: authenticate success, global_id 4202 2024-09-16T04:38:56.817 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.805+0000 7f3372450640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:38:56.817 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.805+0000 7f336affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:56.817 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.805+0000 7f3372450640 10 monclient: get_monmap_and_config success 2024-09-16T04:38:56.817 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.805+0000 7f336affd640 1 -- 172.21.15.46:0/2684040834 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335c032810 con 0x7f336c09fed0 2024-09-16T04:38:56.817 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.805+0000 7f336affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:56.817 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.805+0000 7f3372450640 4 set_mon_vals no callback set 2024-09-16T04:38:56.817 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.805+0000 7f336affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:38:56.818 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.805+0000 7f336affd640 10 monclient: dump: 2024-09-16T04:38:56.818 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-09-16T04:38:56.818 INFO:teuthology.misc.health.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:56.818 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:56.818 INFO:teuthology.misc.health.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:56.818 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:56.818 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:56.818 INFO:teuthology.misc.health.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:56.818 INFO:teuthology.misc.health.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:56.818 INFO:teuthology.misc.health.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:56.818 INFO:teuthology.misc.health.smithi046.stderr: 2024-09-16T04:38:56.818 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.806+0000 7f3372450640 10 monclient: shutdown 2024-09-16T04:38:56.818 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.806+0000 7f3372450640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:56.818 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.806+0000 7f3372450640 1 -- 172.21.15.46:0/2684040834 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336c09fed0 msgr2=0x7f336c09f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:56.818 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.806+0000 7f3372450640 1 --2- 172.21.15.46:0/2684040834 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336c09fed0 0x7f336c09f700 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f335c009870 tx=0x7f335c02a130 comp rx=0 tx=0).stop 2024-09-16T04:38:56.819 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.806+0000 7f3372450640 1 -- 172.21.15.46:0/2684040834 shutdown_connections 2024-09-16T04:38:56.819 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.806+0000 7f3372450640 1 --2- 172.21.15.46:0/2684040834 >> v2:172.21.15.46:3300/0 conn(0x7f336c0a06b0 0x7f336c134dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:56.819 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.806+0000 7f3372450640 1 --2- 172.21.15.46:0/2684040834 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336c09fed0 0x7f336c09f700 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:56.819 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.806+0000 7f3372450640 1 -- 172.21.15.46:0/2684040834 >> 172.21.15.46:0/2684040834 conn(0x7f336c0a57d0 msgr2=0x7f336c0a1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:56.819 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.806+0000 7f3372450640 1 -- 172.21.15.46:0/2684040834 shutdown_connections 2024-09-16T04:38:56.819 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.806+0000 7f3372450640 1 -- 172.21.15.46:0/2684040834 wait complete. 2024-09-16T04:38:56.819 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.807+0000 7f3372450640 10 monclient: build_initial_monmap 2024-09-16T04:38:56.819 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.807+0000 7f3372450640 10 monclient: monmap: 2024-09-16T04:38:56.819 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-09-16T04:38:56.819 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:38:56.819 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-09-16T04:38:56.819 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-09-16T04:38:56.819 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:56.819 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:56.819 INFO:teuthology.misc.health.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:38:56.820 INFO:teuthology.misc.health.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:38:56.820 INFO:teuthology.misc.health.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:38:56.820 INFO:teuthology.misc.health.smithi046.stderr: 2024-09-16T04:38:56.820 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.807+0000 7f3372450640 1 Processor -- start 2024-09-16T04:38:56.820 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.807+0000 7f3372450640 1 -- start start 2024-09-16T04:38:56.820 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.807+0000 7f3372450640 10 monclient: init 2024-09-16T04:38:56.820 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f3372450640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:56.820 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f3372450640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:38:56.821 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f3372450640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f336c09fed0 0x7f336c134660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:56.821 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f3372450640 10 monclient(hunting): picked mon.noname-b con 0x7f336c09fed0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:38:56.821 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f3372450640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f336c0a02b0 0x7f336c134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:56.821 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f3372450640 10 monclient(hunting): picked mon.noname-a con 0x7f336c0a02b0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:38:56.821 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f3372450640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336c0a06b0 0x7f336c1220d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:56.821 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f3372450640 10 monclient(hunting): picked mon.noname-c con 0x7f336c0a06b0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:56.821 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f3372450640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:56.821 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f3372450640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f336c135490 con 0x7f336c0a02b0 2024-09-16T04:38:56.821 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f3372450640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:56.821 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f3372450640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f336c135310 con 0x7f336c09fed0 2024-09-16T04:38:56.821 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f3372450640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:56.821 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f3372450640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f336c0a2180 con 0x7f336c0a06b0 2024-09-16T04:38:56.821 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f3372450640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:56.821 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f3372450640 10 monclient(hunting): authenticate will time out at 735.223244s 2024-09-16T04:38:56.822 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f33709ed640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336c0a06b0 0x7f336c1220d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:56.822 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f336c0a02b0 0x7f336c134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:56.822 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f33709ed640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336c0a06b0 0x7f336c1220d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:60806/0 (socket says 172.21.15.46:60806) 2024-09-16T04:38:56.822 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f33709ed640 1 -- 172.21.15.46:0/293720811 learned_addr learned my addr 172.21.15.46:0/293720811 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:38:56.822 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f33709ed640 10 monclient(hunting): get_auth_request con 0x7f336c0a06b0 auth_method 0 2024-09-16T04:38:56.822 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f33709ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:56.822 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f33709ed640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:56.822 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f33709ed640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:56.822 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336b7fe640 10 monclient(hunting): get_auth_request con 0x7f336c0a02b0 auth_method 0 2024-09-16T04:38:56.822 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:56.822 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:56.822 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:56.822 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:56.822 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:56.823 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:56.823 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f33709ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:56.823 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f33709ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:56.823 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f33709ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:56.823 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336bfff640 1 --2- 172.21.15.46:0/293720811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f336c09fed0 0x7f336c134660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:56.823 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336bfff640 10 monclient(hunting): get_auth_request con 0x7f336c09fed0 auth_method 0 2024-09-16T04:38:56.823 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:56.823 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:56.823 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:56.823 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336b7fe640 10 monclient(hunting): handle_auth_done global_id 4224 payload 824 2024-09-16T04:38:56.823 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336b7fe640 1 -- 172.21.15.46:0/293720811 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336c0a06b0 msgr2=0x7f336c1220d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:56.823 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336b7fe640 1 --2- 172.21.15.46:0/293720811 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336c0a06b0 0x7f336c1220d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:56.823 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336b7fe640 1 -- 172.21.15.46:0/293720811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f336c09fed0 msgr2=0x7f336c134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:56.823 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336b7fe640 1 --2- 172.21.15.46:0/293720811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f336c09fed0 0x7f336c134660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:56.824 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:38:56.824 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336b7fe640 1 monclient: found mon.noname-a 2024-09-16T04:38:56.824 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:56.824 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:38:56.824 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336b7fe640 1 -- 172.21.15.46:0/293720811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f336c125d80 con 0x7f336c0a02b0 2024-09-16T04:38:56.824 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f336b7fe640 1 --2- 172.21.15.46:0/293720811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f336c0a02b0 0x7f336c134ba0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f336000c370 tx=0x7f336000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:56.824 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.808+0000 7f3368ff9640 1 -- 172.21.15.46:0/293720811 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33600052d0 con 0x7f336c0a02b0 2024-09-16T04:38:56.824 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.809+0000 7f3368ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:56.824 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.809+0000 7f3368ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:38:56.824 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.809+0000 7f3368ff9640 10 monclient: dump: 2024-09-16T04:38:56.824 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-09-16T04:38:56.824 INFO:teuthology.misc.health.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:56.824 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:56.824 INFO:teuthology.misc.health.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:56.825 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:56.825 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:56.825 INFO:teuthology.misc.health.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:56.825 INFO:teuthology.misc.health.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:56.825 INFO:teuthology.misc.health.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:56.825 INFO:teuthology.misc.health.smithi046.stderr: 2024-09-16T04:38:56.825 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.809+0000 7f3368ff9640 10 monclient: _finish_auth 0 2024-09-16T04:38:56.825 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.809+0000 7f3368ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:38:56.825 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.809+0000 7f3368ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:56.825 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.809+0000 7f3368ff9640 1 -- 172.21.15.46:0/293720811 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3360005490 con 0x7f336c0a02b0 2024-09-16T04:38:56.825 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.809+0000 7f3372450640 5 monclient: authenticate success, global_id 4224 2024-09-16T04:38:56.825 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.809+0000 7f3372450640 10 monclient: _renew_subs 2024-09-16T04:38:56.825 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.809+0000 7f3372450640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:38:56.825 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.809+0000 7f3372450640 1 -- 172.21.15.46:0/293720811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f336c124210 con 0x7f336c0a02b0 2024-09-16T04:38:56.825 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.809+0000 7f3368ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:56.826 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.809+0000 7f3368ff9640 1 -- 172.21.15.46:0/293720811 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3360007260 con 0x7f336c0a02b0 2024-09-16T04:38:56.826 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.809+0000 7f3368ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:56.826 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.809+0000 7f3368ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:38:56.826 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.809+0000 7f3368ff9640 10 monclient: dump: 2024-09-16T04:38:56.826 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-09-16T04:38:56.826 INFO:teuthology.misc.health.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:56.826 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:56.826 INFO:teuthology.misc.health.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:56.826 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:56.826 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:56.826 INFO:teuthology.misc.health.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:56.826 INFO:teuthology.misc.health.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:56.826 INFO:teuthology.misc.health.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:56.826 INFO:teuthology.misc.health.smithi046.stderr: 2024-09-16T04:38:56.826 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.809+0000 7f3372450640 10 monclient: _renew_subs 2024-09-16T04:38:56.827 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.809+0000 7f336a7fc640 4 set_mon_vals no callback set 2024-09-16T04:38:56.827 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.809+0000 7f3372450640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:38:56.827 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.809+0000 7f3372450640 1 -- 172.21.15.46:0/293720811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f336c06ef90 con 0x7f336c0a02b0 2024-09-16T04:38:56.827 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.810+0000 7f3372450640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:56.827 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.810+0000 7f3372450640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:56.827 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.810+0000 7f3372450640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:38:56.827 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.810+0000 7f3372450640 1 -- 172.21.15.46:0/293720811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f336c072e30 con 0x7f336c0a02b0 2024-09-16T04:38:56.827 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.810+0000 7f3368ff9640 1 -- 172.21.15.46:0/293720811 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f3360007400 con 0x7f336c0a02b0 2024-09-16T04:38:56.827 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.810+0000 7f3368ff9640 1 --2- 172.21.15.46:0/293720811 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f33580383b0 0x7f335803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:56.827 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.810+0000 7f3368ff9640 1 -- 172.21.15.46:0/293720811 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3993+0+0 (secure 0 0 0) 0x7f336004a8c0 con 0x7f336c0a02b0 2024-09-16T04:38:56.827 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.811+0000 7f336bfff640 1 --2- 172.21.15.46:0/293720811 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f33580383b0 0x7f335803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:56.827 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.811+0000 7f336bfff640 10 monclient: get_auth_request con 0x7f33580383b0 auth_method 0 2024-09-16T04:38:56.827 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.811+0000 7f336bfff640 1 --2- 172.21.15.46:0/293720811 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f33580383b0 0x7f335803a870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f335c034000 tx=0x7f335c02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:56.828 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.815+0000 7f3368ff9640 1 -- 172.21.15.46:0/293720811 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f336c072e30 con 0x7f336c0a02b0 2024-09-16T04:38:56.828 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.815+0000 7f3368ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:56.828 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.815+0000 7f3368ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:38:56.994 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.993+0000 7f3372450640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:38:56.995 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.993+0000 7f3372450640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:38:56.995 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.993+0000 7f3372450640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:38:56.995 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.993+0000 7f3372450640 1 -- 172.21.15.46:0/293720811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f336c09f700 con 0x7f336c0a02b0 2024-09-16T04:38:56.995 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.994+0000 7f3368ff9640 1 -- 172.21.15.46:0/293720811 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+10473 (secure 0 0 0) 0x7f336c09e450 con 0x7f336c0a02b0 2024-09-16T04:38:56.995 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.994+0000 7f3368ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:38:56.995 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.994+0000 7f3368ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:38:56.996 INFO:teuthology.misc.health.smithi046.stdout: 2024-09-16T04:38:56.996 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":15,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","created":"2024-09-16T04:38:39.286708+0000","modified":"2024-09-16T04:38:53.500186+0000","last_up_change":"2024-09-16T04:38:50.487347+0000","last_in_change":"2024-09-16T04:38:42.117316+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:38:51.392177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ff88780-0630-4bb1-9f5a-5331d7724d17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3454452571},{"type":"v1","addr":"172.21.15.46:6809","nonce":3454452571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3454452571},{"type":"v1","addr":"172.21.15.46:6811","nonce":3454452571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3454452571},{"type":"v1","addr":"172.21.15.46:6815","nonce":3454452571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3454452571},{"type":"v1","addr":"172.21.15.46:6813","nonce":3454452571}]},"public_addr":"172.21.15.46:6809/3454452571","cluster_addr":"172.21.15.46:6811/3454452571","heartbeat_back_addr":"172.21.15.46:6815/3454452571","heartbeat_front_addr":"172.21.15.46:6813/3454452571","state":["exists","up"]},{"osd":1,"uuid":"37be2e88-32d0-4a30-8979-899b34f3be17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6817","nonce":4217009477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6819","nonce":4217009477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6823","nonce":4217009477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6821","nonce":4217009477}]},"public_addr":"172.21.15.46:6817/4217009477","cluster_addr":"172.21.15.46:6819/4217009477","heartbeat_back_addr":"172.21.15.46:6823/4217009477","heartbeat_front_addr":"172.21.15.46:6821/4217009477","state":["exists","up"]},{"osd":2,"uuid":"73b9a838-944e-48cd-afab-5260608d1c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6801","nonce":3101080311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6803","nonce":3101080311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6807","nonce":3101080311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6805","nonce":3101080311}]},"public_addr":"172.21.15.46:6801/3101080311","cluster_addr":"172.21.15.46:6803/3101080311","heartbeat_back_addr":"172.21.15.46:6807/3101080311","heartbeat_front_addr":"172.21.15.46:6805/3101080311","state":["exists","up"]},{"osd":3,"uuid":"d0c22455-5685-4ade-b540-3d5428a7cb46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":890769734},{"type":"v1","addr":"172.21.15.114:6811","nonce":890769734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":890769734},{"type":"v1","addr":"172.21.15.114:6813","nonce":890769734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":890769734},{"type":"v1","addr":"172.21.15.114:6817","nonce":890769734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":890769734},{"type":"v1","addr":"172.21.15.114:6815","nonce":890769734}]},"public_addr":"172.21.15.114:6811/890769734","cluster_addr":"172.21.15.114:6813/890769734","heartbeat_back_addr":"172.21.15.114:6817/890769734","heartbeat_front_addr":"172.21.15.114:6815/890769734","state":["exists","up"]},{"osd":4,"uuid":"f37724e4-2796-43c8-975b-4458986c4ea6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":936316557},{"type":"v1","addr":"172.21.15.114:6803","nonce":936316557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":936316557},{"type":"v1","addr":"172.21.15.114:6805","nonce":936316557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":936316557},{"type":"v1","addr":"172.21.15.114:6809","nonce":936316557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":936316557},{"type":"v1","addr":"172.21.15.114:6807","nonce":936316557}]},"public_addr":"172.21.15.114:6803/936316557","cluster_addr":"172.21.15.114:6805/936316557","heartbeat_back_addr":"172.21.15.114:6809/936316557","heartbeat_front_addr":"172.21.15.114:6807/936316557","state":["exists","up"]},{"osd":5,"uuid":"979362db-e173-47a0-93d1-187742938a85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6819","nonce":938868100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6821","nonce":938868100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6825","nonce":938868100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6823","nonce":938868100}]},"public_addr":"172.21.15.114:6819/938868100","cluster_addr":"172.21.15.114:6821/938868100","heartbeat_back_addr":"172.21.15.114:6825/938868100","heartbeat_front_addr":"172.21.15.114:6823/938868100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.647737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.394775+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.463096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.655270+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.652725+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.431803+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:38:56.998 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.996+0000 7f3372450640 1 -- 172.21.15.46:0/293720811 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f33580383b0 msgr2=0x7f335803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:56.998 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.996+0000 7f3372450640 1 --2- 172.21.15.46:0/293720811 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f33580383b0 0x7f335803a870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f335c034000 tx=0x7f335c02e000 comp rx=0 tx=0).stop 2024-09-16T04:38:56.998 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.996+0000 7f3372450640 10 monclient: shutdown 2024-09-16T04:38:56.998 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.996+0000 7f3372450640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:56.998 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.996+0000 7f3372450640 1 -- 172.21.15.46:0/293720811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f336c0a02b0 msgr2=0x7f336c134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:56.998 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.996+0000 7f3372450640 1 --2- 172.21.15.46:0/293720811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f336c0a02b0 0x7f336c134ba0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f336000c370 tx=0x7f336000c830 comp rx=0 tx=0).stop 2024-09-16T04:38:56.998 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.996+0000 7f3372450640 1 -- 172.21.15.46:0/293720811 shutdown_connections 2024-09-16T04:38:56.998 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.996+0000 7f3372450640 1 --2- 172.21.15.46:0/293720811 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336c0a06b0 0x7f336c1220d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:56.999 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.996+0000 7f3372450640 1 --2- 172.21.15.46:0/293720811 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f33580383b0 0x7f335803a870 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:56.999 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.997+0000 7f3372450640 1 --2- 172.21.15.46:0/293720811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f336c0a02b0 0x7f336c134ba0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:56.999 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.997+0000 7f3372450640 1 --2- 172.21.15.46:0/293720811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f336c09fed0 0x7f336c134660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:56.999 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.997+0000 7f3372450640 1 -- 172.21.15.46:0/293720811 >> 172.21.15.46:0/293720811 conn(0x7f336c0a57d0 msgr2=0x7f336c0713d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:56.999 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.997+0000 7f3372450640 1 -- 172.21.15.46:0/293720811 shutdown_connections 2024-09-16T04:38:56.999 INFO:teuthology.misc.health.smithi046.stderr:2024-09-16T04:38:56.997+0000 7f3372450640 1 -- 172.21.15.46:0/293720811 wait complete. 2024-09-16T04:38:57.007 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-16T04:38:57.007 INFO:tasks.ceph:Creating RBD pool 2024-09-16T04:38:57.007 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T04:38:57.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.107+0000 7feea3723640 10 monclient: get_monmap_and_config 2024-09-16T04:38:57.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.107+0000 7feea3723640 10 monclient: build_initial_monmap 2024-09-16T04:38:57.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.107+0000 7feea3723640 10 monclient: monmap: 2024-09-16T04:38:57.108 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:38:57.108 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:57.109 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:57.107777+0000 2024-09-16T04:38:57.109 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:57.107777+0000 2024-09-16T04:38:57.109 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:57.109 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:57.109 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:38:57.109 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:38:57.109 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:38:57.109 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:38:57.109 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:38:57.109 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:57.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.107+0000 7feea3723640 1 Processor -- start 2024-09-16T04:38:57.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea3723640 1 -- start start 2024-09-16T04:38:57.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea3723640 10 monclient: init 2024-09-16T04:38:57.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea3723640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:57.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea3723640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-16T04:38:57.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea3723640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fee9c0a3c10 0x7fee9c09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:57.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea3723640 10 monclient(hunting): picked mon.noname-b con 0x7fee9c0a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:57.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea3723640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fee9c0a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:38:57.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea3723640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fee9c13be90 addr v1:172.21.15.114:6789/0 2024-09-16T04:38:57.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea3723640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee9c09b940 con 0x7fee9c0a3ff0 2024-09-16T04:38:57.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea3723640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee9c09ab00 con 0x7fee9c13be90 2024-09-16T04:38:57.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea3723640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:57.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea3723640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fee9c09b030 con 0x7fee9c0a3c10 2024-09-16T04:38:57.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea3723640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:57.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea3723640 10 monclient(hunting): authenticate will time out at 735.523649s 2024-09-16T04:38:57.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea14bf640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fee9c0a3c10 0x7fee9c09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:57.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea0cbe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fee9c0a3ff0 0x7fee9c0a43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52982/0 (socket says 172.21.15.46:52982) 2024-09-16T04:38:57.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea0cbe640 1 -- 172.21.15.46:0/1391111686 learned_addr learned my addr 172.21.15.46:0/1391111686 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:38:57.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea14bf640 10 monclient(hunting): get_auth_request con 0x7fee9c0a3c10 auth_method 0 2024-09-16T04:38:57.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea14bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:57.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea14bf640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:57.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea14bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:57.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea14bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:57.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea14bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:57.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.108+0000 7feea14bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:57.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7fee93fff640 1 -- 172.21.15.46:0/1391111686 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2733041412 0 0) 0x7fee9c09b940 con 0x7fee9c0a3ff0 2024-09-16T04:38:57.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7feea14bf640 10 monclient(hunting): handle_auth_done global_id 4208 payload 274 2024-09-16T04:38:57.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7feea14bf640 1 -- 172.21.15.46:0/1391111686 >> v1:172.21.15.114:6789/0 conn(0x7fee9c13be90 legacy=0x7fee9c13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:38:57.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7feea14bf640 1 -- 172.21.15.46:0/1391111686 >> v1:172.21.15.46:6789/0 conn(0x7fee9c0a3ff0 legacy=0x7fee9c0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:57.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7feea14bf640 10 monclient: _finish_hunting 0 2024-09-16T04:38:57.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7feea14bf640 1 monclient: found mon.noname-b 2024-09-16T04:38:57.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7feea14bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:57.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7feea14bf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:38:57.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7feea14bf640 1 -- 172.21.15.46:0/1391111686 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee9c09b310 con 0x7fee9c0a3c10 2024-09-16T04:38:57.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7fee93fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:38:57.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7feea14bf640 1 --2- 172.21.15.46:0/1391111686 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fee9c0a3c10 0x7fee9c09bcf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fee980097d0 tx=0x7fee9802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27ca71ce18c1fba9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:57.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7fee93fff640 1 -- 172.21.15.46:0/1391111686 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee98037070 con 0x7fee9c0a3c10 2024-09-16T04:38:57.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7fee93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:57.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7fee93fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:38:57.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7fee93fff640 10 monclient: dump: 2024-09-16T04:38:57.119 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:38:57.119 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:57.119 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:57.119 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:57.119 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:57.119 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:57.119 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:57.119 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:57.119 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:57.119 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:57.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7fee93fff640 10 monclient: _finish_auth 0 2024-09-16T04:38:57.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7fee93fff640 10 monclient: _check_auth_tickets 2024-09-16T04:38:57.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7fee93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:57.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7fee93fff640 1 -- 172.21.15.46:0/1391111686 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee9802adb0 con 0x7fee9c0a3c10 2024-09-16T04:38:57.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7fee93fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:57.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7fee93fff640 1 -- 172.21.15.46:0/1391111686 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee98032660 con 0x7fee9c0a3c10 2024-09-16T04:38:57.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7feea3723640 5 monclient: authenticate success, global_id 4208 2024-09-16T04:38:57.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7feea3723640 10 monclient: get_monmap_and_config success 2024-09-16T04:38:57.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7fee93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:57.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7feea3723640 4 set_mon_vals no callback set 2024-09-16T04:38:57.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7fee93fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:38:57.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.109+0000 7fee93fff640 10 monclient: dump: 2024-09-16T04:38:57.120 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:38:57.120 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:57.120 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:57.121 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:57.121 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:57.121 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:57.121 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:57.121 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:57.121 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:57.121 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:57.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.110+0000 7feea3723640 10 monclient: shutdown 2024-09-16T04:38:57.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.110+0000 7feea3723640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:57.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.110+0000 7feea3723640 1 -- 172.21.15.46:0/1391111686 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fee9c0a3c10 msgr2=0x7fee9c09bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:57.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.110+0000 7feea3723640 1 --2- 172.21.15.46:0/1391111686 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fee9c0a3c10 0x7fee9c09bcf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fee980097d0 tx=0x7fee9802a0a0 comp rx=0 tx=0).stop 2024-09-16T04:38:57.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.110+0000 7feea3723640 1 -- 172.21.15.46:0/1391111686 shutdown_connections 2024-09-16T04:38:57.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.110+0000 7feea3723640 1 --2- 172.21.15.46:0/1391111686 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fee9c0a3c10 0x7fee9c09bcf0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:57.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.110+0000 7feea3723640 1 -- 172.21.15.46:0/1391111686 >> 172.21.15.46:0/1391111686 conn(0x7fee9c0a6340 msgr2=0x7fee9c06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:57.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.110+0000 7feea3723640 1 -- 172.21.15.46:0/1391111686 shutdown_connections 2024-09-16T04:38:57.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.110+0000 7feea3723640 1 -- 172.21.15.46:0/1391111686 wait complete. 2024-09-16T04:38:57.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.111+0000 7feea3723640 10 monclient: build_initial_monmap 2024-09-16T04:38:57.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.111+0000 7feea3723640 10 monclient: monmap: 2024-09-16T04:38:57.122 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:38:57.122 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:38:57.122 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:38:57.122 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:38:57.122 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:57.122 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:57.122 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:38:57.122 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:38:57.122 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:38:57.122 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:57.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.111+0000 7feea3723640 1 Processor -- start 2024-09-16T04:38:57.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.111+0000 7feea3723640 1 -- start start 2024-09-16T04:38:57.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.111+0000 7feea3723640 10 monclient: init 2024-09-16T04:38:57.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.111+0000 7feea3723640 10 monclient: _reopen_session rank -1 2024-09-16T04:38:57.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.111+0000 7feea3723640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:38:57.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.111+0000 7feea3723640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee9c0a3c10 0x7fee9c136990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:57.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.111+0000 7feea3723640 10 monclient(hunting): picked mon.noname-b con 0x7fee9c0a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:38:57.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.111+0000 7feea3723640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee9c0a3ff0 0x7fee9c136ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:57.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.111+0000 7feea3723640 10 monclient(hunting): picked mon.noname-a con 0x7fee9c0a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:38:57.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.111+0000 7feea3723640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fee9c13be90 0x7fee9c138b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:57.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.111+0000 7feea3723640 10 monclient(hunting): picked mon.noname-c con 0x7fee9c13be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:57.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.111+0000 7feea3723640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:57.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.111+0000 7feea3723640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fee9c0a6b40 con 0x7fee9c0a3ff0 2024-09-16T04:38:57.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.111+0000 7feea3723640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:57.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.111+0000 7feea3723640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fee9c0a69c0 con 0x7fee9c0a3c10 2024-09-16T04:38:57.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.111+0000 7feea3723640 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:57.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.111+0000 7feea3723640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fee9c13f9a0 con 0x7fee9c13be90 2024-09-16T04:38:57.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.111+0000 7feea3723640 10 monclient(hunting): _renew_subs 2024-09-16T04:38:57.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.111+0000 7feea3723640 10 monclient(hunting): authenticate will time out at 735.526981s 2024-09-16T04:38:57.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea1cc0640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fee9c13be90 0x7fee9c138b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:57.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea0cbe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee9c0a3ff0 0x7fee9c136ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:57.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea0cbe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee9c0a3ff0 0x7fee9c136ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59130/0 (socket says 172.21.15.46:59130) 2024-09-16T04:38:57.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea1cc0640 10 monclient(hunting): get_auth_request con 0x7fee9c13be90 auth_method 0 2024-09-16T04:38:57.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea0cbe640 1 -- 172.21.15.46:0/805791843 learned_addr learned my addr 172.21.15.46:0/805791843 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:38:57.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea1cc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:57.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea1cc0640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:57.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea1cc0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:57.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea0cbe640 10 monclient(hunting): get_auth_request con 0x7fee9c0a3ff0 auth_method 0 2024-09-16T04:38:57.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea0cbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:57.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea0cbe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:57.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea0cbe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:57.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea1cc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:57.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea1cc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:57.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea1cc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:57.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea14bf640 1 --2- 172.21.15.46:0/805791843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee9c0a3c10 0x7fee9c136990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:57.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea0cbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:57.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea0cbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:57.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea0cbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:57.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea14bf640 10 monclient(hunting): get_auth_request con 0x7fee9c0a3c10 auth_method 0 2024-09-16T04:38:57.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea14bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:57.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea14bf640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:57.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea14bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:57.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea14bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:57.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea14bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:57.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea14bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:57.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea1cc0640 10 monclient(hunting): handle_auth_done global_id 4211 payload 824 2024-09-16T04:38:57.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea1cc0640 1 -- 172.21.15.46:0/805791843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee9c0a3c10 msgr2=0x7fee9c136990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:57.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea1cc0640 1 --2- 172.21.15.46:0/805791843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee9c0a3c10 0x7fee9c136990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:57.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea1cc0640 1 -- 172.21.15.46:0/805791843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee9c0a3ff0 msgr2=0x7fee9c136ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:57.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea1cc0640 1 --2- 172.21.15.46:0/805791843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee9c0a3ff0 0x7fee9c136ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:57.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea1cc0640 10 monclient: _finish_hunting 0 2024-09-16T04:38:57.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea1cc0640 1 monclient: found mon.noname-c 2024-09-16T04:38:57.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea1cc0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:57.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea1cc0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:38:57.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea1cc0640 1 -- 172.21.15.46:0/805791843 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee9c120470 con 0x7fee9c13be90 2024-09-16T04:38:57.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea0cbe640 1 --2- 172.21.15.46:0/805791843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee9c0a3ff0 0x7fee9c136ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:38:57.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea1cc0640 1 --2- 172.21.15.46:0/805791843 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fee9c13be90 0x7fee9c138b60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fee9400c780 tx=0x7fee9400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:57.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7fee91ffb640 1 -- 172.21.15.46:0/805791843 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee94016020 con 0x7fee9c13be90 2024-09-16T04:38:57.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7fee91ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:57.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7fee91ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:38:57.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7fee91ffb640 10 monclient: dump: 2024-09-16T04:38:57.127 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:38:57.127 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:57.127 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:57.127 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:57.127 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:57.127 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:57.127 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:57.127 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:57.127 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:57.127 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:57.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7fee91ffb640 10 monclient: _finish_auth 0 2024-09-16T04:38:57.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7fee91ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:38:57.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7fee91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:57.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7fee91ffb640 1 -- 172.21.15.46:0/805791843 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee9400a840 con 0x7fee9c13be90 2024-09-16T04:38:57.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7fee91ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:57.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7fee91ffb640 1 -- 172.21.15.46:0/805791843 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9400ac30 con 0x7fee9c13be90 2024-09-16T04:38:57.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea3723640 5 monclient: authenticate success, global_id 4211 2024-09-16T04:38:57.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7fee91ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:57.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7fee91ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:38:57.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7fee91ffb640 10 monclient: dump: 2024-09-16T04:38:57.128 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:38:57.128 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:57.128 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:57.128 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:57.128 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:57.128 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:57.129 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7fee937fe640 4 set_mon_vals no callback set 2024-09-16T04:38:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea3723640 10 monclient: _renew_subs 2024-09-16T04:38:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea3723640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:38:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.112+0000 7feea3723640 1 -- 172.21.15.46:0/805791843 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee9c0a8dd0 con 0x7fee9c13be90 2024-09-16T04:38:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.113+0000 7feea3723640 10 monclient: _renew_subs 2024-09-16T04:38:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.113+0000 7feea3723640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:38:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.113+0000 7feea3723640 1 -- 172.21.15.46:0/805791843 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fee9c120190 con 0x7fee9c13be90 2024-09-16T04:38:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.113+0000 7fee91ffb640 1 -- 172.21.15.46:0/805791843 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fee94019020 con 0x7fee9c13be90 2024-09-16T04:38:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.113+0000 7fee91ffb640 1 --2- 172.21.15.46:0/805791843 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fee800383b0 0x7fee8003a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.113+0000 7fee91ffb640 1 -- 172.21.15.46:0/805791843 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3993+0+0 (secure 0 0 0) 0x7fee9404b1f0 con 0x7fee9c13be90 2024-09-16T04:38:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.113+0000 7feea3723640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.113+0000 7feea3723640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.114+0000 7feea3723640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:38:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.114+0000 7feea3723640 1 -- 172.21.15.46:0/805791843 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee9c09bd70 con 0x7fee9c13be90 2024-09-16T04:38:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.114+0000 7feea14bf640 1 --2- 172.21.15.46:0/805791843 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fee800383b0 0x7fee8003a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.114+0000 7feea14bf640 10 monclient: get_auth_request con 0x7fee800383b0 auth_method 0 2024-09-16T04:38:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.114+0000 7feea14bf640 1 --2- 172.21.15.46:0/805791843 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fee800383b0 0x7fee8003a870 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fee98034000 tx=0x7fee9802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.119+0000 7fee91ffb640 1 -- 172.21.15.46:0/805791843 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fee94020030 con 0x7fee9c13be90 2024-09-16T04:38:57.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.119+0000 7fee91ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:38:57.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.119+0000 7fee91ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:38:57.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.294+0000 7feea3723640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T04:38:57.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.294+0000 7feea3723640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T04:38:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.294+0000 7feea3723640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:38:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.294+0000 7feea3723640 1 -- 172.21.15.46:0/805791843 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7fee9c13b040 con 0x7fee9c13be90 2024-09-16T04:38:57.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.512+0000 7fee91ffb640 1 -- 172.21.15.46:0/805791843 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) ==== 109+0+0 (secure 0 0 0) 0x7fee94011380 con 0x7fee9c13be90 2024-09-16T04:38:57.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.512+0000 7fee91ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T04:38:57.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.512+0000 7fee91ffb640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-16T04:38:57.514 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'rbd' created 2024-09-16T04:38:57.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.516+0000 7feea3723640 1 -- 172.21.15.46:0/805791843 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fee800383b0 msgr2=0x7fee8003a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:57.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.516+0000 7feea3723640 1 --2- 172.21.15.46:0/805791843 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fee800383b0 0x7fee8003a870 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fee98034000 tx=0x7fee9802e000 comp rx=0 tx=0).stop 2024-09-16T04:38:57.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.516+0000 7feea3723640 10 monclient: shutdown 2024-09-16T04:38:57.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.516+0000 7feea3723640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.516+0000 7feea3723640 1 -- 172.21.15.46:0/805791843 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fee9c13be90 msgr2=0x7fee9c138b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.516+0000 7feea3723640 1 --2- 172.21.15.46:0/805791843 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fee9c13be90 0x7fee9c138b60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fee9400c780 tx=0x7fee9400cc40 comp rx=0 tx=0).stop 2024-09-16T04:38:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.516+0000 7feea3723640 1 -- 172.21.15.46:0/805791843 shutdown_connections 2024-09-16T04:38:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.516+0000 7feea3723640 1 --2- 172.21.15.46:0/805791843 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fee9c13be90 0x7fee9c138b60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.516+0000 7feea3723640 1 --2- 172.21.15.46:0/805791843 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fee800383b0 0x7fee8003a870 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.516+0000 7feea3723640 1 --2- 172.21.15.46:0/805791843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee9c0a3ff0 0x7fee9c136ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.516+0000 7feea3723640 1 --2- 172.21.15.46:0/805791843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee9c0a3c10 0x7fee9c136990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.516+0000 7feea3723640 1 -- 172.21.15.46:0/805791843 >> 172.21.15.46:0/805791843 conn(0x7fee9c0a6340 msgr2=0x7fee9c0a2240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.516+0000 7feea3723640 1 -- 172.21.15.46:0/805791843 shutdown_connections 2024-09-16T04:38:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.516+0000 7feea3723640 1 -- 172.21.15.46:0/805791843 wait complete. 2024-09-16T04:38:57.528 DEBUG:teuthology.orchestra.run.smithi046:> rbd --cluster ceph pool init rbd 2024-09-16T04:38:57.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.591+0000 7fc29dd64d80 10 monclient: get_monmap_and_config 2024-09-16T04:38:57.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.591+0000 7fc29dd64d80 10 monclient: build_initial_monmap 2024-09-16T04:38:57.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.591+0000 7fc29dd64d80 10 monclient: monmap: 2024-09-16T04:38:57.593 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:38:57.594 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:57.594 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:57.592332+0000 2024-09-16T04:38:57.594 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:57.592332+0000 2024-09-16T04:38:57.594 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:57.594 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:57.594 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:38:57.594 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:38:57.594 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:38:57.594 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:38:57.594 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:38:57.594 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:57.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.592+0000 7fc29dd64d80 1 Processor -- start 2024-09-16T04:38:57.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.592+0000 7fc29dd64d80 1 -- start start 2024-09-16T04:38:57.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.592+0000 7fc29dd64d80 10 monclient: init 2024-09-16T04:38:57.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dd64d80 10 monclient: _reopen_session rank -1 2024-09-16T04:38:57.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dd64d80 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-16T04:38:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dd64d80 1 --2- >> v2:172.21.15.114:3300/0 conn(0x55fd7ba96990 0x55fd7ba96d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dd64d80 10 monclient(hunting): picked mon.noname-c con 0x55fd7ba96990 addr v2:172.21.15.114:3300/0 2024-09-16T04:38:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dd64d80 1 --2- >> v2:172.21.15.46:3300/0 conn(0x55fd7baabe00 0x55fd7ba972b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dd64d80 10 monclient(hunting): picked mon.noname-a con 0x55fd7baabe00 addr v2:172.21.15.46:3300/0 2024-09-16T04:38:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dd64d80 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x55fd7bb47340 0x55fd7bb47770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dd64d80 10 monclient(hunting): picked mon.noname-b con 0x55fd7bb47340 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dd64d80 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dd64d80 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x55fd7b929930 con 0x55fd7baabe00 2024-09-16T04:38:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dd64d80 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dd64d80 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x55fd7b984c80 con 0x55fd7ba96990 2024-09-16T04:38:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dd64d80 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dd64d80 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x55fd7b92b540 con 0x55fd7bb47340 2024-09-16T04:38:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dd64d80 10 monclient(hunting): _renew_subs 2024-09-16T04:38:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dd64d80 10 monclient(hunting): authenticate will time out at 736.008140s 2024-09-16T04:38:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dabe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x55fd7bb47340 0x55fd7bb47770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:57.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc297fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x55fd7baabe00 0x55fd7ba972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:57.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dabe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x55fd7bb47340 0x55fd7bb47770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:60838/0 (socket says 172.21.15.46:60838) 2024-09-16T04:38:57.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dabe640 1 -- 172.21.15.46:0/3192033765 learned_addr learned my addr 172.21.15.46:0/3192033765 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:38:57.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dabe640 10 monclient(hunting): get_auth_request con 0x55fd7bb47340 auth_method 0 2024-09-16T04:38:57.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dabe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:57.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dabe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:57.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dabe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:57.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc297fff640 10 monclient(hunting): get_auth_request con 0x55fd7baabe00 auth_method 0 2024-09-16T04:38:57.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc297fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:57.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc297fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:57.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc297fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:57.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dabe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:57.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dabe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:57.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dabe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:57.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc297fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:57.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc297fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:57.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc297fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:57.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29c85c640 1 --2- 172.21.15.46:0/3192033765 >> v2:172.21.15.114:3300/0 conn(0x55fd7ba96990 0x55fd7ba96d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:57.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29c85c640 10 monclient(hunting): get_auth_request con 0x55fd7ba96990 auth_method 0 2024-09-16T04:38:57.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29c85c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:57.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29c85c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:57.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29c85c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:57.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29c85c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:57.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29c85c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:57.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29c85c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:57.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dabe640 10 monclient(hunting): handle_auth_done global_id 4214 payload 274 2024-09-16T04:38:57.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dabe640 1 -- 172.21.15.46:0/3192033765 >> v2:172.21.15.114:3300/0 conn(0x55fd7ba96990 msgr2=0x55fd7ba96d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:38:57.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dabe640 1 --2- 172.21.15.46:0/3192033765 >> v2:172.21.15.114:3300/0 conn(0x55fd7ba96990 0x55fd7ba96d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:57.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dabe640 1 -- 172.21.15.46:0/3192033765 >> v2:172.21.15.46:3300/0 conn(0x55fd7baabe00 msgr2=0x55fd7ba972b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:38:57.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dabe640 1 --2- 172.21.15.46:0/3192033765 >> v2:172.21.15.46:3300/0 conn(0x55fd7baabe00 0x55fd7ba972b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:57.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dabe640 10 monclient: _finish_hunting 0 2024-09-16T04:38:57.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dabe640 1 monclient: found mon.noname-b 2024-09-16T04:38:57.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dabe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:57.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dabe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:38:57.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc29dabe640 1 -- 172.21.15.46:0/3192033765 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd7b91b050 con 0x55fd7bb47340 2024-09-16T04:38:57.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.593+0000 7fc297fff640 1 --2- 172.21.15.46:0/3192033765 >> v2:172.21.15.46:3300/0 conn(0x55fd7baabe00 0x55fd7ba972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:38:57.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.595+0000 7fc29dabe640 1 --2- 172.21.15.46:0/3192033765 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x55fd7bb47340 0x55fd7bb47770 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x55fd7ba97a00 tx=0x7fc290045ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c01248d4e091206e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:57.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.595+0000 7fc2977fe640 1 -- 172.21.15.46:0/3192033765 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc290051470 con 0x55fd7bb47340 2024-09-16T04:38:57.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.595+0000 7fc2977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.595+0000 7fc2977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:38:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.595+0000 7fc2977fe640 10 monclient: dump: 2024-09-16T04:38:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:38:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:57.607 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.595+0000 7fc2977fe640 10 monclient: _finish_auth 0 2024-09-16T04:38:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.595+0000 7fc2977fe640 10 monclient: _check_auth_tickets 2024-09-16T04:38:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.595+0000 7fc2977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.595+0000 7fc29dd64d80 5 monclient: authenticate success, global_id 4214 2024-09-16T04:38:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.595+0000 7fc29dd64d80 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:38:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.595+0000 7fc2977fe640 1 -- 172.21.15.46:0/3192033765 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc290057070 con 0x55fd7bb47340 2024-09-16T04:38:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.595+0000 7fc2977fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.595+0000 7fc29dd64d80 10 monclient: get_monmap_and_config success 2024-09-16T04:38:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.595+0000 7fc29dd64d80 4 set_mon_vals no callback set 2024-09-16T04:38:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.595+0000 7fc2977fe640 1 -- 172.21.15.46:0/3192033765 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc290051830 con 0x55fd7bb47340 2024-09-16T04:38:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.595+0000 7fc2977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.595+0000 7fc2977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:38:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.595+0000 7fc2977fe640 10 monclient: dump: 2024-09-16T04:38:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:38:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:57.609 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:57.609 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:57.609 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:57.609 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:57.609 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:57.609 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:57.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.596+0000 7fc29dd64d80 10 monclient: shutdown 2024-09-16T04:38:57.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.596+0000 7fc29dd64d80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:38:57.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.596+0000 7fc29dd64d80 1 -- 172.21.15.46:0/3192033765 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x55fd7bb47340 msgr2=0x55fd7bb47770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:57.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.596+0000 7fc29dd64d80 1 --2- 172.21.15.46:0/3192033765 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x55fd7bb47340 0x55fd7bb47770 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x55fd7ba97a00 tx=0x7fc290045ab0 comp rx=0 tx=0).stop 2024-09-16T04:38:57.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.596+0000 7fc29dd64d80 1 -- 172.21.15.46:0/3192033765 shutdown_connections 2024-09-16T04:38:57.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.596+0000 7fc29dd64d80 1 --2- 172.21.15.46:0/3192033765 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x55fd7bb47340 0x55fd7bb47770 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:57.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.596+0000 7fc29dd64d80 1 --2- 172.21.15.46:0/3192033765 >> v2:172.21.15.46:3300/0 conn(0x55fd7baabe00 0x55fd7ba972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:57.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.596+0000 7fc29dd64d80 1 --2- 172.21.15.46:0/3192033765 >> v2:172.21.15.114:3300/0 conn(0x55fd7ba96990 0x55fd7ba96d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:57.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.596+0000 7fc29dd64d80 1 -- 172.21.15.46:0/3192033765 >> 172.21.15.46:0/3192033765 conn(0x55fd7ba97f20 msgr2=0x55fd7ba98330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:38:57.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.596+0000 7fc29dd64d80 1 -- 172.21.15.46:0/3192033765 shutdown_connections 2024-09-16T04:38:57.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.596+0000 7fc29dd64d80 1 -- 172.21.15.46:0/3192033765 wait complete. 2024-09-16T04:38:57.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.596+0000 7fc29dd64d80 10 monclient: build_initial_monmap 2024-09-16T04:38:57.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.596+0000 7fc29dd64d80 10 monclient: monmap: 2024-09-16T04:38:57.610 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:38:57.610 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:38:57.610 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:38:57.610 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:38:57.610 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:38:57.610 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:57.610 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:38:57.610 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:38:57.610 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:38:57.611 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:57.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29dd64d80 1 Processor -- start 2024-09-16T04:38:57.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29dd64d80 1 -- start start 2024-09-16T04:38:57.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29dd64d80 10 monclient: init 2024-09-16T04:38:57.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29dd64d80 10 monclient: _reopen_session rank -1 2024-09-16T04:38:57.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29dd64d80 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:38:57.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29dd64d80 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x55fd7ba96990 0x55fd7bb32ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:57.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29dd64d80 10 monclient(hunting): picked mon.noname-c con 0x55fd7ba96990 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:38:57.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29dd64d80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd7baabe00 0x55fd7bb331e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:57.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29dd64d80 10 monclient(hunting): picked mon.noname-a con 0x55fd7baabe00 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:38:57.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29dd64d80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd7bb47340 0x55fd7bb33720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:57.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29dd64d80 10 monclient(hunting): picked mon.noname-b con 0x55fd7bb47340 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:38:57.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29dd64d80 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:57.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29dd64d80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fd7bab63f0 con 0x55fd7baabe00 2024-09-16T04:38:57.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29dd64d80 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:57.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29dd64d80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fd7ba977f0 con 0x55fd7bb47340 2024-09-16T04:38:57.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29dd64d80 10 monclient(hunting): start opening mon connection 2024-09-16T04:38:57.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29dd64d80 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x55fd7baab140 con 0x55fd7ba96990 2024-09-16T04:38:57.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29dd64d80 10 monclient(hunting): _renew_subs 2024-09-16T04:38:57.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29dd64d80 10 monclient(hunting): authenticate will time out at 736.012579s 2024-09-16T04:38:57.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29c85c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x55fd7ba96990 0x55fd7bb32ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:57.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29c85c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x55fd7ba96990 0x55fd7bb32ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:60850/0 (socket says 172.21.15.46:60850) 2024-09-16T04:38:57.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc297fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd7baabe00 0x55fd7bb331e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:57.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29c85c640 1 -- 172.21.15.46:0/2377508486 learned_addr learned my addr 172.21.15.46:0/2377508486 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:38:57.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29c85c640 10 monclient(hunting): get_auth_request con 0x55fd7ba96990 auth_method 0 2024-09-16T04:38:57.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29c85c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:57.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29c85c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:57.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29c85c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:57.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc297fff640 10 monclient(hunting): get_auth_request con 0x55fd7baabe00 auth_method 0 2024-09-16T04:38:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc297fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:38:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc297fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:38:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc297fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:38:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29c85c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29c85c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29c85c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc297fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:38:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc297fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:38:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc297fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:38:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29dabe640 1 --2- 172.21.15.46:0/2377508486 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd7bb47340 0x55fd7bb33720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29c85c640 10 monclient(hunting): handle_auth_done global_id 4217 payload 824 2024-09-16T04:38:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29c85c640 1 -- 172.21.15.46:0/2377508486 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd7bb47340 msgr2=0x55fd7bb33720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29c85c640 1 --2- 172.21.15.46:0/2377508486 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fd7bb47340 0x55fd7bb33720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29c85c640 1 -- 172.21.15.46:0/2377508486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd7baabe00 msgr2=0x55fd7bb331e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:38:57.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29c85c640 1 --2- 172.21.15.46:0/2377508486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd7baabe00 0x55fd7bb331e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:38:57.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29c85c640 10 monclient: _finish_hunting 0 2024-09-16T04:38:57.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29c85c640 1 monclient: found mon.noname-c 2024-09-16T04:38:57.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29c85c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:38:57.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29c85c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:38:57.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc29c85c640 1 -- 172.21.15.46:0/2377508486 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd7b904ce0 con 0x55fd7ba96990 2024-09-16T04:38:57.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.597+0000 7fc297fff640 1 --2- 172.21.15.46:0/2377508486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd7baabe00 0x55fd7bb331e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:38:57.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc29c85c640 1 --2- 172.21.15.46:0/2377508486 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x55fd7ba96990 0x55fd7bb32ca0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc2880087e0 tx=0x7fc2880088d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:57.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc294ff9640 1 -- 172.21.15.46:0/2377508486 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc288012790 con 0x55fd7ba96990 2024-09-16T04:38:57.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc294ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:57.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc294ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:38:57.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc294ff9640 10 monclient: dump: 2024-09-16T04:38:57.614 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:38:57.614 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:57.614 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:57.615 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:57.615 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:57.615 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:57.615 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:57.615 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:57.615 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:57.615 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:57.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc294ff9640 10 monclient: _finish_auth 0 2024-09-16T04:38:57.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc294ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:38:57.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc294ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:38:57.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc294ff9640 1 -- 172.21.15.46:0/2377508486 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc288018070 con 0x55fd7ba96990 2024-09-16T04:38:57.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc29dd64d80 5 monclient: authenticate success, global_id 4217 2024-09-16T04:38:57.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc294ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:38:57.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc29dd64d80 10 monclient: _renew_subs 2024-09-16T04:38:57.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc294ff9640 1 -- 172.21.15.46:0/2377508486 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc288012b30 con 0x55fd7ba96990 2024-09-16T04:38:57.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc29dd64d80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:38:57.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc29dd64d80 1 -- 172.21.15.46:0/2377508486 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fd7bb35200 con 0x55fd7ba96990 2024-09-16T04:38:57.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc294ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:38:57.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc294ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:38:57.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc294ff9640 10 monclient: dump: 2024-09-16T04:38:57.616 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:38:57.616 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:38:57.616 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:57.616 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:38:57.616 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:38:57.616 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:38:57.616 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:38:57.616 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:38:57.617 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:38:57.617 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:38:57.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc296ffd640 4 set_mon_vals no callback set 2024-09-16T04:38:57.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc29dd64d80 10 monclient: _renew_subs 2024-09-16T04:38:57.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc29dd64d80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:38:57.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.598+0000 7fc29dd64d80 1 -- 172.21.15.46:0/2377508486 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x55fd7bb363d0 con 0x55fd7ba96990 2024-09-16T04:38:57.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.599+0000 7fc294ff9640 1 -- 172.21.15.46:0/2377508486 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fc288012cd0 con 0x55fd7ba96990 2024-09-16T04:38:57.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.599+0000 7fc294ff9640 1 --2- 172.21.15.46:0/2377508486 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc27c036ca0 0x7fc27c039160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:57.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.599+0000 7fc294ff9640 1 -- 172.21.15.46:0/2377508486 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4335+0+0 (secure 0 0 0) 0x7fc28804a4e0 con 0x55fd7ba96990 2024-09-16T04:38:57.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.599+0000 7fc29dd64d80 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T04:38:57.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.599+0000 7fc29dd64d80 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T04:38:57.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.599+0000 7fc29dd64d80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:38:57.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.599+0000 7fc29dd64d80 1 -- 172.21.15.46:0/2377508486 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x55fd7bb33c60 con 0x55fd7ba96990 2024-09-16T04:38:57.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.599+0000 7fc297fff640 1 --2- 172.21.15.46:0/2377508486 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc27c036ca0 0x7fc27c039160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:57.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.599+0000 7fc297fff640 10 monclient: get_auth_request con 0x7fc27c036ca0 auth_method 0 2024-09-16T04:38:57.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:57.600+0000 7fc297fff640 1 --2- 172.21.15.46:0/2377508486 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc27c036ca0 0x7fc27c039160 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x55fd7bb342a0 tx=0x7fc28c00c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:58.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.518+0000 7fc294ff9640 1 -- 172.21.15.46:0/2377508486 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v17) ==== 141+0+0 (secure 0 0 0) 0x7fc288013dc0 con 0x55fd7ba96990 2024-09-16T04:38:58.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.518+0000 7fc294ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T04:38:58.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.518+0000 7fc294ff9640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-16T04:38:58.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.519+0000 7fc29dd64d80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:38:58.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.519+0000 7fc29dd64d80 1 -- 172.21.15.46:0/2377508486 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc290057070 con 0x55fd7ba96990 2024-09-16T04:38:58.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.519+0000 7fc294ff9640 1 -- 172.21.15.46:0/2377508486 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fc28801c410 con 0x55fd7ba96990 2024-09-16T04:38:58.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.519+0000 7fc294ff9640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-16T04:38:58.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.519+0000 7fc296ffd640 10 monclient: _renew_subs 2024-09-16T04:38:58.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.519+0000 7fc296ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:38:58.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.519+0000 7fc296ffd640 1 -- 172.21.15.46:0/2377508486 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=17}) -- 0x7fc274002ae0 con 0x55fd7ba96990 2024-09-16T04:38:58.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.519+0000 7fc294ff9640 1 -- 172.21.15.46:0/2377508486 <== mon.2 v2:172.21.15.46:3301/0 8 ==== osd_map(17..17 src has 1..17) ==== 658+0+0 (secure 0 0 0) 0x7fc28801b370 con 0x55fd7ba96990 2024-09-16T04:38:58.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.519+0000 7fc29dd64d80 1 --2- 172.21.15.46:0/2377508486 >> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] conn(0x55fd7bb60d40 0x55fd7bb63130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:38:58.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.520+0000 7fc29dd64d80 1 -- 172.21.15.46:0/2377508486 --> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x55fd7bb63670 con 0x55fd7bb60d40 2024-09-16T04:38:58.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.520+0000 7fc29dabe640 1 --2- 172.21.15.46:0/2377508486 >> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] conn(0x55fd7bb60d40 0x55fd7bb63130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:38:58.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.520+0000 7fc29dabe640 10 monclient: get_auth_request con 0x55fd7bb60d40 auth_method 0 2024-09-16T04:38:58.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.522+0000 7fc29dabe640 1 --2- 172.21.15.46:0/2377508486 >> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] conn(0x55fd7bb60d40 0x55fd7bb63130 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:38:58.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.529+0000 7fc29dabe640 1 -- 172.21.15.46:0/2377508486 <== osd.5 v2:172.21.15.114:6818/938868100 1 ==== osd_op_reply(1 rbd_trash [create] v17'1 uv1 ondisk = 0) ==== 153+0+0 (crc 0 0 0) 0x7fc290052700 con 0x55fd7bb60d40 2024-09-16T04:38:58.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.531+0000 7fc29dd64d80 1 -- 172.21.15.46:0/2377508486 --> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x55fd7bbb55c0 con 0x55fd7bb60d40 2024-09-16T04:38:58.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.532+0000 7fc29dabe640 1 -- 172.21.15.46:0/2377508486 <== osd.5 v2:172.21.15.114:6818/938868100 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fc290052700 con 0x55fd7bb60d40 2024-09-16T04:38:58.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.532+0000 7fc29dd64d80 1 -- 172.21.15.46:0/2377508486 --> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x55fd7bbb46a0 con 0x55fd7bb60d40 2024-09-16T04:38:58.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.532+0000 7fc29dabe640 1 -- 172.21.15.46:0/2377508486 <== osd.5 v2:172.21.15.114:6818/938868100 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fc290052700 con 0x55fd7bb60d40 2024-09-16T04:38:58.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.532+0000 7fc2967fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:38:58.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:58.532+0000 7fc2967fc640 1 -- 172.21.15.46:0/2377508486 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7fc278001f00 con 0x55fd7ba96990 2024-09-16T04:38:59.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:59.522+0000 7fc294ff9640 1 -- 172.21.15.46:0/2377508486 <== mon.2 v2:172.21.15.46:3301/0 9 ==== pool_op_reply(tid 4 (0) Success v18) ==== 55+0+0 (secure 0 0 0) 0x7fc28801b570 con 0x55fd7ba96990 2024-09-16T04:38:59.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:59.522+0000 7fc294ff9640 10 monclient: _renew_subs 2024-09-16T04:38:59.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:59.522+0000 7fc294ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:38:59.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:59.522+0000 7fc294ff9640 1 -- 172.21.15.46:0/2377508486 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=18}) -- 0x7fc27c03ce90 con 0x55fd7ba96990 2024-09-16T04:38:59.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:59.523+0000 7fc294ff9640 1 -- 172.21.15.46:0/2377508486 <== mon.2 v2:172.21.15.46:3301/0 10 ==== osd_map(18..18 src has 1..18) ==== 626+0+0 (secure 0 0 0) 0x7fc28802b060 con 0x55fd7ba96990 2024-09-16T04:38:59.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:59.523+0000 7fc296ffd640 1 -- 172.21.15.46:0/2377508486 --> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) -- 0x7fc2740056a0 con 0x55fd7bb60d40 2024-09-16T04:38:59.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:59.527+0000 7fc29dabe640 1 -- 172.21.15.46:0/2377508486 <== osd.5 v2:172.21.15.114:6818/938868100 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v18'1 uv1 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7fc290052700 con 0x55fd7bb60d40 2024-09-16T04:38:59.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:59.527+0000 7fc2967fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:38:59.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:38:59.527+0000 7fc2967fc640 1 -- 172.21.15.46:0/2377508486 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v18) -- 0x7fc278003350 con 0x55fd7ba96990 2024-09-16T04:39:00.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.524+0000 7fc294ff9640 1 -- 172.21.15.46:0/2377508486 <== mon.2 v2:172.21.15.46:3301/0 11 ==== pool_op_reply(tid 6 (0) Success v19) ==== 43+0+0 (secure 0 0 0) 0x7fc28801ad40 con 0x55fd7ba96990 2024-09-16T04:39:00.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.524+0000 7fc294ff9640 10 monclient: _renew_subs 2024-09-16T04:39:00.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.524+0000 7fc294ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:00.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.524+0000 7fc294ff9640 1 -- 172.21.15.46:0/2377508486 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=19}) -- 0x7fc27c03ce90 con 0x55fd7ba96990 2024-09-16T04:39:00.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.524+0000 7fc294ff9640 1 -- 172.21.15.46:0/2377508486 <== mon.2 v2:172.21.15.46:3301/0 12 ==== osd_map(19..19 src has 1..19) ==== 654+0+0 (secure 0 0 0) 0x7fc288034060 con 0x55fd7ba96990 2024-09-16T04:39:00.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.524+0000 7fc296ffd640 1 -- 172.21.15.46:0/2377508486 --> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) -- 0x7fc274005a40 con 0x55fd7bb60d40 2024-09-16T04:39:00.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.530+0000 7fc29dabe640 1 -- 172.21.15.46:0/2377508486 <== osd.5 v2:172.21.15.114:6818/938868100 5 ==== osd_op_reply(7 rbd_info [write 0~19] v19'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc290052700 con 0x55fd7bb60d40 2024-09-16T04:39:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.531+0000 7fc29dd64d80 1 -- 172.21.15.46:0/2377508486 >> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] conn(0x55fd7bb60d40 msgr2=0x55fd7bb63130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.531+0000 7fc29dd64d80 1 --2- 172.21.15.46:0/2377508486 >> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] conn(0x55fd7bb60d40 0x55fd7bb63130 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.531+0000 7fc29dd64d80 1 -- 172.21.15.46:0/2377508486 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc27c036ca0 msgr2=0x7fc27c039160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.531+0000 7fc29dd64d80 1 --2- 172.21.15.46:0/2377508486 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc27c036ca0 0x7fc27c039160 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x55fd7bb342a0 tx=0x7fc28c00c040 comp rx=0 tx=0).stop 2024-09-16T04:39:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.531+0000 7fc29dd64d80 10 monclient: shutdown 2024-09-16T04:39:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.531+0000 7fc29dd64d80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.531+0000 7fc29dd64d80 1 -- 172.21.15.46:0/2377508486 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x55fd7ba96990 msgr2=0x55fd7bb32ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.531+0000 7fc29dd64d80 1 --2- 172.21.15.46:0/2377508486 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x55fd7ba96990 0x55fd7bb32ca0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc2880087e0 tx=0x7fc2880088d0 comp rx=0 tx=0).stop 2024-09-16T04:39:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.531+0000 7fc29dabe640 1 -- 172.21.15.46:0/2377508486 reap_dead start 2024-09-16T04:39:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.531+0000 7fc29dd64d80 1 -- 172.21.15.46:0/2377508486 shutdown_connections 2024-09-16T04:39:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.531+0000 7fc29dd64d80 1 -- 172.21.15.46:0/2377508486 >> 172.21.15.46:0/2377508486 conn(0x55fd7ba97f20 msgr2=0x55fd7bb4a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.531+0000 7fc29dd64d80 1 -- 172.21.15.46:0/2377508486 shutdown_connections 2024-09-16T04:39:00.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.531+0000 7fc29dd64d80 1 -- 172.21.15.46:0/2377508486 wait complete. 2024-09-16T04:39:00.537 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T04:39:00.538 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T04:39:00.538 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T04:39:00.538 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T04:39:00.538 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T04:39:00.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.650+0000 7f66f161e640 10 monclient: get_monmap_and_config 2024-09-16T04:39:00.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.650+0000 7f66f161e640 10 monclient: build_initial_monmap 2024-09-16T04:39:00.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.650+0000 7f66f161e640 10 monclient: monmap: 2024-09-16T04:39:00.651 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:00.651 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:00.652 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:00.650662+0000 2024-09-16T04:39:00.652 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:00.650662+0000 2024-09-16T04:39:00.652 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:00.652 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:00.652 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:00.652 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:00.652 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:00.652 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:00.652 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:00.652 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:00.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.650+0000 7f66f161e640 1 Processor -- start 2024-09-16T04:39:00.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.650+0000 7f66f161e640 1 -- start start 2024-09-16T04:39:00.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.650+0000 7f66f161e640 10 monclient: init 2024-09-16T04:39:00.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66f161e640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:00.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66f161e640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-16T04:39:00.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66f161e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f66ec0a3c10 0x7f66ec09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:00.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66f161e640 10 monclient(hunting): picked mon.noname-c con 0x7f66ec0a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:00.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66f161e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f66ec0a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:00.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66f161e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f66ec13be90 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:00.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66f161e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66ec09ab70 con 0x7f66ec13be90 2024-09-16T04:39:00.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66f161e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66ec0a4d60 con 0x7f66ec0a3ff0 2024-09-16T04:39:00.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66f161e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:00.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66f161e640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f66ec09b030 con 0x7f66ec0a3c10 2024-09-16T04:39:00.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66f161e640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:00.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66f161e640 10 monclient(hunting): authenticate will time out at 739.066400s 2024-09-16T04:39:00.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66eb7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f66ec13be90 0x7f66ec13e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52988/0 (socket says 172.21.15.46:52988) 2024-09-16T04:39:00.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66eb7fe640 1 -- 172.21.15.46:0/2065102147 learned_addr learned my addr 172.21.15.46:0/2065102147 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:00.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66e9ffb640 1 -- 172.21.15.46:0/2065102147 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 543753108 0 0) 0x7f66ec09ab70 con 0x7f66ec13be90 2024-09-16T04:39:00.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66e9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:39:00.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66e9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:00.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:00.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66eaffd640 1 --2- 172.21.15.46:0/2065102147 >> v2:172.21.15.114:3300/0 conn(0x7f66ec0a3c10 0x7f66ec09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:00.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66e9ffb640 10 monclient(hunting): my global_id is 4239 2024-09-16T04:39:00.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66e9ffb640 1 -- 172.21.15.46:0/2065102147 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66d4003520 con 0x7f66ec13be90 2024-09-16T04:39:00.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66eaffd640 10 monclient(hunting): get_auth_request con 0x7f66ec0a3c10 auth_method 0 2024-09-16T04:39:00.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:00.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:00.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:00.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:00.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.651+0000 7f66eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:00.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:00.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 1 -- 172.21.15.46:0/2065102147 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1307257397 0 0) 0x7f66d4003520 con 0x7f66ec13be90 2024-09-16T04:39:00.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 1 -- 172.21.15.46:0/2065102147 >> v2:172.21.15.114:3300/0 conn(0x7f66ec0a3c10 msgr2=0x7f66ec09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:00.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 1 --2- 172.21.15.46:0/2065102147 >> v2:172.21.15.114:3300/0 conn(0x7f66ec0a3c10 0x7f66ec09bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:00.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 1 -- 172.21.15.46:0/2065102147 >> v1:172.21.15.114:6789/0 conn(0x7f66ec0a3ff0 legacy=0x7f66ec0a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:00.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:39:00.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:39:00.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:00.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:39:00.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 1 -- 172.21.15.46:0/2065102147 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ec09b310 con 0x7f66ec13be90 2024-09-16T04:39:00.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 10 monclient: _finish_auth 0 2024-09-16T04:39:00.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:39:00.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:00.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 1 -- 172.21.15.46:0/2065102147 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f66e40019f0 con 0x7f66ec13be90 2024-09-16T04:39:00.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66f161e640 5 monclient: authenticate success, global_id 4239 2024-09-16T04:39:00.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66f161e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:00.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:00.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:39:00.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 10 monclient: dump: 2024-09-16T04:39:00.666 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:00.666 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:00.666 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:00.666 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:00.666 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:00.667 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:00.667 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:00.667 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:00.667 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:00.667 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:00.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:39:00.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:00.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 1 -- 172.21.15.46:0/2065102147 >> v1:172.21.15.46:6789/0 conn(0x7f66ec13be90 legacy=0x7f66ec13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:00.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:39:00.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 1 --2- 172.21.15.46:0/2065102147 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f66d40039b0 0x7f66d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:00.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 10 monclient(hunting): picked mon.c con 0x7f66d40039b0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:00.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 1 --2- 172.21.15.46:0/2065102147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66d4004800 0x7f66d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:00.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 10 monclient(hunting): picked mon.a con 0x7f66d4004800 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:00.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 1 --2- 172.21.15.46:0/2065102147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66d40052e0 0x7f66d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:00.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 10 monclient(hunting): picked mon.b con 0x7f66d40052e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:00.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:00.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 1 -- 172.21.15.46:0/2065102147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f66ec09b030 con 0x7f66d4004800 2024-09-16T04:39:00.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:00.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 1 -- 172.21.15.46:0/2065102147 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f66d400c1b0 con 0x7f66d40052e0 2024-09-16T04:39:00.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:00.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 1 -- 172.21.15.46:0/2065102147 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f66d400c3b0 con 0x7f66d40039b0 2024-09-16T04:39:00.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:00.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66e9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:39:00.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66f161e640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:00.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66ea7fc640 1 --2- 172.21.15.46:0/2065102147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66d4004800 0x7f66d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:00.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66eaffd640 1 --2- 172.21.15.46:0/2065102147 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f66d40039b0 0x7f66d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:00.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66ea7fc640 10 monclient(hunting): get_auth_request con 0x7f66d4004800 auth_method 0 2024-09-16T04:39:00.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66ea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:00.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:00.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66ea7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:00.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66eaffd640 10 monclient(hunting): get_auth_request con 0x7f66d40039b0 auth_method 0 2024-09-16T04:39:00.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:00.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:00.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66eaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:00.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66eb7fe640 1 --2- 172.21.15.46:0/2065102147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66d40052e0 0x7f66d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:00.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66ea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:00.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66ea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:00.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66ea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:00.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66eb7fe640 10 monclient(hunting): get_auth_request con 0x7f66d40052e0 auth_method 0 2024-09-16T04:39:00.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:00.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:00.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66eb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:00.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:00.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:00.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:00.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:00.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:00.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:00.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66ea7fc640 10 monclient(hunting): handle_auth_done global_id 4239 payload 293 2024-09-16T04:39:00.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66ea7fc640 1 -- 172.21.15.46:0/2065102147 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f66d40039b0 msgr2=0x7f66d4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:00.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66ea7fc640 1 --2- 172.21.15.46:0/2065102147 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f66d40039b0 0x7f66d4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:00.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66ea7fc640 1 -- 172.21.15.46:0/2065102147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66d40052e0 msgr2=0x7f66d400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:00.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66ea7fc640 1 --2- 172.21.15.46:0/2065102147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66d40052e0 0x7f66d400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:00.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66ea7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:39:00.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66ea7fc640 1 monclient: found mon.a 2024-09-16T04:39:00.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:00.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66ea7fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:00.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66ea7fc640 1 -- 172.21.15.46:0/2065102147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66d4003520 con 0x7f66d4004800 2024-09-16T04:39:00.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66eb7fe640 1 -- 172.21.15.46:0/2065102147 reap_dead start 2024-09-16T04:39:00.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66ea7fc640 10 monclient: _finish_auth 0 2024-09-16T04:39:00.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66ea7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:39:00.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.652+0000 7f66ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:00.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.653+0000 7f66ea7fc640 1 --2- 172.21.15.46:0/2065102147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66d4004800 0x7f66d4004c80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f66e0001720 tx=0x7f66d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ab6dbb50b5671a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:00.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.653+0000 7f66e9ffb640 1 -- 172.21.15.46:0/2065102147 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e0001370 con 0x7f66d4004800 2024-09-16T04:39:00.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.653+0000 7f66e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:00.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.653+0000 7f66e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:00.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.653+0000 7f66e9ffb640 10 monclient: dump: 2024-09-16T04:39:00.671 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:00.672 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:00.672 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:00.672 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:00.672 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:00.672 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:00.672 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:00.672 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:00.672 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:00.672 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:00.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.653+0000 7f66f161e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:00.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.653+0000 7f66e9ffb640 1 -- 172.21.15.46:0/2065102147 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66e00052d0 con 0x7f66d4004800 2024-09-16T04:39:00.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.653+0000 7f66e9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:00.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.653+0000 7f66f161e640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:00.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.653+0000 7f66e9ffb640 1 -- 172.21.15.46:0/2065102147 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e002ab90 con 0x7f66d4004800 2024-09-16T04:39:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.653+0000 7f66f161e640 4 set_mon_vals no callback set 2024-09-16T04:39:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.653+0000 7f66e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.653+0000 7f66e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.653+0000 7f66e9ffb640 10 monclient: dump: 2024-09-16T04:39:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:00.673 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:00.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.654+0000 7f66f161e640 10 monclient: shutdown 2024-09-16T04:39:00.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.654+0000 7f66f161e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:00.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.654+0000 7f66f161e640 1 -- 172.21.15.46:0/2065102147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66d4004800 msgr2=0x7f66d4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:00.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.654+0000 7f66f161e640 1 --2- 172.21.15.46:0/2065102147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66d4004800 0x7f66d4004c80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f66e0001720 tx=0x7f66d4005830 comp rx=0 tx=0).stop 2024-09-16T04:39:00.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.654+0000 7f66f161e640 1 -- 172.21.15.46:0/2065102147 shutdown_connections 2024-09-16T04:39:00.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.654+0000 7f66f161e640 1 --2- 172.21.15.46:0/2065102147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66d4004800 0x7f66d4004c80 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:00.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.654+0000 7f66f161e640 1 -- 172.21.15.46:0/2065102147 >> 172.21.15.46:0/2065102147 conn(0x7f66ec0a5d10 msgr2=0x7f66ec0a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:00.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.654+0000 7f66f161e640 1 -- 172.21.15.46:0/2065102147 shutdown_connections 2024-09-16T04:39:00.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.654+0000 7f66f161e640 1 -- 172.21.15.46:0/2065102147 wait complete. 2024-09-16T04:39:00.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.654+0000 7f66f161e640 10 monclient: build_initial_monmap 2024-09-16T04:39:00.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.654+0000 7f66f161e640 10 monclient: monmap: 2024-09-16T04:39:00.674 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:00.674 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:00.674 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:00.675 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.655+0000 7f66f161e640 1 Processor -- start 2024-09-16T04:39:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.655+0000 7f66f161e640 1 -- start start 2024-09-16T04:39:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.655+0000 7f66f161e640 10 monclient: init 2024-09-16T04:39:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.655+0000 7f66f161e640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.655+0000 7f66f161e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:39:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.655+0000 7f66f161e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f66d4004800 0x7f66ec137470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.655+0000 7f66f161e640 10 monclient(hunting): picked mon.noname-c con 0x7f66d4004800 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:00.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.655+0000 7f66f161e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66ec11fcd0 0x7f66ec120110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.655+0000 7f66f161e640 10 monclient(hunting): picked mon.noname-a con 0x7f66ec11fcd0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.655+0000 7f66f161e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66ec120650 0x7f66ec1d2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66f161e640 10 monclient(hunting): picked mon.noname-b con 0x7f66ec120650 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66f161e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66f161e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f66ec0a4580 con 0x7f66ec11fcd0 2024-09-16T04:39:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66f161e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66f161e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f66ec0a4400 con 0x7f66ec120650 2024-09-16T04:39:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66f161e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66f161e640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f66ec0a4700 con 0x7f66d4004800 2024-09-16T04:39:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66f161e640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66f161e640 10 monclient(hunting): authenticate will time out at 739.071056s 2024-09-16T04:39:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eaffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f66d4004800 0x7f66ec137470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eaffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f66d4004800 0x7f66ec137470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:60878/0 (socket says 172.21.15.46:60878) 2024-09-16T04:39:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eaffd640 1 -- 172.21.15.46:0/3950555644 learned_addr learned my addr 172.21.15.46:0/3950555644 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:00.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eaffd640 10 monclient(hunting): get_auth_request con 0x7f66d4004800 auth_method 0 2024-09-16T04:39:00.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66ea7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66ec11fcd0 0x7f66ec120110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:00.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:00.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:00.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:00.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66ea7fc640 10 monclient(hunting): get_auth_request con 0x7f66ec11fcd0 auth_method 0 2024-09-16T04:39:00.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66ea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:00.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:00.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:00.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eb7fe640 1 --2- 172.21.15.46:0/3950555644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66ec120650 0x7f66ec1d2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:00.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eb7fe640 10 monclient(hunting): get_auth_request con 0x7f66ec120650 auth_method 0 2024-09-16T04:39:00.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:00.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:00.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:00.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:00.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:00.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:00.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66ea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:00.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66ea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:00.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66ea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:00.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:00.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:00.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:00.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eaffd640 10 monclient(hunting): handle_auth_done global_id 4220 payload 824 2024-09-16T04:39:00.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eaffd640 1 -- 172.21.15.46:0/3950555644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66ec120650 msgr2=0x7f66ec1d2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:00.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eaffd640 1 --2- 172.21.15.46:0/3950555644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66ec120650 0x7f66ec1d2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:00.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eaffd640 1 -- 172.21.15.46:0/3950555644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66ec11fcd0 msgr2=0x7f66ec120110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:00.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eaffd640 1 --2- 172.21.15.46:0/3950555644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66ec11fcd0 0x7f66ec120110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:00.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eaffd640 10 monclient: _finish_hunting 0 2024-09-16T04:39:00.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eaffd640 1 monclient: found mon.noname-c 2024-09-16T04:39:00.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:00.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:39:00.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eaffd640 1 -- 172.21.15.46:0/3950555644 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ec0a0d70 con 0x7f66d4004800 2024-09-16T04:39:00.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66eaffd640 1 --2- 172.21.15.46:0/3950555644 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f66d4004800 0x7f66ec137470 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f66dc011960 tx=0x7f66dc011e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:00.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66d37fe640 1 -- 172.21.15.46:0/3950555644 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66dc00fa90 con 0x7f66d4004800 2024-09-16T04:39:00.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:00.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66d37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:39:00.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66d37fe640 10 monclient: dump: 2024-09-16T04:39:00.679 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:00.679 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:00.679 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:00.680 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:00.680 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:00.680 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:00.680 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:00.680 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:00.680 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:00.680 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:00.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66d37fe640 10 monclient: _finish_auth 0 2024-09-16T04:39:00.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66d37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:39:00.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:00.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66d37fe640 1 -- 172.21.15.46:0/3950555644 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66dc00fc30 con 0x7f66d4004800 2024-09-16T04:39:00.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66d37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:00.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66d37fe640 1 -- 172.21.15.46:0/3950555644 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66dc014440 con 0x7f66d4004800 2024-09-16T04:39:00.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66f161e640 5 monclient: authenticate success, global_id 4220 2024-09-16T04:39:00.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66f161e640 10 monclient: _renew_subs 2024-09-16T04:39:00.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66f161e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:00.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66f161e640 1 -- 172.21.15.46:0/3950555644 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66ec1235a0 con 0x7f66d4004800 2024-09-16T04:39:00.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:00.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:00.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66d37fe640 10 monclient: dump: 2024-09-16T04:39:00.681 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:00.681 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:00.681 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:00.681 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:00.681 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:00.681 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:00.681 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:00.682 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:00.682 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:00.682 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:00.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66e97fa640 4 set_mon_vals no callback set 2024-09-16T04:39:00.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66f161e640 10 monclient: _renew_subs 2024-09-16T04:39:00.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66f161e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:00.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.656+0000 7f66f161e640 1 -- 172.21.15.46:0/3950555644 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f66ec123b60 con 0x7f66d4004800 2024-09-16T04:39:00.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.657+0000 7f66d37fe640 1 -- 172.21.15.46:0/3950555644 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f66dc012040 con 0x7f66d4004800 2024-09-16T04:39:00.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.658+0000 7f66f161e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:00.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.658+0000 7f66d37fe640 1 --2- 172.21.15.46:0/3950555644 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f66d80383b0 0x7f66d803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:00.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.658+0000 7f66f161e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:00.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.658+0000 7f66f161e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:00.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.658+0000 7f66f161e640 1 -- 172.21.15.46:0/3950555644 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66ec09bcf0 con 0x7f66d4004800 2024-09-16T04:39:00.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.658+0000 7f66d37fe640 1 -- 172.21.15.46:0/3950555644 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f66dc04bbe0 con 0x7f66d4004800 2024-09-16T04:39:00.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.658+0000 7f66ea7fc640 1 --2- 172.21.15.46:0/3950555644 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f66d80383b0 0x7f66d803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:00.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.658+0000 7f66ea7fc640 10 monclient: get_auth_request con 0x7f66d80383b0 auth_method 0 2024-09-16T04:39:00.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.658+0000 7f66ea7fc640 1 --2- 172.21.15.46:0/3950555644 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f66d80383b0 0x7f66d803a870 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f66e0032000 tx=0x7f66e002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:00.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.663+0000 7f66d37fe640 1 -- 172.21.15.46:0/3950555644 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f66ec0a0d70 con 0x7f66d4004800 2024-09-16T04:39:00.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.663+0000 7f66d37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:00.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.663+0000 7f66d37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:00.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.886+0000 7f66f161e640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-16T04:39:00.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.886+0000 7f66f161e640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T04:39:00.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.886+0000 7f66f161e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:00.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.886+0000 7f66f161e640 1 -- 172.21.15.46:0/3950555644 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f66ec069150 con 0x7f66d4004800 2024-09-16T04:39:00.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.888+0000 7f66d37fe640 1 -- 172.21.15.46:0/3950555644 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+87489 (secure 0 0 0) 0x7f66dc019030 con 0x7f66d4004800 2024-09-16T04:39:00.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.888+0000 7f66d37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T04:39:00.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.888+0000 7f66d37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:00.891 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:39:00.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.890+0000 7f66f161e640 1 -- 172.21.15.46:0/3950555644 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f66d80383b0 msgr2=0x7f66d803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:00.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.890+0000 7f66f161e640 1 --2- 172.21.15.46:0/3950555644 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f66d80383b0 0x7f66d803a870 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f66e0032000 tx=0x7f66e002e000 comp rx=0 tx=0).stop 2024-09-16T04:39:00.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.890+0000 7f66f161e640 10 monclient: shutdown 2024-09-16T04:39:00.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.890+0000 7f66f161e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:00.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.890+0000 7f66f161e640 1 -- 172.21.15.46:0/3950555644 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f66d4004800 msgr2=0x7f66ec137470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:00.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.890+0000 7f66f161e640 1 --2- 172.21.15.46:0/3950555644 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f66d4004800 0x7f66ec137470 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f66dc011960 tx=0x7f66dc011e20 comp rx=0 tx=0).stop 2024-09-16T04:39:00.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.890+0000 7f66f161e640 1 -- 172.21.15.46:0/3950555644 shutdown_connections 2024-09-16T04:39:00.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.890+0000 7f66f161e640 1 --2- 172.21.15.46:0/3950555644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66ec120650 0x7f66ec1d2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:00.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.890+0000 7f66f161e640 1 --2- 172.21.15.46:0/3950555644 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f66d80383b0 0x7f66d803a870 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:00.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.890+0000 7f66f161e640 1 --2- 172.21.15.46:0/3950555644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66ec11fcd0 0x7f66ec120110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:00.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.890+0000 7f66f161e640 1 --2- 172.21.15.46:0/3950555644 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f66d4004800 0x7f66ec137470 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:00.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.890+0000 7f66f161e640 1 -- 172.21.15.46:0/3950555644 >> 172.21.15.46:0/3950555644 conn(0x7f66ec0a5d10 msgr2=0x7f66ec0a2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:00.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.890+0000 7f66f161e640 1 -- 172.21.15.46:0/3950555644 shutdown_connections 2024-09-16T04:39:00.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:00.890+0000 7f66f161e640 1 -- 172.21.15.46:0/3950555644 wait complete. 2024-09-16T04:39:00.902 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":5,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3521677055},{"type":"v1","addr":"172.21.15.114:6801","nonce":3521677055}]},"active_addr":"172.21.15.114:6801/3521677055","active_change":"2024-09-16T04:38:43.341616+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":724185640}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":3900319779}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":4176446344}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":3697581891}]}]} 2024-09-16T04:39:00.904 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T04:39:00.904 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T04:39:00.904 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:39:01.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.004+0000 7ff86af31640 10 monclient: get_monmap_and_config 2024-09-16T04:39:01.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.004+0000 7ff86af31640 10 monclient: build_initial_monmap 2024-09-16T04:39:01.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.004+0000 7ff86af31640 10 monclient: monmap: 2024-09-16T04:39:01.005 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:01.005 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.005 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:01.004611+0000 2024-09-16T04:39:01.005 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:01.004611+0000 2024-09-16T04:39:01.006 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:01.006 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.006 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:01.006 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:01.006 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:01.006 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:01.006 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:01.006 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.004+0000 7ff86af31640 1 Processor -- start 2024-09-16T04:39:01.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.004+0000 7ff86af31640 1 -- start start 2024-09-16T04:39:01.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.004+0000 7ff86af31640 10 monclient: init 2024-09-16T04:39:01.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff86af31640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:01.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff86af31640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-16T04:39:01.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff86af31640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff8640a3c10 0x7ff86409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff86af31640 10 monclient(hunting): picked mon.noname-b con 0x7ff8640a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:01.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff86af31640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff8640a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:01.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff86af31640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff86413be90 0x7ff86413e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff86af31640 10 monclient(hunting): picked mon.noname-c con 0x7ff86413be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff86af31640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff86409ab70 con 0x7ff8640a3ff0 2024-09-16T04:39:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff86af31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff86af31640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ff8640a4d60 con 0x7ff86413be90 2024-09-16T04:39:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff86af31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff86af31640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7ff86409b030 con 0x7ff8640a3c10 2024-09-16T04:39:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff86af31640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff86af31640 10 monclient(hunting): authenticate will time out at 739.420434s 2024-09-16T04:39:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff868ccd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff8640a3c10 0x7ff86409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff868ccd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff8640a3c10 0x7ff86409bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:60886/0 (socket says 172.21.15.46:60886) 2024-09-16T04:39:01.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff868ccd640 1 -- 172.21.15.46:0/1708129869 learned_addr learned my addr 172.21.15.46:0/1708129869 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:01.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff868ccd640 10 monclient(hunting): get_auth_request con 0x7ff8640a3c10 auth_method 0 2024-09-16T04:39:01.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff868ccd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff868ccd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff868ccd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff8694ce640 1 --2- 172.21.15.46:0/1708129869 >> v2:172.21.15.114:3300/0 conn(0x7ff86413be90 0x7ff86413e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff8694ce640 10 monclient(hunting): get_auth_request con 0x7ff86413be90 auth_method 0 2024-09-16T04:39:01.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff8694ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff8694ce640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff8694ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff868ccd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff868ccd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff868ccd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff8694ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff8694ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff8694ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff868ccd640 10 monclient(hunting): handle_auth_done global_id 4223 payload 274 2024-09-16T04:39:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff868ccd640 1 -- 172.21.15.46:0/1708129869 >> v2:172.21.15.114:3300/0 conn(0x7ff86413be90 msgr2=0x7ff86413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff868ccd640 1 --2- 172.21.15.46:0/1708129869 >> v2:172.21.15.114:3300/0 conn(0x7ff86413be90 0x7ff86413e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff868ccd640 1 -- 172.21.15.46:0/1708129869 >> v1:172.21.15.114:6789/0 conn(0x7ff8640a3ff0 legacy=0x7ff8640a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff868ccd640 10 monclient: _finish_hunting 0 2024-09-16T04:39:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff868ccd640 1 monclient: found mon.noname-b 2024-09-16T04:39:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff868ccd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff868ccd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:39:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.005+0000 7ff868ccd640 1 -- 172.21.15.46:0/1708129869 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff86409b310 con 0x7ff8640a3c10 2024-09-16T04:39:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.006+0000 7ff868ccd640 1 --2- 172.21.15.46:0/1708129869 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff8640a3c10 0x7ff86409bcf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff858009870 tx=0x7ff85802a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6978c4f049b171ce server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.006+0000 7ff8637fe640 1 -- 172.21.15.46:0/1708129869 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff858037070 con 0x7ff8640a3c10 2024-09-16T04:39:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.006+0000 7ff8637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.006+0000 7ff8637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.006+0000 7ff8637fe640 10 monclient: dump: 2024-09-16T04:39:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.019 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.006+0000 7ff8637fe640 10 monclient: _finish_auth 0 2024-09-16T04:39:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.006+0000 7ff8637fe640 10 monclient: _check_auth_tickets 2024-09-16T04:39:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.006+0000 7ff8637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.006+0000 7ff8637fe640 1 -- 172.21.15.46:0/1708129869 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff858032440 con 0x7ff8640a3c10 2024-09-16T04:39:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.006+0000 7ff86af31640 5 monclient: authenticate success, global_id 4223 2024-09-16T04:39:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.006+0000 7ff8637fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.006+0000 7ff86af31640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.006+0000 7ff8637fe640 1 -- 172.21.15.46:0/1708129869 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff858032810 con 0x7ff8640a3c10 2024-09-16T04:39:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.006+0000 7ff86af31640 4 set_mon_vals no callback set 2024-09-16T04:39:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.006+0000 7ff8637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.006+0000 7ff8637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.006+0000 7ff8637fe640 10 monclient: dump: 2024-09-16T04:39:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.020 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.007+0000 7ff86af31640 10 monclient: shutdown 2024-09-16T04:39:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.007+0000 7ff86af31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.007+0000 7ff86af31640 1 -- 172.21.15.46:0/1708129869 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff8640a3c10 msgr2=0x7ff86409bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.007+0000 7ff86af31640 1 --2- 172.21.15.46:0/1708129869 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff8640a3c10 0x7ff86409bcf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff858009870 tx=0x7ff85802a130 comp rx=0 tx=0).stop 2024-09-16T04:39:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.007+0000 7ff86af31640 1 -- 172.21.15.46:0/1708129869 shutdown_connections 2024-09-16T04:39:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.007+0000 7ff86af31640 1 --2- 172.21.15.46:0/1708129869 >> v2:172.21.15.114:3300/0 conn(0x7ff86413be90 0x7ff86413e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.007+0000 7ff86af31640 1 --2- 172.21.15.46:0/1708129869 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff8640a3c10 0x7ff86409bcf0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.007+0000 7ff86af31640 1 -- 172.21.15.46:0/1708129869 >> 172.21.15.46:0/1708129869 conn(0x7ff8640a5d10 msgr2=0x7ff8640a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.007+0000 7ff86af31640 1 -- 172.21.15.46:0/1708129869 shutdown_connections 2024-09-16T04:39:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.007+0000 7ff86af31640 1 -- 172.21.15.46:0/1708129869 wait complete. 2024-09-16T04:39:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.007+0000 7ff86af31640 10 monclient: build_initial_monmap 2024-09-16T04:39:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.007+0000 7ff86af31640 10 monclient: monmap: 2024-09-16T04:39:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:01.022 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.008+0000 7ff86af31640 1 Processor -- start 2024-09-16T04:39:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.008+0000 7ff86af31640 1 -- start start 2024-09-16T04:39:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.008+0000 7ff86af31640 10 monclient: init 2024-09-16T04:39:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.008+0000 7ff86af31640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.008+0000 7ff86af31640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:39:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.008+0000 7ff86af31640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8640a3c10 0x7ff864133cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.008+0000 7ff86af31640 10 monclient(hunting): picked mon.noname-a con 0x7ff8640a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.008+0000 7ff86af31640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff8640a3ff0 0x7ff864134230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.008+0000 7ff86af31640 10 monclient(hunting): picked mon.noname-c con 0x7ff8640a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:01.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.008+0000 7ff86af31640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff86413be90 0x7ff8641cba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.008+0000 7ff86af31640 10 monclient(hunting): picked mon.noname-b con 0x7ff86413be90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:01.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.008+0000 7ff86af31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.008+0000 7ff86af31640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff86406e550 con 0x7ff8640a3c10 2024-09-16T04:39:01.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.008+0000 7ff86af31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.008+0000 7ff86af31640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff86406e3d0 con 0x7ff86413be90 2024-09-16T04:39:01.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.008+0000 7ff86af31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.008+0000 7ff86af31640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7ff86413e7c0 con 0x7ff8640a3ff0 2024-09-16T04:39:01.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.008+0000 7ff86af31640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:01.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.008+0000 7ff86af31640 10 monclient(hunting): authenticate will time out at 739.423924s 2024-09-16T04:39:01.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff868ccd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8640a3c10 0x7ff864133cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff863fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff8640a3ff0 0x7ff864134230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff868ccd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8640a3c10 0x7ff864133cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59184/0 (socket says 172.21.15.46:59184) 2024-09-16T04:39:01.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff863fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff8640a3ff0 0x7ff864134230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:60902/0 (socket says 172.21.15.46:60902) 2024-09-16T04:39:01.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff868ccd640 1 -- 172.21.15.46:0/3709708527 learned_addr learned my addr 172.21.15.46:0/3709708527 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:01.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff868ccd640 10 monclient(hunting): get_auth_request con 0x7ff8640a3c10 auth_method 0 2024-09-16T04:39:01.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff868ccd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff868ccd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff868ccd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff863fff640 10 monclient(hunting): get_auth_request con 0x7ff8640a3ff0 auth_method 0 2024-09-16T04:39:01.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff863fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff863fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff863fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff8694ce640 1 --2- 172.21.15.46:0/3709708527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff86413be90 0x7ff8641cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff8694ce640 10 monclient(hunting): get_auth_request con 0x7ff86413be90 auth_method 0 2024-09-16T04:39:01.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff8694ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff8694ce640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff8694ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff868ccd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff868ccd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff868ccd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff863fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff863fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff863fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff8694ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff8694ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff8694ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff868ccd640 10 monclient(hunting): handle_auth_done global_id 4245 payload 824 2024-09-16T04:39:01.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff868ccd640 1 -- 172.21.15.46:0/3709708527 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff8640a3ff0 msgr2=0x7ff864134230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff868ccd640 1 --2- 172.21.15.46:0/3709708527 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff8640a3ff0 0x7ff864134230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff868ccd640 1 -- 172.21.15.46:0/3709708527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff86413be90 msgr2=0x7ff8641cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff868ccd640 1 --2- 172.21.15.46:0/3709708527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff86413be90 0x7ff8641cba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff868ccd640 10 monclient: _finish_hunting 0 2024-09-16T04:39:01.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff868ccd640 1 monclient: found mon.noname-a 2024-09-16T04:39:01.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff868ccd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:01.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff868ccd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:01.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff868ccd640 1 -- 172.21.15.46:0/3709708527 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff86413c9a0 con 0x7ff8640a3c10 2024-09-16T04:39:01.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff868ccd640 1 --2- 172.21.15.46:0/3709708527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8640a3c10 0x7ff864133cf0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff858000c00 tx=0x7ff8580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff8617fa640 1 -- 172.21.15.46:0/3709708527 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff858041020 con 0x7ff8640a3c10 2024-09-16T04:39:01.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff8617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff8617fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:01.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff8617fa640 10 monclient: dump: 2024-09-16T04:39:01.026 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.027 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.027 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.027 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.027 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.027 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.027 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.027 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.027 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.027 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff8617fa640 10 monclient: _finish_auth 0 2024-09-16T04:39:01.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff8617fa640 10 monclient: _check_auth_tickets 2024-09-16T04:39:01.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff8617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:01.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff8617fa640 1 -- 172.21.15.46:0/3709708527 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff858030640 con 0x7ff8640a3c10 2024-09-16T04:39:01.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff86af31640 5 monclient: authenticate success, global_id 4245 2024-09-16T04:39:01.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff86af31640 10 monclient: _renew_subs 2024-09-16T04:39:01.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff86af31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:01.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff86af31640 1 -- 172.21.15.46:0/3709708527 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff864138820 con 0x7ff8640a3c10 2024-09-16T04:39:01.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff8617fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:01.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff8617fa640 1 -- 172.21.15.46:0/3709708527 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff858037040 con 0x7ff8640a3c10 2024-09-16T04:39:01.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff8617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff8617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:01.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff8617fa640 10 monclient: dump: 2024-09-16T04:39:01.028 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.028 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.029 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.029 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.029 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.029 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.029 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.029 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.029 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.029 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff86af31640 10 monclient: _renew_subs 2024-09-16T04:39:01.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff862ffd640 4 set_mon_vals no callback set 2024-09-16T04:39:01.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff86af31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:01.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.009+0000 7ff86af31640 1 -- 172.21.15.46:0/3709708527 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff86413c4e0 con 0x7ff8640a3c10 2024-09-16T04:39:01.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.010+0000 7ff8617fa640 1 -- 172.21.15.46:0/3709708527 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7ff858044020 con 0x7ff8640a3c10 2024-09-16T04:39:01.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.010+0000 7ff86af31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:01.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.010+0000 7ff86af31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:01.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.010+0000 7ff86af31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:01.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.010+0000 7ff86af31640 1 -- 172.21.15.46:0/3709708527 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff86409bcf0 con 0x7ff8640a3c10 2024-09-16T04:39:01.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.010+0000 7ff8617fa640 1 --2- 172.21.15.46:0/3709708527 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff8480383b0 0x7ff84803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.010+0000 7ff8617fa640 1 -- 172.21.15.46:0/3709708527 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7ff85806bec0 con 0x7ff8640a3c10 2024-09-16T04:39:01.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.011+0000 7ff863fff640 1 --2- 172.21.15.46:0/3709708527 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff8480383b0 0x7ff84803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.011+0000 7ff863fff640 10 monclient: get_auth_request con 0x7ff8480383b0 auth_method 0 2024-09-16T04:39:01.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.011+0000 7ff863fff640 1 --2- 172.21.15.46:0/3709708527 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff8480383b0 0x7ff84803a870 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff86409fc30 tx=0x7ff850009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.015+0000 7ff8617fa640 1 -- 172.21.15.46:0/3709708527 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff85804b030 con 0x7ff8640a3c10 2024-09-16T04:39:01.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.015+0000 7ff8617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:01.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.015+0000 7ff8617fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:01.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.191+0000 7ff86af31640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:39:01.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.191+0000 7ff86af31640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:39:01.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.191+0000 7ff86af31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:01.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.191+0000 7ff86af31640 1 -- 172.21.15.46:0/3709708527 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff864069150 con 0x7ff8640a3c10 2024-09-16T04:39:01.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.192+0000 7ff8617fa640 1 -- 172.21.15.46:0/3709708527 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+12377 (secure 0 0 0) 0x7ff8580459f0 con 0x7ff8640a3c10 2024-09-16T04:39:01.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.192+0000 7ff8617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:39:01.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.192+0000 7ff8617fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:01.193 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:39:01.193 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":19,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","created":"2024-09-16T04:38:39.286708+0000","modified":"2024-09-16T04:39:00.519938+0000","last_up_change":"2024-09-16T04:38:50.487347+0000","last_in_change":"2024-09-16T04:38:42.117316+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:38:51.392177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:38:57.324878+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ff88780-0630-4bb1-9f5a-5331d7724d17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3454452571},{"type":"v1","addr":"172.21.15.46:6809","nonce":3454452571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3454452571},{"type":"v1","addr":"172.21.15.46:6811","nonce":3454452571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3454452571},{"type":"v1","addr":"172.21.15.46:6815","nonce":3454452571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3454452571},{"type":"v1","addr":"172.21.15.46:6813","nonce":3454452571}]},"public_addr":"172.21.15.46:6809/3454452571","cluster_addr":"172.21.15.46:6811/3454452571","heartbeat_back_addr":"172.21.15.46:6815/3454452571","heartbeat_front_addr":"172.21.15.46:6813/3454452571","state":["exists","up"]},{"osd":1,"uuid":"37be2e88-32d0-4a30-8979-899b34f3be17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6817","nonce":4217009477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6819","nonce":4217009477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6823","nonce":4217009477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6821","nonce":4217009477}]},"public_addr":"172.21.15.46:6817/4217009477","cluster_addr":"172.21.15.46:6819/4217009477","heartbeat_back_addr":"172.21.15.46:6823/4217009477","heartbeat_front_addr":"172.21.15.46:6821/4217009477","state":["exists","up"]},{"osd":2,"uuid":"73b9a838-944e-48cd-afab-5260608d1c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6801","nonce":3101080311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6803","nonce":3101080311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6807","nonce":3101080311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6805","nonce":3101080311}]},"public_addr":"172.21.15.46:6801/3101080311","cluster_addr":"172.21.15.46:6803/3101080311","heartbeat_back_addr":"172.21.15.46:6807/3101080311","heartbeat_front_addr":"172.21.15.46:6805/3101080311","state":["exists","up"]},{"osd":3,"uuid":"d0c22455-5685-4ade-b540-3d5428a7cb46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":890769734},{"type":"v1","addr":"172.21.15.114:6811","nonce":890769734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":890769734},{"type":"v1","addr":"172.21.15.114:6813","nonce":890769734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":890769734},{"type":"v1","addr":"172.21.15.114:6817","nonce":890769734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":890769734},{"type":"v1","addr":"172.21.15.114:6815","nonce":890769734}]},"public_addr":"172.21.15.114:6811/890769734","cluster_addr":"172.21.15.114:6813/890769734","heartbeat_back_addr":"172.21.15.114:6817/890769734","heartbeat_front_addr":"172.21.15.114:6815/890769734","state":["exists","up"]},{"osd":4,"uuid":"f37724e4-2796-43c8-975b-4458986c4ea6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":936316557},{"type":"v1","addr":"172.21.15.114:6803","nonce":936316557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":936316557},{"type":"v1","addr":"172.21.15.114:6805","nonce":936316557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":936316557},{"type":"v1","addr":"172.21.15.114:6809","nonce":936316557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":936316557},{"type":"v1","addr":"172.21.15.114:6807","nonce":936316557}]},"public_addr":"172.21.15.114:6803/936316557","cluster_addr":"172.21.15.114:6805/936316557","heartbeat_back_addr":"172.21.15.114:6809/936316557","heartbeat_front_addr":"172.21.15.114:6807/936316557","state":["exists","up"]},{"osd":5,"uuid":"979362db-e173-47a0-93d1-187742938a85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6819","nonce":938868100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6821","nonce":938868100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6825","nonce":938868100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6823","nonce":938868100}]},"public_addr":"172.21.15.114:6819/938868100","cluster_addr":"172.21.15.114:6821/938868100","heartbeat_back_addr":"172.21.15.114:6825/938868100","heartbeat_front_addr":"172.21.15.114:6823/938868100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.647737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.394775+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.463096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.655270+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.652725+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.431803+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:39:01.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.194+0000 7ff86af31640 1 -- 172.21.15.46:0/3709708527 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff8480383b0 msgr2=0x7ff84803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.194+0000 7ff86af31640 1 --2- 172.21.15.46:0/3709708527 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff8480383b0 0x7ff84803a870 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff86409fc30 tx=0x7ff850009040 comp rx=0 tx=0).stop 2024-09-16T04:39:01.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.194+0000 7ff86af31640 10 monclient: shutdown 2024-09-16T04:39:01.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.194+0000 7ff86af31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:01.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.194+0000 7ff86af31640 1 -- 172.21.15.46:0/3709708527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8640a3c10 msgr2=0x7ff864133cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.194+0000 7ff86af31640 1 --2- 172.21.15.46:0/3709708527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8640a3c10 0x7ff864133cf0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff858000c00 tx=0x7ff8580046c0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.194+0000 7ff86af31640 1 -- 172.21.15.46:0/3709708527 shutdown_connections 2024-09-16T04:39:01.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.194+0000 7ff86af31640 1 --2- 172.21.15.46:0/3709708527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff86413be90 0x7ff8641cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.194+0000 7ff86af31640 1 --2- 172.21.15.46:0/3709708527 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff8640a3ff0 0x7ff864134230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.194+0000 7ff86af31640 1 --2- 172.21.15.46:0/3709708527 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff8480383b0 0x7ff84803a870 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.194+0000 7ff86af31640 1 --2- 172.21.15.46:0/3709708527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8640a3c10 0x7ff864133cf0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.194+0000 7ff86af31640 1 -- 172.21.15.46:0/3709708527 >> 172.21.15.46:0/3709708527 conn(0x7ff8640a5d10 msgr2=0x7ff86413b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:01.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.194+0000 7ff86af31640 1 -- 172.21.15.46:0/3709708527 shutdown_connections 2024-09-16T04:39:01.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.194+0000 7ff86af31640 1 -- 172.21.15.46:0/3709708527 wait complete. 2024-09-16T04:39:01.206 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T04:39:01.206 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:39:01.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.306+0000 7f29a9ec9640 10 monclient: get_monmap_and_config 2024-09-16T04:39:01.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.306+0000 7f29a9ec9640 10 monclient: build_initial_monmap 2024-09-16T04:39:01.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.306+0000 7f29a9ec9640 10 monclient: monmap: 2024-09-16T04:39:01.308 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:01.308 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.308 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:01.307131+0000 2024-09-16T04:39:01.308 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:01.307131+0000 2024-09-16T04:39:01.308 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:01.308 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.308 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:01.308 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:01.308 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:01.308 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:01.309 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:01.309 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.307+0000 7f29a9ec9640 1 Processor -- start 2024-09-16T04:39:01.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.307+0000 7f29a9ec9640 1 -- start start 2024-09-16T04:39:01.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.307+0000 7f29a9ec9640 10 monclient: init 2024-09-16T04:39:01.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.307+0000 7f29a9ec9640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:01.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.307+0000 7f29a9ec9640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-16T04:39:01.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.307+0000 7f29a9ec9640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f29a40a3c10 0x7f29a409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.307+0000 7f29a9ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f29a40a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:01.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.307+0000 7f29a9ec9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f29a40a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:01.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a9ec9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f29a413be90 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:01.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a9ec9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29a409b940 con 0x7f29a40a3ff0 2024-09-16T04:39:01.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a9ec9640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29a409ab00 con 0x7f29a413be90 2024-09-16T04:39:01.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a9ec9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a9ec9640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f29a409b030 con 0x7f29a40a3c10 2024-09-16T04:39:01.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a9ec9640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:01.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a9ec9640 10 monclient(hunting): authenticate will time out at 739.723097s 2024-09-16T04:39:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a37fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f29a40a3c10 0x7f29a409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a2ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f29a40a3ff0 0x7f29a40a43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52992/0 (socket says 172.21.15.46:52992) 2024-09-16T04:39:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a2ffd640 1 -- 172.21.15.46:0/3331744823 learned_addr learned my addr 172.21.15.46:0/3331744823 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a37fe640 10 monclient(hunting): get_auth_request con 0x7f29a40a3c10 auth_method 0 2024-09-16T04:39:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a37fe640 10 monclient(hunting): handle_auth_done global_id 4229 payload 274 2024-09-16T04:39:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a37fe640 1 -- 172.21.15.46:0/3331744823 >> v1:172.21.15.114:6789/0 conn(0x7f29a413be90 legacy=0x7f29a413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a37fe640 1 -- 172.21.15.46:0/3331744823 >> v1:172.21.15.46:6789/0 conn(0x7f29a40a3ff0 legacy=0x7f29a40a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:39:01.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a37fe640 1 monclient: found mon.noname-b 2024-09-16T04:39:01.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a27fc640 1 -- 172.21.15.46:0/3331744823 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2615990403 0 0) 0x7f29a409b940 con 0x7f29a40a3ff0 2024-09-16T04:39:01.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:01.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:39:01.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a37fe640 1 -- 172.21.15.46:0/3331744823 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a409b310 con 0x7f29a40a3c10 2024-09-16T04:39:01.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.308+0000 7f29a27fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:39:01.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a37fe640 1 --2- 172.21.15.46:0/3331744823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f29a40a3c10 0x7f29a409bcf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f29940097d0 tx=0x7f299402a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9964d8abf2090e9b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a27fc640 1 -- 172.21.15.46:0/3331744823 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2994037070 con 0x7f29a40a3c10 2024-09-16T04:39:01.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:01.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a27fc640 10 monclient: dump: 2024-09-16T04:39:01.320 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.320 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.320 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.320 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.320 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.320 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.320 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.320 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.320 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.321 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a27fc640 10 monclient: _finish_auth 0 2024-09-16T04:39:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:39:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a27fc640 1 -- 172.21.15.46:0/3331744823 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f299402adb0 con 0x7f29a40a3c10 2024-09-16T04:39:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a9ec9640 5 monclient: authenticate success, global_id 4229 2024-09-16T04:39:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a9ec9640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a27fc640 1 -- 172.21.15.46:0/3331744823 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2994032660 con 0x7f29a40a3c10 2024-09-16T04:39:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a9ec9640 4 set_mon_vals no callback set 2024-09-16T04:39:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a27fc640 10 monclient: dump: 2024-09-16T04:39:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.322 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a9ec9640 10 monclient: shutdown 2024-09-16T04:39:01.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a9ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:01.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a9ec9640 1 -- 172.21.15.46:0/3331744823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f29a40a3c10 msgr2=0x7f29a409bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a9ec9640 1 --2- 172.21.15.46:0/3331744823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f29a40a3c10 0x7f29a409bcf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f29940097d0 tx=0x7f299402a0a0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a9ec9640 1 -- 172.21.15.46:0/3331744823 shutdown_connections 2024-09-16T04:39:01.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a9ec9640 1 --2- 172.21.15.46:0/3331744823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f29a40a3c10 0x7f29a409bcf0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a9ec9640 1 -- 172.21.15.46:0/3331744823 >> 172.21.15.46:0/3331744823 conn(0x7f29a40a6340 msgr2=0x7f29a406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:01.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a9ec9640 1 -- 172.21.15.46:0/3331744823 shutdown_connections 2024-09-16T04:39:01.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.309+0000 7f29a9ec9640 1 -- 172.21.15.46:0/3331744823 wait complete. 2024-09-16T04:39:01.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.310+0000 7f29a9ec9640 10 monclient: build_initial_monmap 2024-09-16T04:39:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.310+0000 7f29a9ec9640 10 monclient: monmap: 2024-09-16T04:39:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:01.324 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.310+0000 7f29a9ec9640 1 Processor -- start 2024-09-16T04:39:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a9ec9640 1 -- start start 2024-09-16T04:39:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a9ec9640 10 monclient: init 2024-09-16T04:39:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a9ec9640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a9ec9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:39:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a9ec9640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f29a40a3c10 0x7f29a41cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a9ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f29a40a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a9ec9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29a40a3ff0 0x7f29a4136e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a9ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f29a40a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a9ec9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29a413be90 0x7f29a4138ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a9ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f29a413be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a9ec9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a9ec9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f29a40a6b40 con 0x7f29a413be90 2024-09-16T04:39:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a9ec9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a9ec9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f29a40a69c0 con 0x7f29a40a3ff0 2024-09-16T04:39:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a9ec9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a9ec9640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f29a413f9a0 con 0x7f29a40a3c10 2024-09-16T04:39:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a9ec9640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a9ec9640 10 monclient(hunting): authenticate will time out at 739.726563s 2024-09-16T04:39:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a37fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f29a40a3c10 0x7f29a41cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a37fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f29a40a3c10 0x7f29a41cbb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:60924/0 (socket says 172.21.15.46:60924) 2024-09-16T04:39:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a37fe640 1 -- 172.21.15.46:0/403153523 learned_addr learned my addr 172.21.15.46:0/403153523 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a37fe640 10 monclient(hunting): get_auth_request con 0x7f29a40a3c10 auth_method 0 2024-09-16T04:39:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a3fff640 1 --2- 172.21.15.46:0/403153523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29a413be90 0x7f29a4138ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a3fff640 10 monclient(hunting): get_auth_request con 0x7f29a413be90 auth_method 0 2024-09-16T04:39:01.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a2ffd640 1 --2- 172.21.15.46:0/403153523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29a40a3ff0 0x7f29a4136e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a2ffd640 10 monclient(hunting): get_auth_request con 0x7f29a40a3ff0 auth_method 0 2024-09-16T04:39:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a37fe640 10 monclient(hunting): handle_auth_done global_id 4232 payload 824 2024-09-16T04:39:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a37fe640 1 -- 172.21.15.46:0/403153523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29a40a3ff0 msgr2=0x7f29a4136e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a37fe640 1 --2- 172.21.15.46:0/403153523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29a40a3ff0 0x7f29a4136e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a37fe640 1 -- 172.21.15.46:0/403153523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29a413be90 msgr2=0x7f29a4138ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a37fe640 1 --2- 172.21.15.46:0/403153523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29a413be90 0x7f29a4138ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:39:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a37fe640 1 monclient: found mon.noname-c 2024-09-16T04:39:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:39:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.311+0000 7f29a37fe640 1 -- 172.21.15.46:0/403153523 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a41cca50 con 0x7f29a40a3c10 2024-09-16T04:39:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f29a37fe640 1 --2- 172.21.15.46:0/403153523 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f29a40a3c10 0x7f29a41cbb30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f29940097d0 tx=0x7f29940046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f298bfff640 1 -- 172.21.15.46:0/403153523 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2994037040 con 0x7f29a40a3c10 2024-09-16T04:39:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f298bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f298bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:39:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f298bfff640 10 monclient: dump: 2024-09-16T04:39:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.330 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f298bfff640 10 monclient: _finish_auth 0 2024-09-16T04:39:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f298bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:39:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f298bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f298bfff640 1 -- 172.21.15.46:0/403153523 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2994004030 con 0x7f29a40a3c10 2024-09-16T04:39:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f29a9ec9640 5 monclient: authenticate success, global_id 4232 2024-09-16T04:39:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f29a9ec9640 10 monclient: _renew_subs 2024-09-16T04:39:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f29a9ec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f29a9ec9640 1 -- 172.21.15.46:0/403153523 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29a413b5f0 con 0x7f29a40a3c10 2024-09-16T04:39:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f298bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f298bfff640 1 -- 172.21.15.46:0/403153523 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2994006e10 con 0x7f29a40a3c10 2024-09-16T04:39:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f298bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f298bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f298bfff640 10 monclient: dump: 2024-09-16T04:39:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.332 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.332 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f29a9ec9640 10 monclient: _renew_subs 2024-09-16T04:39:01.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f29a1ffb640 4 set_mon_vals no callback set 2024-09-16T04:39:01.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f29a9ec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:01.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.312+0000 7f29a9ec9640 1 -- 172.21.15.46:0/403153523 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f29a411ff50 con 0x7f29a40a3c10 2024-09-16T04:39:01.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.313+0000 7f298bfff640 1 -- 172.21.15.46:0/403153523 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f299403d650 con 0x7f29a40a3c10 2024-09-16T04:39:01.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.313+0000 7f298bfff640 1 --2- 172.21.15.46:0/403153523 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f29900383b0 0x7f299003a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.313+0000 7f298bfff640 1 -- 172.21.15.46:0/403153523 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f299406beb0 con 0x7f29a40a3c10 2024-09-16T04:39:01.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.313+0000 7f29a9ec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:01.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.313+0000 7f29a9ec9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:01.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.313+0000 7f29a9ec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:01.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.313+0000 7f29a9ec9640 1 -- 172.21.15.46:0/403153523 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29a409bcf0 con 0x7f29a40a3c10 2024-09-16T04:39:01.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.313+0000 7f29a2ffd640 1 --2- 172.21.15.46:0/403153523 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f29900383b0 0x7f299003a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.313+0000 7f29a2ffd640 10 monclient: get_auth_request con 0x7f29900383b0 auth_method 0 2024-09-16T04:39:01.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.314+0000 7f29a2ffd640 1 --2- 172.21.15.46:0/403153523 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f29900383b0 0x7f299003a870 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f29a40a1890 tx=0x7f2998007d50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.318+0000 7f298bfff640 1 -- 172.21.15.46:0/403153523 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2994042020 con 0x7f29a40a3c10 2024-09-16T04:39:01.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.318+0000 7f298bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:01.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.318+0000 7f298bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:01.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.495+0000 7f29a9ec9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:39:01.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.495+0000 7f29a9ec9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:39:01.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.495+0000 7f29a9ec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:01.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.495+0000 7f29a9ec9640 1 -- 172.21.15.46:0/403153523 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f29a413f770 con 0x7f29a40a3c10 2024-09-16T04:39:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.496+0000 7f298bfff640 1 -- 172.21.15.46:0/403153523 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+12377 (secure 0 0 0) 0x7f2994004400 con 0x7f29a40a3c10 2024-09-16T04:39:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.496+0000 7f298bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:39:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.496+0000 7f298bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:01.497 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:39:01.497 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":19,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","created":"2024-09-16T04:38:39.286708+0000","modified":"2024-09-16T04:39:00.519938+0000","last_up_change":"2024-09-16T04:38:50.487347+0000","last_in_change":"2024-09-16T04:38:42.117316+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:38:51.392177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:38:57.324878+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ff88780-0630-4bb1-9f5a-5331d7724d17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3454452571},{"type":"v1","addr":"172.21.15.46:6809","nonce":3454452571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3454452571},{"type":"v1","addr":"172.21.15.46:6811","nonce":3454452571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3454452571},{"type":"v1","addr":"172.21.15.46:6815","nonce":3454452571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3454452571},{"type":"v1","addr":"172.21.15.46:6813","nonce":3454452571}]},"public_addr":"172.21.15.46:6809/3454452571","cluster_addr":"172.21.15.46:6811/3454452571","heartbeat_back_addr":"172.21.15.46:6815/3454452571","heartbeat_front_addr":"172.21.15.46:6813/3454452571","state":["exists","up"]},{"osd":1,"uuid":"37be2e88-32d0-4a30-8979-899b34f3be17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6817","nonce":4217009477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6819","nonce":4217009477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6823","nonce":4217009477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6821","nonce":4217009477}]},"public_addr":"172.21.15.46:6817/4217009477","cluster_addr":"172.21.15.46:6819/4217009477","heartbeat_back_addr":"172.21.15.46:6823/4217009477","heartbeat_front_addr":"172.21.15.46:6821/4217009477","state":["exists","up"]},{"osd":2,"uuid":"73b9a838-944e-48cd-afab-5260608d1c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6801","nonce":3101080311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6803","nonce":3101080311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6807","nonce":3101080311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6805","nonce":3101080311}]},"public_addr":"172.21.15.46:6801/3101080311","cluster_addr":"172.21.15.46:6803/3101080311","heartbeat_back_addr":"172.21.15.46:6807/3101080311","heartbeat_front_addr":"172.21.15.46:6805/3101080311","state":["exists","up"]},{"osd":3,"uuid":"d0c22455-5685-4ade-b540-3d5428a7cb46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":890769734},{"type":"v1","addr":"172.21.15.114:6811","nonce":890769734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":890769734},{"type":"v1","addr":"172.21.15.114:6813","nonce":890769734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":890769734},{"type":"v1","addr":"172.21.15.114:6817","nonce":890769734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":890769734},{"type":"v1","addr":"172.21.15.114:6815","nonce":890769734}]},"public_addr":"172.21.15.114:6811/890769734","cluster_addr":"172.21.15.114:6813/890769734","heartbeat_back_addr":"172.21.15.114:6817/890769734","heartbeat_front_addr":"172.21.15.114:6815/890769734","state":["exists","up"]},{"osd":4,"uuid":"f37724e4-2796-43c8-975b-4458986c4ea6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":936316557},{"type":"v1","addr":"172.21.15.114:6803","nonce":936316557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":936316557},{"type":"v1","addr":"172.21.15.114:6805","nonce":936316557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":936316557},{"type":"v1","addr":"172.21.15.114:6809","nonce":936316557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":936316557},{"type":"v1","addr":"172.21.15.114:6807","nonce":936316557}]},"public_addr":"172.21.15.114:6803/936316557","cluster_addr":"172.21.15.114:6805/936316557","heartbeat_back_addr":"172.21.15.114:6809/936316557","heartbeat_front_addr":"172.21.15.114:6807/936316557","state":["exists","up"]},{"osd":5,"uuid":"979362db-e173-47a0-93d1-187742938a85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6819","nonce":938868100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6821","nonce":938868100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6825","nonce":938868100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6823","nonce":938868100}]},"public_addr":"172.21.15.114:6819/938868100","cluster_addr":"172.21.15.114:6821/938868100","heartbeat_back_addr":"172.21.15.114:6825/938868100","heartbeat_front_addr":"172.21.15.114:6823/938868100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.647737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.394775+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.463096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.655270+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.652725+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.431803+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:39:01.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.498+0000 7f29a9ec9640 1 -- 172.21.15.46:0/403153523 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f29900383b0 msgr2=0x7f299003a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.498+0000 7f29a9ec9640 1 --2- 172.21.15.46:0/403153523 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f29900383b0 0x7f299003a870 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f29a40a1890 tx=0x7f2998007d50 comp rx=0 tx=0).stop 2024-09-16T04:39:01.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.498+0000 7f29a9ec9640 10 monclient: shutdown 2024-09-16T04:39:01.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.498+0000 7f29a9ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:01.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.498+0000 7f29a9ec9640 1 -- 172.21.15.46:0/403153523 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f29a40a3c10 msgr2=0x7f29a41cbb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.498+0000 7f29a9ec9640 1 --2- 172.21.15.46:0/403153523 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f29a40a3c10 0x7f29a41cbb30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f29940097d0 tx=0x7f29940046c0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.498+0000 7f29a9ec9640 1 -- 172.21.15.46:0/403153523 shutdown_connections 2024-09-16T04:39:01.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.498+0000 7f29a9ec9640 1 --2- 172.21.15.46:0/403153523 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f29900383b0 0x7f299003a870 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.498+0000 7f29a9ec9640 1 --2- 172.21.15.46:0/403153523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29a413be90 0x7f29a4138ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.498+0000 7f29a9ec9640 1 --2- 172.21.15.46:0/403153523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29a40a3ff0 0x7f29a4136e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.498+0000 7f29a9ec9640 1 --2- 172.21.15.46:0/403153523 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f29a40a3c10 0x7f29a41cbb30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.498+0000 7f29a9ec9640 1 -- 172.21.15.46:0/403153523 >> 172.21.15.46:0/403153523 conn(0x7f29a40a6340 msgr2=0x7f29a40a0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:01.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.498+0000 7f29a9ec9640 1 -- 172.21.15.46:0/403153523 shutdown_connections 2024-09-16T04:39:01.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.498+0000 7f29a9ec9640 1 -- 172.21.15.46:0/403153523 wait complete. 2024-09-16T04:39:01.511 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T04:39:01.511 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T04:39:01.511 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T04:39:01.512 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T04:39:01.512 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T04:39:01.513 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T04:39:01.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.629+0000 7f3e7bb0f640 10 monclient: get_monmap_and_config 2024-09-16T04:39:01.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.629+0000 7f3e7bb0f640 10 monclient: build_initial_monmap 2024-09-16T04:39:01.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.629+0000 7f3e7bb0f640 10 monclient: monmap: 2024-09-16T04:39:01.631 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:01.631 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.631 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:01.630325+0000 2024-09-16T04:39:01.631 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:01.630325+0000 2024-09-16T04:39:01.631 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:01.631 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.632 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:01.632 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:01.632 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:01.632 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:01.632 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:01.632 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.630+0000 7f8da8331640 10 monclient: get_monmap_and_config 2024-09-16T04:39:01.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.630+0000 7f8da8331640 10 monclient: build_initial_monmap 2024-09-16T04:39:01.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.630+0000 7f8da8331640 10 monclient: monmap: 2024-09-16T04:39:01.633 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:01.633 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.633 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:01.630569+0000 2024-09-16T04:39:01.633 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:01.630569+0000 2024-09-16T04:39:01.633 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:01.633 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.633 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:01.633 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:01.633 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:01.633 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:01.633 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:01.633 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.630+0000 7f3e7bb0f640 1 Processor -- start 2024-09-16T04:39:01.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.630+0000 7f8da8331640 1 Processor -- start 2024-09-16T04:39:01.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.630+0000 7f8da8331640 1 -- start start 2024-09-16T04:39:01.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.630+0000 7f8da8331640 10 monclient: init 2024-09-16T04:39:01.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.630+0000 7f3e7bb0f640 1 -- start start 2024-09-16T04:39:01.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.630+0000 7f3e7bb0f640 10 monclient: init 2024-09-16T04:39:01.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.631+0000 7f3e7bb0f640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:01.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.631+0000 7f3e7bb0f640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-16T04:39:01.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.631+0000 7f3e7bb0f640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3e740a3c10 0x7f3e7409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.631+0000 7f3e7bb0f640 10 monclient(hunting): picked mon.noname-c con 0x7f3e740a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:01.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.631+0000 7f8da8331640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:01.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.631+0000 7f8da8331640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-16T04:39:01.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.631+0000 7f8da8331640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8da00a3c10 0x7f8da009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.631+0000 7f3e7bb0f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e740a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:01.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.631+0000 7f3e7bb0f640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e7413be90 0x7f3e7413e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.631+0000 7f8da8331640 10 monclient(hunting): picked mon.noname-a con 0x7f8da00a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:01.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.631+0000 7f8da8331640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8da00a3ff0 0x7f8da00a4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.631+0000 7f3e790aa640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3e740a3ff0 0x7f3e740a4400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52994/0 (socket says 172.21.15.46:52994) 2024-09-16T04:39:01.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.631+0000 7f3e7bb0f640 10 monclient(hunting): picked mon.noname-b con 0x7f3e7413be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:01.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e7bb0f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e7409ab70 con 0x7f3e740a3ff0 2024-09-16T04:39:01.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e7bb0f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e7bb0f640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f3e740a4d60 con 0x7f3e740a3c10 2024-09-16T04:39:01.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e7bb0f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e7bb0f640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f3e7409b030 con 0x7f3e7413be90 2024-09-16T04:39:01.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da60cd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8da00a3c10 0x7f8da009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da60cd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8da00a3c10 0x7f8da009bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59202/0 (socket says 172.21.15.46:59202) 2024-09-16T04:39:01.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e790aa640 1 -- 172.21.15.46:0/3915074180 learned_addr learned my addr 172.21.15.46:0/3915074180 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:01.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e798ab640 1 --2- 172.21.15.46:0/3915074180 >> v2:172.21.15.114:3300/0 conn(0x7f3e740a3c10 0x7f3e7409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e7a0ac640 1 --2- 172.21.15.46:0/3915074180 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e7413be90 0x7f3e7413e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da60cd640 1 -- 172.21.15.46:0/2670202249 learned_addr learned my addr 172.21.15.46:0/2670202249 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:01.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e7bb0f640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:01.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e7bb0f640 10 monclient(hunting): authenticate will time out at 740.047282s 2024-09-16T04:39:01.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da8331640 10 monclient(hunting): picked mon.noname-c con 0x7f8da00a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:01.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da8331640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8da01369d0 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:01.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da58cc640 1 --2- 172.21.15.46:0/2670202249 >> v2:172.21.15.114:3300/0 conn(0x7f8da00a3ff0 0x7f8da00a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da8331640 1 -- 172.21.15.46:0/2670202249 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8da009ab70 con 0x7f8da01369d0 2024-09-16T04:39:01.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da8331640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da8331640 1 -- 172.21.15.46:0/2670202249 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f8da00a4d60 con 0x7f8da00a3c10 2024-09-16T04:39:01.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da8331640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da8331640 1 -- 172.21.15.46:0/2670202249 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f8da009b030 con 0x7f8da00a3ff0 2024-09-16T04:39:01.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da8331640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:01.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e798ab640 10 monclient(hunting): get_auth_request con 0x7f3e740a3c10 auth_method 0 2024-09-16T04:39:01.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e798ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e798ab640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e798ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e7a0ac640 10 monclient(hunting): get_auth_request con 0x7f3e7413be90 auth_method 0 2024-09-16T04:39:01.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e7a0ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e7a0ac640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e7a0ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e7a0ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e7a0ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e7a0ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e798ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e798ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e798ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e788a9640 1 -- 172.21.15.46:0/3915074180 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1191642779 0 0) 0x7f3e7409ab70 con 0x7f3e740a3ff0 2024-09-16T04:39:01.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e788a9640 10 monclient(hunting): _negotiate 2024-09-16T04:39:01.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e788a9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e788a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e788a9640 10 monclient(hunting): my global_id is 4254 2024-09-16T04:39:01.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f3e788a9640 1 -- 172.21.15.46:0/3915074180 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e60003520 con 0x7f3e740a3ff0 2024-09-16T04:39:01.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da8331640 10 monclient(hunting): authenticate will time out at 740.047678s 2024-09-16T04:39:01.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da60cd640 10 monclient(hunting): get_auth_request con 0x7f8da00a3c10 auth_method 0 2024-09-16T04:39:01.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da60cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da60cd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da60cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da60cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da60cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da60cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da58cc640 10 monclient(hunting): get_auth_request con 0x7f8da00a3ff0 auth_method 0 2024-09-16T04:39:01.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da58cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da58cc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da58cc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da50cb640 1 -- 172.21.15.46:0/2670202249 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3679373095 0 0) 0x7f8da009ab70 con 0x7f8da01369d0 2024-09-16T04:39:01.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da50cb640 10 monclient(hunting): _negotiate 2024-09-16T04:39:01.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da50cb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.633+0000 7f3e7a0ac640 10 monclient(hunting): handle_auth_done global_id 4235 payload 274 2024-09-16T04:39:01.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.633+0000 7f3e7a0ac640 1 -- 172.21.15.46:0/3915074180 >> v2:172.21.15.114:3300/0 conn(0x7f3e740a3c10 msgr2=0x7f3e7409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:01.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.633+0000 7f3e7a0ac640 1 --2- 172.21.15.46:0/3915074180 >> v2:172.21.15.114:3300/0 conn(0x7f3e740a3c10 0x7f3e7409bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.633+0000 7f3e7a0ac640 1 -- 172.21.15.46:0/3915074180 >> v1:172.21.15.46:6789/0 conn(0x7f3e740a3ff0 legacy=0x7f3e740a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.633+0000 7f3e7a0ac640 10 monclient: _finish_hunting 0 2024-09-16T04:39:01.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.633+0000 7f3e7a0ac640 1 monclient: found mon.noname-b 2024-09-16T04:39:01.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.633+0000 7f3e7a0ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:01.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.633+0000 7f3e7a0ac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:39:01.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.633+0000 7f3e7a0ac640 1 -- 172.21.15.46:0/3915074180 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e7409b310 con 0x7f3e7413be90 2024-09-16T04:39:01.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da50cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da50cb640 10 monclient(hunting): my global_id is 4257 2024-09-16T04:39:01.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.632+0000 7f8da50cb640 1 -- 172.21.15.46:0/2670202249 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d8c003520 con 0x7f8da01369d0 2024-09-16T04:39:01.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.633+0000 7f8da60cd640 10 monclient(hunting): handle_auth_done global_id 4260 payload 274 2024-09-16T04:39:01.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.633+0000 7f8da60cd640 1 -- 172.21.15.46:0/2670202249 >> v2:172.21.15.114:3300/0 conn(0x7f8da00a3ff0 msgr2=0x7f8da00a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:01.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.633+0000 7f8da60cd640 1 --2- 172.21.15.46:0/2670202249 >> v2:172.21.15.114:3300/0 conn(0x7f8da00a3ff0 0x7f8da00a4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.633+0000 7f8da60cd640 1 -- 172.21.15.46:0/2670202249 >> v1:172.21.15.46:6789/0 conn(0x7f8da01369d0 legacy=0x7f8da0136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.633+0000 7f8da60cd640 10 monclient: _finish_hunting 0 2024-09-16T04:39:01.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.633+0000 7f8da60cd640 1 monclient: found mon.noname-a 2024-09-16T04:39:01.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.633+0000 7f8da60cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:01.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.633+0000 7f8da60cd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:01.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.633+0000 7f8da60cd640 1 -- 172.21.15.46:0/2670202249 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da009b310 con 0x7f8da00a3c10 2024-09-16T04:39:01.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f3e7a0ac640 1 --2- 172.21.15.46:0/3915074180 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e7413be90 0x7f3e7413e280 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c00b050 tx=0x7f3e6c02a7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88606875c154a9e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f3e788a9640 1 -- 172.21.15.46:0/3915074180 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c031600 con 0x7f3e7413be90 2024-09-16T04:39:01.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f8da60cd640 1 --2- 172.21.15.46:0/2670202249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da00a3c10 0x7f8da009bcf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8da006edc0 tx=0x7f8d9c032920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbe337579503dbf3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f3e788a9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f3e788a9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:01.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f3e788a9640 10 monclient: dump: 2024-09-16T04:39:01.653 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.653 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.653 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.654 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.654 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.654 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.654 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.654 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.654 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.654 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f3e788a9640 10 monclient: _finish_auth 0 2024-09-16T04:39:01.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f3e788a9640 10 monclient: _check_auth_tickets 2024-09-16T04:39:01.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f3e788a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:01.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f3e788a9640 1 -- 172.21.15.46:0/3915074180 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e6c0317a0 con 0x7f3e7413be90 2024-09-16T04:39:01.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f3e7bb0f640 5 monclient: authenticate success, global_id 4235 2024-09-16T04:39:01.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f3e7bb0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:01.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f3e788a9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:01.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f8da50cb640 1 -- 172.21.15.46:0/2670202249 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c035070 con 0x7f8da00a3c10 2024-09-16T04:39:01.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f8da50cb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f8da50cb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:01.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f8da50cb640 10 monclient: dump: 2024-09-16T04:39:01.655 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.655 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.655 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.655 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.655 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.655 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.655 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.655 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.656 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.656 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f8da50cb640 10 monclient: _finish_auth 0 2024-09-16T04:39:01.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f3e788a9640 1 -- 172.21.15.46:0/3915074180 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c031b70 con 0x7f3e7413be90 2024-09-16T04:39:01.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f8da50cb640 10 monclient: _check_auth_tickets 2024-09-16T04:39:01.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f8da50cb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:01.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f3e7bb0f640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:01.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f8da8331640 5 monclient: authenticate success, global_id 4260 2024-09-16T04:39:01.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f8da8331640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:01.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f3e7bb0f640 4 set_mon_vals no callback set 2024-09-16T04:39:01.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f3e788a9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f3e788a9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:01.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f8da50cb640 1 -- 172.21.15.46:0/2670202249 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d9c03e070 con 0x7f8da00a3c10 2024-09-16T04:39:01.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f8da50cb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:01.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f8da50cb640 1 -- 172.21.15.46:0/2670202249 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c0389e0 con 0x7f8da00a3c10 2024-09-16T04:39:01.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f8da50cb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f8da50cb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:01.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f3e788a9640 10 monclient: dump: 2024-09-16T04:39:01.659 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.659 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.659 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.659 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.659 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.659 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.659 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.659 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.659 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.659 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f8da50cb640 10 monclient: dump: 2024-09-16T04:39:01.660 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.660 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.660 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.660 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.660 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.660 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.660 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.660 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.660 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.660 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f8da8331640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:01.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.634+0000 7f8da8331640 4 set_mon_vals no callback set 2024-09-16T04:39:01.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.635+0000 7f3e7bb0f640 10 monclient: shutdown 2024-09-16T04:39:01.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.635+0000 7f3e7bb0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:01.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.635+0000 7f3e7bb0f640 1 -- 172.21.15.46:0/3915074180 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e7413be90 msgr2=0x7f3e7413e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.635+0000 7f3e7bb0f640 1 --2- 172.21.15.46:0/3915074180 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e7413be90 0x7f3e7413e280 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c00b050 tx=0x7f3e6c02a7a0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.635+0000 7f8da8331640 10 monclient: shutdown 2024-09-16T04:39:01.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.635+0000 7f8da8331640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:01.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.635+0000 7f8da8331640 1 -- 172.21.15.46:0/2670202249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da00a3c10 msgr2=0x7f8da009bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.635+0000 7f3e7bb0f640 1 -- 172.21.15.46:0/3915074180 shutdown_connections 2024-09-16T04:39:01.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.635+0000 7f3e7bb0f640 1 --2- 172.21.15.46:0/3915074180 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e7413be90 0x7f3e7413e280 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.635+0000 7f3e7bb0f640 1 --2- 172.21.15.46:0/3915074180 >> v2:172.21.15.114:3300/0 conn(0x7f3e740a3c10 0x7f3e7409bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.635+0000 7f3e7bb0f640 1 -- 172.21.15.46:0/3915074180 >> 172.21.15.46:0/3915074180 conn(0x7f3e740a5d10 msgr2=0x7f3e740a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:01.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.635+0000 7f8da8331640 1 --2- 172.21.15.46:0/2670202249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da00a3c10 0x7f8da009bcf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8da006edc0 tx=0x7f8d9c032920 comp rx=0 tx=0).stop 2024-09-16T04:39:01.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.635+0000 7f8da8331640 1 -- 172.21.15.46:0/2670202249 shutdown_connections 2024-09-16T04:39:01.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.635+0000 7f8da8331640 1 --2- 172.21.15.46:0/2670202249 >> v2:172.21.15.114:3300/0 conn(0x7f8da00a3ff0 0x7f8da00a4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.635+0000 7f3e7bb0f640 1 -- 172.21.15.46:0/3915074180 shutdown_connections 2024-09-16T04:39:01.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.635+0000 7f3e7bb0f640 1 -- 172.21.15.46:0/3915074180 wait complete. 2024-09-16T04:39:01.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.635+0000 7f8da8331640 1 --2- 172.21.15.46:0/2670202249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da00a3c10 0x7f8da009bcf0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.635+0000 7f8da8331640 1 -- 172.21.15.46:0/2670202249 >> 172.21.15.46:0/2670202249 conn(0x7f8da00a5d10 msgr2=0x7f8da00a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:01.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.635+0000 7f8da8331640 1 -- 172.21.15.46:0/2670202249 shutdown_connections 2024-09-16T04:39:01.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.635+0000 7f8da8331640 1 -- 172.21.15.46:0/2670202249 wait complete. 2024-09-16T04:39:01.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.636+0000 7f3e7bb0f640 10 monclient: build_initial_monmap 2024-09-16T04:39:01.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.636+0000 7f3e7bb0f640 10 monclient: monmap: 2024-09-16T04:39:01.665 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:01.665 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:01.665 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:01.666 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:01.666 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:01.666 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.666 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:01.666 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:01.666 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:01.666 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.636+0000 7f8da8331640 10 monclient: build_initial_monmap 2024-09-16T04:39:01.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.636+0000 7f8da8331640 10 monclient: monmap: 2024-09-16T04:39:01.667 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:01.667 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:01.667 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:01.667 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:01.667 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:01.667 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.667 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:01.667 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:01.667 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:01.667 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.636+0000 7f3e7bb0f640 1 Processor -- start 2024-09-16T04:39:01.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.636+0000 7f3e7bb0f640 1 -- start start 2024-09-16T04:39:01.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.636+0000 7f3e7bb0f640 10 monclient: init 2024-09-16T04:39:01.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f8da8331640 1 Processor -- start 2024-09-16T04:39:01.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f8da8331640 1 -- start start 2024-09-16T04:39:01.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f8da8331640 10 monclient: init 2024-09-16T04:39:01.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e7bb0f640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:01.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e7bb0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:39:01.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e7bb0f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e740a3c10 0x7f3e74136af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e7bb0f640 10 monclient(hunting): picked mon.noname-a con 0x7f3e740a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:01.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e7bb0f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e740a3ff0 0x7f3e74137030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e7bb0f640 10 monclient(hunting): picked mon.noname-b con 0x7f3e740a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:01.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e7bb0f640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e7413be90 0x7f3e74139c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e790aa640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e740a3ff0 0x7f3e74137030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e798ab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e740a3c10 0x7f3e74136af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e7a0ac640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e7413be90 0x7f3e74139c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e790aa640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e740a3ff0 0x7f3e74137030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.46:38204/0 (socket says 172.21.15.46:38204) 2024-09-16T04:39:01.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e7a0ac640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e7413be90 0x7f3e74139c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:60940/0 (socket says 172.21.15.46:60940) 2024-09-16T04:39:01.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e7bb0f640 10 monclient(hunting): picked mon.noname-c con 0x7f3e7413be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:01.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e7bb0f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e7bb0f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7406e550 con 0x7f3e740a3c10 2024-09-16T04:39:01.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e7bb0f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e7bb0f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7406e3d0 con 0x7f3e740a3ff0 2024-09-16T04:39:01.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e7bb0f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e798ab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e740a3c10 0x7f3e74136af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59210/0 (socket says 172.21.15.46:59210) 2024-09-16T04:39:01.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e7bb0f640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f3e7413e7c0 con 0x7f3e7413be90 2024-09-16T04:39:01.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e7bb0f640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:01.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e798ab640 1 -- 172.21.15.46:0/1899872277 learned_addr learned my addr 172.21.15.46:0/1899872277 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:01.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e7bb0f640 10 monclient(hunting): authenticate will time out at 740.052878s 2024-09-16T04:39:01.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e798ab640 10 monclient(hunting): get_auth_request con 0x7f3e740a3c10 auth_method 0 2024-09-16T04:39:01.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e798ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e798ab640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e798ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e790aa640 10 monclient(hunting): get_auth_request con 0x7f3e740a3ff0 auth_method 0 2024-09-16T04:39:01.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e790aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e790aa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f8da8331640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:01.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f8da8331640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:39:01.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.637+0000 7f3e790aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f8da8331640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da00a3c10 0x7f8da0131f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e798ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e798ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f8da8331640 10 monclient(hunting): picked mon.noname-a con 0x7f8da00a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:01.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e798ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e7a0ac640 10 monclient(hunting): get_auth_request con 0x7f3e7413be90 auth_method 0 2024-09-16T04:39:01.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e7a0ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f8da8331640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8da00a3ff0 0x7f8da01324b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e7a0ac640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e7a0ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e790aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e790aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e790aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e7a0ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e7a0ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e7a0ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e790aa640 10 monclient(hunting): handle_auth_done global_id 4270 payload 824 2024-09-16T04:39:01.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e790aa640 1 -- 172.21.15.46:0/1899872277 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e7413be90 msgr2=0x7f3e74139c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e790aa640 1 --2- 172.21.15.46:0/1899872277 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e7413be90 0x7f3e74139c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e790aa640 1 -- 172.21.15.46:0/1899872277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e740a3c10 msgr2=0x7f3e74136af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f8da60cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da00a3c10 0x7f8da0131f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f8da60cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da00a3c10 0x7f8da0131f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59218/0 (socket says 172.21.15.46:59218) 2024-09-16T04:39:01.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f8da8331640 10 monclient(hunting): picked mon.noname-b con 0x7f8da00a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:01.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f8da60cd640 1 -- 172.21.15.46:0/1000634446 learned_addr learned my addr 172.21.15.46:0/1000634446 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:01.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f8da8331640 1 --2- 172.21.15.46:0/1000634446 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8da01369d0 0x7f8da01c3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e790aa640 1 --2- 172.21.15.46:0/1899872277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e740a3c10 0x7f3e74136af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e790aa640 10 monclient: _finish_hunting 0 2024-09-16T04:39:01.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e790aa640 1 monclient: found mon.noname-b 2024-09-16T04:39:01.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e790aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:01.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e790aa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3300/0 2024-09-16T04:39:01.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e790aa640 1 -- 172.21.15.46:0/1899872277 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e741212e0 con 0x7f3e740a3ff0 2024-09-16T04:39:01.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f8da58cc640 1 --2- 172.21.15.46:0/1000634446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8da00a3ff0 0x7f8da01324b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.638+0000 7f3e798ab640 1 --2- 172.21.15.46:0/1899872277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e740a3c10 0x7f3e74136af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:01.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da8331640 10 monclient(hunting): picked mon.noname-c con 0x7f8da01369d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:01.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da8331640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da8331640 1 -- 172.21.15.46:0/1000634446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8da006e3d0 con 0x7f8da00a3c10 2024-09-16T04:39:01.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da8331640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da8331640 1 -- 172.21.15.46:0/1000634446 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8da00a4940 con 0x7f8da00a3ff0 2024-09-16T04:39:01.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da8331640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da8331640 1 -- 172.21.15.46:0/1000634446 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f8da006e550 con 0x7f8da01369d0 2024-09-16T04:39:01.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e790aa640 1 --2- 172.21.15.46:0/1899872277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e740a3ff0 0x7f3e74137030 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3e6400c430 tx=0x7f3e6400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da8331640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:01.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da8331640 10 monclient(hunting): authenticate will time out at 740.054221s 2024-09-16T04:39:01.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da68ce640 1 --2- 172.21.15.46:0/1000634446 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8da01369d0 0x7f8da01c3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da60cd640 10 monclient(hunting): get_auth_request con 0x7f8da00a3c10 auth_method 0 2024-09-16T04:39:01.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da60cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e6a7fc640 1 -- 172.21.15.46:0/1899872277 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e64017070 con 0x7f3e740a3ff0 2024-09-16T04:39:01.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e6a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e6a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:01.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e6a7fc640 10 monclient: dump: 2024-09-16T04:39:01.684 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.684 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.684 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.684 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.684 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.684 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.684 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.684 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.684 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.685 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e6a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:39:01.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da60cd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da60cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da58cc640 10 monclient(hunting): get_auth_request con 0x7f8da00a3ff0 auth_method 0 2024-09-16T04:39:01.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da58cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da58cc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e6a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:39:01.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:01.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e6a7fc640 1 -- 172.21.15.46:0/1899872277 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e64005ce0 con 0x7f3e740a3ff0 2024-09-16T04:39:01.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e6a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:01.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e6a7fc640 1 -- 172.21.15.46:0/1899872277 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e640072f0 con 0x7f3e740a3ff0 2024-09-16T04:39:01.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e6a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e6a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:01.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da58cc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da60cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da60cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da60cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da58cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da58cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da58cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da68ce640 10 monclient(hunting): get_auth_request con 0x7f8da01369d0 auth_method 0 2024-09-16T04:39:01.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da68ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da68ce640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da68ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da60cd640 10 monclient(hunting): handle_auth_done global_id 4266 payload 824 2024-09-16T04:39:01.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da60cd640 1 -- 172.21.15.46:0/1000634446 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8da01369d0 msgr2=0x7f8da01c3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da60cd640 1 --2- 172.21.15.46:0/1000634446 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8da01369d0 0x7f8da01c3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da60cd640 1 -- 172.21.15.46:0/1000634446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8da00a3ff0 msgr2=0x7f8da01324b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da60cd640 1 --2- 172.21.15.46:0/1000634446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8da00a3ff0 0x7f8da01324b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da60cd640 10 monclient: _finish_hunting 0 2024-09-16T04:39:01.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da60cd640 1 monclient: found mon.noname-a 2024-09-16T04:39:01.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da60cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:01.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da60cd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:01.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f8da60cd640 1 -- 172.21.15.46:0/1000634446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da00a0400 con 0x7f8da00a3c10 2024-09-16T04:39:01.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8da58cc640 1 --2- 172.21.15.46:0/1000634446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8da00a3ff0 0x7f8da01324b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:01.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8da68ce640 1 --2- 172.21.15.46:0/1000634446 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8da01369d0 0x7f8da01c3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:39:01.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8da60cd640 1 --2- 172.21.15.46:0/1000634446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da00a3c10 0x7f8da0131f70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c032960 tx=0x7f8d9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e6a7fc640 10 monclient: dump: 2024-09-16T04:39:01.689 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.689 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.689 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.689 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.689 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.690 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.690 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.690 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.690 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.690 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e6bfff640 4 set_mon_vals no callback set 2024-09-16T04:39:01.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e7bb0f640 5 monclient: authenticate success, global_id 4270 2024-09-16T04:39:01.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e7bb0f640 10 monclient: _renew_subs 2024-09-16T04:39:01.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e7bb0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:01.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8d96ffd640 1 -- 172.21.15.46:0/1000634446 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c048070 con 0x7f8da00a3c10 2024-09-16T04:39:01.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8d96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8d96ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:01.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8d96ffd640 10 monclient: dump: 2024-09-16T04:39:01.691 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.691 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.691 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.691 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.691 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.691 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.691 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.691 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.691 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.691 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8d96ffd640 10 monclient: _finish_auth 0 2024-09-16T04:39:01.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8d96ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:39:01.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8d96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:01.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e7bb0f640 1 -- 172.21.15.46:0/1899872277 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e740a0400 con 0x7f3e740a3ff0 2024-09-16T04:39:01.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e7bb0f640 10 monclient: _renew_subs 2024-09-16T04:39:01.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e7bb0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:01.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.639+0000 7f3e7bb0f640 1 -- 172.21.15.46:0/1899872277 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e741215c0 con 0x7f3e740a3ff0 2024-09-16T04:39:01.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8d96ffd640 1 -- 172.21.15.46:0/1000634446 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d9c003c40 con 0x7f8da00a3c10 2024-09-16T04:39:01.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8d96ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:01.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8d96ffd640 1 -- 172.21.15.46:0/1000634446 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c0385e0 con 0x7f8da00a3c10 2024-09-16T04:39:01.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8d96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8d96ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:01.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8d96ffd640 10 monclient: dump: 2024-09-16T04:39:01.694 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.694 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.694 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.694 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.694 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.694 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.695 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.695 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.695 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.695 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8da8331640 5 monclient: authenticate success, global_id 4266 2024-09-16T04:39:01.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8da8331640 10 monclient: _renew_subs 2024-09-16T04:39:01.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8da8331640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:01.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8da8331640 1 -- 172.21.15.46:0/1000634446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8da00711d0 con 0x7f8da00a3c10 2024-09-16T04:39:01.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8da8331640 10 monclient: _renew_subs 2024-09-16T04:39:01.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8da8331640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:01.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8da8331640 1 -- 172.21.15.46:0/1000634446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8da009f730 con 0x7f8da00a3c10 2024-09-16T04:39:01.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.640+0000 7f8da48ca640 4 set_mon_vals no callback set 2024-09-16T04:39:01.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.641+0000 7f3e7bb0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:01.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.641+0000 7f3e7bb0f640 1 -- 172.21.15.46:0/1899872277 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3e7409bcf0 con 0x7f3e740a3ff0 2024-09-16T04:39:01.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.641+0000 7f3e6a7fc640 1 -- 172.21.15.46:0/1899872277 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f3e6401a070 con 0x7f3e740a3ff0 2024-09-16T04:39:01.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.641+0000 7f8da8331640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:01.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.641+0000 7f8da8331640 1 -- 172.21.15.46:0/1000634446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8da0071b00 con 0x7f8da00a3c10 2024-09-16T04:39:01.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.642+0000 7f3e6a7fc640 1 --2- 172.21.15.46:0/1899872277 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3e58038400 0x7f3e5803a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.642+0000 7f3e6a7fc640 1 -- 172.21.15.46:0/1899872277 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f3e6404aaa0 con 0x7f3e740a3ff0 2024-09-16T04:39:01.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.642+0000 7f3e6a7fc640 1 --2- 172.21.15.46:0/1899872277 >> [v2:172.21.15.46:6808/3454452571,v1:172.21.15.46:6809/3454452571] conn(0x7f3e5803e270 0x7f3e580406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.642+0000 7f3e6a7fc640 1 -- 172.21.15.46:0/1899872277 --> [v2:172.21.15.46:6808/3454452571,v1:172.21.15.46:6809/3454452571] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3e64015e70 con 0x7f3e5803e270 2024-09-16T04:39:01.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.642+0000 7f3e6a7fc640 1 -- 172.21.15.46:0/1899872277 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f3e64051050 con 0x7f3e740a3ff0 2024-09-16T04:39:01.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.642+0000 7f3e6a7fc640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-16T04:39:01.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.642+0000 7f3e798ab640 1 --2- 172.21.15.46:0/1899872277 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3e58038400 0x7f3e5803a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.642+0000 7f3e798ab640 10 monclient: get_auth_request con 0x7f3e58038400 auth_method 0 2024-09-16T04:39:01.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.642+0000 7f3e7a0ac640 1 --2- 172.21.15.46:0/1899872277 >> [v2:172.21.15.46:6808/3454452571,v1:172.21.15.46:6809/3454452571] conn(0x7f3e5803e270 0x7f3e580406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.642+0000 7f3e7a0ac640 10 monclient: get_auth_request con 0x7f3e5803e270 auth_method 0 2024-09-16T04:39:01.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.642+0000 7f8d96ffd640 1 -- 172.21.15.46:0/1000634446 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f8d9c053020 con 0x7f8da00a3c10 2024-09-16T04:39:01.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.642+0000 7f8d96ffd640 1 --2- 172.21.15.46:0/1000634446 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8d840383b0 0x7f8d8403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.643+0000 7f8d96ffd640 1 -- 172.21.15.46:0/1000634446 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f8d9c074130 con 0x7f8da00a3c10 2024-09-16T04:39:01.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.643+0000 7f3e798ab640 1 --2- 172.21.15.46:0/1899872277 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3e58038400 0x7f3e5803a8c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3e700112c0 tx=0x7f3e70009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.643+0000 7f8d96ffd640 1 --2- 172.21.15.46:0/1000634446 >> [v2:172.21.15.114:6802/936316557,v1:172.21.15.114:6803/936316557] conn(0x7f8d8403e220 0x7f8d84040660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.643+0000 7f3e7a0ac640 1 --2- 172.21.15.46:0/1899872277 >> [v2:172.21.15.46:6808/3454452571,v1:172.21.15.46:6809/3454452571] conn(0x7f3e5803e270 0x7f3e580406b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.643+0000 7f8da58cc640 1 --2- 172.21.15.46:0/1000634446 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8d840383b0 0x7f8d8403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.643+0000 7f8d96ffd640 1 -- 172.21.15.46:0/1000634446 --> [v2:172.21.15.114:6802/936316557,v1:172.21.15.114:6803/936316557] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8d84040d30 con 0x7f8d8403e220 2024-09-16T04:39:01.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.643+0000 7f8da58cc640 10 monclient: get_auth_request con 0x7f8d840383b0 auth_method 0 2024-09-16T04:39:01.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.643+0000 7f8d96ffd640 1 -- 172.21.15.46:0/1000634446 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f8d9c07b050 con 0x7f8da00a3c10 2024-09-16T04:39:01.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.643+0000 7f8d96ffd640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-16T04:39:01.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.643+0000 7f8da68ce640 1 --2- 172.21.15.46:0/1000634446 >> [v2:172.21.15.114:6802/936316557,v1:172.21.15.114:6803/936316557] conn(0x7f8d8403e220 0x7f8d84040660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.643+0000 7f8da68ce640 10 monclient: get_auth_request con 0x7f8d8403e220 auth_method 0 2024-09-16T04:39:01.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.643+0000 7f8da58cc640 1 --2- 172.21.15.46:0/1000634446 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8d840383b0 0x7f8d8403a870 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8da009fcd0 tx=0x7f8d90008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.643+0000 7f8da68ce640 1 --2- 172.21.15.46:0/1000634446 >> [v2:172.21.15.114:6802/936316557,v1:172.21.15.114:6803/936316557] conn(0x7f8d8403e220 0x7f8d84040660 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.644+0000 7f3e6a7fc640 1 -- 172.21.15.46:0/1899872277 <== osd.0 v2:172.21.15.46:6808/3454452571 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3e64015e70 con 0x7f3e5803e270 2024-09-16T04:39:01.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.644+0000 7f8d96ffd640 1 -- 172.21.15.46:0/1000634446 <== osd.4 v2:172.21.15.114:6802/936316557 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8d84040d30 con 0x7f8d8403e220 2024-09-16T04:39:01.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.671+0000 7f2e85fd8640 10 monclient: get_monmap_and_config 2024-09-16T04:39:01.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.671+0000 7f2e85fd8640 10 monclient: build_initial_monmap 2024-09-16T04:39:01.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.671+0000 7f2e85fd8640 10 monclient: monmap: 2024-09-16T04:39:01.705 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:01.705 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.705 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:01.671900+0000 2024-09-16T04:39:01.705 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:01.671900+0000 2024-09-16T04:39:01.705 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:01.706 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.706 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:01.706 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:01.706 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:01.706 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:01.706 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:01.706 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.672+0000 7f2e85fd8640 1 Processor -- start 2024-09-16T04:39:01.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.672+0000 7f2e85fd8640 1 -- start start 2024-09-16T04:39:01.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.672+0000 7f2e85fd8640 10 monclient: init 2024-09-16T04:39:01.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.672+0000 7f2e85fd8640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:01.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.672+0000 7f2e85fd8640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-16T04:39:01.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.672+0000 7f2e85fd8640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2e800a3c10 0x7f2e8009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.672+0000 7f2e85fd8640 10 monclient(hunting): picked mon.noname-c con 0x7f2e800a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:01.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.672+0000 7f2e85fd8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e800a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:01.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.672+0000 7f2e85fd8640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2e8013be90 0x7f2e8013e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e85fd8640 10 monclient(hunting): picked mon.noname-b con 0x7f2e8013be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:01.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e85fd8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e8009ab70 con 0x7f2e800a3ff0 2024-09-16T04:39:01.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e85fd8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e85fd8640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f2e800a4d60 con 0x7f2e800a3c10 2024-09-16T04:39:01.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e85fd8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e85fd8640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f2e8009b030 con 0x7f2e8013be90 2024-09-16T04:39:01.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e85fd8640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:01.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e7ffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2e8013be90 0x7f2e8013e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e85fd8640 10 monclient(hunting): authenticate will time out at 740.088250s 2024-09-16T04:39:01.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e7ffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2e8013be90 0x7f2e8013e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:60968/0 (socket says 172.21.15.46:60968) 2024-09-16T04:39:01.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e7ffff640 1 -- 172.21.15.46:0/2407908308 learned_addr learned my addr 172.21.15.46:0/2407908308 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:01.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e7ffff640 10 monclient(hunting): get_auth_request con 0x7f2e8013be90 auth_method 0 2024-09-16T04:39:01.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e7f7fe640 1 --2- 172.21.15.46:0/2407908308 >> v2:172.21.15.114:3300/0 conn(0x7f2e800a3c10 0x7f2e8009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e7ffff640 10 monclient(hunting): handle_auth_done global_id 4244 payload 274 2024-09-16T04:39:01.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e7ffff640 1 -- 172.21.15.46:0/2407908308 >> v2:172.21.15.114:3300/0 conn(0x7f2e800a3c10 msgr2=0x7f2e8009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:01.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e7ffff640 1 --2- 172.21.15.46:0/2407908308 >> v2:172.21.15.114:3300/0 conn(0x7f2e800a3c10 0x7f2e8009bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e7ffff640 1 -- 172.21.15.46:0/2407908308 >> v1:172.21.15.114:6789/0 conn(0x7f2e800a3ff0 legacy=0x7f2e800a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e7ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:39:01.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e7ffff640 1 monclient: found mon.noname-b 2024-09-16T04:39:01.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:01.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e7ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:39:01.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.673+0000 7f2e7ffff640 1 -- 172.21.15.46:0/2407908308 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e8009b310 con 0x7f2e8013be90 2024-09-16T04:39:01.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.674+0000 7f2e7ffff640 1 --2- 172.21.15.46:0/2407908308 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2e8013be90 0x7f2e8013e280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2e78009e50 tx=0x7f2e7802a500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f003bda94b4ccf2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.674+0000 7f2e7e7fc640 1 -- 172.21.15.46:0/2407908308 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e78036070 con 0x7f2e8013be90 2024-09-16T04:39:01.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.674+0000 7f2e7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.674+0000 7f2e7e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:01.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.674+0000 7f2e7e7fc640 10 monclient: dump: 2024-09-16T04:39:01.709 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.709 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.709 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.709 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.709 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.710 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.675+0000 7f2e7e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:39:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.675+0000 7f2e7e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:39:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.675+0000 7f2e7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.675+0000 7f2e85fd8640 5 monclient: authenticate success, global_id 4244 2024-09-16T04:39:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.675+0000 7f2e85fd8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.675+0000 7f2e7e7fc640 1 -- 172.21.15.46:0/2407908308 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e78031600 con 0x7f2e8013be90 2024-09-16T04:39:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.675+0000 7f2e7e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.675+0000 7f2e85fd8640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.675+0000 7f2e85fd8640 4 set_mon_vals no callback set 2024-09-16T04:39:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.675+0000 7f2e7e7fc640 1 -- 172.21.15.46:0/2407908308 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e780319d0 con 0x7f2e8013be90 2024-09-16T04:39:01.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.675+0000 7f3e7bb0f640 1 -- 172.21.15.46:0/1899872277 --> [v2:172.21.15.46:6808/3454452571,v1:172.21.15.46:6809/3454452571] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3e7406daa0 con 0x7f3e5803e270 2024-09-16T04:39:01.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.675+0000 7f2e7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.675+0000 7f2e7e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:01.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.675+0000 7f2e7e7fc640 10 monclient: dump: 2024-09-16T04:39:01.712 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.712 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.712 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.712 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.712 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.712 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.713 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.713 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.713 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.713 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.676+0000 7f2e85fd8640 10 monclient: shutdown 2024-09-16T04:39:01.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.676+0000 7f2e85fd8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:01.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.676+0000 7f2e85fd8640 1 -- 172.21.15.46:0/2407908308 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2e8013be90 msgr2=0x7f2e8013e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.676+0000 7f2e85fd8640 1 --2- 172.21.15.46:0/2407908308 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2e8013be90 0x7f2e8013e280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2e78009e50 tx=0x7f2e7802a500 comp rx=0 tx=0).stop 2024-09-16T04:39:01.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.676+0000 7f2e85fd8640 1 -- 172.21.15.46:0/2407908308 shutdown_connections 2024-09-16T04:39:01.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.676+0000 7f2e85fd8640 1 --2- 172.21.15.46:0/2407908308 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2e8013be90 0x7f2e8013e280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.676+0000 7f2e85fd8640 1 --2- 172.21.15.46:0/2407908308 >> v2:172.21.15.114:3300/0 conn(0x7f2e800a3c10 0x7f2e8009bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.676+0000 7f2e85fd8640 1 -- 172.21.15.46:0/2407908308 >> 172.21.15.46:0/2407908308 conn(0x7f2e800a5d10 msgr2=0x7f2e800a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:01.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.676+0000 7f8da8331640 1 -- 172.21.15.46:0/1000634446 --> [v2:172.21.15.114:6802/936316557,v1:172.21.15.114:6803/936316557] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8da009bcf0 con 0x7f8d8403e220 2024-09-16T04:39:01.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.676+0000 7f3e6a7fc640 1 -- 172.21.15.46:0/1899872277 <== osd.0 v2:172.21.15.46:6808/3454452571 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3e7406daa0 con 0x7f3e5803e270 2024-09-16T04:39:01.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.676+0000 7f2e85fd8640 1 -- 172.21.15.46:0/2407908308 shutdown_connections 2024-09-16T04:39:01.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.676+0000 7f8d96ffd640 1 -- 172.21.15.46:0/1000634446 <== osd.4 v2:172.21.15.114:6802/936316557 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8da009bcf0 con 0x7f8d8403e220 2024-09-16T04:39:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.676+0000 7f2e85fd8640 1 -- 172.21.15.46:0/2407908308 wait complete. 2024-09-16T04:39:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.676+0000 7f51fe7c6640 10 monclient: get_monmap_and_config 2024-09-16T04:39:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.676+0000 7f51fe7c6640 10 monclient: build_initial_monmap 2024-09-16T04:39:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.676+0000 7f51fe7c6640 10 monclient: monmap: 2024-09-16T04:39:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:01.677254+0000 2024-09-16T04:39:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:01.677254+0000 2024-09-16T04:39:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:01.716 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.676+0000 7f3e7bb0f640 1 -- 172.21.15.46:0/1899872277 >> [v2:172.21.15.46:6808/3454452571,v1:172.21.15.46:6809/3454452571] conn(0x7f3e5803e270 msgr2=0x7f3e580406b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.676+0000 7f3e7bb0f640 1 --2- 172.21.15.46:0/1899872277 >> [v2:172.21.15.46:6808/3454452571,v1:172.21.15.46:6809/3454452571] conn(0x7f3e5803e270 0x7f3e580406b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f3e7bb0f640 1 -- 172.21.15.46:0/1899872277 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3e58038400 msgr2=0x7f3e5803a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7fc15eaa6640 10 monclient: get_monmap_and_config 2024-09-16T04:39:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7fc15eaa6640 10 monclient: build_initial_monmap 2024-09-16T04:39:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7fc15eaa6640 10 monclient: monmap: 2024-09-16T04:39:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:01.677556+0000 2024-09-16T04:39:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:01.677556+0000 2024-09-16T04:39:01.718 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:01.718 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.718 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:01.718 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:01.718 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:01.718 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:01.718 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:01.718 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f3e7bb0f640 1 --2- 172.21.15.46:0/1899872277 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3e58038400 0x7f3e5803a8c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3e700112c0 tx=0x7f3e70009190 comp rx=0 tx=0).stop 2024-09-16T04:39:01.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f3e7bb0f640 10 monclient: shutdown 2024-09-16T04:39:01.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f3e7bb0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:01.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f3e7bb0f640 1 -- 172.21.15.46:0/1899872277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e740a3ff0 msgr2=0x7f3e74137030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f3e7bb0f640 1 --2- 172.21.15.46:0/1899872277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e740a3ff0 0x7f3e74137030 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3e6400c430 tx=0x7f3e6400c8f0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f8da8331640 1 -- 172.21.15.46:0/1000634446 >> [v2:172.21.15.114:6802/936316557,v1:172.21.15.114:6803/936316557] conn(0x7f8d8403e220 msgr2=0x7f8d84040660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f8da8331640 1 --2- 172.21.15.46:0/1000634446 >> [v2:172.21.15.114:6802/936316557,v1:172.21.15.114:6803/936316557] conn(0x7f8d8403e220 0x7f8d84040660 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f3e7a0ac640 1 -- 172.21.15.46:0/1899872277 reap_dead start 2024-09-16T04:39:01.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f2e85fd8640 10 monclient: build_initial_monmap 2024-09-16T04:39:01.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f3e7bb0f640 1 -- 172.21.15.46:0/1899872277 shutdown_connections 2024-09-16T04:39:01.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f8da8331640 1 -- 172.21.15.46:0/1000634446 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8d840383b0 msgr2=0x7f8d8403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f3e7bb0f640 1 -- 172.21.15.46:0/1899872277 >> 172.21.15.46:0/1899872277 conn(0x7f3e740a5d10 msgr2=0x7f3e740a2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:01.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f2e85fd8640 10 monclient: monmap: 2024-09-16T04:39:01.721 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:01.721 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:01.721 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:01.721 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:01.721 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:01.721 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.721 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:01.722 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:01.722 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:01.722 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f8da8331640 1 --2- 172.21.15.46:0/1000634446 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8d840383b0 0x7f8d8403a870 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8da009fcd0 tx=0x7f8d90008040 comp rx=0 tx=0).stop 2024-09-16T04:39:01.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f8da8331640 10 monclient: shutdown 2024-09-16T04:39:01.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f8da8331640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:01.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f8da8331640 1 -- 172.21.15.46:0/1000634446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da00a3c10 msgr2=0x7f8da0131f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f8da8331640 1 --2- 172.21.15.46:0/1000634446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da00a3c10 0x7f8da0131f70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c032960 tx=0x7f8d9c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f8da68ce640 1 -- 172.21.15.46:0/1000634446 reap_dead start 2024-09-16T04:39:01.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f3e7bb0f640 1 -- 172.21.15.46:0/1899872277 shutdown_connections 2024-09-16T04:39:01.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f84ac359640 10 monclient: get_monmap_and_config 2024-09-16T04:39:01.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f84ac359640 10 monclient: build_initial_monmap 2024-09-16T04:39:01.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f84ac359640 10 monclient: monmap: 2024-09-16T04:39:01.723 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:01.723 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.723 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:01.678069+0000 2024-09-16T04:39:01.723 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:01.678069+0000 2024-09-16T04:39:01.723 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:01.724 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f8da8331640 1 -- 172.21.15.46:0/1000634446 shutdown_connections 2024-09-16T04:39:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f8da8331640 1 -- 172.21.15.46:0/1000634446 >> 172.21.15.46:0/1000634446 conn(0x7f8da00a5d10 msgr2=0x7f8da0135e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f3e7bb0f640 1 -- 172.21.15.46:0/1899872277 wait complete. 2024-09-16T04:39:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f8da8331640 1 -- 172.21.15.46:0/1000634446 shutdown_connections 2024-09-16T04:39:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f8da8331640 1 -- 172.21.15.46:0/1000634446 wait complete. 2024-09-16T04:39:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.677+0000 7f51fe7c6640 1 Processor -- start 2024-09-16T04:39:01.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.678+0000 7f51fe7c6640 1 -- start start 2024-09-16T04:39:01.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.678+0000 7f51fe7c6640 10 monclient: init 2024-09-16T04:39:01.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.678+0000 7f2e85fd8640 1 Processor -- start 2024-09-16T04:39:01.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.678+0000 7fc15eaa6640 1 Processor -- start 2024-09-16T04:39:01.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.678+0000 7fc15eaa6640 1 -- start start 2024-09-16T04:39:01.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.678+0000 7fc15eaa6640 10 monclient: init 2024-09-16T04:39:01.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.678+0000 7f84ac359640 1 Processor -- start 2024-09-16T04:39:01.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.678+0000 7f2e85fd8640 1 -- start start 2024-09-16T04:39:01.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.678+0000 7f2e85fd8640 10 monclient: init 2024-09-16T04:39:01.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.678+0000 7f51fe7c6640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:01.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.678+0000 7f51fe7c6640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-16T04:39:01.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.678+0000 7f51fe7c6640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f51f80a3c10 0x7f51f809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.678+0000 7f51fe7c6640 10 monclient(hunting): picked mon.noname-c con 0x7f51f80a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:01.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.678+0000 7f51fe7c6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f51f80a3ff0 0x7f51f80a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.678+0000 7f84ac359640 1 -- start start 2024-09-16T04:39:01.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.678+0000 7f84ac359640 10 monclient: init 2024-09-16T04:39:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.678+0000 7f51fe7c6640 10 monclient(hunting): picked mon.noname-b con 0x7f51f80a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51fe7c6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f51f81369d0 0x7f51f8136e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51fe7c6640 10 monclient(hunting): picked mon.noname-a con 0x7f51f81369d0 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51fe7c6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51fe7c6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f51f809ab70 con 0x7f51f81369d0 2024-09-16T04:39:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51f77fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f51f80a3ff0 0x7f51f80a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51fcd63640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f51f81369d0 0x7f51f8136e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51f77fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f51f80a3ff0 0x7f51f80a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:60972/0 (socket says 172.21.15.46:60972) 2024-09-16T04:39:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51f77fe640 1 -- 172.21.15.46:0/1655353592 learned_addr learned my addr 172.21.15.46:0/1655353592 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51fe7c6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51fe7c6640 1 -- 172.21.15.46:0/1655353592 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f51f80a4d60 con 0x7f51f80a3c10 2024-09-16T04:39:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51fe7c6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51fe7c6640 1 -- 172.21.15.46:0/1655353592 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f51f809b030 con 0x7f51f80a3ff0 2024-09-16T04:39:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51fe7c6640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51fe7c6640 10 monclient(hunting): authenticate will time out at 740.094321s 2024-09-16T04:39:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51f77fe640 10 monclient(hunting): get_auth_request con 0x7f51f80a3ff0 auth_method 0 2024-09-16T04:39:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51fcd63640 10 monclient(hunting): get_auth_request con 0x7f51f81369d0 auth_method 0 2024-09-16T04:39:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51fcd63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51fcd63640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51fcd63640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51f7fff640 1 --2- 172.21.15.46:0/1655353592 >> v2:172.21.15.114:3300/0 conn(0x7f51f80a3c10 0x7f51f809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51f7fff640 10 monclient(hunting): get_auth_request con 0x7f51f80a3c10 auth_method 0 2024-09-16T04:39:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51fcd63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51fcd63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51fcd63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15eaa6640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:01.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15eaa6640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-16T04:39:01.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15eaa6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc15813b1a0 0x7fc15809afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15eaa6640 10 monclient(hunting): picked mon.noname-a con 0x7fc15813b1a0 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:01.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15eaa6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc15813b580 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:01.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15eaa6640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc1580a4c20 0x7fc1581423f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15eaa6640 10 monclient(hunting): picked mon.noname-c con 0x7fc1580a4c20 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:01.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15eaa6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1580a58d0 con 0x7fc15813b580 2024-09-16T04:39:01.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15eaa6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15eaa6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc15809bd90 con 0x7fc15813b1a0 2024-09-16T04:39:01.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15eaa6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15eaa6640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fc15809c070 con 0x7fc1580a4c20 2024-09-16T04:39:01.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15eaa6640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:01.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15eaa6640 10 monclient(hunting): authenticate will time out at 740.094293s 2024-09-16T04:39:01.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15d043640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc1580a4c20 0x7fc1581423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15d043640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc1580a4c20 0x7fc1581423f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.46:38234/0 (socket says 172.21.15.46:38234) 2024-09-16T04:39:01.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15d043640 1 -- 172.21.15.46:0/3637207044 learned_addr learned my addr 172.21.15.46:0/3637207044 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:01.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15d043640 10 monclient(hunting): get_auth_request con 0x7fc1580a4c20 auth_method 0 2024-09-16T04:39:01.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15d043640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15d043640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15d043640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15c842640 1 --2- 172.21.15.46:0/3637207044 >> v2:172.21.15.46:3300/0 conn(0x7fc15813b1a0 0x7fc15809afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15c842640 10 monclient(hunting): get_auth_request con 0x7fc15813b1a0 auth_method 0 2024-09-16T04:39:01.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15c842640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15c842640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15c842640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15d043640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15d043640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15d043640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15c842640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15c842640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15c842640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7fc15d043640 10 monclient(hunting): handle_auth_done global_id 4282 payload 274 2024-09-16T04:39:01.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84ac359640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:01.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e85fd8640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:01.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e85fd8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:39:01.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e85fd8640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2e800a3c10 0x7f2e80136770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e7f7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2e800a3c10 0x7f2e80136770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e7f7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2e800a3c10 0x7f2e80136770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:60984/0 (socket says 172.21.15.46:60984) 2024-09-16T04:39:01.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e7f7fe640 1 -- 172.21.15.46:0/1846870334 learned_addr learned my addr 172.21.15.46:0/1846870334 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:01.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e85fd8640 10 monclient(hunting): picked mon.noname-c con 0x7f2e800a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:01.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e85fd8640 1 --2- 172.21.15.46:0/1846870334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e800a3ff0 0x7f2e80136cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e85fd8640 10 monclient(hunting): picked mon.noname-b con 0x7f2e800a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:01.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e85fd8640 1 --2- 172.21.15.46:0/1846870334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e8013be90 0x7f2e80139c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e7effd640 1 --2- 172.21.15.46:0/1846870334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e800a3ff0 0x7f2e80136cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e85fd8640 10 monclient(hunting): picked mon.noname-a con 0x7f2e8013be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:01.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e85fd8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e85fd8640 1 -- 172.21.15.46:0/1846870334 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8006e550 con 0x7f2e8013be90 2024-09-16T04:39:01.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7fc15d043640 1 -- 172.21.15.46:0/3637207044 >> v2:172.21.15.46:3300/0 conn(0x7fc15813b1a0 msgr2=0x7fc15809afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:01.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7fc15d043640 1 --2- 172.21.15.46:0/3637207044 >> v2:172.21.15.46:3300/0 conn(0x7fc15813b1a0 0x7fc15809afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84ac359640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-16T04:39:01.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e85fd8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e85fd8640 1 -- 172.21.15.46:0/1846870334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8006e3d0 con 0x7f2e800a3ff0 2024-09-16T04:39:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e85fd8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e85fd8640 1 -- 172.21.15.46:0/1846870334 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f2e8013e7c0 con 0x7f2e800a3c10 2024-09-16T04:39:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e85fd8640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e85fd8640 10 monclient(hunting): authenticate will time out at 740.094847s 2024-09-16T04:39:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e7f7fe640 10 monclient(hunting): get_auth_request con 0x7f2e800a3c10 auth_method 0 2024-09-16T04:39:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e7effd640 10 monclient(hunting): get_auth_request con 0x7f2e800a3ff0 auth_method 0 2024-09-16T04:39:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e7effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f2e7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84ac359640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f84a40a29b0 0x7f84a40734a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f2e7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7fc15d043640 1 -- 172.21.15.46:0/3637207044 >> v1:172.21.15.114:6789/0 conn(0x7fc15813b580 legacy=0x7fc15813b990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:39:01.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7fc15d043640 10 monclient: _finish_hunting 0 2024-09-16T04:39:01.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7fc15d043640 1 monclient: found mon.noname-c 2024-09-16T04:39:01.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84ac359640 10 monclient(hunting): picked mon.noname-b con 0x7f84a40a29b0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:01.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7fc15d043640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:01.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84ac359640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f84a40739e0 0x7f84a406cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7fc15d043640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:39:01.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84ac359640 10 monclient(hunting): picked mon.noname-a con 0x7f84a40739e0 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:01.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7fc15d043640 1 -- 172.21.15.46:0/3637207044 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc158143940 con 0x7fc1580a4c20 2024-09-16T04:39:01.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84ac359640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84a4073de0 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:01.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f2e7ffff640 1 --2- 172.21.15.46:0/1846870334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e8013be90 0x7f2e80139c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f2e7ffff640 10 monclient(hunting): get_auth_request con 0x7f2e8013be90 auth_method 0 2024-09-16T04:39:01.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f2e7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f2e7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f2e7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84aa0f5640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f84a40a29b0 0x7f84a40734a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84aa0f5640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f84a40a29b0 0x7f84a40734a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:60996/0 (socket says 172.21.15.46:60996) 2024-09-16T04:39:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84aa0f5640 1 -- 172.21.15.46:0/2835082602 learned_addr learned my addr 172.21.15.46:0/2835082602 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.679+0000 7f51f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f51f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f51f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f51f77fe640 10 monclient(hunting): handle_auth_done global_id 4247 payload 274 2024-09-16T04:39:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f51f77fe640 1 -- 172.21.15.46:0/1655353592 >> v2:172.21.15.114:3300/0 conn(0x7f51f80a3c10 msgr2=0x7f51f809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f51f77fe640 1 --2- 172.21.15.46:0/1655353592 >> v2:172.21.15.114:3300/0 conn(0x7f51f80a3c10 0x7f51f809bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f51f77fe640 1 -- 172.21.15.46:0/1655353592 >> v2:172.21.15.46:3300/0 conn(0x7f51f81369d0 msgr2=0x7f51f8136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:01.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f2e7effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f2e7effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f2e7effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f2e7f7fe640 10 monclient(hunting): handle_auth_done global_id 4250 payload 824 2024-09-16T04:39:01.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84ac359640 1 -- 172.21.15.46:0/2835082602 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84a40a34b0 con 0x7f84a4073de0 2024-09-16T04:39:01.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f51f77fe640 1 --2- 172.21.15.46:0/1655353592 >> v2:172.21.15.46:3300/0 conn(0x7f51f81369d0 0x7f51f8136e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f51f77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:39:01.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f51f77fe640 1 monclient: found mon.noname-b 2024-09-16T04:39:01.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f51f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:01.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f51f77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:39:01.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f51f77fe640 1 -- 172.21.15.46:0/1655353592 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f809b310 con 0x7f51f80a3ff0 2024-09-16T04:39:01.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84ac359640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84ac359640 1 -- 172.21.15.46:0/2835082602 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f84a406dec0 con 0x7f84a40739e0 2024-09-16T04:39:01.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84ac359640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84ac359640 1 -- 172.21.15.46:0/2835082602 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f84a40a2d90 con 0x7f84a40a29b0 2024-09-16T04:39:01.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84ac359640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:01.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84ac359640 10 monclient(hunting): authenticate will time out at 740.095680s 2024-09-16T04:39:01.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84aa0f5640 10 monclient(hunting): get_auth_request con 0x7f84a40a29b0 auth_method 0 2024-09-16T04:39:01.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84aa0f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84aa0f5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84aa0f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84aa0f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84aa0f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f2e7f7fe640 1 -- 172.21.15.46:0/1846870334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e800a3ff0 msgr2=0x7f2e80136cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f2e7f7fe640 1 --2- 172.21.15.46:0/1846870334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e800a3ff0 0x7f2e80136cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f2e7f7fe640 1 -- 172.21.15.46:0/1846870334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e8013be90 msgr2=0x7f2e80139c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f2e7f7fe640 1 --2- 172.21.15.46:0/1846870334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e8013be90 0x7f2e80139c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f2e7f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:39:01.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f2e7f7fe640 1 monclient: found mon.noname-c 2024-09-16T04:39:01.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f2e7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:01.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f2e7f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:39:01.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f2e7f7fe640 1 -- 172.21.15.46:0/1846870334 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e800a0400 con 0x7f2e800a3c10 2024-09-16T04:39:01.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f84aa0f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f84a98f4640 1 --2- 172.21.15.46:0/2835082602 >> v2:172.21.15.46:3300/0 conn(0x7f84a40739e0 0x7f84a406cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f84aa0f5640 10 monclient(hunting): handle_auth_done global_id 4253 payload 274 2024-09-16T04:39:01.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f84aa0f5640 1 -- 172.21.15.46:0/2835082602 >> v2:172.21.15.46:3300/0 conn(0x7f84a40739e0 msgr2=0x7f84a406cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:01.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f84aa0f5640 1 --2- 172.21.15.46:0/2835082602 >> v2:172.21.15.46:3300/0 conn(0x7f84a40739e0 0x7f84a406cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f84aa0f5640 1 -- 172.21.15.46:0/2835082602 >> v1:172.21.15.46:6789/0 conn(0x7f84a4073de0 legacy=0x7f84a406d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f84aa0f5640 10 monclient: _finish_hunting 0 2024-09-16T04:39:01.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.680+0000 7f2e7f7fe640 1 --2- 172.21.15.46:0/1846870334 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2e800a3c10 0x7f2e80136770 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2e7000c960 tx=0x7f2e7000ce20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f84aa0f5640 1 monclient: found mon.noname-b 2024-09-16T04:39:01.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f84aa0f5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:01.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f84aa0f5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:39:01.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f84aa0f5640 1 -- 172.21.15.46:0/2835082602 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84a409e4a0 con 0x7f84a40a29b0 2024-09-16T04:39:01.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f2e67fff640 1 -- 172.21.15.46:0/1846870334 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e70015070 con 0x7f2e800a3c10 2024-09-16T04:39:01.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f2e67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f2e67fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:39:01.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f2e67fff640 10 monclient: dump: 2024-09-16T04:39:01.749 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.749 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.749 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.749 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.749 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.749 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.750 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.750 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.750 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.750 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f84a98f4640 10 monclient: get_auth_request con 0x7f84a40739e0 auth_method 0 2024-09-16T04:39:01.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f84a98f4640 1 --2- 172.21.15.46:0/2835082602 >> v2:172.21.15.46:3300/0 conn(0x7f84a40739e0 0x7f84a406cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:39:01.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f2e67fff640 10 monclient: _finish_auth 0 2024-09-16T04:39:01.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f2e67fff640 10 monclient: _check_auth_tickets 2024-09-16T04:39:01.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f2e67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:01.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f2e67fff640 1 -- 172.21.15.46:0/1846870334 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e70004240 con 0x7f2e800a3c10 2024-09-16T04:39:01.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f2e67fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:01.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f2e67fff640 1 -- 172.21.15.46:0/1846870334 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e700076e0 con 0x7f2e800a3c10 2024-09-16T04:39:01.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f2e67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f2e67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:01.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f2e67fff640 10 monclient: dump: 2024-09-16T04:39:01.752 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.752 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.752 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.752 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.752 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.752 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.752 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.752 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.752 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.752 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f2e85fd8640 5 monclient: authenticate success, global_id 4250 2024-09-16T04:39:01.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f2e85fd8640 10 monclient: _renew_subs 2024-09-16T04:39:01.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f2e85fd8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:01.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f2e85fd8640 1 -- 172.21.15.46:0/1846870334 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e8013c490 con 0x7f2e800a3c10 2024-09-16T04:39:01.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f51f77fe640 1 --2- 172.21.15.46:0/1655353592 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f51f80a3ff0 0x7f51f80a43d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f51f806be80 tx=0x7f51ec02a160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=153fbd3ec24448a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7f2e7dffb640 4 set_mon_vals no callback set 2024-09-16T04:39:01.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.681+0000 7fc15d043640 1 --2- 172.21.15.46:0/3637207044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1580a4c20 0x7fc1581423f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc1580a57d0 tx=0x7fc15002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e4a6c9e571aef3a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7fc1577fe640 1 -- 172.21.15.46:0/3637207044 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc150037070 con 0x7fc1580a4c20 2024-09-16T04:39:01.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7fc1577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7fc1577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:39:01.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7fc1577fe640 10 monclient: dump: 2024-09-16T04:39:01.755 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.755 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.755 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.755 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.755 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.755 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.755 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.756 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.756 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.756 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7fc1577fe640 10 monclient: _finish_auth 0 2024-09-16T04:39:01.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7fc1577fe640 10 monclient: _check_auth_tickets 2024-09-16T04:39:01.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7fc1577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:01.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7fc1577fe640 1 -- 172.21.15.46:0/3637207044 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc150032440 con 0x7fc1580a4c20 2024-09-16T04:39:01.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7fc1577fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:01.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7fc1577fe640 1 -- 172.21.15.46:0/3637207044 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc150032810 con 0x7fc1580a4c20 2024-09-16T04:39:01.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7fc1577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7fc1577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:01.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7fc1577fe640 10 monclient: dump: 2024-09-16T04:39:01.756 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.756 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.757 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.757 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.757 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.757 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.757 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.757 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.757 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.757 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f51f6ffd640 1 -- 172.21.15.46:0/1655353592 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec037070 con 0x7f51f80a3ff0 2024-09-16T04:39:01.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f51f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f51f6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:01.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f51f6ffd640 10 monclient: dump: 2024-09-16T04:39:01.758 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.758 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.758 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.758 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.758 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.758 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.758 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.759 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.759 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.759 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7fc15eaa6640 5 monclient: authenticate success, global_id 4282 2024-09-16T04:39:01.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f51f6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:39:01.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f51f6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:39:01.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f51f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:01.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f51f6ffd640 1 -- 172.21.15.46:0/1655353592 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51ec032440 con 0x7f51f80a3ff0 2024-09-16T04:39:01.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f51f6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:01.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f51f6ffd640 1 -- 172.21.15.46:0/1655353592 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec032850 con 0x7f51f80a3ff0 2024-09-16T04:39:01.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f51f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f51f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:01.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f51f6ffd640 10 monclient: dump: 2024-09-16T04:39:01.760 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.760 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.760 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.760 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.760 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.760 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.760 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.760 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.760 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.760 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f84aa0f5640 1 --2- 172.21.15.46:0/2835082602 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f84a40a29b0 0x7f84a40734a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f849c009e50 tx=0x7f849c02a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3be44c9d23605f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f2e85fd8640 10 monclient: _renew_subs 2024-09-16T04:39:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f2e85fd8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f2e85fd8640 1 -- 172.21.15.46:0/1846870334 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e80121040 con 0x7f2e800a3c10 2024-09-16T04:39:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f51fe7c6640 5 monclient: authenticate success, global_id 4247 2024-09-16T04:39:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f51fe7c6640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f51fe7c6640 4 set_mon_vals no callback set 2024-09-16T04:39:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7fc15eaa6640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7fc15eaa6640 4 set_mon_vals no callback set 2024-09-16T04:39:01.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f84a90f3640 1 -- 172.21.15.46:0/2835082602 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f849c036070 con 0x7f84a40a29b0 2024-09-16T04:39:01.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f84a90f3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f84a90f3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:01.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f84a90f3640 10 monclient: dump: 2024-09-16T04:39:01.763 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.763 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.763 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.764 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.764 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.764 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.764 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.764 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.764 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.764 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f84a90f3640 10 monclient: _finish_auth 0 2024-09-16T04:39:01.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f84a90f3640 10 monclient: _check_auth_tickets 2024-09-16T04:39:01.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f84a90f3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:01.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f84a90f3640 1 -- 172.21.15.46:0/2835082602 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f849c031600 con 0x7f84a40a29b0 2024-09-16T04:39:01.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f84ac359640 5 monclient: authenticate success, global_id 4253 2024-09-16T04:39:01.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.682+0000 7f84ac359640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:01.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f84a90f3640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:01.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f84a90f3640 1 -- 172.21.15.46:0/2835082602 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f849c036070 con 0x7f84a40a29b0 2024-09-16T04:39:01.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f84a90f3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f84a90f3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:01.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f84a90f3640 10 monclient: dump: 2024-09-16T04:39:01.765 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.765 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.765 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.765 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.765 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.765 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.765 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.765 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.766 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.766 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f84ac359640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:01.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f84ac359640 4 set_mon_vals no callback set 2024-09-16T04:39:01.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f2e67fff640 1 -- 172.21.15.46:0/1846870334 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f2e70007880 con 0x7f2e800a3c10 2024-09-16T04:39:01.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7fc15eaa6640 10 monclient: shutdown 2024-09-16T04:39:01.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7fc15eaa6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:01.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7fc15eaa6640 1 -- 172.21.15.46:0/3637207044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1580a4c20 msgr2=0x7fc1581423f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7fc15eaa6640 1 --2- 172.21.15.46:0/3637207044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1580a4c20 0x7fc1581423f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc1580a57d0 tx=0x7fc15002a1a0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f51fe7c6640 10 monclient: shutdown 2024-09-16T04:39:01.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f2e85fd8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:01.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f2e85fd8640 1 -- 172.21.15.46:0/1846870334 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2e801213e0 con 0x7f2e800a3c10 2024-09-16T04:39:01.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f2e67fff640 1 --2- 172.21.15.46:0/1846870334 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2e6c038400 0x7f2e6c03a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7fc15eaa6640 1 -- 172.21.15.46:0/3637207044 shutdown_connections 2024-09-16T04:39:01.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7fc15eaa6640 1 --2- 172.21.15.46:0/3637207044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1580a4c20 0x7fc1581423f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f51fe7c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:01.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f51fe7c6640 1 -- 172.21.15.46:0/1655353592 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f51f80a3ff0 msgr2=0x7f51f80a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7fc15eaa6640 1 --2- 172.21.15.46:0/3637207044 >> v2:172.21.15.46:3300/0 conn(0x7fc15813b1a0 0x7fc15809afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7fc15eaa6640 1 -- 172.21.15.46:0/3637207044 >> 172.21.15.46:0/3637207044 conn(0x7fc1580a5d10 msgr2=0x7fc1580a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:01.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f51fe7c6640 1 --2- 172.21.15.46:0/1655353592 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f51f80a3ff0 0x7f51f80a43d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f51f806be80 tx=0x7f51ec02a160 comp rx=0 tx=0).stop 2024-09-16T04:39:01.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f51fe7c6640 1 -- 172.21.15.46:0/1655353592 shutdown_connections 2024-09-16T04:39:01.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f51fe7c6640 1 --2- 172.21.15.46:0/1655353592 >> v2:172.21.15.46:3300/0 conn(0x7f51f81369d0 0x7f51f8136e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f51fe7c6640 1 --2- 172.21.15.46:0/1655353592 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f51f80a3ff0 0x7f51f80a43d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f51fe7c6640 1 --2- 172.21.15.46:0/1655353592 >> v2:172.21.15.114:3300/0 conn(0x7f51f80a3c10 0x7f51f809bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f84ac359640 10 monclient: shutdown 2024-09-16T04:39:01.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f84ac359640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:01.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f51fe7c6640 1 -- 172.21.15.46:0/1655353592 >> 172.21.15.46:0/1655353592 conn(0x7f51f80a5d10 msgr2=0x7f51f80a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:01.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f84ac359640 1 -- 172.21.15.46:0/2835082602 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f84a40a29b0 msgr2=0x7f84a40734a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.683+0000 7f84ac359640 1 --2- 172.21.15.46:0/2835082602 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f84a40a29b0 0x7f84a40734a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f849c009e50 tx=0x7f849c02a5f0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.684+0000 7f51fe7c6640 1 -- 172.21.15.46:0/1655353592 shutdown_connections 2024-09-16T04:39:01.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.684+0000 7f84ac359640 1 -- 172.21.15.46:0/2835082602 shutdown_connections 2024-09-16T04:39:01.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.684+0000 7f84ac359640 1 --2- 172.21.15.46:0/2835082602 >> v2:172.21.15.46:3300/0 conn(0x7f84a40739e0 0x7f84a406cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.684+0000 7f84ac359640 1 --2- 172.21.15.46:0/2835082602 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f84a40a29b0 0x7f84a40734a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.684+0000 7f84ac359640 1 -- 172.21.15.46:0/2835082602 >> 172.21.15.46:0/2835082602 conn(0x7f84a40a3aa0 msgr2=0x7f84a40a3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:01.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.684+0000 7f51fe7c6640 1 -- 172.21.15.46:0/1655353592 wait complete. 2024-09-16T04:39:01.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.684+0000 7f2e67fff640 1 -- 172.21.15.46:0/1846870334 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f2e70010b80 con 0x7f2e800a3c10 2024-09-16T04:39:01.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.684+0000 7f84ac359640 1 -- 172.21.15.46:0/2835082602 shutdown_connections 2024-09-16T04:39:01.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.684+0000 7f84ac359640 1 -- 172.21.15.46:0/2835082602 wait complete. 2024-09-16T04:39:01.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.684+0000 7f2e67fff640 1 --2- 172.21.15.46:0/1846870334 >> [v2:172.21.15.114:6810/890769734,v1:172.21.15.114:6811/890769734] conn(0x7f2e6c03e270 0x7f2e6c0406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.684+0000 7f2e67fff640 1 -- 172.21.15.46:0/1846870334 --> [v2:172.21.15.114:6810/890769734,v1:172.21.15.114:6811/890769734] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2e70003e70 con 0x7f2e6c03e270 2024-09-16T04:39:01.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.684+0000 7f2e67fff640 1 -- 172.21.15.46:0/1846870334 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f2e70050060 con 0x7f2e800a3c10 2024-09-16T04:39:01.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.684+0000 7f2e67fff640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-16T04:39:01.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.684+0000 7f2e7effd640 1 --2- 172.21.15.46:0/1846870334 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2e6c038400 0x7f2e6c03a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.684+0000 7f2e7effd640 10 monclient: get_auth_request con 0x7f2e6c038400 auth_method 0 2024-09-16T04:39:01.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.684+0000 7f2e7ffff640 1 --2- 172.21.15.46:0/1846870334 >> [v2:172.21.15.114:6810/890769734,v1:172.21.15.114:6811/890769734] conn(0x7f2e6c03e270 0x7f2e6c0406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.684+0000 7f2e7ffff640 10 monclient: get_auth_request con 0x7f2e6c03e270 auth_method 0 2024-09-16T04:39:01.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.684+0000 7fc15eaa6640 1 -- 172.21.15.46:0/3637207044 shutdown_connections 2024-09-16T04:39:01.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.684+0000 7fc15eaa6640 1 -- 172.21.15.46:0/3637207044 wait complete. 2024-09-16T04:39:01.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.685+0000 7f51fe7c6640 10 monclient: build_initial_monmap 2024-09-16T04:39:01.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.685+0000 7f51fe7c6640 10 monclient: monmap: 2024-09-16T04:39:01.774 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:01.774 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:01.774 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:01.774 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:01.774 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:01.775 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.775 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:01.775 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:01.775 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:01.775 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.685+0000 7f84ac359640 10 monclient: build_initial_monmap 2024-09-16T04:39:01.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.685+0000 7f84ac359640 10 monclient: monmap: 2024-09-16T04:39:01.775 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:01.775 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:01.775 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:01.775 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:01.775 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:01.776 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.776 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:01.776 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:01.776 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:01.776 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.685+0000 7f2e7ffff640 1 --2- 172.21.15.46:0/1846870334 >> [v2:172.21.15.114:6810/890769734,v1:172.21.15.114:6811/890769734] conn(0x7f2e6c03e270 0x7f2e6c0406b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.685+0000 7f2e7effd640 1 --2- 172.21.15.46:0/1846870334 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2e6c038400 0x7f2e6c03a8c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2e8013b3d0 tx=0x7f2e74009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.685+0000 7f51fe7c6640 1 Processor -- start 2024-09-16T04:39:01.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.685+0000 7fc15eaa6640 10 monclient: build_initial_monmap 2024-09-16T04:39:01.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.685+0000 7fc15eaa6640 10 monclient: monmap: 2024-09-16T04:39:01.777 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:01.777 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:01.777 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:01.777 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:01.777 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:01.777 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.777 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:01.777 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:01.778 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:01.778 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.685+0000 7f84ac359640 1 Processor -- start 2024-09-16T04:39:01.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.685+0000 7f84ac359640 1 -- start start 2024-09-16T04:39:01.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.685+0000 7f84ac359640 10 monclient: init 2024-09-16T04:39:01.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.685+0000 7f51fe7c6640 1 -- start start 2024-09-16T04:39:01.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.685+0000 7f51fe7c6640 10 monclient: init 2024-09-16T04:39:01.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.685+0000 7f2e67fff640 1 -- 172.21.15.46:0/1846870334 <== osd.3 v2:172.21.15.114:6810/890769734 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2e70003e70 con 0x7f2e6c03e270 2024-09-16T04:39:01.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7fc15eaa6640 1 Processor -- start 2024-09-16T04:39:01.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fe7c6640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:01.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fe7c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:39:01.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fe7c6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51f80a3c10 0x7f51f81cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fe7c6640 10 monclient(hunting): picked mon.noname-b con 0x7f51f80a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:01.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fe7c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f80a3ff0 0x7f51f81cc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fe7c6640 10 monclient(hunting): picked mon.noname-a con 0x7f51f80a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:01.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fe7c6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f51f81369d0 0x7f51f81231f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fe7c6640 10 monclient(hunting): picked mon.noname-c con 0x7f51f81369d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:01.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fe7c6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fe7c6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f51f80a4910 con 0x7f51f80a3ff0 2024-09-16T04:39:01.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fe7c6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fe7c6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f51f806e3d0 con 0x7f51f80a3c10 2024-09-16T04:39:01.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fe7c6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fe7c6640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f51f8127ac0 con 0x7f51f81369d0 2024-09-16T04:39:01.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fe7c6640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:01.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51f77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f80a3ff0 0x7f51f81cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fe7c6640 10 monclient(hunting): authenticate will time out at 740.101671s 2024-09-16T04:39:01.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51f77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f80a3ff0 0x7f51f81cc490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59272/0 (socket says 172.21.15.46:59272) 2024-09-16T04:39:01.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51f77fe640 1 -- 172.21.15.46:0/1398851665 learned_addr learned my addr 172.21.15.46:0/1398851665 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:01.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51f77fe640 10 monclient(hunting): get_auth_request con 0x7f51f80a3ff0 auth_method 0 2024-09-16T04:39:01.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fcd63640 1 --2- 172.21.15.46:0/1398851665 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f51f81369d0 0x7f51f81231f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7fc15eaa6640 1 -- start start 2024-09-16T04:39:01.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7fc15eaa6640 10 monclient: init 2024-09-16T04:39:01.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fcd63640 10 monclient(hunting): get_auth_request con 0x7f51f81369d0 auth_method 0 2024-09-16T04:39:01.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fcd63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fcd63640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fcd63640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fcd63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fcd63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.686+0000 7f51fcd63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f51f7fff640 1 --2- 172.21.15.46:0/1398851665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51f80a3c10 0x7f51f81cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f51f7fff640 10 monclient(hunting): get_auth_request con 0x7f51f80a3c10 auth_method 0 2024-09-16T04:39:01.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f51f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f51f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f51f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15eaa6640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:01.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15eaa6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:39:01.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15eaa6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc1580a4c20 0x7fc158071a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15eaa6640 10 monclient(hunting): picked mon.noname-c con 0x7fc1580a4c20 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:01.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15eaa6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc15813b1a0 0x7fc158071f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15eaa6640 10 monclient(hunting): picked mon.noname-a con 0x7fc15813b1a0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:01.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15eaa6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc15813b580 0x7fc1580724d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15eaa6640 10 monclient(hunting): picked mon.noname-b con 0x7fc15813b580 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:01.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15eaa6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15eaa6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc158142cc0 con 0x7fc15813b1a0 2024-09-16T04:39:01.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15eaa6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15eaa6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc158142b40 con 0x7fc15813b580 2024-09-16T04:39:01.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15eaa6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15eaa6640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fc158142e40 con 0x7fc1580a4c20 2024-09-16T04:39:01.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15eaa6640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:01.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f51f77fe640 10 monclient(hunting): handle_auth_done global_id 4281 payload 824 2024-09-16T04:39:01.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f51f77fe640 1 -- 172.21.15.46:0/1398851665 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f51f81369d0 msgr2=0x7f51f81231f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15eaa6640 10 monclient(hunting): authenticate will time out at 740.102249s 2024-09-16T04:39:01.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84ac359640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:01.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84ac359640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:39:01.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84ac359640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84a40739e0 0x7f84a41cc860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84ac359640 10 monclient(hunting): picked mon.noname-a con 0x7f84a40739e0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:01.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84ac359640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84a4073de0 0x7f84a41c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84ac359640 10 monclient(hunting): picked mon.noname-b con 0x7f84a4073de0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:01.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84ac359640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f84a40a29b0 0x7f84a41cfda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84ac359640 10 monclient(hunting): picked mon.noname-c con 0x7f84a40a29b0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:01.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84ac359640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84ac359640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f84a409f120 con 0x7f84a40739e0 2024-09-16T04:39:01.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84ac359640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84ac359640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f84a40a36f0 con 0x7f84a4073de0 2024-09-16T04:39:01.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84ac359640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84ac359640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f84a409db30 con 0x7f84a40a29b0 2024-09-16T04:39:01.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84ac359640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:01.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84ac359640 10 monclient(hunting): authenticate will time out at 740.102516s 2024-09-16T04:39:01.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f51f77fe640 1 --2- 172.21.15.46:0/1398851665 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f51f81369d0 0x7f51f81231f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f51f77fe640 1 -- 172.21.15.46:0/1398851665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51f80a3c10 msgr2=0x7f51f81cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f51f77fe640 1 --2- 172.21.15.46:0/1398851665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51f80a3c10 0x7f51f81cbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f51f77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:39:01.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f51f77fe640 1 monclient: found mon.noname-a 2024-09-16T04:39:01.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f51f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:01.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f51f77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:01.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f51f77fe640 1 -- 172.21.15.46:0/1398851665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f8125950 con 0x7f51f80a3ff0 2024-09-16T04:39:01.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f51fcd63640 1 --2- 172.21.15.46:0/1398851665 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f51f81369d0 0x7f51f81231f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:01.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15d043640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc15813b580 0x7fc1580724d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15d043640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc15813b580 0x7fc1580724d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.46:38262/0 (socket says 172.21.15.46:38262) 2024-09-16T04:39:01.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15d043640 1 -- 172.21.15.46:0/2228195892 learned_addr learned my addr 172.21.15.46:0/2228195892 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:01.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15d043640 10 monclient(hunting): get_auth_request con 0x7fc15813b580 auth_method 0 2024-09-16T04:39:01.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15d043640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15d043640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15d043640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84aa0f5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84a40739e0 0x7f84a41cc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc157fff640 1 --2- 172.21.15.46:0/2228195892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc15813b1a0 0x7fc158071f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84aa0f5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84a40739e0 0x7f84a41cc860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59296/0 (socket says 172.21.15.46:59296) 2024-09-16T04:39:01.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84aa0f5640 1 -- 172.21.15.46:0/4238002721 learned_addr learned my addr 172.21.15.46:0/4238002721 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:01.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84aa0f5640 10 monclient(hunting): get_auth_request con 0x7f84a40739e0 auth_method 0 2024-09-16T04:39:01.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84aa0f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84aa0f5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84aa0f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84a98f4640 1 --2- 172.21.15.46:0/4238002721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84a4073de0 0x7f84a41c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15d043640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15d043640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15d043640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc157fff640 10 monclient(hunting): get_auth_request con 0x7fc15813b1a0 auth_method 0 2024-09-16T04:39:01.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc157fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc157fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc157fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84aa8f6640 1 --2- 172.21.15.46:0/4238002721 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f84a40a29b0 0x7f84a41cfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84aa0f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84aa0f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f84aa0f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15d043640 10 monclient(hunting): handle_auth_done global_id 4291 payload 824 2024-09-16T04:39:01.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15c842640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc1580a4c20 0x7fc158071a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7fc15d043640 1 -- 172.21.15.46:0/2228195892 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc1580a4c20 msgr2=0x7fc158071a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84a98f4640 10 monclient(hunting): get_auth_request con 0x7f84a4073de0 auth_method 0 2024-09-16T04:39:01.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84a98f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84a98f4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84a98f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84aa8f6640 10 monclient(hunting): get_auth_request con 0x7f84a40a29b0 auth_method 0 2024-09-16T04:39:01.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84aa8f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84aa8f6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84aa8f6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84aa8f6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84aa8f6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84aa8f6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84aa0f5640 10 monclient(hunting): handle_auth_done global_id 4284 payload 824 2024-09-16T04:39:01.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84aa0f5640 1 -- 172.21.15.46:0/4238002721 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f84a40a29b0 msgr2=0x7f84a41cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.687+0000 7f51f7fff640 1 --2- 172.21.15.46:0/1398851665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51f80a3c10 0x7f51f81cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:39:01.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84aa0f5640 1 --2- 172.21.15.46:0/4238002721 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f84a40a29b0 0x7f84a41cfda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84aa0f5640 1 -- 172.21.15.46:0/4238002721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84a4073de0 msgr2=0x7f84a41c9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84aa0f5640 1 --2- 172.21.15.46:0/4238002721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84a4073de0 0x7f84a41c9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84aa0f5640 10 monclient: _finish_hunting 0 2024-09-16T04:39:01.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84aa0f5640 1 monclient: found mon.noname-a 2024-09-16T04:39:01.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f51f77fe640 1 --2- 172.21.15.46:0/1398851665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f80a3ff0 0x7f51f81cc490 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f51ec009960 tx=0x7f51ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84aa0f5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:01.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84aa0f5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:01.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84aa0f5640 1 -- 172.21.15.46:0/4238002721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84a41b1750 con 0x7f84a40739e0 2024-09-16T04:39:01.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84a98f4640 1 --2- 172.21.15.46:0/4238002721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84a4073de0 0x7f84a41c9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:39:01.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84aa8f6640 1 --2- 172.21.15.46:0/4238002721 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f84a40a29b0 0x7f84a41cfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:01.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7fc15d043640 1 --2- 172.21.15.46:0/2228195892 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc1580a4c20 0x7fc158071a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7fc15d043640 1 -- 172.21.15.46:0/2228195892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc15813b1a0 msgr2=0x7fc158071f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7fc15d043640 1 --2- 172.21.15.46:0/2228195892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc15813b1a0 0x7fc158071f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7f84aa0f5640 1 --2- 172.21.15.46:0/4238002721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84a40739e0 0x7f84a41cc860 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f849c034040 tx=0x7f849c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7fc15d043640 10 monclient: _finish_hunting 0 2024-09-16T04:39:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7fc15d043640 1 monclient: found mon.noname-b 2024-09-16T04:39:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7fc15d043640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7fc15d043640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3300/0 2024-09-16T04:39:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7fc15d043640 1 -- 172.21.15.46:0/2228195892 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc15813e760 con 0x7fc15813b580 2024-09-16T04:39:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7fc15c842640 10 monclient: get_auth_request con 0x7fc1580a4c20 auth_method 0 2024-09-16T04:39:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7fc15c842640 1 --2- 172.21.15.46:0/2228195892 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc1580a4c20 0x7fc158071a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:39:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f51f4ff9640 1 -- 172.21.15.46:0/1398851665 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec037040 con 0x7f51f80a3ff0 2024-09-16T04:39:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f51f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f51f4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f51f4ff9640 10 monclient: dump: 2024-09-16T04:39:01.799 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.799 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.799 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.799 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.799 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.799 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.799 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.799 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.799 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.799 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.688+0000 7fc157fff640 1 --2- 172.21.15.46:0/2228195892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc15813b1a0 0x7fc158071f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:39:01.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7fc15d043640 1 --2- 172.21.15.46:0/2228195892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc15813b580 0x7fc1580724d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc15002a1e0 tx=0x7fc1500046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f84a2ffd640 1 -- 172.21.15.46:0/4238002721 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f849c041070 con 0x7f84a40739e0 2024-09-16T04:39:01.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f84a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f51f4ff9640 10 monclient: _finish_auth 0 2024-09-16T04:39:01.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f51f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:39:01.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f51f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:01.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f51f4ff9640 1 -- 172.21.15.46:0/1398851665 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51ec042070 con 0x7f51f80a3ff0 2024-09-16T04:39:01.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f51fe7c6640 5 monclient: authenticate success, global_id 4281 2024-09-16T04:39:01.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f84a2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:01.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f84a2ffd640 10 monclient: dump: 2024-09-16T04:39:01.801 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.801 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.801 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.801 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.801 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.801 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.801 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.801 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.802 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.802 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f84a2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:39:01.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f84a2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:39:01.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f84a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:01.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f84a2ffd640 1 -- 172.21.15.46:0/4238002721 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f849c007a10 con 0x7f84a40739e0 2024-09-16T04:39:01.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f84ac359640 5 monclient: authenticate success, global_id 4284 2024-09-16T04:39:01.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f51fe7c6640 10 monclient: _renew_subs 2024-09-16T04:39:01.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f51fe7c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:01.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f51fe7c6640 1 -- 172.21.15.46:0/1398851665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f81270e0 con 0x7f51f80a3ff0 2024-09-16T04:39:01.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f84a2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:01.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f84a2ffd640 1 -- 172.21.15.46:0/4238002721 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f849c036070 con 0x7f84a40739e0 2024-09-16T04:39:01.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f84a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f84a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:01.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f84a2ffd640 10 monclient: dump: 2024-09-16T04:39:01.803 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.804 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f84a88f2640 4 set_mon_vals no callback set 2024-09-16T04:39:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f84ac359640 10 monclient: _renew_subs 2024-09-16T04:39:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f84ac359640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f84ac359640 1 -- 172.21.15.46:0/4238002721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84a4121510 con 0x7f84a40739e0 2024-09-16T04:39:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f84ac359640 10 monclient: _renew_subs 2024-09-16T04:39:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f84ac359640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f84ac359640 1 -- 172.21.15.46:0/4238002721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84a40a58e0 con 0x7f84a40739e0 2024-09-16T04:39:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.690+0000 7fc1557fa640 1 -- 172.21.15.46:0/2228195892 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc150041020 con 0x7fc15813b580 2024-09-16T04:39:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.690+0000 7fc1557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.690+0000 7fc1557fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.690+0000 7fc1557fa640 10 monclient: dump: 2024-09-16T04:39:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.806 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.806 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.806 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.806 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.806 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.806 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.690+0000 7fc1557fa640 10 monclient: _finish_auth 0 2024-09-16T04:39:01.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.690+0000 7fc1557fa640 10 monclient: _check_auth_tickets 2024-09-16T04:39:01.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.690+0000 7fc1557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:01.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.690+0000 7fc15eaa6640 5 monclient: authenticate success, global_id 4291 2024-09-16T04:39:01.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.690+0000 7fc15eaa6640 10 monclient: _renew_subs 2024-09-16T04:39:01.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.690+0000 7fc15eaa6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:01.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.690+0000 7fc15eaa6640 1 -- 172.21.15.46:0/2228195892 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc15813ea40 con 0x7fc15813b580 2024-09-16T04:39:01.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.690+0000 7fc15eaa6640 10 monclient: _renew_subs 2024-09-16T04:39:01.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.690+0000 7fc15eaa6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:01.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.690+0000 7fc15eaa6640 1 -- 172.21.15.46:0/2228195892 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1581425b0 con 0x7fc15813b580 2024-09-16T04:39:01.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f51f4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:01.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.690+0000 7fc1557fa640 1 -- 172.21.15.46:0/2228195892 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc150032440 con 0x7fc15813b580 2024-09-16T04:39:01.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.690+0000 7fc1557fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:01.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.690+0000 7fc156ffd640 4 set_mon_vals no callback set 2024-09-16T04:39:01.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f51f4ff9640 1 -- 172.21.15.46:0/1398851665 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec030bf0 con 0x7f51f80a3ff0 2024-09-16T04:39:01.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f51f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f51f4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:01.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f51f4ff9640 10 monclient: dump: 2024-09-16T04:39:01.809 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.809 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.809 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.809 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.809 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.809 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.809 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.809 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.809 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.809 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f51fe7c6640 10 monclient: _renew_subs 2024-09-16T04:39:01.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f51fe7c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:01.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f51f67fc640 4 set_mon_vals no callback set 2024-09-16T04:39:01.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.689+0000 7f51fe7c6640 1 -- 172.21.15.46:0/1398851665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f80a0400 con 0x7f51f80a3ff0 2024-09-16T04:39:01.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.690+0000 7f51de7fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:01.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.690+0000 7f51de7fc640 1 -- 172.21.15.46:0/1398851665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f51f80aa300 con 0x7f51f80a3ff0 2024-09-16T04:39:01.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.691+0000 7f84a0ff9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:01.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.691+0000 7f84a0ff9640 1 -- 172.21.15.46:0/4238002721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8488000f80 con 0x7f84a40739e0 2024-09-16T04:39:01.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.691+0000 7fc1557fa640 1 -- 172.21.15.46:0/2228195892 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc150041020 con 0x7fc15813b580 2024-09-16T04:39:01.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.691+0000 7fc1557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.691+0000 7fc1557fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:01.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.691+0000 7fc1557fa640 10 monclient: dump: 2024-09-16T04:39:01.811 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.811 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.811 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.811 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.811 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.811 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.811 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.811 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.812 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.812 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.691+0000 7fc15eaa6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:01.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.691+0000 7fc15eaa6640 1 -- 172.21.15.46:0/2228195892 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc138000f80 con 0x7fc15813b580 2024-09-16T04:39:01.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.691+0000 7f84a2ffd640 1 -- 172.21.15.46:0/4238002721 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f849c044070 con 0x7f84a40739e0 2024-09-16T04:39:01.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.691+0000 7f84a2ffd640 1 --2- 172.21.15.46:0/4238002721 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8490038400 0x7f849003a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.691+0000 7f84a2ffd640 1 -- 172.21.15.46:0/4238002721 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f849c06c420 con 0x7f84a40739e0 2024-09-16T04:39:01.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7f51f4ff9640 1 -- 172.21.15.46:0/1398851665 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f51ec047460 con 0x7f51f80a3ff0 2024-09-16T04:39:01.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7f84a2ffd640 1 --2- 172.21.15.46:0/4238002721 >> [v2:172.21.15.46:6816/4217009477,v1:172.21.15.46:6817/4217009477] conn(0x7f849003e200 0x7f8490040640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7f84a98f4640 1 --2- 172.21.15.46:0/4238002721 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8490038400 0x7f849003a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7f84a2ffd640 1 -- 172.21.15.46:0/4238002721 --> [v2:172.21.15.46:6816/4217009477,v1:172.21.15.46:6817/4217009477] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f849c031600 con 0x7f849003e200 2024-09-16T04:39:01.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7f84a2ffd640 1 -- 172.21.15.46:0/4238002721 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f849c073050 con 0x7f84a40739e0 2024-09-16T04:39:01.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7f84a2ffd640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-16T04:39:01.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7f84a98f4640 10 monclient: get_auth_request con 0x7f8490038400 auth_method 0 2024-09-16T04:39:01.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7f51f4ff9640 1 --2- 172.21.15.46:0/1398851665 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f51e40383b0 0x7f51e403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7f84aa8f6640 1 --2- 172.21.15.46:0/4238002721 >> [v2:172.21.15.46:6816/4217009477,v1:172.21.15.46:6817/4217009477] conn(0x7f849003e200 0x7f8490040640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7f51f4ff9640 1 -- 172.21.15.46:0/1398851665 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f51ec06ccf0 con 0x7f51f80a3ff0 2024-09-16T04:39:01.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7f84aa8f6640 10 monclient: get_auth_request con 0x7f849003e200 auth_method 0 2024-09-16T04:39:01.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7f51f4ff9640 1 --2- 172.21.15.46:0/1398851665 >> [v2:172.21.15.46:6800/3101080311,v1:172.21.15.46:6801/3101080311] conn(0x7f51e403e220 0x7f51e4040660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7f51f4ff9640 1 -- 172.21.15.46:0/1398851665 --> [v2:172.21.15.46:6800/3101080311,v1:172.21.15.46:6801/3101080311] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f51ec03ee50 con 0x7f51e403e220 2024-09-16T04:39:01.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7f51f4ff9640 1 -- 172.21.15.46:0/1398851665 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f51ec06d0d0 con 0x7f51f80a3ff0 2024-09-16T04:39:01.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7f51f4ff9640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-16T04:39:01.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7fc1557fa640 1 -- 172.21.15.46:0/2228195892 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fc150044020 con 0x7fc15813b580 2024-09-16T04:39:01.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7f51f7fff640 1 --2- 172.21.15.46:0/1398851665 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f51e40383b0 0x7f51e403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7f51f7fff640 10 monclient: get_auth_request con 0x7f51e40383b0 auth_method 0 2024-09-16T04:39:01.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7f51fcd63640 1 --2- 172.21.15.46:0/1398851665 >> [v2:172.21.15.46:6800/3101080311,v1:172.21.15.46:6801/3101080311] conn(0x7f51e403e220 0x7f51e4040660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7f84aa8f6640 1 --2- 172.21.15.46:0/4238002721 >> [v2:172.21.15.46:6816/4217009477,v1:172.21.15.46:6817/4217009477] conn(0x7f849003e200 0x7f8490040640 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7fc1557fa640 1 --2- 172.21.15.46:0/2228195892 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc1440387b0 0x7fc14403ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.692+0000 7f51fcd63640 10 monclient: get_auth_request con 0x7f51e403e220 auth_method 0 2024-09-16T04:39:01.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.693+0000 7fc1557fa640 1 -- 172.21.15.46:0/2228195892 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fc15003d420 con 0x7fc15813b580 2024-09-16T04:39:01.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.693+0000 7fc1557fa640 1 --2- 172.21.15.46:0/2228195892 >> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] conn(0x7fc14403e620 0x7fc144040a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.693+0000 7fc1557fa640 1 -- 172.21.15.46:0/2228195892 --> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc144041130 con 0x7fc14403e620 2024-09-16T04:39:01.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.693+0000 7fc1557fa640 1 -- 172.21.15.46:0/2228195892 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fc15003a530 con 0x7fc15813b580 2024-09-16T04:39:01.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.693+0000 7fc1557fa640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-16T04:39:01.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.693+0000 7fc157fff640 1 --2- 172.21.15.46:0/2228195892 >> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] conn(0x7fc14403e620 0x7fc144040a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.693+0000 7f51f7fff640 1 --2- 172.21.15.46:0/1398851665 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f51e40383b0 0x7f51e403a870 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009b90 tx=0x7f51e8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.693+0000 7f51fcd63640 1 --2- 172.21.15.46:0/1398851665 >> [v2:172.21.15.46:6800/3101080311,v1:172.21.15.46:6801/3101080311] conn(0x7f51e403e220 0x7f51e4040660 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.693+0000 7fc157fff640 10 monclient: get_auth_request con 0x7fc14403e620 auth_method 0 2024-09-16T04:39:01.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.693+0000 7f84a2ffd640 1 -- 172.21.15.46:0/4238002721 <== osd.1 v2:172.21.15.46:6816/4217009477 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f849c031600 con 0x7f849003e200 2024-09-16T04:39:01.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.694+0000 7fc157fff640 1 --2- 172.21.15.46:0/2228195892 >> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] conn(0x7fc14403e620 0x7fc144040a60 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.820 INFO:teuthology.orchestra.run.smithi046.stdout:51539607558 2024-09-16T04:39:01.820 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:39:01.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.694+0000 7f51f4ff9640 1 -- 172.21.15.46:0/1398851665 <== osd.2 v2:172.21.15.46:6800/3101080311 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f51ec03ee50 con 0x7f51e403e220 2024-09-16T04:39:01.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.694+0000 7fc1557fa640 1 -- 172.21.15.46:0/2228195892 <== osd.5 v2:172.21.15.114:6818/938868100 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc144041130 con 0x7fc14403e620 2024-09-16T04:39:01.822 INFO:teuthology.orchestra.run.smithi046.stdout:51539607558 2024-09-16T04:39:01.822 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T04:39:01.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.703+0000 7f2e85fd8640 1 -- 172.21.15.46:0/1846870334 --> [v2:172.21.15.114:6810/890769734,v1:172.21.15.114:6811/890769734] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2e801371f0 con 0x7f2e6c03e270 2024-09-16T04:39:01.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.704+0000 7fc15c842640 1 --2- 172.21.15.46:0/2228195892 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc1440387b0 0x7fc14403ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.704+0000 7fc15c842640 10 monclient: get_auth_request con 0x7fc1440387b0 auth_method 0 2024-09-16T04:39:01.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.704+0000 7f2e67fff640 1 -- 172.21.15.46:0/1846870334 <== osd.3 v2:172.21.15.114:6810/890769734 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2e801371f0 con 0x7f2e6c03e270 2024-09-16T04:39:01.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.704+0000 7f84a98f4640 1 --2- 172.21.15.46:0/4238002721 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8490038400 0x7f849003a8c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f84a4051070 tx=0x7f8498008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.704+0000 7f2e85fd8640 1 -- 172.21.15.46:0/1846870334 >> [v2:172.21.15.114:6810/890769734,v1:172.21.15.114:6811/890769734] conn(0x7f2e6c03e270 msgr2=0x7f2e6c0406b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.704+0000 7f2e85fd8640 1 --2- 172.21.15.46:0/1846870334 >> [v2:172.21.15.114:6810/890769734,v1:172.21.15.114:6811/890769734] conn(0x7f2e6c03e270 0x7f2e6c0406b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.704+0000 7fc15c842640 1 --2- 172.21.15.46:0/2228195892 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc1440387b0 0x7fc14403ac70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc148007520 tx=0x7fc14800b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.704+0000 7f2e85fd8640 1 -- 172.21.15.46:0/1846870334 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2e6c038400 msgr2=0x7f2e6c03a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.704+0000 7f2e85fd8640 1 --2- 172.21.15.46:0/1846870334 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2e6c038400 0x7f2e6c03a8c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2e8013b3d0 tx=0x7f2e74009040 comp rx=0 tx=0).stop 2024-09-16T04:39:01.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.705+0000 7f2e85fd8640 10 monclient: shutdown 2024-09-16T04:39:01.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.705+0000 7f2e85fd8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:01.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.705+0000 7f2e85fd8640 1 -- 172.21.15.46:0/1846870334 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2e800a3c10 msgr2=0x7f2e80136770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.705+0000 7f2e85fd8640 1 --2- 172.21.15.46:0/1846870334 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2e800a3c10 0x7f2e80136770 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2e7000c960 tx=0x7f2e7000ce20 comp rx=0 tx=0).stop 2024-09-16T04:39:01.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.705+0000 7f2e7ffff640 1 -- 172.21.15.46:0/1846870334 reap_dead start 2024-09-16T04:39:01.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.705+0000 7f2e85fd8640 1 -- 172.21.15.46:0/1846870334 shutdown_connections 2024-09-16T04:39:01.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.705+0000 7f2e85fd8640 1 -- 172.21.15.46:0/1846870334 >> 172.21.15.46:0/1846870334 conn(0x7f2e800a5d10 msgr2=0x7f2e8009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:01.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.705+0000 7f2e85fd8640 1 -- 172.21.15.46:0/1846870334 shutdown_connections 2024-09-16T04:39:01.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.705+0000 7f2e85fd8640 1 -- 172.21.15.46:0/1846870334 wait complete. 2024-09-16T04:39:01.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.710+0000 7fc15eaa6640 1 -- 172.21.15.46:0/2228195892 --> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc138002bf0 con 0x7fc14403e620 2024-09-16T04:39:01.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.710+0000 7fc1557fa640 1 -- 172.21.15.46:0/2228195892 <== osd.5 v2:172.21.15.114:6818/938868100 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc138002bf0 con 0x7fc14403e620 2024-09-16T04:39:01.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.710+0000 7fc15eaa6640 1 -- 172.21.15.46:0/2228195892 >> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] conn(0x7fc14403e620 msgr2=0x7fc144040a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.710+0000 7fc15eaa6640 1 --2- 172.21.15.46:0/2228195892 >> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] conn(0x7fc14403e620 0x7fc144040a60 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.710+0000 7fc15eaa6640 1 -- 172.21.15.46:0/2228195892 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc1440387b0 msgr2=0x7fc14403ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.711+0000 7fc15eaa6640 1 --2- 172.21.15.46:0/2228195892 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc1440387b0 0x7fc14403ac70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc148007520 tx=0x7fc14800b040 comp rx=0 tx=0).stop 2024-09-16T04:39:01.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.711+0000 7fc15eaa6640 10 monclient: shutdown 2024-09-16T04:39:01.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.711+0000 7fc15eaa6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:01.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.711+0000 7fc15eaa6640 1 -- 172.21.15.46:0/2228195892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc15813b580 msgr2=0x7fc1580724d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.711+0000 7fc15eaa6640 1 --2- 172.21.15.46:0/2228195892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc15813b580 0x7fc1580724d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc15002a1e0 tx=0x7fc1500046c0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.711+0000 7fc15d043640 1 -- 172.21.15.46:0/2228195892 reap_dead start 2024-09-16T04:39:01.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.711+0000 7fc15eaa6640 1 -- 172.21.15.46:0/2228195892 shutdown_connections 2024-09-16T04:39:01.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.711+0000 7fc15eaa6640 1 -- 172.21.15.46:0/2228195892 >> 172.21.15.46:0/2228195892 conn(0x7fc1580a5d10 msgr2=0x7fc1580a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:01.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.711+0000 7fc15eaa6640 1 -- 172.21.15.46:0/2228195892 shutdown_connections 2024-09-16T04:39:01.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.711+0000 7fc15eaa6640 1 -- 172.21.15.46:0/2228195892 wait complete. 2024-09-16T04:39:01.830 INFO:teuthology.orchestra.run.smithi046.stdout:51539607558 2024-09-16T04:39:01.830 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:39:01.831 INFO:teuthology.orchestra.run.smithi046.stdout:47244640262 2024-09-16T04:39:01.831 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T04:39:01.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.725+0000 7f84a0ff9640 1 -- 172.21.15.46:0/4238002721 --> [v2:172.21.15.46:6816/4217009477,v1:172.21.15.46:6817/4217009477] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8488002c70 con 0x7f849003e200 2024-09-16T04:39:01.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.725+0000 7f84a2ffd640 1 -- 172.21.15.46:0/4238002721 <== osd.1 v2:172.21.15.46:6816/4217009477 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8488002c70 con 0x7f849003e200 2024-09-16T04:39:01.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.725+0000 7f84a0ff9640 1 -- 172.21.15.46:0/4238002721 >> [v2:172.21.15.46:6816/4217009477,v1:172.21.15.46:6817/4217009477] conn(0x7f849003e200 msgr2=0x7f8490040640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.725+0000 7f84a0ff9640 1 --2- 172.21.15.46:0/4238002721 >> [v2:172.21.15.46:6816/4217009477,v1:172.21.15.46:6817/4217009477] conn(0x7f849003e200 0x7f8490040640 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.726+0000 7f84a0ff9640 1 -- 172.21.15.46:0/4238002721 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8490038400 msgr2=0x7f849003a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.726+0000 7f84a0ff9640 1 --2- 172.21.15.46:0/4238002721 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8490038400 0x7f849003a8c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f84a4051070 tx=0x7f8498008040 comp rx=0 tx=0).stop 2024-09-16T04:39:01.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.726+0000 7f84a0ff9640 10 monclient: shutdown 2024-09-16T04:39:01.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.726+0000 7f84a0ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:01.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.726+0000 7f84a0ff9640 1 -- 172.21.15.46:0/4238002721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84a40739e0 msgr2=0x7f84a41cc860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.726+0000 7f84a0ff9640 1 --2- 172.21.15.46:0/4238002721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84a40739e0 0x7f84a41cc860 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f849c034040 tx=0x7f849c004a40 comp rx=0 tx=0).stop 2024-09-16T04:39:01.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.726+0000 7f84aa8f6640 1 -- 172.21.15.46:0/4238002721 reap_dead start 2024-09-16T04:39:01.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.726+0000 7f84a0ff9640 1 -- 172.21.15.46:0/4238002721 shutdown_connections 2024-09-16T04:39:01.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.726+0000 7f84a0ff9640 1 -- 172.21.15.46:0/4238002721 >> 172.21.15.46:0/4238002721 conn(0x7f84a40a3aa0 msgr2=0x7f84a40a2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:01.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.726+0000 7f84a0ff9640 1 -- 172.21.15.46:0/4238002721 shutdown_connections 2024-09-16T04:39:01.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.726+0000 7f84a0ff9640 1 -- 172.21.15.46:0/4238002721 wait complete. 2024-09-16T04:39:01.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.726+0000 7f51de7fc640 1 -- 172.21.15.46:0/1398851665 --> [v2:172.21.15.46:6800/3101080311,v1:172.21.15.46:6801/3101080311] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f51f8067e40 con 0x7f51e403e220 2024-09-16T04:39:01.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.726+0000 7f51f4ff9640 1 -- 172.21.15.46:0/1398851665 <== osd.2 v2:172.21.15.46:6800/3101080311 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f51f8067e40 con 0x7f51e403e220 2024-09-16T04:39:01.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.726+0000 7f51de7fc640 1 -- 172.21.15.46:0/1398851665 >> [v2:172.21.15.46:6800/3101080311,v1:172.21.15.46:6801/3101080311] conn(0x7f51e403e220 msgr2=0x7f51e4040660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.726+0000 7f51de7fc640 1 --2- 172.21.15.46:0/1398851665 >> [v2:172.21.15.46:6800/3101080311,v1:172.21.15.46:6801/3101080311] conn(0x7f51e403e220 0x7f51e4040660 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.726+0000 7f51de7fc640 1 -- 172.21.15.46:0/1398851665 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f51e40383b0 msgr2=0x7f51e403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.727+0000 7f51de7fc640 1 --2- 172.21.15.46:0/1398851665 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f51e40383b0 0x7f51e403a870 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009b90 tx=0x7f51e8008040 comp rx=0 tx=0).stop 2024-09-16T04:39:01.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.727+0000 7f51de7fc640 10 monclient: shutdown 2024-09-16T04:39:01.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.727+0000 7f51de7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:01.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.727+0000 7f51de7fc640 1 -- 172.21.15.46:0/1398851665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f80a3ff0 msgr2=0x7f51f81cc490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.727+0000 7f51de7fc640 1 --2- 172.21.15.46:0/1398851665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f80a3ff0 0x7f51f81cc490 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f51ec009960 tx=0x7f51ec0046c0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.727+0000 7f51fcd63640 1 -- 172.21.15.46:0/1398851665 reap_dead start 2024-09-16T04:39:01.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.727+0000 7f51de7fc640 1 -- 172.21.15.46:0/1398851665 shutdown_connections 2024-09-16T04:39:01.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.727+0000 7f51de7fc640 1 -- 172.21.15.46:0/1398851665 >> 172.21.15.46:0/1398851665 conn(0x7f51f80a5d10 msgr2=0x7f51f809e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:01.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.727+0000 7f51de7fc640 1 -- 172.21.15.46:0/1398851665 shutdown_connections 2024-09-16T04:39:01.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.727+0000 7f51de7fc640 1 -- 172.21.15.46:0/1398851665 wait complete. 2024-09-16T04:39:01.836 INFO:teuthology.orchestra.run.smithi046.stdout:47244640261 2024-09-16T04:39:01.837 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T04:39:01.837 INFO:teuthology.orchestra.run.smithi046.stdout:47244640261 2024-09-16T04:39:01.838 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:39:01.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.978+0000 7fcb6de9b640 10 monclient: get_monmap_and_config 2024-09-16T04:39:01.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.978+0000 7f53fb2ac640 10 monclient: get_monmap_and_config 2024-09-16T04:39:01.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.978+0000 7f53fb2ac640 10 monclient: build_initial_monmap 2024-09-16T04:39:01.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.978+0000 7f53fb2ac640 10 monclient: monmap: 2024-09-16T04:39:01.980 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:01.980 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.980 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:01.979379+0000 2024-09-16T04:39:01.981 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:01.979379+0000 2024-09-16T04:39:01.981 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:01.981 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.981 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:01.981 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:01.981 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:01.981 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:01.981 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:01.981 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.978+0000 7fcb6de9b640 10 monclient: build_initial_monmap 2024-09-16T04:39:01.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.978+0000 7fcb6de9b640 10 monclient: monmap: 2024-09-16T04:39:01.982 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:01.982 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.982 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:01.979341+0000 2024-09-16T04:39:01.982 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:01.979341+0000 2024-09-16T04:39:01.982 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:01.982 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.982 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:01.982 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:01.982 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:01.982 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:01.982 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:01.982 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.979+0000 7f53fb2ac640 1 Processor -- start 2024-09-16T04:39:01.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.979+0000 7fcb6de9b640 1 Processor -- start 2024-09-16T04:39:01.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.979+0000 7f53fb2ac640 1 -- start start 2024-09-16T04:39:01.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.979+0000 7fcb6de9b640 1 -- start start 2024-09-16T04:39:01.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.979+0000 7fcb6de9b640 10 monclient: init 2024-09-16T04:39:01.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.979+0000 7f53fb2ac640 10 monclient: init 2024-09-16T04:39:01.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb6de9b640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:01.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7f53fb2ac640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:01.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7f53fb2ac640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-16T04:39:01.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb6de9b640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-16T04:39:01.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7f53fb2ac640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f53f40a3c10 0x7f53f409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb6de9b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb680a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb6de9b640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcb680a3ff0 0x7fcb680a4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb6de9b640 10 monclient(hunting): picked mon.noname-c con 0x7fcb680a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb6de9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcb6813be90 0x7fcb6813e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb6de9b640 10 monclient(hunting): picked mon.noname-a con 0x7fcb6813be90 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb6de9b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb6809ab70 con 0x7fcb680a3c10 2024-09-16T04:39:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb6de9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb6de9b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fcb680a4d60 con 0x7fcb6813be90 2024-09-16T04:39:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb6de9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb6de9b640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fcb6809b030 con 0x7fcb680a3ff0 2024-09-16T04:39:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb6de9b640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:01.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb6de9b640 10 monclient(hunting): authenticate will time out at 740.395595s 2024-09-16T04:39:01.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7f53fb2ac640 10 monclient(hunting): picked mon.noname-c con 0x7f53f40a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:01.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7f53fb2ac640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f53f40a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:01.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7f53fb2ac640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53f413be90 0x7f53f413e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:01.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7f53fb2ac640 10 monclient(hunting): picked mon.noname-b con 0x7f53f413be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:01.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb66ffd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcb680a3ff0 0x7fcb680a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb66ffd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcb680a3ff0 0x7fcb680a4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.46:38290/0 (socket says 172.21.15.46:38290) 2024-09-16T04:39:01.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb66ffd640 1 -- 172.21.15.46:0/735007660 learned_addr learned my addr 172.21.15.46:0/735007660 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:01.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb66ffd640 10 monclient(hunting): get_auth_request con 0x7fcb680a3ff0 auth_method 0 2024-09-16T04:39:01.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7fcb67fff640 1 --2- 172.21.15.46:0/735007660 >> v2:172.21.15.46:3300/0 conn(0x7fcb6813be90 0x7fcb6813e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7fcb66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7fcb66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7fcb66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7fcb67fff640 10 monclient(hunting): get_auth_request con 0x7fcb6813be90 auth_method 0 2024-09-16T04:39:01.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7fcb67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7fcb67fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7fcb67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7f53fb2ac640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53f409ab70 con 0x7f53f40a3ff0 2024-09-16T04:39:01.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7f53fb2ac640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7fcb66ffd640 10 monclient(hunting): handle_auth_done global_id 4297 payload 274 2024-09-16T04:39:01.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7fcb66ffd640 1 -- 172.21.15.46:0/735007660 >> v2:172.21.15.46:3300/0 conn(0x7fcb6813be90 msgr2=0x7fcb6813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:01.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7fcb66ffd640 1 --2- 172.21.15.46:0/735007660 >> v2:172.21.15.46:3300/0 conn(0x7fcb6813be90 0x7fcb6813e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7fcb66ffd640 1 -- 172.21.15.46:0/735007660 >> v1:172.21.15.114:6789/0 conn(0x7fcb680a3c10 legacy=0x7fcb6809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:01.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7f53fb2ac640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f53f40a4d60 con 0x7f53f40a3c10 2024-09-16T04:39:01.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7f53fb2ac640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:01.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7f53fb2ac640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f53f409b030 con 0x7f53f413be90 2024-09-16T04:39:01.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7f53fb2ac640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:01.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7f53fb2ac640 10 monclient(hunting): authenticate will time out at 740.395692s 2024-09-16T04:39:01.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7f53f9048640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f53f40a3c10 0x7f53f409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:01.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7f53f9048640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f53f40a3c10 0x7f53f409bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.46:38296/0 (socket says 172.21.15.46:38296) 2024-09-16T04:39:01.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7f53f9048640 1 -- 172.21.15.46:0/570356805 learned_addr learned my addr 172.21.15.46:0/570356805 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:01.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7f53f9048640 10 monclient(hunting): get_auth_request con 0x7f53f40a3c10 auth_method 0 2024-09-16T04:39:01.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7f53f9048640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:01.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7f53f9048640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:01.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.980+0000 7f53f9048640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:01.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7f53f9048640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:01.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7f53f9048640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:01.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7f53f9048640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:01.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7f53f9048640 10 monclient(hunting): handle_auth_done global_id 4300 payload 274 2024-09-16T04:39:01.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.982+0000 7fcb66ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:39:01.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.982+0000 7fcb66ffd640 1 monclient: found mon.noname-c 2024-09-16T04:39:01.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.982+0000 7fcb66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:01.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.982+0000 7fcb66ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:39:01.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.982+0000 7fcb66ffd640 1 -- 172.21.15.46:0/735007660 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb6809b310 con 0x7fcb680a3ff0 2024-09-16T04:39:01.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7f53f9048640 1 -- 172.21.15.46:0/570356805 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53f413be90 msgr2=0x7f53f413e280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:39:01.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7f53f9048640 1 --2- 172.21.15.46:0/570356805 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53f413be90 0x7f53f413e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:01.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.982+0000 7fcb67fff640 1 --2- 172.21.15.46:0/735007660 >> v2:172.21.15.46:3300/0 conn(0x7fcb6813be90 0x7fcb6813e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:39:01.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7f53f9048640 1 -- 172.21.15.46:0/570356805 >> v1:172.21.15.114:6789/0 conn(0x7f53f40a3ff0 legacy=0x7f53f40a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:01.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7f53ebfff640 1 -- 172.21.15.46:0/570356805 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3704876481 0 0) 0x7f53f409ab70 con 0x7f53f40a3ff0 2024-09-16T04:39:01.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7f53f9048640 10 monclient: _finish_hunting 0 2024-09-16T04:39:01.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7f53f9048640 1 monclient: found mon.noname-c 2024-09-16T04:39:01.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7f53f9048640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:01.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7f53f9048640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:39:01.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7f53f9048640 1 -- 172.21.15.46:0/570356805 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f409b310 con 0x7f53f40a3c10 2024-09-16T04:39:01.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.981+0000 7f53ebfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:39:01.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.982+0000 7f53f9048640 1 --2- 172.21.15.46:0/570356805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53f40a3c10 0x7f53f409bcf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f53f0009870 tx=0x7f53f0032940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1cb144ff01be4c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.982+0000 7fcb66ffd640 1 --2- 172.21.15.46:0/735007660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb680a3ff0 0x7fcb680a4400 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcb6806edc0 tx=0x7fcb5c02a160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d759a35d1e8e78a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:01.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7f53ebfff640 1 -- 172.21.15.46:0/570356805 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f003c020 con 0x7f53f40a3c10 2024-09-16T04:39:01.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7f53ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb667fc640 1 -- 172.21.15.46:0/735007660 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c037070 con 0x7fcb680a3ff0 2024-09-16T04:39:01.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:01.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:39:01.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb667fc640 10 monclient: dump: 2024-09-16T04:39:01.997 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.997 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.997 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.997 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.998 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.998 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.998 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.998 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.998 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.998 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb667fc640 10 monclient: _finish_auth 0 2024-09-16T04:39:01.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb667fc640 10 monclient: _check_auth_tickets 2024-09-16T04:39:01.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7f53ebfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:39:01.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7f53ebfff640 10 monclient: dump: 2024-09-16T04:39:01.998 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:01.999 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:01.999 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.999 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:01.999 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:01.999 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:01.999 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:01.999 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:01.999 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:01.999 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:01.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7f53ebfff640 10 monclient: _finish_auth 0 2024-09-16T04:39:01.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7f53ebfff640 10 monclient: _check_auth_tickets 2024-09-16T04:39:01.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7f53ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:01.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7f53fb2ac640 5 monclient: authenticate success, global_id 4300 2024-09-16T04:39:01.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7f53fb2ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:02.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:02.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb667fc640 1 -- 172.21.15.46:0/735007660 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb5c032440 con 0x7fcb680a3ff0 2024-09-16T04:39:02.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb667fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:02.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb667fc640 1 -- 172.21.15.46:0/735007660 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c032810 con 0x7fcb680a3ff0 2024-09-16T04:39:02.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:02.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb667fc640 10 monclient: dump: 2024-09-16T04:39:02.000 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.000 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.000 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.001 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.001 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.001 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.001 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.001 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.001 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.001 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb6de9b640 5 monclient: authenticate success, global_id 4297 2024-09-16T04:39:02.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb6de9b640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:02.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb6de9b640 4 set_mon_vals no callback set 2024-09-16T04:39:02.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7f53ebfff640 1 -- 172.21.15.46:0/570356805 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53f0038640 con 0x7f53f40a3c10 2024-09-16T04:39:02.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7f53ebfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:02.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7f53fb2ac640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:02.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7f53fb2ac640 4 set_mon_vals no callback set 2024-09-16T04:39:02.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7f53ebfff640 1 -- 172.21.15.46:0/570356805 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f0038a10 con 0x7f53f40a3c10 2024-09-16T04:39:02.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7f53ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7f53ebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:02.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7f53ebfff640 10 monclient: dump: 2024-09-16T04:39:02.003 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.003 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.003 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.003 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.003 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.003 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.003 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.003 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.003 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.003 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb6de9b640 10 monclient: shutdown 2024-09-16T04:39:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb6de9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb6de9b640 1 -- 172.21.15.46:0/735007660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb680a3ff0 msgr2=0x7fcb680a4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb6de9b640 1 --2- 172.21.15.46:0/735007660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb680a3ff0 0x7fcb680a4400 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcb6806edc0 tx=0x7fcb5c02a160 comp rx=0 tx=0).stop 2024-09-16T04:39:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb6de9b640 1 -- 172.21.15.46:0/735007660 shutdown_connections 2024-09-16T04:39:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb6de9b640 1 --2- 172.21.15.46:0/735007660 >> v2:172.21.15.46:3300/0 conn(0x7fcb6813be90 0x7fcb6813e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb6de9b640 1 --2- 172.21.15.46:0/735007660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb680a3ff0 0x7fcb680a4400 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.983+0000 7fcb6de9b640 1 -- 172.21.15.46:0/735007660 >> 172.21.15.46:0/735007660 conn(0x7fcb680a5d10 msgr2=0x7fcb680a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:02.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.984+0000 7fcb6de9b640 1 -- 172.21.15.46:0/735007660 shutdown_connections 2024-09-16T04:39:02.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.984+0000 7f53fb2ac640 10 monclient: shutdown 2024-09-16T04:39:02.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.984+0000 7fcb6de9b640 1 -- 172.21.15.46:0/735007660 wait complete. 2024-09-16T04:39:02.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.984+0000 7f53fb2ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:02.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.984+0000 7f53fb2ac640 1 -- 172.21.15.46:0/570356805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53f40a3c10 msgr2=0x7f53f409bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.984+0000 7f53fb2ac640 1 --2- 172.21.15.46:0/570356805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53f40a3c10 0x7f53f409bcf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f53f0009870 tx=0x7f53f0032940 comp rx=0 tx=0).stop 2024-09-16T04:39:02.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.984+0000 7f53fb2ac640 1 -- 172.21.15.46:0/570356805 shutdown_connections 2024-09-16T04:39:02.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.984+0000 7f53fb2ac640 1 --2- 172.21.15.46:0/570356805 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53f413be90 0x7f53f413e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.984+0000 7f53fb2ac640 1 --2- 172.21.15.46:0/570356805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53f40a3c10 0x7f53f409bcf0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.984+0000 7f53fb2ac640 1 -- 172.21.15.46:0/570356805 >> 172.21.15.46:0/570356805 conn(0x7f53f40a5d10 msgr2=0x7f53f40a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:02.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.984+0000 7f53fb2ac640 1 -- 172.21.15.46:0/570356805 shutdown_connections 2024-09-16T04:39:02.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.984+0000 7f53fb2ac640 1 -- 172.21.15.46:0/570356805 wait complete. 2024-09-16T04:39:02.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.984+0000 7fcb6de9b640 10 monclient: build_initial_monmap 2024-09-16T04:39:02.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.984+0000 7f53fb2ac640 10 monclient: build_initial_monmap 2024-09-16T04:39:02.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.984+0000 7fcb6de9b640 10 monclient: monmap: 2024-09-16T04:39:02.007 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:02.007 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:02.007 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:02.007 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:02.007 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:02.007 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.008 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:02.008 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:02.008 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:02.008 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.984+0000 7f53fb2ac640 10 monclient: monmap: 2024-09-16T04:39:02.008 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:02.008 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:02.008 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:02.008 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:02.008 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:02.009 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.985+0000 7fcb6de9b640 1 Processor -- start 2024-09-16T04:39:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.985+0000 7f53fb2ac640 1 Processor -- start 2024-09-16T04:39:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.985+0000 7f53fb2ac640 1 -- start start 2024-09-16T04:39:02.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.985+0000 7f53fb2ac640 10 monclient: init 2024-09-16T04:39:02.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.985+0000 7fcb6de9b640 1 -- start start 2024-09-16T04:39:02.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.985+0000 7fcb6de9b640 10 monclient: init 2024-09-16T04:39:02.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.985+0000 7f53fb2ac640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:02.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.985+0000 7f53fb2ac640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:39:02.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.985+0000 7f53fb2ac640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53f40a3c10 0x7f53f4137670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb6de9b640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:02.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53fb2ac640 10 monclient(hunting): picked mon.noname-c con 0x7f53f40a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:02.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53fb2ac640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53f40a3ff0 0x7f53f406f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb6de9b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:39:02.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb6de9b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb680a3c10 0x7fcb6809f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb6de9b640 10 monclient(hunting): picked mon.noname-c con 0x7fcb680a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:02.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb6de9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb680a3ff0 0x7fcb680a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb6de9b640 10 monclient(hunting): picked mon.noname-b con 0x7fcb680a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:02.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb6de9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb6813be90 0x7fcb680a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb6de9b640 10 monclient(hunting): picked mon.noname-a con 0x7fcb6813be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:02.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb6de9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb6de9b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6806e550 con 0x7fcb6813be90 2024-09-16T04:39:02.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb6de9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53f9048640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53f40a3c10 0x7f53f4137670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53f9048640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53f40a3c10 0x7f53f4137670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:32814/0 (socket says 172.21.15.46:32814) 2024-09-16T04:39:02.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53f9048640 1 -- 172.21.15.46:0/4025507077 learned_addr learned my addr 172.21.15.46:0/4025507077 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:02.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53fb2ac640 10 monclient(hunting): picked mon.noname-a con 0x7f53f40a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:02.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53fb2ac640 1 --2- 172.21.15.46:0/4025507077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53f413be90 0x7f53f406f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb6de9b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6806e3d0 con 0x7fcb680a3ff0 2024-09-16T04:39:02.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb6de9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53fb2ac640 10 monclient(hunting): picked mon.noname-b con 0x7f53f413be90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:02.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53fb2ac640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53fb2ac640 1 -- 172.21.15.46:0/4025507077 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f53f406e550 con 0x7f53f40a3ff0 2024-09-16T04:39:02.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53fb2ac640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53fb2ac640 1 -- 172.21.15.46:0/4025507077 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f53f406e3d0 con 0x7f53f413be90 2024-09-16T04:39:02.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53fb2ac640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53fb2ac640 1 -- 172.21.15.46:0/4025507077 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f53f406ef30 con 0x7f53f40a3c10 2024-09-16T04:39:02.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53fb2ac640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:02.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53fb2ac640 10 monclient(hunting): authenticate will time out at 740.401449s 2024-09-16T04:39:02.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53f9048640 10 monclient(hunting): get_auth_request con 0x7f53f40a3c10 auth_method 0 2024-09-16T04:39:02.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53f9048640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53f9048640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53f9048640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53f9048640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53f9048640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53f9048640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:02.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53f8847640 1 --2- 172.21.15.46:0/4025507077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53f40a3ff0 0x7f53f406f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53f8847640 10 monclient(hunting): get_auth_request con 0x7f53f40a3ff0 auth_method 0 2024-09-16T04:39:02.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53f8847640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53f8847640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53f8847640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53f8847640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb6de9b640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fcb6813e7c0 con 0x7fcb680a3c10 2024-09-16T04:39:02.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb6de9b640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:02.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb6de9b640 10 monclient(hunting): authenticate will time out at 740.401119s 2024-09-16T04:39:02.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb677fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb680a3c10 0x7fcb6809f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb677fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb680a3c10 0x7fcb6809f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:32826/0 (socket says 172.21.15.46:32826) 2024-09-16T04:39:02.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb677fe640 1 -- 172.21.15.46:0/464899078 learned_addr learned my addr 172.21.15.46:0/464899078 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:02.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb677fe640 10 monclient(hunting): get_auth_request con 0x7fcb680a3c10 auth_method 0 2024-09-16T04:39:02.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb677fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb67fff640 1 --2- 172.21.15.46:0/464899078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb6813be90 0x7fcb680a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:02.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb66ffd640 1 --2- 172.21.15.46:0/464899078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb680a3ff0 0x7fcb680a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb66ffd640 10 monclient(hunting): get_auth_request con 0x7fcb680a3ff0 auth_method 0 2024-09-16T04:39:02.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:02.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53f8847640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7f53f8847640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:02.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.986+0000 7fcb677fe640 10 monclient(hunting): handle_auth_done global_id 4262 payload 824 2024-09-16T04:39:02.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7f53f8847640 10 monclient(hunting): handle_auth_done global_id 4293 payload 824 2024-09-16T04:39:02.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb677fe640 1 -- 172.21.15.46:0/464899078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb680a3ff0 msgr2=0x7fcb680a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb677fe640 1 --2- 172.21.15.46:0/464899078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb680a3ff0 0x7fcb680a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb677fe640 1 -- 172.21.15.46:0/464899078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb6813be90 msgr2=0x7fcb680a1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb677fe640 1 --2- 172.21.15.46:0/464899078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb6813be90 0x7fcb680a1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb677fe640 10 monclient: _finish_hunting 0 2024-09-16T04:39:02.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb677fe640 1 monclient: found mon.noname-c 2024-09-16T04:39:02.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:02.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb677fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:39:02.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb677fe640 1 -- 172.21.15.46:0/464899078 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb680a0400 con 0x7fcb680a3c10 2024-09-16T04:39:02.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7f53f8847640 1 -- 172.21.15.46:0/4025507077 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53f40a3c10 msgr2=0x7f53f4137670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb67fff640 10 monclient: get_auth_request con 0x7fcb6813be90 auth_method 0 2024-09-16T04:39:02.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb67fff640 1 --2- 172.21.15.46:0/464899078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb6813be90 0x7fcb680a1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:39:02.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7f53f8847640 1 --2- 172.21.15.46:0/4025507077 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53f40a3c10 0x7f53f4137670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7f53f8847640 1 -- 172.21.15.46:0/4025507077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53f413be90 msgr2=0x7f53f406f880 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T04:39:02.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7f53f8847640 1 --2- 172.21.15.46:0/4025507077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53f413be90 0x7f53f406f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb677fe640 1 --2- 172.21.15.46:0/464899078 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb680a3c10 0x7fcb6809f860 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcb5800c780 tx=0x7fcb5800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:02.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7f53f8847640 10 monclient: _finish_hunting 0 2024-09-16T04:39:02.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7f53f8847640 1 monclient: found mon.noname-a 2024-09-16T04:39:02.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7f53f8847640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:02.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7f53f8847640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:02.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7f53f8847640 1 -- 172.21.15.46:0/4025507077 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f40712a0 con 0x7f53f40a3ff0 2024-09-16T04:39:02.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb4ffff640 1 -- 172.21.15.46:0/464899078 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58016020 con 0x7fcb680a3c10 2024-09-16T04:39:02.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb4ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:39:02.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb4ffff640 10 monclient: dump: 2024-09-16T04:39:02.024 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.024 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.024 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.024 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.024 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.025 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.025 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.025 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.025 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.025 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb4ffff640 10 monclient: _finish_auth 0 2024-09-16T04:39:02.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb4ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:39:02.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:02.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb4ffff640 1 -- 172.21.15.46:0/464899078 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb580050d0 con 0x7fcb680a3c10 2024-09-16T04:39:02.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb4ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:02.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb4ffff640 1 -- 172.21.15.46:0/464899078 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58007020 con 0x7fcb680a3c10 2024-09-16T04:39:02.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:02.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb4ffff640 10 monclient: dump: 2024-09-16T04:39:02.025 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.026 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.026 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.026 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.026 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.026 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.026 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.026 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.026 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.026 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb6de9b640 5 monclient: authenticate success, global_id 4262 2024-09-16T04:39:02.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb6de9b640 10 monclient: _renew_subs 2024-09-16T04:39:02.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb65ffb640 4 set_mon_vals no callback set 2024-09-16T04:39:02.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb6de9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:02.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7fcb6de9b640 1 -- 172.21.15.46:0/464899078 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb6809eac0 con 0x7fcb680a3c10 2024-09-16T04:39:02.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7f53f8847640 1 --2- 172.21.15.46:0/4025507077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53f40a3ff0 0x7f53f406f340 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f53e400c400 tx=0x7f53e400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:02.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7f53e9ffb640 1 -- 172.21.15.46:0/4025507077 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e4017070 con 0x7f53f40a3ff0 2024-09-16T04:39:02.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7f53e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.987+0000 7f53e9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:02.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.988+0000 7f53e9ffb640 10 monclient: dump: 2024-09-16T04:39:02.029 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.029 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.029 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.029 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.029 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.029 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.029 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.030 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.030 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.030 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.988+0000 7f53e9ffb640 10 monclient: _finish_auth 0 2024-09-16T04:39:02.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.988+0000 7f53e9ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:39:02.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.988+0000 7f53e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:02.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.988+0000 7f53e9ffb640 1 -- 172.21.15.46:0/4025507077 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53e40035f0 con 0x7f53f40a3ff0 2024-09-16T04:39:02.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.988+0000 7f53e9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:02.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.988+0000 7f53e9ffb640 1 -- 172.21.15.46:0/4025507077 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e40039e0 con 0x7f53f40a3ff0 2024-09-16T04:39:02.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.988+0000 7f53e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.988+0000 7f53e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:02.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.988+0000 7f53e9ffb640 10 monclient: dump: 2024-09-16T04:39:02.030 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.030 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.031 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.031 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.031 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.031 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.031 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.031 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.031 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.031 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.988+0000 7f53fb2ac640 5 monclient: authenticate success, global_id 4293 2024-09-16T04:39:02.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.988+0000 7f53fb2ac640 10 monclient: _renew_subs 2024-09-16T04:39:02.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.988+0000 7f53fb2ac640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:02.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.988+0000 7f53fb2ac640 1 -- 172.21.15.46:0/4025507077 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53f411fab0 con 0x7f53f40a3ff0 2024-09-16T04:39:02.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.988+0000 7f53eb7fe640 4 set_mon_vals no callback set 2024-09-16T04:39:02.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.989+0000 7f53e9ffb640 1 -- 172.21.15.46:0/4025507077 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f53e401a070 con 0x7f53f40a3ff0 2024-09-16T04:39:02.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.989+0000 7f53fb2ac640 10 monclient: _renew_subs 2024-09-16T04:39:02.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.988+0000 7fcb6de9b640 10 monclient: _renew_subs 2024-09-16T04:39:02.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.988+0000 7fcb6de9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:02.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.988+0000 7fcb6de9b640 1 -- 172.21.15.46:0/464899078 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb6809f080 con 0x7fcb680a3c10 2024-09-16T04:39:02.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.989+0000 7fcb4ffff640 1 -- 172.21.15.46:0/464899078 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fcb580071c0 con 0x7fcb680a3c10 2024-09-16T04:39:02.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.989+0000 7fcb4ffff640 1 --2- 172.21.15.46:0/464899078 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcb54038400 0x7fcb5403a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.989+0000 7f53e9ffb640 1 --2- 172.21.15.46:0/4025507077 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f53d8038430 0x7f53d803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.990+0000 7f53fb2ac640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:02.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.990+0000 7f53fb2ac640 1 -- 172.21.15.46:0/4025507077 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53f4071020 con 0x7f53f40a3ff0 2024-09-16T04:39:02.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.990+0000 7fcb4ffff640 1 -- 172.21.15.46:0/464899078 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fcb58012820 con 0x7fcb680a3c10 2024-09-16T04:39:02.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.990+0000 7fcb66ffd640 1 --2- 172.21.15.46:0/464899078 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcb54038400 0x7fcb5403a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.990+0000 7fcb66ffd640 10 monclient: get_auth_request con 0x7fcb54038400 auth_method 0 2024-09-16T04:39:02.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.990+0000 7fcb4dffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:02.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.990+0000 7fcb4dffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:02.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.990+0000 7fcb4dffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:02.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.990+0000 7fcb4dffb640 1 -- 172.21.15.46:0/464899078 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb680a4400 con 0x7fcb680a3c10 2024-09-16T04:39:02.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.990+0000 7f53e9ffb640 1 -- 172.21.15.46:0/4025507077 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f53e40119b0 con 0x7f53f40a3ff0 2024-09-16T04:39:02.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.990+0000 7f53f9048640 1 --2- 172.21.15.46:0/4025507077 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f53d8038430 0x7f53d803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.990+0000 7f53f9048640 10 monclient: get_auth_request con 0x7f53d8038430 auth_method 0 2024-09-16T04:39:02.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.991+0000 7f53f9048640 1 --2- 172.21.15.46:0/4025507077 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f53d8038430 0x7f53d803a8f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f53f0000c00 tx=0x7f53f0044040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:02.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.991+0000 7fcb66ffd640 1 --2- 172.21.15.46:0/464899078 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcb54038400 0x7fcb5403a8c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c034000 tx=0x7fcb5c02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:02.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.991+0000 7f53fb2ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:02.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.991+0000 7f53fb2ac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:02.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.991+0000 7f53fb2ac640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:02.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.991+0000 7f53fb2ac640 1 -- 172.21.15.46:0/4025507077 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53f406d930 con 0x7f53f40a3ff0 2024-09-16T04:39:02.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.995+0000 7fcb4ffff640 1 -- 172.21.15.46:0/464899078 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fcb680a0400 con 0x7fcb680a3c10 2024-09-16T04:39:02.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.995+0000 7fcb4ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:02.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.995+0000 7fcb4ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:02.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.996+0000 7f53e9ffb640 1 -- 172.21.15.46:0/4025507077 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f53f406d930 con 0x7f53f40a3ff0 2024-09-16T04:39:02.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.996+0000 7f53e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:02.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:01.996+0000 7f53e9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:02.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.020+0000 7f11dbcc9640 10 monclient: get_monmap_and_config 2024-09-16T04:39:02.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.020+0000 7f11dbcc9640 10 monclient: build_initial_monmap 2024-09-16T04:39:02.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.020+0000 7f11dbcc9640 10 monclient: monmap: 2024-09-16T04:39:02.040 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:02.040 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.041 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:02.020626+0000 2024-09-16T04:39:02.041 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:02.020626+0000 2024-09-16T04:39:02.041 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:02.041 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.041 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:02.041 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:02.041 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:02.041 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:02.041 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:02.041 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.020+0000 7f11dbcc9640 1 Processor -- start 2024-09-16T04:39:02.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.021+0000 7f11dbcc9640 1 -- start start 2024-09-16T04:39:02.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.021+0000 7f11dbcc9640 10 monclient: init 2024-09-16T04:39:02.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.021+0000 7f11dbcc9640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:02.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.021+0000 7f11dbcc9640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-16T04:39:02.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.021+0000 7f11dbcc9640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11d40a3c10 0x7f11d409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.021+0000 7f11dbcc9640 10 monclient(hunting): picked mon.noname-b con 0x7f11d40a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:02.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.021+0000 7f11dbcc9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f11d40a3ff0 0x7f11d40a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.021+0000 7f11dbcc9640 10 monclient(hunting): picked mon.noname-a con 0x7f11d40a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:02.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.021+0000 7f11dbcc9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f11d41369d0 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:02.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.021+0000 7f11dbcc9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11d409b940 con 0x7f11d41369d0 2024-09-16T04:39:02.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.021+0000 7f11dbcc9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.021+0000 7f11dbcc9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f11d409ab00 con 0x7f11d40a3ff0 2024-09-16T04:39:02.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.021+0000 7f11dbcc9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.021+0000 7f11dbcc9640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f11d409b030 con 0x7f11d40a3c10 2024-09-16T04:39:02.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.021+0000 7f11dbcc9640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:02.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.021+0000 7f11dbcc9640 10 monclient(hunting): authenticate will time out at 740.436920s 2024-09-16T04:39:02.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.021+0000 7f11d9264640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f11d40a3ff0 0x7f11d40a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9264640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f11d40a3ff0 0x7f11d40a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59342/0 (socket says 172.21.15.46:59342) 2024-09-16T04:39:02.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9264640 1 -- 172.21.15.46:0/1008915994 learned_addr learned my addr 172.21.15.46:0/1008915994 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:02.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9264640 10 monclient(hunting): get_auth_request con 0x7f11d40a3ff0 auth_method 0 2024-09-16T04:39:02.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9a65640 1 --2- 172.21.15.46:0/1008915994 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11d40a3c10 0x7f11d409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9264640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9264640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9264640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9a65640 10 monclient(hunting): get_auth_request con 0x7f11d40a3c10 auth_method 0 2024-09-16T04:39:02.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9a65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9a65640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9a65640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9264640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9264640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9264640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:02.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9a65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9a65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9a65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:02.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9264640 10 monclient(hunting): handle_auth_done global_id 4296 payload 274 2024-09-16T04:39:02.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9264640 1 -- 172.21.15.46:0/1008915994 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11d40a3c10 msgr2=0x7f11d409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:02.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9264640 1 --2- 172.21.15.46:0/1008915994 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11d40a3c10 0x7f11d409bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9264640 1 -- 172.21.15.46:0/1008915994 >> v1:172.21.15.46:6789/0 conn(0x7f11d41369d0 legacy=0x7f11d4136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9264640 10 monclient: _finish_hunting 0 2024-09-16T04:39:02.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9264640 1 monclient: found mon.noname-a 2024-09-16T04:39:02.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9264640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:02.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9264640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:02.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7f11d9264640 1 -- 172.21.15.46:0/1008915994 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d409b310 con 0x7f11d40a3ff0 2024-09-16T04:39:02.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7fb5455f4640 10 monclient: get_monmap_and_config 2024-09-16T04:39:02.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7fb5455f4640 10 monclient: build_initial_monmap 2024-09-16T04:39:02.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.022+0000 7fb5455f4640 10 monclient: monmap: 2024-09-16T04:39:02.045 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:02.045 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:02.023300+0000 2024-09-16T04:39:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:02.023300+0000 2024-09-16T04:39:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:02.046 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.023+0000 7fb5455f4640 1 Processor -- start 2024-09-16T04:39:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.023+0000 7fb5455f4640 1 -- start start 2024-09-16T04:39:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.023+0000 7fb5455f4640 10 monclient: init 2024-09-16T04:39:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.023+0000 7f11d9264640 1 --2- 172.21.15.46:0/1008915994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d40a3ff0 0x7f11d40a43d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f11c4009870 tx=0x7f11c402a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f5028b5120313ed server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7f11d8a63640 1 -- 172.21.15.46:0/1008915994 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c4037070 con 0x7f11d40a3ff0 2024-09-16T04:39:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7f11d8a63640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7f11d8a63640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7f11d8a63640 10 monclient: dump: 2024-09-16T04:39:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.049 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.049 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7f11d8a63640 10 monclient: _finish_auth 0 2024-09-16T04:39:02.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7f11d8a63640 10 monclient: _check_auth_tickets 2024-09-16T04:39:02.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7f11d8a63640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:02.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7f11d8a63640 1 -- 172.21.15.46:0/1008915994 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11c4032440 con 0x7f11d40a3ff0 2024-09-16T04:39:02.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7f11d8a63640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:02.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7f11d8a63640 1 -- 172.21.15.46:0/1008915994 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c4032810 con 0x7f11d40a3ff0 2024-09-16T04:39:02.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb5455f4640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:02.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb5455f4640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-16T04:39:02.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb5455f4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb540130900 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:02.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb5455f4640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb540136a60 0x7fb540138e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb5455f4640 10 monclient(hunting): picked mon.noname-b con 0x7fb540136a60 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:02.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb5455f4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb540139390 0x7fb54013b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb5455f4640 10 monclient(hunting): picked mon.noname-a con 0x7fb540139390 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:02.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb5455f4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5401313f0 con 0x7fb540130900 2024-09-16T04:39:02.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb5455f4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb5455f4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb5400a16e0 con 0x7fb540139390 2024-09-16T04:39:02.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb5455f4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb5455f4640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fb54009d5b0 con 0x7fb540136a60 2024-09-16T04:39:02.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb5455f4640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:02.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb5455f4640 10 monclient(hunting): authenticate will time out at 740.439587s 2024-09-16T04:39:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7f11d8a63640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7f11d8a63640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7f11d8a63640 10 monclient: dump: 2024-09-16T04:39:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.052 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7f11dbcc9640 5 monclient: authenticate success, global_id 4296 2024-09-16T04:39:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7f11dbcc9640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7f11dbcc9640 4 set_mon_vals no callback set 2024-09-16T04:39:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb53effd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb540130900 0x7fb540130ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53034/0 (socket says 172.21.15.46:53034) 2024-09-16T04:39:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb53effd640 1 -- 172.21.15.46:0/3152695427 learned_addr learned my addr 172.21.15.46:0/3152695427 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb53f7fe640 1 --2- 172.21.15.46:0/3152695427 >> v2:172.21.15.46:3300/0 conn(0x7fb540139390 0x7fb54013b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb53f7fe640 10 monclient(hunting): get_auth_request con 0x7fb540139390 auth_method 0 2024-09-16T04:39:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb53f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb53f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb5367fc640 1 --2- 172.21.15.46:0/3152695427 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb540136a60 0x7fb540138e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb53f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb5367fc640 10 monclient(hunting): get_auth_request con 0x7fb540136a60 auth_method 0 2024-09-16T04:39:02.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb5367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb5367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb5367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb5367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.024+0000 7fb5367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7fb5367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:02.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7fb53f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7fb53f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7f11dbcc9640 10 monclient: shutdown 2024-09-16T04:39:02.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7fb53f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:02.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7fb5367fc640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-09-16T04:39:02.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7fb5367fc640 1 -- 172.21.15.46:0/3152695427 >> v2:172.21.15.46:3300/0 conn(0x7fb540139390 msgr2=0x7fb54013b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:02.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7fb5367fc640 1 --2- 172.21.15.46:0/3152695427 >> v2:172.21.15.46:3300/0 conn(0x7fb540139390 0x7fb54013b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7fb5367fc640 1 -- 172.21.15.46:0/3152695427 >> v1:172.21.15.46:6789/0 conn(0x7fb540130900 legacy=0x7fb540130ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:02.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7fb5367fc640 10 monclient: _finish_hunting 0 2024-09-16T04:39:02.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7fb5367fc640 1 monclient: found mon.noname-b 2024-09-16T04:39:02.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7fb5367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:02.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7fb5367fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:39:02.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7fb5367fc640 1 -- 172.21.15.46:0/3152695427 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb54009dae0 con 0x7fb540136a60 2024-09-16T04:39:02.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7f11dbcc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:02.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7f11dbcc9640 1 -- 172.21.15.46:0/1008915994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d40a3ff0 msgr2=0x7f11d40a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7f11dbcc9640 1 --2- 172.21.15.46:0/1008915994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d40a3ff0 0x7f11d40a43d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f11c4009870 tx=0x7f11c402a130 comp rx=0 tx=0).stop 2024-09-16T04:39:02.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7f11dbcc9640 1 -- 172.21.15.46:0/1008915994 shutdown_connections 2024-09-16T04:39:02.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7f11dbcc9640 1 --2- 172.21.15.46:0/1008915994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d40a3ff0 0x7f11d40a43d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7f11dbcc9640 1 --2- 172.21.15.46:0/1008915994 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11d40a3c10 0x7f11d409bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7f11dbcc9640 1 -- 172.21.15.46:0/1008915994 >> 172.21.15.46:0/1008915994 conn(0x7f11d40a6340 msgr2=0x7f11d406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:02.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7f11dbcc9640 1 -- 172.21.15.46:0/1008915994 shutdown_connections 2024-09-16T04:39:02.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.025+0000 7f11dbcc9640 1 -- 172.21.15.46:0/1008915994 wait complete. 2024-09-16T04:39:02.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7f7df314b640 10 monclient: get_monmap_and_config 2024-09-16T04:39:02.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7f7df314b640 10 monclient: build_initial_monmap 2024-09-16T04:39:02.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7f7df314b640 10 monclient: monmap: 2024-09-16T04:39:02.058 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:02.058 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.058 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:02.026559+0000 2024-09-16T04:39:02.058 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:02.026559+0000 2024-09-16T04:39:02.058 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:02.058 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.058 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:02.058 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:02.058 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:02.058 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:02.058 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:02.058 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7fb5367fc640 1 --2- 172.21.15.46:0/3152695427 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb540136a60 0x7fb540138e50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb52800dbc0 tx=0x7fb528029f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3112b8b2c3edb261 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:02.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7f6557df8640 10 monclient: get_monmap_and_config 2024-09-16T04:39:02.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7f6557df8640 10 monclient: build_initial_monmap 2024-09-16T04:39:02.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7f6557df8640 10 monclient: monmap: 2024-09-16T04:39:02.059 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:02.059 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.059 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:02.027038+0000 2024-09-16T04:39:02.059 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:02.027038+0000 2024-09-16T04:39:02.059 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:02.060 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.060 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:02.060 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:02.060 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:02.060 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:02.060 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:02.060 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7fb53e7fc640 1 -- 172.21.15.46:0/3152695427 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52802aa00 con 0x7fb540136a60 2024-09-16T04:39:02.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7fb53e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7fb53e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:02.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7fb53e7fc640 10 monclient: dump: 2024-09-16T04:39:02.061 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.061 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.061 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.061 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.061 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.061 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.061 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.061 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.061 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.061 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7fb53e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:39:02.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7fb53e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:39:02.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7fb53e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:02.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7fb5455f4640 5 monclient: authenticate success, global_id 4271 2024-09-16T04:39:02.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7fb5455f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:02.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7fb53e7fc640 1 -- 172.21.15.46:0/3152695427 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb52802aba0 con 0x7fb540136a60 2024-09-16T04:39:02.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7fb53e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:02.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7fb5455f4640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:02.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7fb5455f4640 4 set_mon_vals no callback set 2024-09-16T04:39:02.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7f11dbcc9640 10 monclient: build_initial_monmap 2024-09-16T04:39:02.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.026+0000 7f11dbcc9640 10 monclient: monmap: 2024-09-16T04:39:02.063 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:02.063 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:02.063 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:02.063 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:02.063 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:02.063 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.063 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:02.063 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:02.063 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:02.063 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7fb53e7fc640 1 -- 172.21.15.46:0/3152695427 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52802aa00 con 0x7fb540136a60 2024-09-16T04:39:02.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7fb53e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7fb53e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:02.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7fb53e7fc640 10 monclient: dump: 2024-09-16T04:39:02.064 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.064 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.064 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.064 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.064 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.064 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.064 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.064 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.065 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.065 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7f7df314b640 1 Processor -- start 2024-09-16T04:39:02.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7f7df314b640 1 -- start start 2024-09-16T04:39:02.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7f7df314b640 10 monclient: init 2024-09-16T04:39:02.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7f11dbcc9640 1 Processor -- start 2024-09-16T04:39:02.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7f6557df8640 1 Processor -- start 2024-09-16T04:39:02.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7f11dbcc9640 1 -- start start 2024-09-16T04:39:02.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7f11dbcc9640 10 monclient: init 2024-09-16T04:39:02.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7fb5455f4640 10 monclient: shutdown 2024-09-16T04:39:02.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7fb5455f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:02.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7fb5455f4640 1 -- 172.21.15.46:0/3152695427 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb540136a60 msgr2=0x7fb540138e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7fb5455f4640 1 --2- 172.21.15.46:0/3152695427 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb540136a60 0x7fb540138e50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb52800dbc0 tx=0x7fb528029f00 comp rx=0 tx=0).stop 2024-09-16T04:39:02.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7fb5455f4640 1 -- 172.21.15.46:0/3152695427 shutdown_connections 2024-09-16T04:39:02.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7f6557df8640 1 -- start start 2024-09-16T04:39:02.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7f6557df8640 10 monclient: init 2024-09-16T04:39:02.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7fb5455f4640 1 --2- 172.21.15.46:0/3152695427 >> v2:172.21.15.46:3300/0 conn(0x7fb540139390 0x7fb54013b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7fb5455f4640 1 --2- 172.21.15.46:0/3152695427 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb540136a60 0x7fb540138e50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7fb5455f4640 1 -- 172.21.15.46:0/3152695427 >> 172.21.15.46:0/3152695427 conn(0x7fb5400a5960 msgr2=0x7fb5400a1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:02.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.027+0000 7fb5455f4640 1 -- 172.21.15.46:0/3152695427 shutdown_connections 2024-09-16T04:39:02.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f7df314b640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:02.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f7df314b640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-16T04:39:02.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f7df314b640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7dec0a3c10 0x7f7dec09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11dbcc9640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:02.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11dbcc9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:39:02.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11dbcc9640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11d40a3c10 0x7f11d411f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11dbcc9640 10 monclient(hunting): picked mon.noname-c con 0x7f11d40a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:02.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f7df314b640 10 monclient(hunting): picked mon.noname-c con 0x7f7dec0a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:02.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11dbcc9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11d40a3ff0 0x7f11d411fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11dbcc9640 10 monclient(hunting): picked mon.noname-b con 0x7f11d40a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:02.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f7df314b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7dec0a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:02.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f7df314b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7dec13be90 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:02.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11dbcc9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d41369d0 0x7f11d41202e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11dbcc9640 10 monclient(hunting): picked mon.noname-a con 0x7f11d41369d0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:02.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11dbcc9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11dbcc9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f11d40a4910 con 0x7f11d41369d0 2024-09-16T04:39:02.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11dbcc9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11dbcc9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f11d40a69c0 con 0x7f11d40a3ff0 2024-09-16T04:39:02.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11dbcc9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11dbcc9640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f11d41223f0 con 0x7f11d40a3c10 2024-09-16T04:39:02.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11d9a65640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11d40a3c10 0x7f11d411f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f7df314b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dec09ab70 con 0x7f7dec13be90 2024-09-16T04:39:02.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11d9a65640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11d40a3c10 0x7f11d411f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:32860/0 (socket says 172.21.15.46:32860) 2024-09-16T04:39:02.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11d9a65640 1 -- 172.21.15.46:0/4061370527 learned_addr learned my addr 172.21.15.46:0/4061370527 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:02.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11dbcc9640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:02.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f7df16e8640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7dec13be90 0x7f7dec13e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53040/0 (socket says 172.21.15.46:53040) 2024-09-16T04:39:02.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11dbcc9640 10 monclient(hunting): authenticate will time out at 740.443452s 2024-09-16T04:39:02.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f7df16e8640 1 -- 172.21.15.46:0/3602861136 learned_addr learned my addr 172.21.15.46:0/3602861136 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:02.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11d9264640 1 --2- 172.21.15.46:0/4061370527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11d40a3ff0 0x7f11d411fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11d9a65640 10 monclient(hunting): get_auth_request con 0x7f11d40a3c10 auth_method 0 2024-09-16T04:39:02.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11d9a65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11d9a65640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11d9a65640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f7df314b640 1 -- 172.21.15.46:0/3602861136 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dec0a4d60 con 0x7f7dec0a3ff0 2024-09-16T04:39:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f7df314b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f7df314b640 1 -- 172.21.15.46:0/3602861136 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f7dec09b030 con 0x7f7dec0a3c10 2024-09-16T04:39:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f7df314b640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f7df314b640 10 monclient(hunting): authenticate will time out at 740.443653s 2024-09-16T04:39:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11da266640 1 --2- 172.21.15.46:0/4061370527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d41369d0 0x7f11d41202e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11da266640 10 monclient(hunting): get_auth_request con 0x7f11d41369d0 auth_method 0 2024-09-16T04:39:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11da266640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11da266640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11da266640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11d9a65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11d9a65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11d9a65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:02.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7fb5455f4640 1 -- 172.21.15.46:0/3152695427 wait complete. 2024-09-16T04:39:02.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f6557df8640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:02.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f6557df8640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-16T04:39:02.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f6557df8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65500a2b90 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:02.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11da266640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11da266640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11da266640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:02.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11d9a65640 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2024-09-16T04:39:02.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11d9a65640 1 -- 172.21.15.46:0/4061370527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11d40a3ff0 msgr2=0x7f11d411fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11d9a65640 1 --2- 172.21.15.46:0/4061370527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11d40a3ff0 0x7f11d411fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.028+0000 7f11d9a65640 1 -- 172.21.15.46:0/4061370527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d41369d0 msgr2=0x7f11d41202e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 1 -- 172.21.15.46:0/3602861136 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2592203746 0 0) 0x7f7dec09ab70 con 0x7f7dec13be90 2024-09-16T04:39:02.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:39:02.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7df0ee7640 1 --2- 172.21.15.46:0/3602861136 >> v2:172.21.15.114:3300/0 conn(0x7f7dec0a3c10 0x7f7dec09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 10 monclient(hunting): my global_id is 4308 2024-09-16T04:39:02.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 1 -- 172.21.15.46:0/3602861136 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7dd4003520 con 0x7f7dec13be90 2024-09-16T04:39:02.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f11d9a65640 1 --2- 172.21.15.46:0/4061370527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d41369d0 0x7f11d41202e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f11d9a65640 10 monclient: _finish_hunting 0 2024-09-16T04:39:02.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f11d9a65640 1 monclient: found mon.noname-c 2024-09-16T04:39:02.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f11d9a65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:02.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f11d9a65640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:39:02.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f11d9a65640 1 -- 172.21.15.46:0/4061370527 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d4135fa0 con 0x7f11d40a3c10 2024-09-16T04:39:02.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f11da266640 1 --2- 172.21.15.46:0/4061370527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d41369d0 0x7f11d41202e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:02.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f6557df8640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f655009dca0 0x7f655013bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7fb5455f4640 10 monclient: build_initial_monmap 2024-09-16T04:39:02.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f6557df8640 10 monclient(hunting): picked mon.noname-c con 0x7f655009dca0 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:02.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f6555b94640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f65500a2b90 0x7f655009d590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.46:59998/0 (socket says 172.21.15.46:59998) 2024-09-16T04:39:02.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f6557df8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f655013c2a0 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:02.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f6555b94640 1 -- 172.21.15.46:0/1006641787 learned_addr learned my addr 172.21.15.46:0/1006641787 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:02.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7fb5455f4640 10 monclient: monmap: 2024-09-16T04:39:02.084 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:02.084 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:02.084 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:02.084 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:02.084 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:02.084 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.084 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:02.084 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:02.084 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:02.084 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f11d9a65640 1 --2- 172.21.15.46:0/4061370527 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11d40a3c10 0x7f11d411f860 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f11d0009f50 tx=0x7f11d0017560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:02.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 1 -- 172.21.15.46:0/3602861136 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 120882822 0 0) 0x7f7dd4003520 con 0x7f7dec13be90 2024-09-16T04:39:02.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 1 -- 172.21.15.46:0/3602861136 >> v2:172.21.15.114:3300/0 conn(0x7f7dec0a3c10 msgr2=0x7f7dec09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:02.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 1 --2- 172.21.15.46:0/3602861136 >> v2:172.21.15.114:3300/0 conn(0x7f7dec0a3c10 0x7f7dec09bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 1 -- 172.21.15.46:0/3602861136 >> v1:172.21.15.114:6789/0 conn(0x7f7dec0a3ff0 legacy=0x7f7dec0a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f6557df8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f655006de50 con 0x7f655013c2a0 2024-09-16T04:39:02.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f6557df8640 1 -- 172.21.15.46:0/1006641787 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65500a3630 con 0x7f65500a2b90 2024-09-16T04:39:02.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f6557df8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f6557df8640 1 -- 172.21.15.46:0/1006641787 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f65500a2f70 con 0x7f655009dca0 2024-09-16T04:39:02.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f6557df8640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:02.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f6557df8640 10 monclient(hunting): authenticate will time out at 740.444565s 2024-09-16T04:39:02.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:39:02.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:39:02.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:02.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:39:02.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 1 -- 172.21.15.46:0/3602861136 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dec09b310 con 0x7f7dec13be90 2024-09-16T04:39:02.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 10 monclient: _finish_auth 0 2024-09-16T04:39:02.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:39:02.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:02.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 1 -- 172.21.15.46:0/3602861136 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f7de40019f0 con 0x7f7dec13be90 2024-09-16T04:39:02.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:39:02.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 10 monclient: dump: 2024-09-16T04:39:02.088 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.088 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.088 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.088 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.088 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.088 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.088 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.088 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.088 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.088 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:39:02.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:02.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 1 -- 172.21.15.46:0/3602861136 >> v1:172.21.15.46:6789/0 conn(0x7f7dec13be90 legacy=0x7f7dec13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:39:02.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 1 --2- 172.21.15.46:0/3602861136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dd40039b0 0x7f7dd4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 10 monclient(hunting): picked mon.b con 0x7f7dd40039b0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:02.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 1 --2- 172.21.15.46:0/3602861136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dd4004800 0x7f7dd4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 10 monclient(hunting): picked mon.a con 0x7f7dd4004800 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:02.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f7deb7fe640 1 --2- 172.21.15.46:0/3602861136 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7dd4005410 0x7f7dd400ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f7debfff640 1 --2- 172.21.15.46:0/3602861136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dd4004800 0x7f7dd4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f7deb7fe640 10 monclient(hunting): picked mon.c con 0x7f7dd4005410 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:02.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f7deb7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f7deb7fe640 1 -- 172.21.15.46:0/3602861136 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7dec09b030 con 0x7f7dd4004800 2024-09-16T04:39:02.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f7deb7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f7deb7fe640 1 -- 172.21.15.46:0/3602861136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd400c320 con 0x7f7dd40039b0 2024-09-16T04:39:02.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7f6554b92640 1 -- 172.21.15.46:0/1006641787 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2196739968 0 0) 0x7f65500a3630 con 0x7f65500a2b90 2024-09-16T04:39:02.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient(hunting): _negotiate 2024-09-16T04:39:02.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient(hunting): my global_id is 4315 2024-09-16T04:39:02.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 1 -- 172.21.15.46:0/1006641787 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f653c003520 con 0x7f65500a2b90 2024-09-16T04:39:02.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.029+0000 7fb5455f4640 1 Processor -- start 2024-09-16T04:39:02.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7fb5455f4640 1 -- start start 2024-09-16T04:39:02.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7fb5455f4640 10 monclient: init 2024-09-16T04:39:02.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f11ca7fc640 1 -- 172.21.15.46:0/4061370527 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0024070 con 0x7f11d40a3c10 2024-09-16T04:39:02.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f11ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f11ca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:39:02.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f11ca7fc640 10 monclient: dump: 2024-09-16T04:39:02.091 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.091 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.091 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.092 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.092 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.092 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.092 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.092 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.092 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.092 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f11ca7fc640 10 monclient: _finish_auth 0 2024-09-16T04:39:02.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f11ca7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:39:02.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f11ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:02.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f11ca7fc640 1 -- 172.21.15.46:0/4061370527 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11d001f600 con 0x7f11d40a3c10 2024-09-16T04:39:02.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f11ca7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:02.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 1 -- 172.21.15.46:0/1006641787 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3719149469 0 0) 0x7f655006de50 con 0x7f655013c2a0 2024-09-16T04:39:02.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient(hunting): _negotiate 2024-09-16T04:39:02.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7fb5455f4640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:02.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7fb5455f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:39:02.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f11cbfff640 4 set_mon_vals no callback set 2024-09-16T04:39:02.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f11ca7fc640 1 -- 172.21.15.46:0/4061370527 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d001f9f0 con 0x7f11d40a3c10 2024-09-16T04:39:02.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f11ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f11ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:02.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f11ca7fc640 10 monclient: dump: 2024-09-16T04:39:02.094 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.094 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.094 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.095 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.095 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.095 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.095 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.095 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.095 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.095 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient(hunting): my global_id is 4311 2024-09-16T04:39:02.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 1 -- 172.21.15.46:0/1006641787 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65500a3630 con 0x7f655013c2a0 2024-09-16T04:39:02.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 1 -- 172.21.15.46:0/1006641787 <== mon.1 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2028658984 0 0) 0x7f653c003520 con 0x7f65500a2b90 2024-09-16T04:39:02.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f11dbcc9640 5 monclient: authenticate success, global_id 4274 2024-09-16T04:39:02.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 1 -- 172.21.15.46:0/1006641787 >> v2:172.21.15.114:3300/0 conn(0x7f655009dca0 msgr2=0x7f655013bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:02.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 1 --2- 172.21.15.46:0/1006641787 >> v2:172.21.15.114:3300/0 conn(0x7f655009dca0 0x7f655013bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 1 -- 172.21.15.46:0/1006641787 >> v1:172.21.15.46:6789/0 conn(0x7f655013c2a0 legacy=0x7f655013e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient: _finish_hunting 0 2024-09-16T04:39:02.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f11dbcc9640 10 monclient: _renew_subs 2024-09-16T04:39:02.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 1 monclient: found mon.noname-c-legacy 2024-09-16T04:39:02.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.114:6789/0 2024-09-16T04:39:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 1 -- 172.21.15.46:0/1006641787 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65500a38a0 con 0x7f65500a2b90 2024-09-16T04:39:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient: _finish_auth 0 2024-09-16T04:39:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient: _check_auth_tickets 2024-09-16T04:39:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 1 -- 172.21.15.46:0/1006641787 <== mon.1 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f654c0015f0 con 0x7f65500a2b90 2024-09-16T04:39:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T04:39:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient: dump: 2024-09-16T04:39:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.099 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T04:39:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 1 -- 172.21.15.46:0/1006641787 >> v1:172.21.15.114:6789/0 conn(0x7f65500a2b90 legacy=0x7f655009d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:39:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 1 --2- 172.21.15.46:0/1006641787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f653c003b90 0x7f653c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient(hunting): picked mon.b con 0x7f653c003b90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 1 --2- 172.21.15.46:0/1006641787 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f653c004800 0x7f653c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient(hunting): picked mon.c con 0x7f653c004800 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:02.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 1 --2- 172.21.15.46:0/1006641787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f653c005410 0x7f653c014280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient(hunting): picked mon.a con 0x7f653c005410 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:02.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 1 -- 172.21.15.46:0/1006641787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f65500a2f70 con 0x7f653c005410 2024-09-16T04:39:02.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 1 -- 172.21.15.46:0/1006641787 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f653c0059c0 con 0x7f653c003b90 2024-09-16T04:39:02.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 1 -- 172.21.15.46:0/1006641787 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f653c005bc0 con 0x7f653c004800 2024-09-16T04:39:02.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:02.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6554b92640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:39:02.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6556395640 1 --2- 172.21.15.46:0/1006641787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f653c005410 0x7f653c014280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6556395640 10 monclient(hunting): get_auth_request con 0x7f653c005410 auth_method 0 2024-09-16T04:39:02.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6556395640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6556395640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6556395640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6555b94640 1 --2- 172.21.15.46:0/1006641787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f653c003b90 0x7f653c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6555393640 1 --2- 172.21.15.46:0/1006641787 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f653c004800 0x7f653c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6555b94640 10 monclient(hunting): get_auth_request con 0x7f653c003b90 auth_method 0 2024-09-16T04:39:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7fb5455f4640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb540130900 0x7fb5401c7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7fb5455f4640 10 monclient(hunting): picked mon.noname-c con 0x7fb540130900 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7fb5455f4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb540139390 0x7fb54012fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6555b94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6555b94640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6555b94640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6555393640 10 monclient(hunting): get_auth_request con 0x7f653c004800 auth_method 0 2024-09-16T04:39:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6555393640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6555393640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6555393640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6556395640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6556395640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6556395640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7fb5455f4640 10 monclient(hunting): picked mon.noname-a con 0x7fb540139390 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7fb5455f4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb54012ff50 0x7fb5401326d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7fb5455f4640 10 monclient(hunting): picked mon.noname-b con 0x7fb54012ff50 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7fb5455f4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7fb5455f4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb5400a2410 con 0x7fb540139390 2024-09-16T04:39:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7fb5455f4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7fb5455f4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb5400a2290 con 0x7fb54012ff50 2024-09-16T04:39:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7fb5455f4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7fb5455f4640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fb5400a0fe0 con 0x7fb540130900 2024-09-16T04:39:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7fb5455f4640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7fb5455f4640 10 monclient(hunting): authenticate will time out at 740.445894s 2024-09-16T04:39:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb5367fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb540139390 0x7fb54012fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6555393640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6555393640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6555393640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:02.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6555b94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f6555b94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f6555b94640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:02.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f11dbcc9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:02.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f6555393640 10 monclient(hunting): handle_auth_done global_id 4315 payload 293 2024-09-16T04:39:02.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f11dbcc9640 1 -- 172.21.15.46:0/4061370527 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d40a0400 con 0x7f11d40a3c10 2024-09-16T04:39:02.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f6555393640 1 -- 172.21.15.46:0/1006641787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f653c003b90 msgr2=0x7f653c003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:02.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f6555393640 1 --2- 172.21.15.46:0/1006641787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f653c003b90 0x7f653c003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f11dbcc9640 10 monclient: _renew_subs 2024-09-16T04:39:02.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f11dbcc9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:02.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f11dbcc9640 1 -- 172.21.15.46:0/4061370527 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f11d4122a00 con 0x7f11d40a3c10 2024-09-16T04:39:02.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f6555393640 1 -- 172.21.15.46:0/1006641787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f653c005410 msgr2=0x7f653c014280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:02.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb5367fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb540139390 0x7fb54012fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59388/0 (socket says 172.21.15.46:59388) 2024-09-16T04:39:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f6555393640 1 --2- 172.21.15.46:0/1006641787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f653c005410 0x7f653c014280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f6555393640 10 monclient: _finish_hunting 0 2024-09-16T04:39:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f6555393640 1 monclient: found mon.c 2024-09-16T04:39:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f6555393640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f6555393640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f6555393640 1 -- 172.21.15.46:0/1006641787 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653c003520 con 0x7f653c004800 2024-09-16T04:39:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f6555393640 10 monclient: _finish_auth 0 2024-09-16T04:39:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f6555393640 10 monclient: _check_auth_tickets 2024-09-16T04:39:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f6555393640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f6556395640 1 -- 172.21.15.46:0/1006641787 reap_dead start 2024-09-16T04:39:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f6557df8640 5 monclient: authenticate success, global_id 4315 2024-09-16T04:39:02.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f6557df8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:02.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb5367fc640 1 -- 172.21.15.46:0/2888558010 learned_addr learned my addr 172.21.15.46:0/2888558010 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:02.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f7deb7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f7deb7fe640 1 -- 172.21.15.46:0/3602861136 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f7dd400c520 con 0x7f7dd4005410 2024-09-16T04:39:02.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f7deb7fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:02.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f7deb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:39:02.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f7debfff640 10 monclient(hunting): get_auth_request con 0x7f7dd4004800 auth_method 0 2024-09-16T04:39:02.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f7debfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f7debfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f7debfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:02.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f7debfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f7debfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f7debfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:02.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f7df0ee7640 1 --2- 172.21.15.46:0/3602861136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dd40039b0 0x7f7dd4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f7debfff640 10 monclient(hunting): handle_auth_done global_id 4308 payload 293 2024-09-16T04:39:02.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f7debfff640 1 -- 172.21.15.46:0/3602861136 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7dd4005410 msgr2=0x7f7dd400ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:02.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.030+0000 7f7df16e8640 1 --2- 172.21.15.46:0/3602861136 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7dd4005410 0x7f7dd400ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f7debfff640 1 --2- 172.21.15.46:0/3602861136 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7dd4005410 0x7f7dd400ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f7debfff640 1 -- 172.21.15.46:0/3602861136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dd40039b0 msgr2=0x7f7dd4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:02.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f7debfff640 1 --2- 172.21.15.46:0/3602861136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dd40039b0 0x7f7dd4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f7debfff640 10 monclient: _finish_hunting 0 2024-09-16T04:39:02.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f7debfff640 1 monclient: found mon.a 2024-09-16T04:39:02.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f7debfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:02.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f7debfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:02.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f7debfff640 1 -- 172.21.15.46:0/3602861136 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dec09b310 con 0x7f7dd4004800 2024-09-16T04:39:02.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f7debfff640 10 monclient: _finish_auth 0 2024-09-16T04:39:02.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f7debfff640 10 monclient: _check_auth_tickets 2024-09-16T04:39:02.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7f7debfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:02.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.032+0000 7f7df314b640 5 monclient: authenticate success, global_id 4308 2024-09-16T04:39:02.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.032+0000 7f7df16e8640 10 monclient: get_auth_request con 0x7f7dd4005410 auth_method 0 2024-09-16T04:39:02.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.032+0000 7f7df16e8640 1 --2- 172.21.15.46:0/3602861136 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7dd4005410 0x7f7dd400ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:39:02.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.032+0000 7f7df16e8640 1 -- 172.21.15.46:0/3602861136 reap_dead start 2024-09-16T04:39:02.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.032+0000 7f7df314b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:02.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.032+0000 7f7df0ee7640 10 monclient: get_auth_request con 0x7f7dd40039b0 auth_method 0 2024-09-16T04:39:02.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.032+0000 7f7df0ee7640 1 --2- 172.21.15.46:0/3602861136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dd40039b0 0x7f7dd4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:39:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb5367fc640 10 monclient(hunting): get_auth_request con 0x7fb540139390 auth_method 0 2024-09-16T04:39:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb5367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb5367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb5367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53effd640 1 --2- 172.21.15.46:0/2888558010 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb540130900 0x7fb5401c7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53f7fe640 1 --2- 172.21.15.46:0/2888558010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb54012ff50 0x7fb5401326d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53effd640 10 monclient(hunting): get_auth_request con 0x7fb540130900 auth_method 0 2024-09-16T04:39:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.032+0000 7f6555393640 1 --2- 172.21.15.46:0/1006641787 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f653c004800 0x7f653c004c80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6540009860 tx=0x7f653c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=796183847c7e968d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:02.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb5367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb5367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb5367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:02.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53f7fe640 10 monclient(hunting): get_auth_request con 0x7fb54012ff50 auth_method 0 2024-09-16T04:39:02.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53effd640 10 monclient(hunting): handle_auth_done global_id 4277 payload 824 2024-09-16T04:39:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53effd640 1 -- 172.21.15.46:0/2888558010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb54012ff50 msgr2=0x7fb5401326d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53effd640 1 --2- 172.21.15.46:0/2888558010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb54012ff50 0x7fb5401326d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53effd640 1 -- 172.21.15.46:0/2888558010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb540139390 msgr2=0x7fb54012fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53effd640 1 --2- 172.21.15.46:0/2888558010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb540139390 0x7fb54012fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53effd640 10 monclient: _finish_hunting 0 2024-09-16T04:39:02.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53effd640 1 monclient: found mon.noname-c 2024-09-16T04:39:02.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.032+0000 7f11dbcc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:02.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:02.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:39:02.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.031+0000 7fb53effd640 1 -- 172.21.15.46:0/2888558010 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb540135fd0 con 0x7fb540130900 2024-09-16T04:39:02.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.032+0000 7fb5367fc640 1 --2- 172.21.15.46:0/2888558010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb540139390 0x7fb54012fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:02.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.032+0000 7fb53effd640 1 --2- 172.21.15.46:0/2888558010 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb540130900 0x7fb5401c7790 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb53000c3e0 tx=0x7fb53000c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:02.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.032+0000 7f11dbcc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:02.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.032+0000 7f11dbcc9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:02.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f7debfff640 1 --2- 172.21.15.46:0/3602861136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dd4004800 0x7f7dd4004c80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7de0001750 tx=0x7f7dd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=454320755f0511d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:02.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.032+0000 7f11dbcc9640 1 -- 172.21.15.46:0/4061370527 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11d409bcf0 con 0x7f11d40a3c10 2024-09-16T04:39:02.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7fb537fff640 1 -- 172.21.15.46:0/2888558010 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb530017070 con 0x7fb540130900 2024-09-16T04:39:02.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7fb537fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7fb537fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:39:02.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7fb537fff640 10 monclient: dump: 2024-09-16T04:39:02.117 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.117 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.117 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.117 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.117 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.117 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.118 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.118 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.118 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.118 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7fb537fff640 10 monclient: _finish_auth 0 2024-09-16T04:39:02.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7fb537fff640 10 monclient: _check_auth_tickets 2024-09-16T04:39:02.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7fb537fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:02.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7fb5455f4640 5 monclient: authenticate success, global_id 4277 2024-09-16T04:39:02.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7fb5455f4640 10 monclient: _renew_subs 2024-09-16T04:39:02.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7fb5455f4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:02.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7fb5455f4640 1 -- 172.21.15.46:0/2888558010 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5401342a0 con 0x7fb540130900 2024-09-16T04:39:02.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f6554b92640 1 -- 172.21.15.46:0/1006641787 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6540037070 con 0x7f653c004800 2024-09-16T04:39:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f6554b92640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f6554b92640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f6554b92640 10 monclient: dump: 2024-09-16T04:39:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.119 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f6554b92640 1 -- 172.21.15.46:0/1006641787 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f654002aa60 con 0x7f653c004800 2024-09-16T04:39:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f6554b92640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f6554b92640 1 -- 172.21.15.46:0/1006641787 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6540009460 con 0x7f653c004800 2024-09-16T04:39:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f6554b92640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f6554b92640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f6554b92640 10 monclient: dump: 2024-09-16T04:39:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.121 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7fb537fff640 1 -- 172.21.15.46:0/2888558010 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb530005ce0 con 0x7fb540130900 2024-09-16T04:39:02.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f6557df8640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:02.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7fb537fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:02.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7fb537fff640 1 -- 172.21.15.46:0/2888558010 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb530007310 con 0x7fb540130900 2024-09-16T04:39:02.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f6557df8640 4 set_mon_vals no callback set 2024-09-16T04:39:02.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7fb537fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7fb537fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:02.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7fb537fff640 10 monclient: dump: 2024-09-16T04:39:02.122 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.122 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.122 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.122 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.123 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.123 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.123 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.123 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.123 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.123 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f11ca7fc640 1 -- 172.21.15.46:0/4061370527 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f11d0027070 con 0x7f11d40a3c10 2024-09-16T04:39:02.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7fb5455f4640 10 monclient: _renew_subs 2024-09-16T04:39:02.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7fb5455f4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:02.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7fb5455f4640 1 -- 172.21.15.46:0/2888558010 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb540134e00 con 0x7fb540130900 2024-09-16T04:39:02.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f7deb7fe640 1 -- 172.21.15.46:0/3602861136 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de0001370 con 0x7f7dd4004800 2024-09-16T04:39:02.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f7deb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f7deb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:02.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f7deb7fe640 10 monclient: dump: 2024-09-16T04:39:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.125 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f7deb7fe640 1 -- 172.21.15.46:0/3602861136 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7de00052d0 con 0x7f7dd4004800 2024-09-16T04:39:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f7df314b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:02.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f11ca7fc640 1 --2- 172.21.15.46:0/4061370527 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f11b80383b0 0x7f11b803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7f11ca7fc640 1 -- 172.21.15.46:0/4061370527 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f11d0053b20 con 0x7f11d40a3c10 2024-09-16T04:39:02.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.034+0000 7f6557df8640 10 monclient: shutdown 2024-09-16T04:39:02.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.034+0000 7f6557df8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:02.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.034+0000 7f6557df8640 1 -- 172.21.15.46:0/1006641787 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f653c004800 msgr2=0x7f653c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.034+0000 7f6557df8640 1 --2- 172.21.15.46:0/1006641787 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f653c004800 0x7f653c004c80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6540009860 tx=0x7f653c005830 comp rx=0 tx=0).stop 2024-09-16T04:39:02.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.034+0000 7f11d9264640 1 --2- 172.21.15.46:0/4061370527 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f11b80383b0 0x7f11b803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.034+0000 7f11d9264640 10 monclient: get_auth_request con 0x7f11b80383b0 auth_method 0 2024-09-16T04:39:02.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.033+0000 7fb53d7fa640 4 set_mon_vals no callback set 2024-09-16T04:39:02.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.034+0000 7fb537fff640 1 -- 172.21.15.46:0/2888558010 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fb53001a070 con 0x7fb540130900 2024-09-16T04:39:02.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.034+0000 7f7deb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:02.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.034+0000 7f7deb7fe640 1 -- 172.21.15.46:0/3602861136 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de002abc0 con 0x7f7dd4004800 2024-09-16T04:39:02.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.034+0000 7f7df314b640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:02.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.034+0000 7f7df314b640 4 set_mon_vals no callback set 2024-09-16T04:39:02.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.034+0000 7fb537fff640 1 --2- 172.21.15.46:0/2888558010 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb520038400 0x7fb52003a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.034+0000 7fb537fff640 1 -- 172.21.15.46:0/2888558010 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fb53004aa80 con 0x7fb540130900 2024-09-16T04:39:02.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.034+0000 7f6557df8640 1 -- 172.21.15.46:0/1006641787 shutdown_connections 2024-09-16T04:39:02.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.034+0000 7f6557df8640 1 --2- 172.21.15.46:0/1006641787 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f653c004800 0x7f653c004c80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.034+0000 7f6557df8640 1 -- 172.21.15.46:0/1006641787 >> 172.21.15.46:0/1006641787 conn(0x7f65500a7d00 msgr2=0x7f65500a3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:02.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.034+0000 7fb5367fc640 1 --2- 172.21.15.46:0/2888558010 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb520038400 0x7fb52003a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.034+0000 7fb5367fc640 10 monclient: get_auth_request con 0x7fb520038400 auth_method 0 2024-09-16T04:39:02.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.035+0000 7f11d9264640 1 --2- 172.21.15.46:0/4061370527 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f11b80383b0 0x7f11b803a870 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f11c4034000 tx=0x7f11c402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:02.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.035+0000 7f6557df8640 1 -- 172.21.15.46:0/1006641787 shutdown_connections 2024-09-16T04:39:02.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.035+0000 7f6557df8640 1 -- 172.21.15.46:0/1006641787 wait complete. 2024-09-16T04:39:02.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.035+0000 7f7deb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.035+0000 7f7deb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:02.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.035+0000 7f7deb7fe640 10 monclient: dump: 2024-09-16T04:39:02.130 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.130 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.131 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.131 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.131 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.131 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.131 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.131 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.131 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.131 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.035+0000 7fb5455f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:02.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.035+0000 7fb5367fc640 1 --2- 172.21.15.46:0/2888558010 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb520038400 0x7fb52003a8c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb528000c00 tx=0x7fb52802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:02.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.035+0000 7fb5455f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:02.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.035+0000 7f7df314b640 10 monclient: shutdown 2024-09-16T04:39:02.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.035+0000 7fb5455f4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:02.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.035+0000 7fb5455f4640 1 -- 172.21.15.46:0/2888558010 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb524005080 con 0x7fb540130900 2024-09-16T04:39:02.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.035+0000 7f7df314b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:02.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.035+0000 7f7df314b640 1 -- 172.21.15.46:0/3602861136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dd4004800 msgr2=0x7f7dd4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.035+0000 7f7df314b640 1 --2- 172.21.15.46:0/3602861136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dd4004800 0x7f7dd4004c80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7de0001750 tx=0x7f7dd4005830 comp rx=0 tx=0).stop 2024-09-16T04:39:02.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.035+0000 7f7df314b640 1 -- 172.21.15.46:0/3602861136 shutdown_connections 2024-09-16T04:39:02.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.036+0000 7f7df314b640 1 --2- 172.21.15.46:0/3602861136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dd4004800 0x7f7dd4004c80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.036+0000 7f7df314b640 1 -- 172.21.15.46:0/3602861136 >> 172.21.15.46:0/3602861136 conn(0x7f7dec0a5d10 msgr2=0x7f7dec0a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:02.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.036+0000 7f7df314b640 1 -- 172.21.15.46:0/3602861136 shutdown_connections 2024-09-16T04:39:02.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.036+0000 7f6557df8640 10 monclient: build_initial_monmap 2024-09-16T04:39:02.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.036+0000 7f6557df8640 10 monclient: monmap: 2024-09-16T04:39:02.134 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:02.134 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:02.134 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:02.134 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:02.134 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:02.134 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.135 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:02.135 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:02.135 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:02.135 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.036+0000 7f7df314b640 1 -- 172.21.15.46:0/3602861136 wait complete. 2024-09-16T04:39:02.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.036+0000 7f6557df8640 1 Processor -- start 2024-09-16T04:39:02.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.037+0000 7f6557df8640 1 -- start start 2024-09-16T04:39:02.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.037+0000 7f6557df8640 10 monclient: init 2024-09-16T04:39:02.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.037+0000 7f7df314b640 10 monclient: build_initial_monmap 2024-09-16T04:39:02.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.037+0000 7f7df314b640 10 monclient: monmap: 2024-09-16T04:39:02.136 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:02.136 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:02.136 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:02.136 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:02.136 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:02.136 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.137 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:02.137 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:02.137 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:02.137 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.037+0000 7f6557df8640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:02.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.037+0000 7f6557df8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:39:02.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f7df314b640 1 Processor -- start 2024-09-16T04:39:02.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6557df8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f653c004800 0x7f65501c7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6557df8640 10 monclient(hunting): picked mon.noname-a con 0x7f653c004800 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:02.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6557df8640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f655013b420 0x7f65500aa6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f7df314b640 1 -- start start 2024-09-16T04:39:02.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6555b94640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f653c004800 0x7f65501c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6555b94640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f653c004800 0x7f65501c7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59394/0 (socket says 172.21.15.46:59394) 2024-09-16T04:39:02.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f7df314b640 10 monclient: init 2024-09-16T04:39:02.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6555393640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f655013b420 0x7f65500aa6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6555393640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f655013b420 0x7f65500aa6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:32894/0 (socket says 172.21.15.46:32894) 2024-09-16T04:39:02.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6557df8640 10 monclient(hunting): picked mon.noname-c con 0x7f655013b420 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:02.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6555b94640 1 -- 172.21.15.46:0/478125436 learned_addr learned my addr 172.21.15.46:0/478125436 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:02.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6557df8640 1 --2- 172.21.15.46:0/478125436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f655013b860 0x7f65500aac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6557df8640 10 monclient(hunting): picked mon.noname-b con 0x7f655013b860 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:02.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6557df8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6557df8640 1 -- 172.21.15.46:0/478125436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f655006e570 con 0x7f653c004800 2024-09-16T04:39:02.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6557df8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6557df8640 1 -- 172.21.15.46:0/478125436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f655009e080 con 0x7f655013b860 2024-09-16T04:39:02.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6557df8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6557df8640 1 -- 172.21.15.46:0/478125436 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f655006e6f0 con 0x7f655013b420 2024-09-16T04:39:02.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6557df8640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:02.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6557df8640 10 monclient(hunting): authenticate will time out at 740.453639s 2024-09-16T04:39:02.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6555b94640 10 monclient(hunting): get_auth_request con 0x7f653c004800 auth_method 0 2024-09-16T04:39:02.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6555b94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6555b94640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6555b94640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6555393640 10 monclient(hunting): get_auth_request con 0x7f655013b420 auth_method 0 2024-09-16T04:39:02.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6556395640 1 --2- 172.21.15.46:0/478125436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f655013b860 0x7f65500aac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6555393640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6555393640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6555393640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6556395640 10 monclient(hunting): get_auth_request con 0x7f655013b860 auth_method 0 2024-09-16T04:39:02.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6556395640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6556395640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6556395640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6555b94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6555b94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f7df314b640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:02.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f7df314b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:39:02.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f7df314b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7dd4004800 0x7f7dec1c80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6555b94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:02.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6556395640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6556395640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6556395640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:02.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f7df0ee7640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7dd4004800 0x7f7dec1c80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6555393640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f7df314b640 10 monclient(hunting): picked mon.noname-c con 0x7f7dd4004800 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:02.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f7df314b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dec135100 0x7f7dec1cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6555393640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f7df314b640 10 monclient(hunting): picked mon.noname-b con 0x7f7dec135100 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:02.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.038+0000 7f6555393640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:02.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f7df314b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dec135580 0x7f7dec1cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f6556395640 10 monclient(hunting): handle_auth_done global_id 4321 payload 824 2024-09-16T04:39:02.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f6556395640 1 -- 172.21.15.46:0/478125436 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f655013b420 msgr2=0x7f65500aa6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f6556395640 1 --2- 172.21.15.46:0/478125436 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f655013b420 0x7f65500aa6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f6556395640 1 -- 172.21.15.46:0/478125436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f653c004800 msgr2=0x7f65501c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7debfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dec135100 0x7f7dec1cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7debfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dec135100 0x7f7dec1cb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.46:38396/0 (socket says 172.21.15.46:38396) 2024-09-16T04:39:02.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f6556395640 1 --2- 172.21.15.46:0/478125436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f653c004800 0x7f65501c7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f6556395640 10 monclient: _finish_hunting 0 2024-09-16T04:39:02.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df16e8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dec135580 0x7f7dec1cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df0ee7640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7dd4004800 0x7f7dec1c80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:32896/0 (socket says 172.21.15.46:32896) 2024-09-16T04:39:02.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df16e8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dec135580 0x7f7dec1cbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59404/0 (socket says 172.21.15.46:59404) 2024-09-16T04:39:02.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7debfff640 1 -- 172.21.15.46:0/337506250 learned_addr learned my addr 172.21.15.46:0/337506250 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:02.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df314b640 10 monclient(hunting): picked mon.noname-a con 0x7f7dec135580 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:02.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df314b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df314b640 1 -- 172.21.15.46:0/337506250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7dec0a4580 con 0x7f7dec135580 2024-09-16T04:39:02.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df314b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df314b640 1 -- 172.21.15.46:0/337506250 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7dec0a4400 con 0x7f7dec135100 2024-09-16T04:39:02.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df314b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:02.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df314b640 1 -- 172.21.15.46:0/337506250 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f7dec0a4700 con 0x7f7dd4004800 2024-09-16T04:39:02.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df314b640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:02.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df314b640 10 monclient(hunting): authenticate will time out at 740.455677s 2024-09-16T04:39:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f6556395640 1 monclient: found mon.noname-b 2024-09-16T04:39:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f6556395640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f6556395640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3300/0 2024-09-16T04:39:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f6556395640 1 -- 172.21.15.46:0/478125436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65501c8460 con 0x7f655013b860 2024-09-16T04:39:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f6556395640 1 --2- 172.21.15.46:0/478125436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f655013b860 0x7f65500aac10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f65500741d0 tx=0x7f654800ff70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f65467fc640 1 -- 172.21.15.46:0/478125436 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6548019070 con 0x7f655013b860 2024-09-16T04:39:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f65467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7debfff640 10 monclient(hunting): get_auth_request con 0x7f7dec135100 auth_method 0 2024-09-16T04:39:02.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7debfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f65467fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:02.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7debfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7debfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df0ee7640 10 monclient(hunting): get_auth_request con 0x7f7dd4004800 auth_method 0 2024-09-16T04:39:02.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df0ee7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df0ee7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df0ee7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df16e8640 10 monclient(hunting): get_auth_request con 0x7f7dec135580 auth_method 0 2024-09-16T04:39:02.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df16e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:02.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df16e8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:02.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df16e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:02.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df0ee7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df0ee7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7df0ee7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:02.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7debfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:02.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7debfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:02.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f7debfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:02.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f7debfff640 10 monclient(hunting): handle_auth_done global_id 4324 payload 824 2024-09-16T04:39:02.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f7debfff640 1 -- 172.21.15.46:0/337506250 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7dd4004800 msgr2=0x7f7dec1c80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f7debfff640 1 --2- 172.21.15.46:0/337506250 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7dd4004800 0x7f7dec1c80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f7debfff640 1 -- 172.21.15.46:0/337506250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dec135580 msgr2=0x7f7dec1cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f7debfff640 1 --2- 172.21.15.46:0/337506250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dec135580 0x7f7dec1cbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f7debfff640 10 monclient: _finish_hunting 0 2024-09-16T04:39:02.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f7debfff640 1 monclient: found mon.noname-b 2024-09-16T04:39:02.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f7debfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:02.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f7debfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3300/0 2024-09-16T04:39:02.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f7debfff640 1 -- 172.21.15.46:0/337506250 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dec1203b0 con 0x7f7dec135100 2024-09-16T04:39:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f65467fc640 10 monclient: dump: 2024-09-16T04:39:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.157 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f65467fc640 10 monclient: _finish_auth 0 2024-09-16T04:39:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f65467fc640 10 monclient: _check_auth_tickets 2024-09-16T04:39:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f65467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f7df16e8640 1 --2- 172.21.15.46:0/337506250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dec135580 0x7f7dec1cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:39:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f7df0ee7640 1 --2- 172.21.15.46:0/337506250 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7dd4004800 0x7f7dec1c80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f7debfff640 1 --2- 172.21.15.46:0/337506250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dec135100 0x7f7dec1cb630 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7de0005040 tx=0x7f7de0001120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:02.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f65467fc640 1 -- 172.21.15.46:0/478125436 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6548006e20 con 0x7f655013b860 2024-09-16T04:39:02.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f65467fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:02.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f65467fc640 1 -- 172.21.15.46:0/478125436 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6548003bb0 con 0x7f655013b860 2024-09-16T04:39:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f65467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f65467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f65467fc640 10 monclient: dump: 2024-09-16T04:39:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.159 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f6547fff640 4 set_mon_vals no callback set 2024-09-16T04:39:02.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f6557df8640 5 monclient: authenticate success, global_id 4321 2024-09-16T04:39:02.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f6557df8640 10 monclient: _renew_subs 2024-09-16T04:39:02.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f6557df8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:02.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f6557df8640 1 -- 172.21.15.46:0/478125436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6550122380 con 0x7f655013b860 2024-09-16T04:39:02.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f6557df8640 10 monclient: _renew_subs 2024-09-16T04:39:02.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f6557df8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:02.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.039+0000 7f6557df8640 1 -- 172.21.15.46:0/478125436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6550123d90 con 0x7f655013b860 2024-09-16T04:39:02.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.040+0000 7f65467fc640 1 -- 172.21.15.46:0/478125436 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f65480050d0 con 0x7f655013b860 2024-09-16T04:39:02.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f65467fc640 1 --2- 172.21.15.46:0/478125436 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f65340383b0 0x7f653403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f65467fc640 1 -- 172.21.15.46:0/478125436 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f654804abe0 con 0x7f655013b860 2024-09-16T04:39:02.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f6557df8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:02.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f6557df8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:02.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f6557df8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:02.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f6555b94640 1 --2- 172.21.15.46:0/478125436 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f65340383b0 0x7f653403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f6557df8640 1 -- 172.21.15.46:0/478125436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65500a5590 con 0x7f655013b860 2024-09-16T04:39:02.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f6555b94640 10 monclient: get_auth_request con 0x7f65340383b0 auth_method 0 2024-09-16T04:39:02.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.041+0000 7f6555b94640 1 --2- 172.21.15.46:0/478125436 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f65340383b0 0x7f653403a870 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f654c0019d0 tx=0x7f654c001000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:02.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.043+0000 7f11ca7fc640 1 -- 172.21.15.46:0/4061370527 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f11d001a070 con 0x7f11d40a3c10 2024-09-16T04:39:02.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.043+0000 7f11ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:02.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.043+0000 7f11ca7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:02.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.046+0000 7f7de97fa640 1 -- 172.21.15.46:0/337506250 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de0005480 con 0x7f7dec135100 2024-09-16T04:39:02.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.046+0000 7f7de97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.046+0000 7f7de97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:02.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.046+0000 7f7de97fa640 10 monclient: dump: 2024-09-16T04:39:02.162 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.162 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.162 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.162 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.162 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.162 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.163 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.163 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.163 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.163 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.046+0000 7f7de97fa640 10 monclient: _finish_auth 0 2024-09-16T04:39:02.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.046+0000 7f7de97fa640 10 monclient: _check_auth_tickets 2024-09-16T04:39:02.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.046+0000 7f7de97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:02.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.046+0000 7f7de97fa640 1 -- 172.21.15.46:0/337506250 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7de0007240 con 0x7f7dec135100 2024-09-16T04:39:02.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.046+0000 7f7de97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:02.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.046+0000 7f7deaffd640 4 set_mon_vals no callback set 2024-09-16T04:39:02.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.046+0000 7f7df314b640 5 monclient: authenticate success, global_id 4324 2024-09-16T04:39:02.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.046+0000 7f7de97fa640 1 -- 172.21.15.46:0/337506250 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de0007610 con 0x7f7dec135100 2024-09-16T04:39:02.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.046+0000 7f7df314b640 10 monclient: _renew_subs 2024-09-16T04:39:02.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.046+0000 7f7df314b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.046+0000 7f7df314b640 1 -- 172.21.15.46:0/337506250 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dec121210 con 0x7f7dec135100 2024-09-16T04:39:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.046+0000 7f65467fc640 1 -- 172.21.15.46:0/478125436 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f65480053c0 con 0x7f655013b860 2024-09-16T04:39:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.046+0000 7f65467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.046+0000 7f65467fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:02.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.047+0000 7f7de97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:02.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.047+0000 7f7de97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:02.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.047+0000 7f7de97fa640 10 monclient: dump: 2024-09-16T04:39:02.166 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:02.166 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:02.166 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.166 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:02.166 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:02.166 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:02.166 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:02.166 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:02.167 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.047+0000 7f7df314b640 10 monclient: _renew_subs 2024-09-16T04:39:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.047+0000 7f7df314b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.047+0000 7f7df314b640 1 -- 172.21.15.46:0/337506250 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dec1217d0 con 0x7f7dec135100 2024-09-16T04:39:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.048+0000 7f7de97fa640 1 -- 172.21.15.46:0/337506250 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f7de00067d0 con 0x7f7dec135100 2024-09-16T04:39:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.048+0000 7f7de97fa640 1 --2- 172.21.15.46:0/337506250 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7dd8038400 0x7f7dd803a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.048+0000 7f7de97fa640 1 -- 172.21.15.46:0/337506250 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f7de0073c90 con 0x7f7dec135100 2024-09-16T04:39:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.048+0000 7f7df314b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.048+0000 7f7df314b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.048+0000 7f7df314b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.048+0000 7f7df314b640 1 -- 172.21.15.46:0/337506250 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dec069150 con 0x7f7dec135100 2024-09-16T04:39:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.048+0000 7f7df0ee7640 1 --2- 172.21.15.46:0/337506250 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7dd8038400 0x7f7dd803a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.048+0000 7f7df0ee7640 10 monclient: get_auth_request con 0x7f7dd8038400 auth_method 0 2024-09-16T04:39:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.049+0000 7f7df0ee7640 1 --2- 172.21.15.46:0/337506250 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7dd8038400 0x7f7dd803a8c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc006fd0 tx=0x7f7ddc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.052+0000 7fb537fff640 1 -- 172.21.15.46:0/2888558010 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb530011960 con 0x7fb540130900 2024-09-16T04:39:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.052+0000 7fb537fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.052+0000 7fb537fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.053+0000 7f7de97fa640 1 -- 172.21.15.46:0/337506250 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7de00527a0 con 0x7f7dec135100 2024-09-16T04:39:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.053+0000 7f7de97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.053+0000 7f7de97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:02.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.197+0000 7f53fb2ac640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T04:39:02.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.197+0000 7f53fb2ac640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T04:39:02.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.197+0000 7f53fb2ac640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:02.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.197+0000 7f53fb2ac640 1 -- 172.21.15.46:0/4025507077 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f53f409bcf0 con 0x7f53f40a3ff0 2024-09-16T04:39:02.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.197+0000 7f53e9ffb640 1 -- 172.21.15.46:0/4025507077 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f53e4011c10 con 0x7f53f40a3ff0 2024-09-16T04:39:02.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.197+0000 7f53e9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T04:39:02.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.197+0000 7f53e9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:02.199 INFO:teuthology.orchestra.run.smithi046.stdout:47244640261 2024-09-16T04:39:02.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.199+0000 7f53fb2ac640 1 -- 172.21.15.46:0/4025507077 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f53d8038430 msgr2=0x7f53d803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.199+0000 7f53fb2ac640 1 --2- 172.21.15.46:0/4025507077 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f53d8038430 0x7f53d803a8f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f53f0000c00 tx=0x7f53f0044040 comp rx=0 tx=0).stop 2024-09-16T04:39:02.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.199+0000 7f53fb2ac640 10 monclient: shutdown 2024-09-16T04:39:02.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.199+0000 7f53fb2ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:02.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.199+0000 7f53fb2ac640 1 -- 172.21.15.46:0/4025507077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53f40a3ff0 msgr2=0x7f53f406f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.199+0000 7f53fb2ac640 1 --2- 172.21.15.46:0/4025507077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53f40a3ff0 0x7f53f406f340 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f53e400c400 tx=0x7f53e400c8c0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.199+0000 7f53fb2ac640 1 -- 172.21.15.46:0/4025507077 shutdown_connections 2024-09-16T04:39:02.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.199+0000 7f53fb2ac640 1 --2- 172.21.15.46:0/4025507077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53f413be90 0x7f53f406f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.199+0000 7f53fb2ac640 1 --2- 172.21.15.46:0/4025507077 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f53d8038430 0x7f53d803a8f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.199+0000 7f53fb2ac640 1 --2- 172.21.15.46:0/4025507077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53f40a3ff0 0x7f53f406f340 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.199+0000 7f53fb2ac640 1 --2- 172.21.15.46:0/4025507077 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53f40a3c10 0x7f53f4137670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.199+0000 7f53fb2ac640 1 -- 172.21.15.46:0/4025507077 >> 172.21.15.46:0/4025507077 conn(0x7f53f40a5d10 msgr2=0x7f53f413b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:02.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.200+0000 7f53fb2ac640 1 -- 172.21.15.46:0/4025507077 shutdown_connections 2024-09-16T04:39:02.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.200+0000 7f53fb2ac640 1 -- 172.21.15.46:0/4025507077 wait complete. 2024-09-16T04:39:02.212 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640261 for osd.5 2024-09-16T04:39:02.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.236+0000 7f7df314b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T04:39:02.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.236+0000 7f7df314b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T04:39:02.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.236+0000 7f7df314b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:02.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.236+0000 7f7df314b640 1 -- 172.21.15.46:0/337506250 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f7dec1c8ac0 con 0x7f7dec135100 2024-09-16T04:39:02.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.237+0000 7f7de97fa640 1 -- 172.21.15.46:0/337506250 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7de0058090 con 0x7f7dec135100 2024-09-16T04:39:02.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.237+0000 7f7de97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T04:39:02.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.237+0000 7f7de97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:02.239 INFO:teuthology.orchestra.run.smithi046.stdout:51539607556 2024-09-16T04:39:02.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.239+0000 7f7df314b640 1 -- 172.21.15.46:0/337506250 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7dd8038400 msgr2=0x7f7dd803a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.239+0000 7f7df314b640 1 --2- 172.21.15.46:0/337506250 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7dd8038400 0x7f7dd803a8c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc006fd0 tx=0x7f7ddc008040 comp rx=0 tx=0).stop 2024-09-16T04:39:02.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.239+0000 7f7df314b640 10 monclient: shutdown 2024-09-16T04:39:02.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.239+0000 7f7df314b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:02.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.239+0000 7f7df314b640 1 -- 172.21.15.46:0/337506250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dec135100 msgr2=0x7f7dec1cb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.239+0000 7f7df314b640 1 --2- 172.21.15.46:0/337506250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dec135100 0x7f7dec1cb630 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7de0005040 tx=0x7f7de0001120 comp rx=0 tx=0).stop 2024-09-16T04:39:02.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.239+0000 7f7df314b640 1 -- 172.21.15.46:0/337506250 shutdown_connections 2024-09-16T04:39:02.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.239+0000 7f7df314b640 1 --2- 172.21.15.46:0/337506250 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7dd8038400 0x7f7dd803a8c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.239+0000 7f7df314b640 1 --2- 172.21.15.46:0/337506250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dec135580 0x7f7dec1cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.239+0000 7f7df314b640 1 --2- 172.21.15.46:0/337506250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dec135100 0x7f7dec1cb630 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.239+0000 7f7df314b640 1 --2- 172.21.15.46:0/337506250 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7dd4004800 0x7f7dec1c80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.239+0000 7f7df314b640 1 -- 172.21.15.46:0/337506250 >> 172.21.15.46:0/337506250 conn(0x7f7dec0a5d10 msgr2=0x7f7dec136de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:02.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.239+0000 7f7df314b640 1 -- 172.21.15.46:0/337506250 shutdown_connections 2024-09-16T04:39:02.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.239+0000 7f7df314b640 1 -- 172.21.15.46:0/337506250 wait complete. 2024-09-16T04:39:02.251 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607556 for osd.0 2024-09-16T04:39:02.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.318+0000 7fcb4dffb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T04:39:02.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.318+0000 7fcb4dffb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T04:39:02.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.318+0000 7fcb4dffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:02.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.318+0000 7fcb4dffb640 1 -- 172.21.15.46:0/464899078 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fcb68073000 con 0x7fcb680a3c10 2024-09-16T04:39:02.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.318+0000 7fcb4ffff640 1 -- 172.21.15.46:0/464899078 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcb5801bd20 con 0x7fcb680a3c10 2024-09-16T04:39:02.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.318+0000 7fcb4ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T04:39:02.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.318+0000 7fcb4ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:02.320 INFO:teuthology.orchestra.run.smithi046.stdout:47244640260 2024-09-16T04:39:02.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.321+0000 7fcb4dffb640 1 -- 172.21.15.46:0/464899078 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcb54038400 msgr2=0x7fcb5403a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.321+0000 7fcb4dffb640 1 --2- 172.21.15.46:0/464899078 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcb54038400 0x7fcb5403a8c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c034000 tx=0x7fcb5c02e000 comp rx=0 tx=0).stop 2024-09-16T04:39:02.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.321+0000 7fcb4dffb640 10 monclient: shutdown 2024-09-16T04:39:02.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.321+0000 7fcb4dffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:02.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.321+0000 7fcb4dffb640 1 -- 172.21.15.46:0/464899078 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb680a3c10 msgr2=0x7fcb6809f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.321+0000 7fcb4dffb640 1 --2- 172.21.15.46:0/464899078 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb680a3c10 0x7fcb6809f860 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcb5800c780 tx=0x7fcb5800cc40 comp rx=0 tx=0).stop 2024-09-16T04:39:02.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.321+0000 7fcb4dffb640 1 -- 172.21.15.46:0/464899078 shutdown_connections 2024-09-16T04:39:02.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.321+0000 7fcb4dffb640 1 --2- 172.21.15.46:0/464899078 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcb54038400 0x7fcb5403a8c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.321+0000 7fcb4dffb640 1 --2- 172.21.15.46:0/464899078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb6813be90 0x7fcb680a1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.321+0000 7fcb4dffb640 1 --2- 172.21.15.46:0/464899078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb680a3ff0 0x7fcb680a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.321+0000 7fcb4dffb640 1 --2- 172.21.15.46:0/464899078 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb680a3c10 0x7fcb6809f860 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.321+0000 7fcb4dffb640 1 -- 172.21.15.46:0/464899078 >> 172.21.15.46:0/464899078 conn(0x7fcb680a5d10 msgr2=0x7fcb6813b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:02.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.321+0000 7fcb4dffb640 1 -- 172.21.15.46:0/464899078 shutdown_connections 2024-09-16T04:39:02.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.321+0000 7fcb4dffb640 1 -- 172.21.15.46:0/464899078 wait complete. 2024-09-16T04:39:02.334 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.2 2024-09-16T04:39:02.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.355+0000 7f11dbcc9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T04:39:02.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.355+0000 7f11dbcc9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T04:39:02.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.355+0000 7f11dbcc9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:02.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.355+0000 7f11dbcc9640 1 -- 172.21.15.46:0/4061370527 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f11d4067e20 con 0x7f11d40a3c10 2024-09-16T04:39:02.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.355+0000 7f11ca7fc640 1 -- 172.21.15.46:0/4061370527 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f11d001fd70 con 0x7f11d40a3c10 2024-09-16T04:39:02.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.355+0000 7f11ca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T04:39:02.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.355+0000 7f11ca7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:02.358 INFO:teuthology.orchestra.run.smithi046.stdout:51539607556 2024-09-16T04:39:02.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.357+0000 7f11dbcc9640 1 -- 172.21.15.46:0/4061370527 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f11b80383b0 msgr2=0x7f11b803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.358+0000 7f11dbcc9640 1 --2- 172.21.15.46:0/4061370527 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f11b80383b0 0x7f11b803a870 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f11c4034000 tx=0x7f11c402e000 comp rx=0 tx=0).stop 2024-09-16T04:39:02.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.358+0000 7f11dbcc9640 10 monclient: shutdown 2024-09-16T04:39:02.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.358+0000 7f11dbcc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:02.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.358+0000 7f11dbcc9640 1 -- 172.21.15.46:0/4061370527 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11d40a3c10 msgr2=0x7f11d411f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.358+0000 7f11dbcc9640 1 --2- 172.21.15.46:0/4061370527 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11d40a3c10 0x7f11d411f860 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f11d0009f50 tx=0x7f11d0017560 comp rx=0 tx=0).stop 2024-09-16T04:39:02.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.358+0000 7f11dbcc9640 1 -- 172.21.15.46:0/4061370527 shutdown_connections 2024-09-16T04:39:02.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.358+0000 7f11dbcc9640 1 --2- 172.21.15.46:0/4061370527 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f11b80383b0 0x7f11b803a870 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.358+0000 7f11dbcc9640 1 --2- 172.21.15.46:0/4061370527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d41369d0 0x7f11d41202e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.358+0000 7f11dbcc9640 1 --2- 172.21.15.46:0/4061370527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11d40a3ff0 0x7f11d411fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.358+0000 7f11dbcc9640 1 --2- 172.21.15.46:0/4061370527 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11d40a3c10 0x7f11d411f860 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.358+0000 7f11dbcc9640 1 -- 172.21.15.46:0/4061370527 >> 172.21.15.46:0/4061370527 conn(0x7f11d40a6340 msgr2=0x7f11d40a23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:02.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.358+0000 7f11dbcc9640 1 -- 172.21.15.46:0/4061370527 shutdown_connections 2024-09-16T04:39:02.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.358+0000 7f11dbcc9640 1 -- 172.21.15.46:0/4061370527 wait complete. 2024-09-16T04:39:02.370 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607556 for osd.3 2024-09-16T04:39:02.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.393+0000 7f6557df8640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T04:39:02.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.393+0000 7f6557df8640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T04:39:02.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.393+0000 7f6557df8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:02.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.393+0000 7f6557df8640 1 -- 172.21.15.46:0/478125436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f65500a50f0 con 0x7f655013b860 2024-09-16T04:39:02.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.394+0000 7f65467fc640 1 -- 172.21.15.46:0/478125436 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f654801c760 con 0x7f655013b860 2024-09-16T04:39:02.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.394+0000 7f65467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T04:39:02.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.394+0000 7f65467fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:02.396 INFO:teuthology.orchestra.run.smithi046.stdout:51539607556 2024-09-16T04:39:02.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.396+0000 7f6557df8640 1 -- 172.21.15.46:0/478125436 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f65340383b0 msgr2=0x7f653403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.396+0000 7f6557df8640 1 --2- 172.21.15.46:0/478125436 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f65340383b0 0x7f653403a870 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f654c0019d0 tx=0x7f654c001000 comp rx=0 tx=0).stop 2024-09-16T04:39:02.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.396+0000 7f6557df8640 10 monclient: shutdown 2024-09-16T04:39:02.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.396+0000 7f6557df8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:02.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.396+0000 7f6557df8640 1 -- 172.21.15.46:0/478125436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f655013b860 msgr2=0x7f65500aac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.396+0000 7f6557df8640 1 --2- 172.21.15.46:0/478125436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f655013b860 0x7f65500aac10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f65500741d0 tx=0x7f654800ff70 comp rx=0 tx=0).stop 2024-09-16T04:39:02.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.396+0000 7f6557df8640 1 -- 172.21.15.46:0/478125436 shutdown_connections 2024-09-16T04:39:02.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.396+0000 7f6557df8640 1 --2- 172.21.15.46:0/478125436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f655013b860 0x7f65500aac10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.396+0000 7f6557df8640 1 --2- 172.21.15.46:0/478125436 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f655013b420 0x7f65500aa6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.396+0000 7f6557df8640 1 --2- 172.21.15.46:0/478125436 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f65340383b0 0x7f653403a870 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.396+0000 7f6557df8640 1 --2- 172.21.15.46:0/478125436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f653c004800 0x7f65501c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.396+0000 7f6557df8640 1 -- 172.21.15.46:0/478125436 >> 172.21.15.46:0/478125436 conn(0x7f65500a7d00 msgr2=0x7f655006cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:02.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.396+0000 7f6557df8640 1 -- 172.21.15.46:0/478125436 shutdown_connections 2024-09-16T04:39:02.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.396+0000 7f6557df8640 1 -- 172.21.15.46:0/478125436 wait complete. 2024-09-16T04:39:02.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.398+0000 7fb5455f4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T04:39:02.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.398+0000 7fb5455f4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T04:39:02.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.398+0000 7fb5455f4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:02.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.398+0000 7fb5455f4640 1 -- 172.21.15.46:0/2888558010 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fb524005370 con 0x7fb540130900 2024-09-16T04:39:02.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.399+0000 7fb537fff640 1 -- 172.21.15.46:0/2888558010 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb530011bc0 con 0x7fb540130900 2024-09-16T04:39:02.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.399+0000 7fb537fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T04:39:02.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.399+0000 7fb537fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:02.402 INFO:teuthology.orchestra.run.smithi046.stdout:47244640260 2024-09-16T04:39:02.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.401+0000 7fb5455f4640 1 -- 172.21.15.46:0/2888558010 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb520038400 msgr2=0x7fb52003a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.401+0000 7fb5455f4640 1 --2- 172.21.15.46:0/2888558010 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb520038400 0x7fb52003a8c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb528000c00 tx=0x7fb52802e000 comp rx=0 tx=0).stop 2024-09-16T04:39:02.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.401+0000 7fb5455f4640 10 monclient: shutdown 2024-09-16T04:39:02.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.401+0000 7fb5455f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:02.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.401+0000 7fb5455f4640 1 -- 172.21.15.46:0/2888558010 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb540130900 msgr2=0x7fb5401c7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:02.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.401+0000 7fb5455f4640 1 --2- 172.21.15.46:0/2888558010 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb540130900 0x7fb5401c7790 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb53000c3e0 tx=0x7fb53000c8a0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.401+0000 7fb5455f4640 1 -- 172.21.15.46:0/2888558010 shutdown_connections 2024-09-16T04:39:02.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.401+0000 7fb5455f4640 1 --2- 172.21.15.46:0/2888558010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb54012ff50 0x7fb5401326d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.401+0000 7fb5455f4640 1 --2- 172.21.15.46:0/2888558010 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb520038400 0x7fb52003a8c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.401+0000 7fb5455f4640 1 --2- 172.21.15.46:0/2888558010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb540139390 0x7fb54012fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.401+0000 7fb5455f4640 1 --2- 172.21.15.46:0/2888558010 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb540130900 0x7fb5401c7790 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:02.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.401+0000 7fb5455f4640 1 -- 172.21.15.46:0/2888558010 >> 172.21.15.46:0/2888558010 conn(0x7fb5400a5960 msgr2=0x7fb54009f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:02.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.401+0000 7fb5455f4640 1 -- 172.21.15.46:0/2888558010 shutdown_connections 2024-09-16T04:39:02.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:02.401+0000 7fb5455f4640 1 -- 172.21.15.46:0/2888558010 wait complete. 2024-09-16T04:39:02.408 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607556 for osd.4 2024-09-16T04:39:02.413 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.1 2024-09-16T04:39:03.213 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T04:39:03.252 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:39:03.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.328+0000 7fd8f872c640 10 monclient: get_monmap_and_config 2024-09-16T04:39:03.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.328+0000 7fd8f872c640 10 monclient: build_initial_monmap 2024-09-16T04:39:03.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.328+0000 7fd8f872c640 10 monclient: monmap: 2024-09-16T04:39:03.330 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:03.330 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.330 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:03.328941+0000 2024-09-16T04:39:03.330 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:03.328941+0000 2024-09-16T04:39:03.330 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:03.330 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.330 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:03.330 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:03.330 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:03.330 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:03.330 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:03.331 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.329+0000 7fd8f872c640 1 Processor -- start 2024-09-16T04:39:03.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.329+0000 7fd8f872c640 1 -- start start 2024-09-16T04:39:03.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.329+0000 7fd8f872c640 10 monclient: init 2024-09-16T04:39:03.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.329+0000 7fd8f872c640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:03.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.329+0000 7fd8f872c640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-16T04:39:03.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.329+0000 7fd8f872c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd8f00a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:03.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.329+0000 7fd8f872c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd8f00a3ff0 0x7fd8f00a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.329+0000 7fd8f872c640 10 monclient(hunting): picked mon.noname-b con 0x7fd8f00a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:03.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.329+0000 7fd8f872c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8f013be90 0x7fd8f013e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.329+0000 7fd8f872c640 10 monclient(hunting): picked mon.noname-a con 0x7fd8f013be90 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:03.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.329+0000 7fd8f872c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8f009ab70 con 0x7fd8f00a3c10 2024-09-16T04:39:03.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.329+0000 7fd8f872c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.329+0000 7fd8f872c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd8f00a4d60 con 0x7fd8f013be90 2024-09-16T04:39:03.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.329+0000 7fd8f872c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.329+0000 7fd8f872c640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd8f009b030 con 0x7fd8f00a3ff0 2024-09-16T04:39:03.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.329+0000 7fd8f872c640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:03.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.329+0000 7fd8f872c640 10 monclient(hunting): authenticate will time out at 741.744896s 2024-09-16T04:39:03.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f6cc9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8f013be90 0x7fd8f013e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f5cc7640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd8f00a3ff0 0x7fd8f00a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f6cc9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8f013be90 0x7fd8f013e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59408/0 (socket says 172.21.15.46:59408) 2024-09-16T04:39:03.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f5cc7640 10 monclient(hunting): get_auth_request con 0x7fd8f00a3ff0 auth_method 0 2024-09-16T04:39:03.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f6cc9640 1 -- 172.21.15.46:0/4208878071 learned_addr learned my addr 172.21.15.46:0/4208878071 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:03.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f5cc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f5cc7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f5cc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f6cc9640 10 monclient(hunting): get_auth_request con 0x7fd8f013be90 auth_method 0 2024-09-16T04:39:03.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f6cc9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f6cc9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f6cc9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f5cc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f5cc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f5cc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f6cc9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f6cc9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f6cc9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f6cc9640 10 monclient(hunting): handle_auth_done global_id 4323 payload 274 2024-09-16T04:39:03.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f6cc9640 1 -- 172.21.15.46:0/4208878071 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd8f00a3ff0 msgr2=0x7fd8f00a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:03.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f6cc9640 1 --2- 172.21.15.46:0/4208878071 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd8f00a3ff0 0x7fd8f00a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f6cc9640 1 -- 172.21.15.46:0/4208878071 >> v1:172.21.15.114:6789/0 conn(0x7fd8f00a3c10 legacy=0x7fd8f009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f6cc9640 10 monclient: _finish_hunting 0 2024-09-16T04:39:03.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f6cc9640 1 monclient: found mon.noname-a 2024-09-16T04:39:03.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f6cc9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:03.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f6cc9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:03.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f6cc9640 1 -- 172.21.15.46:0/4208878071 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f009b310 con 0x7fd8f013be90 2024-09-16T04:39:03.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.330+0000 7fd8f5cc7640 1 --2- 172.21.15.46:0/4208878071 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd8f00a3ff0 0x7fd8f00a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:03.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.331+0000 7fd8f6cc9640 1 --2- 172.21.15.46:0/4208878071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f013be90 0x7fd8f013e280 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8009870 tx=0x7fd8e802a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=105dedddcbdfae9d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:03.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.331+0000 7fd8f54c6640 1 -- 172.21.15.46:0/4208878071 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8037070 con 0x7fd8f013be90 2024-09-16T04:39:03.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.331+0000 7fd8f54c6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.331+0000 7fd8f54c6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:03.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.331+0000 7fd8f54c6640 10 monclient: dump: 2024-09-16T04:39:03.337 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.337 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.337 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.337 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.338 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.338 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.338 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.338 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.338 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.338 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.331+0000 7fd8f54c6640 10 monclient: _finish_auth 0 2024-09-16T04:39:03.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.331+0000 7fd8f54c6640 10 monclient: _check_auth_tickets 2024-09-16T04:39:03.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.331+0000 7fd8f54c6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:03.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.331+0000 7fd8f872c640 5 monclient: authenticate success, global_id 4323 2024-09-16T04:39:03.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.331+0000 7fd8f54c6640 1 -- 172.21.15.46:0/4208878071 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8e8032440 con 0x7fd8f013be90 2024-09-16T04:39:03.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.331+0000 7fd8f872c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:03.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.331+0000 7fd8f54c6640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:03.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.331+0000 7fd8f872c640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.331+0000 7fd8f54c6640 1 -- 172.21.15.46:0/4208878071 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8032810 con 0x7fd8f013be90 2024-09-16T04:39:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.331+0000 7fd8f872c640 4 set_mon_vals no callback set 2024-09-16T04:39:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.331+0000 7fd8f54c6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.331+0000 7fd8f54c6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.331+0000 7fd8f54c6640 10 monclient: dump: 2024-09-16T04:39:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.339 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.331+0000 7fd8f872c640 10 monclient: shutdown 2024-09-16T04:39:03.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.332+0000 7fd8f872c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:03.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.332+0000 7fd8f872c640 1 -- 172.21.15.46:0/4208878071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f013be90 msgr2=0x7fd8f013e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.332+0000 7fd8f872c640 1 --2- 172.21.15.46:0/4208878071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f013be90 0x7fd8f013e280 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8009870 tx=0x7fd8e802a130 comp rx=0 tx=0).stop 2024-09-16T04:39:03.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.332+0000 7fd8f872c640 1 -- 172.21.15.46:0/4208878071 shutdown_connections 2024-09-16T04:39:03.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.332+0000 7fd8f872c640 1 --2- 172.21.15.46:0/4208878071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f013be90 0x7fd8f013e280 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.332+0000 7fd8f872c640 1 --2- 172.21.15.46:0/4208878071 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd8f00a3ff0 0x7fd8f00a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.332+0000 7fd8f872c640 1 -- 172.21.15.46:0/4208878071 >> 172.21.15.46:0/4208878071 conn(0x7fd8f00a5d10 msgr2=0x7fd8f00a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:03.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.332+0000 7fd8f872c640 1 -- 172.21.15.46:0/4208878071 shutdown_connections 2024-09-16T04:39:03.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.332+0000 7fd8f872c640 1 -- 172.21.15.46:0/4208878071 wait complete. 2024-09-16T04:39:03.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.332+0000 7fd8f872c640 10 monclient: build_initial_monmap 2024-09-16T04:39:03.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.332+0000 7fd8f872c640 10 monclient: monmap: 2024-09-16T04:39:03.340 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:03.341 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:03.341 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:03.341 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:03.341 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:03.341 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.341 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:03.341 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:03.341 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:03.341 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f872c640 1 Processor -- start 2024-09-16T04:39:03.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f872c640 1 -- start start 2024-09-16T04:39:03.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f872c640 10 monclient: init 2024-09-16T04:39:03.342 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:39:03.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f872c640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:03.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f872c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:39:03.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f872c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8f00a3c10 0x7fd8f01cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f872c640 10 monclient(hunting): picked mon.noname-b con 0x7fd8f00a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:03.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f872c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd8f00a3ff0 0x7fd8f0136ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f872c640 10 monclient(hunting): picked mon.noname-c con 0x7fd8f00a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:03.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f872c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f013be90 0x7fd8f00a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f872c640 10 monclient(hunting): picked mon.noname-a con 0x7fd8f013be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:03.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f872c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f872c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f006e550 con 0x7fd8f013be90 2024-09-16T04:39:03.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f872c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f872c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f006e3d0 con 0x7fd8f00a3c10 2024-09-16T04:39:03.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f872c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f872c640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd8f013e7c0 con 0x7fd8f00a3ff0 2024-09-16T04:39:03.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f872c640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:03.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f872c640 10 monclient(hunting): authenticate will time out at 741.748638s 2024-09-16T04:39:03.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f5cc7640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd8f00a3ff0 0x7fd8f0136ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f6cc9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f013be90 0x7fd8f00a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f5cc7640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd8f00a3ff0 0x7fd8f0136ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:32920/0 (socket says 172.21.15.46:32920) 2024-09-16T04:39:03.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f5cc7640 1 -- 172.21.15.46:0/3448657720 learned_addr learned my addr 172.21.15.46:0/3448657720 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:03.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f5cc7640 10 monclient(hunting): get_auth_request con 0x7fd8f00a3ff0 auth_method 0 2024-09-16T04:39:03.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f5cc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f5cc7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f5cc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f6cc9640 10 monclient(hunting): get_auth_request con 0x7fd8f013be90 auth_method 0 2024-09-16T04:39:03.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f6cc9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f6cc9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f6cc9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f64c8640 1 --2- 172.21.15.46:0/3448657720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8f00a3c10 0x7fd8f01cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f5cc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f5cc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f5cc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f64c8640 10 monclient(hunting): get_auth_request con 0x7fd8f00a3c10 auth_method 0 2024-09-16T04:39:03.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f64c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f64c8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f64c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f6cc9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f6cc9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.333+0000 7fd8f6cc9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f64c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f64c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f64c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f5cc7640 10 monclient(hunting): handle_auth_done global_id 4289 payload 824 2024-09-16T04:39:03.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f5cc7640 1 -- 172.21.15.46:0/3448657720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8f00a3c10 msgr2=0x7fd8f01cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f5cc7640 1 --2- 172.21.15.46:0/3448657720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8f00a3c10 0x7fd8f01cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f5cc7640 1 -- 172.21.15.46:0/3448657720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f013be90 msgr2=0x7fd8f00a1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f5cc7640 1 --2- 172.21.15.46:0/3448657720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f013be90 0x7fd8f00a1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f5cc7640 10 monclient: _finish_hunting 0 2024-09-16T04:39:03.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f5cc7640 1 monclient: found mon.noname-c 2024-09-16T04:39:03.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f5cc7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:03.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f5cc7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f5cc7640 1 -- 172.21.15.46:0/3448657720 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f013cdb0 con 0x7fd8f00a3ff0 2024-09-16T04:39:03.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f6cc9640 1 --2- 172.21.15.46:0/3448657720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f013be90 0x7fd8f00a1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:03.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f5cc7640 1 --2- 172.21.15.46:0/3448657720 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd8f00a3ff0 0x7fd8f0136ff0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000ebe0 tx=0x7fd8e000c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:03.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8e6ffd640 1 -- 172.21.15.46:0/3448657720 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0017070 con 0x7fd8f00a3ff0 2024-09-16T04:39:03.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8e6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:39:03.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8e6ffd640 10 monclient: dump: 2024-09-16T04:39:03.351 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.351 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.351 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.351 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.351 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.351 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.352 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.352 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.352 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.352 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8e6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:39:03.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8e6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:39:03.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:03.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8e6ffd640 1 -- 172.21.15.46:0/3448657720 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8e000cc30 con 0x7fd8f00a3ff0 2024-09-16T04:39:03.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8e6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:03.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8e6ffd640 1 -- 172.21.15.46:0/3448657720 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0013470 con 0x7fd8f00a3ff0 2024-09-16T04:39:03.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:03.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8e6ffd640 10 monclient: dump: 2024-09-16T04:39:03.353 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.353 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.353 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.353 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.353 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.353 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.353 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.353 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.353 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.353 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f872c640 5 monclient: authenticate success, global_id 4289 2024-09-16T04:39:03.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f872c640 10 monclient: _renew_subs 2024-09-16T04:39:03.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f872c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f872c640 1 -- 172.21.15.46:0/3448657720 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8f013c880 con 0x7fd8f00a3ff0 2024-09-16T04:39:03.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f4cc5640 4 set_mon_vals no callback set 2024-09-16T04:39:03.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f872c640 10 monclient: _renew_subs 2024-09-16T04:39:03.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f872c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.334+0000 7fd8f872c640 1 -- 172.21.15.46:0/3448657720 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8f0121100 con 0x7fd8f00a3ff0 2024-09-16T04:39:03.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.335+0000 7fd8f872c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:03.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.335+0000 7fd8f872c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:03.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.335+0000 7fd8f872c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.335+0000 7fd8f872c640 1 -- 172.21.15.46:0/3448657720 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8f00a43d0 con 0x7fd8f00a3ff0 2024-09-16T04:39:03.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.335+0000 7fd8e6ffd640 1 -- 172.21.15.46:0/3448657720 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fd8e001a070 con 0x7fd8f00a3ff0 2024-09-16T04:39:03.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.335+0000 7fd8e6ffd640 1 --2- 172.21.15.46:0/3448657720 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd8d40383b0 0x7fd8d403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.336+0000 7fd8e6ffd640 1 -- 172.21.15.46:0/3448657720 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fd8e004b5c0 con 0x7fd8f00a3ff0 2024-09-16T04:39:03.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.336+0000 7fd8f64c8640 1 --2- 172.21.15.46:0/3448657720 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd8d40383b0 0x7fd8d403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.336+0000 7fd8f64c8640 10 monclient: get_auth_request con 0x7fd8d40383b0 auth_method 0 2024-09-16T04:39:03.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.336+0000 7fd8f64c8640 1 --2- 172.21.15.46:0/3448657720 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd8d40383b0 0x7fd8d403a870 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec001e70 tx=0x7fd8ec009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:03.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.340+0000 7fd8e6ffd640 1 -- 172.21.15.46:0/3448657720 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd8f013c880 con 0x7fd8f00a3ff0 2024-09-16T04:39:03.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.340+0000 7fd8e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:03.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.340+0000 7fd8e6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:03.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.367+0000 7f105607b640 10 monclient: get_monmap_and_config 2024-09-16T04:39:03.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.367+0000 7f105607b640 10 monclient: build_initial_monmap 2024-09-16T04:39:03.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.367+0000 7f105607b640 10 monclient: monmap: 2024-09-16T04:39:03.369 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:03.369 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.369 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:03.368367+0000 2024-09-16T04:39:03.369 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:03.368367+0000 2024-09-16T04:39:03.369 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:03.369 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.369 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:03.369 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:03.369 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:03.370 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:03.370 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:03.370 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.368+0000 7f105607b640 1 Processor -- start 2024-09-16T04:39:03.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.368+0000 7f105607b640 1 -- start start 2024-09-16T04:39:03.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.368+0000 7f105607b640 10 monclient: init 2024-09-16T04:39:03.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f105607b640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:03.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f105607b640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-16T04:39:03.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f105607b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f10500a3c10 0x7f105009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f105607b640 10 monclient(hunting): picked mon.noname-a con 0x7f10500a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:03.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f105607b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f10500a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:03.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f105607b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f105013be90 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:03.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f105607b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f105009ab70 con 0x7f105013be90 2024-09-16T04:39:03.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f105607b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10500a4d60 con 0x7f10500a3ff0 2024-09-16T04:39:03.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f105607b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f105607b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f105009b030 con 0x7f10500a3c10 2024-09-16T04:39:03.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f105607b640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:03.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f105607b640 10 monclient(hunting): authenticate will time out at 741.784311s 2024-09-16T04:39:03.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f104ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f105013be90 0x7f105013e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53058/0 (socket says 172.21.15.46:53058) 2024-09-16T04:39:03.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f104f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f10500a3c10 0x7f105009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f104ffff640 1 -- 172.21.15.46:0/4293309464 learned_addr learned my addr 172.21.15.46:0/4293309464 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f104f7fe640 10 monclient(hunting): get_auth_request con 0x7f10500a3c10 auth_method 0 2024-09-16T04:39:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f104f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f104f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f104f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f104f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f104f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f104f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f104f7fe640 10 monclient(hunting): handle_auth_done global_id 4329 payload 274 2024-09-16T04:39:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f104f7fe640 1 -- 172.21.15.46:0/4293309464 >> v1:172.21.15.114:6789/0 conn(0x7f10500a3ff0 legacy=0x7f10500a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f104f7fe640 1 -- 172.21.15.46:0/4293309464 >> v1:172.21.15.46:6789/0 conn(0x7f105013be90 legacy=0x7f105013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f104f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:39:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f104f7fe640 1 monclient: found mon.noname-a 2024-09-16T04:39:03.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f104f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:03.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f104f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:03.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f104f7fe640 1 -- 172.21.15.46:0/4293309464 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f105009b310 con 0x7f10500a3c10 2024-09-16T04:39:03.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f104e7fc640 1 -- 172.21.15.46:0/4293309464 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4127083366 0 0) 0x7f105009ab70 con 0x7f105013be90 2024-09-16T04:39:03.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.369+0000 7f104e7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:39:03.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.370+0000 7f104f7fe640 1 --2- 172.21.15.46:0/4293309464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10500a3c10 0x7f105009bcf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f10400097a0 tx=0x7f104002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=808348df543f7cd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:03.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f104e7fc640 1 -- 172.21.15.46:0/4293309464 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1040037070 con 0x7f10500a3c10 2024-09-16T04:39:03.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f104e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f104e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:03.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f104e7fc640 10 monclient: dump: 2024-09-16T04:39:03.376 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.376 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.376 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.376 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.376 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.377 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.377 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.377 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.377 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.377 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f104e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:39:03.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f104e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:39:03.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f104e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:03.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f104e7fc640 1 -- 172.21.15.46:0/4293309464 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1040032440 con 0x7f10500a3c10 2024-09-16T04:39:03.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f105607b640 5 monclient: authenticate success, global_id 4329 2024-09-16T04:39:03.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f105607b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:03.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f104e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:03.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f105607b640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:03.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f105607b640 4 set_mon_vals no callback set 2024-09-16T04:39:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f104e7fc640 1 -- 172.21.15.46:0/4293309464 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104002ae60 con 0x7f10500a3c10 2024-09-16T04:39:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f104e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f104e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f104e7fc640 10 monclient: dump: 2024-09-16T04:39:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.378 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f105607b640 10 monclient: shutdown 2024-09-16T04:39:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f105607b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f105607b640 1 -- 172.21.15.46:0/4293309464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10500a3c10 msgr2=0x7f105009bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f105607b640 1 --2- 172.21.15.46:0/4293309464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10500a3c10 0x7f105009bcf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f10400097a0 tx=0x7f104002a110 comp rx=0 tx=0).stop 2024-09-16T04:39:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f105607b640 1 -- 172.21.15.46:0/4293309464 shutdown_connections 2024-09-16T04:39:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f105607b640 1 --2- 172.21.15.46:0/4293309464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10500a3c10 0x7f105009bcf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f105607b640 1 -- 172.21.15.46:0/4293309464 >> 172.21.15.46:0/4293309464 conn(0x7f10500a5d10 msgr2=0x7f10500a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f105607b640 1 -- 172.21.15.46:0/4293309464 shutdown_connections 2024-09-16T04:39:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.371+0000 7f105607b640 1 -- 172.21.15.46:0/4293309464 wait complete. 2024-09-16T04:39:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.372+0000 7f105607b640 10 monclient: build_initial_monmap 2024-09-16T04:39:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.372+0000 7f105607b640 10 monclient: monmap: 2024-09-16T04:39:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:03.380 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:03.380 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:03.380 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.380 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:03.380 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:03.380 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:03.380 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.380 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:39:03.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.372+0000 7f105607b640 1 Processor -- start 2024-09-16T04:39:03.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f105607b640 1 -- start start 2024-09-16T04:39:03.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f105607b640 10 monclient: init 2024-09-16T04:39:03.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f105607b640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:03.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f105607b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:39:03.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f105607b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10500a3c10 0x7f1050136460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f105607b640 10 monclient(hunting): picked mon.noname-a con 0x7f10500a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:03.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f105607b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f10500a3ff0 0x7f10501369a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f105607b640 10 monclient(hunting): picked mon.noname-c con 0x7f10500a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:03.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f105607b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f105013be90 0x7f1050138b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f105607b640 10 monclient(hunting): picked mon.noname-b con 0x7f105013be90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:03.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f105607b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f105607b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f105006e550 con 0x7f10500a3c10 2024-09-16T04:39:03.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f105607b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f105607b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f105006e3d0 con 0x7f105013be90 2024-09-16T04:39:03.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f105607b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f105607b640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f105013f9a0 con 0x7f10500a3ff0 2024-09-16T04:39:03.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f105607b640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:03.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f105607b640 10 monclient(hunting): authenticate will time out at 741.788551s 2024-09-16T04:39:03.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10500a3c10 0x7f1050136460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10500a3c10 0x7f1050136460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59428/0 (socket says 172.21.15.46:59428) 2024-09-16T04:39:03.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104effd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f10500a3ff0 0x7f10501369a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104f7fe640 1 -- 172.21.15.46:0/3878965616 learned_addr learned my addr 172.21.15.46:0/3878965616 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:03.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104f7fe640 10 monclient(hunting): get_auth_request con 0x7f10500a3c10 auth_method 0 2024-09-16T04:39:03.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104effd640 10 monclient(hunting): get_auth_request con 0x7f10500a3ff0 auth_method 0 2024-09-16T04:39:03.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104ffff640 1 --2- 172.21.15.46:0/3878965616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f105013be90 0x7f1050138b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104ffff640 10 monclient(hunting): get_auth_request con 0x7f105013be90 auth_method 0 2024-09-16T04:39:03.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.373+0000 7f104effd640 10 monclient(hunting): handle_auth_done global_id 4292 payload 824 2024-09-16T04:39:03.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f104effd640 1 -- 172.21.15.46:0/3878965616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f105013be90 msgr2=0x7f1050138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f104effd640 1 --2- 172.21.15.46:0/3878965616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f105013be90 0x7f1050138b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f104effd640 1 -- 172.21.15.46:0/3878965616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10500a3c10 msgr2=0x7f1050136460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f104effd640 1 --2- 172.21.15.46:0/3878965616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10500a3c10 0x7f1050136460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f104effd640 10 monclient: _finish_hunting 0 2024-09-16T04:39:03.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f104effd640 1 monclient: found mon.noname-c 2024-09-16T04:39:03.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f104effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:03.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f104effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f104effd640 1 -- 172.21.15.46:0/3878965616 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1050120170 con 0x7f10500a3ff0 2024-09-16T04:39:03.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f104f7fe640 1 --2- 172.21.15.46:0/3878965616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10500a3c10 0x7f1050136460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:03.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f104effd640 1 --2- 172.21.15.46:0/3878965616 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f10500a3ff0 0x7f10501369a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f104400c390 tx=0x7f104400c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:03.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f1037fff640 1 -- 172.21.15.46:0/3878965616 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1044017070 con 0x7f10500a3ff0 2024-09-16T04:39:03.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f1037fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f1037fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:39:03.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f1037fff640 10 monclient: dump: 2024-09-16T04:39:03.390 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.390 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.390 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.390 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.390 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.390 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.390 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.390 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.390 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.390 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f1037fff640 10 monclient: _finish_auth 0 2024-09-16T04:39:03.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f1037fff640 10 monclient: _check_auth_tickets 2024-09-16T04:39:03.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f1037fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:03.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f1037fff640 1 -- 172.21.15.46:0/3878965616 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10440052d0 con 0x7f10500a3ff0 2024-09-16T04:39:03.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f1037fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:03.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f1037fff640 1 -- 172.21.15.46:0/3878965616 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1044003830 con 0x7f10500a3ff0 2024-09-16T04:39:03.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f105607b640 5 monclient: authenticate success, global_id 4292 2024-09-16T04:39:03.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f1037fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f1037fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:03.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f1037fff640 10 monclient: dump: 2024-09-16T04:39:03.391 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.391 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.392 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.392 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.392 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.392 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.392 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.392 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.392 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.392 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f105607b640 10 monclient: _renew_subs 2024-09-16T04:39:03.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f105607b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f104dffb640 4 set_mon_vals no callback set 2024-09-16T04:39:03.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f105607b640 1 -- 172.21.15.46:0/3878965616 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10500a07c0 con 0x7f10500a3ff0 2024-09-16T04:39:03.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f105607b640 10 monclient: _renew_subs 2024-09-16T04:39:03.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f105607b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.374+0000 7f105607b640 1 -- 172.21.15.46:0/3878965616 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f105013b070 con 0x7f10500a3ff0 2024-09-16T04:39:03.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.375+0000 7f1037fff640 1 -- 172.21.15.46:0/3878965616 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f104401a070 con 0x7f10500a3ff0 2024-09-16T04:39:03.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.375+0000 7f1037fff640 1 --2- 172.21.15.46:0/3878965616 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f103c0383b0 0x7f103c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.375+0000 7f1037fff640 1 -- 172.21.15.46:0/3878965616 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f104404a970 con 0x7f10500a3ff0 2024-09-16T04:39:03.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.375+0000 7f105607b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:03.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.375+0000 7f105607b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:03.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.375+0000 7f105607b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.375+0000 7f105607b640 1 -- 172.21.15.46:0/3878965616 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f105009bcf0 con 0x7f10500a3ff0 2024-09-16T04:39:03.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.376+0000 7f104f7fe640 1 --2- 172.21.15.46:0/3878965616 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f103c0383b0 0x7f103c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.376+0000 7f104f7fe640 10 monclient: get_auth_request con 0x7f103c0383b0 auth_method 0 2024-09-16T04:39:03.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.376+0000 7f104f7fe640 1 --2- 172.21.15.46:0/3878965616 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f103c0383b0 0x7f103c03a870 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f104002a150 tx=0x7f104002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:03.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.381+0000 7f1037fff640 1 -- 172.21.15.46:0/3878965616 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1044019e10 con 0x7f10500a3ff0 2024-09-16T04:39:03.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.381+0000 7f1037fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:03.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.381+0000 7f1037fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:03.409 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T04:39:03.414 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T04:39:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.475+0000 7f8a3bc2e640 10 monclient: get_monmap_and_config 2024-09-16T04:39:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.475+0000 7f8a3bc2e640 10 monclient: build_initial_monmap 2024-09-16T04:39:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.475+0000 7f8a3bc2e640 10 monclient: monmap: 2024-09-16T04:39:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:03.476156+0000 2024-09-16T04:39:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:03.476156+0000 2024-09-16T04:39:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:03.478 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.478 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:03.478 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:03.478 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:03.478 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:03.478 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:03.478 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.476+0000 7f8a3bc2e640 1 Processor -- start 2024-09-16T04:39:03.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.476+0000 7f8a3bc2e640 1 -- start start 2024-09-16T04:39:03.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.476+0000 7f8a3bc2e640 10 monclient: init 2024-09-16T04:39:03.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.477+0000 7f8a3bc2e640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:03.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.477+0000 7f8a3bc2e640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-16T04:39:03.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.477+0000 7f8a3bc2e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8a340a3c10 0x7f8a3409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.477+0000 7f8a3bc2e640 10 monclient(hunting): picked mon.noname-b con 0x7f8a340a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:03.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.477+0000 7f8a3bc2e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8a340a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:03.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.477+0000 7f8a3bc2e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8a3413be90 0x7f8a3413e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.477+0000 7f8a3bc2e640 10 monclient(hunting): picked mon.noname-a con 0x7f8a3413be90 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:03.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.477+0000 7f8a3bc2e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a3409ab70 con 0x7f8a340a3ff0 2024-09-16T04:39:03.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.477+0000 7f8a3a1cb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8a3413be90 0x7f8a3413e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3a1cb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8a3413be90 0x7f8a3413e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59432/0 (socket says 172.21.15.46:59432) 2024-09-16T04:39:03.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3a1cb640 1 -- 172.21.15.46:0/1331577402 learned_addr learned my addr 172.21.15.46:0/1331577402 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:03.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3bc2e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3bc2e640 1 -- 172.21.15.46:0/1331577402 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f8a340a4d60 con 0x7f8a3413be90 2024-09-16T04:39:03.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3bc2e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3bc2e640 1 -- 172.21.15.46:0/1331577402 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f8a3409b030 con 0x7f8a340a3c10 2024-09-16T04:39:03.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3bc2e640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:03.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3bc2e640 10 monclient(hunting): authenticate will time out at 741.893156s 2024-09-16T04:39:03.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3a1cb640 10 monclient(hunting): get_auth_request con 0x7f8a3413be90 auth_method 0 2024-09-16T04:39:03.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3a1cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3a1cb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3a1cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a399ca640 1 --2- 172.21.15.46:0/1331577402 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8a340a3c10 0x7f8a3409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a399ca640 10 monclient(hunting): get_auth_request con 0x7f8a340a3c10 auth_method 0 2024-09-16T04:39:03.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a399ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a399ca640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a399ca640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3a1cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3a1cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3a1cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a399ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a399ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a399ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3a1cb640 10 monclient(hunting): handle_auth_done global_id 4338 payload 274 2024-09-16T04:39:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3a1cb640 1 -- 172.21.15.46:0/1331577402 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8a340a3c10 msgr2=0x7f8a3409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3a1cb640 1 --2- 172.21.15.46:0/1331577402 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8a340a3c10 0x7f8a3409bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3a1cb640 1 -- 172.21.15.46:0/1331577402 >> v1:172.21.15.114:6789/0 conn(0x7f8a340a3ff0 legacy=0x7f8a340a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3a1cb640 10 monclient: _finish_hunting 0 2024-09-16T04:39:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3a1cb640 1 monclient: found mon.noname-a 2024-09-16T04:39:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3a1cb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:03.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3a1cb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:03.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a3a1cb640 1 -- 172.21.15.46:0/1331577402 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a3409b310 con 0x7f8a3413be90 2024-09-16T04:39:03.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.478+0000 7f8a399ca640 1 --2- 172.21.15.46:0/1331577402 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8a340a3c10 0x7f8a3409bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:03.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.480+0000 7f8a3a1cb640 1 --2- 172.21.15.46:0/1331577402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a3413be90 0x7f8a3413e280 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c009e50 tx=0x7f8a2c02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9933a388b59ec94b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:03.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.480+0000 7f8a389c8640 1 -- 172.21.15.46:0/1331577402 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c036070 con 0x7f8a3413be90 2024-09-16T04:39:03.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.480+0000 7f8a389c8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.480+0000 7f8a389c8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:03.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.480+0000 7f8a389c8640 10 monclient: dump: 2024-09-16T04:39:03.497 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.497 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.497 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.497 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.497 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.497 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.497 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.497 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.497 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.497 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.480+0000 7f8a389c8640 10 monclient: _finish_auth 0 2024-09-16T04:39:03.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.480+0000 7f8a389c8640 10 monclient: _check_auth_tickets 2024-09-16T04:39:03.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.480+0000 7f8a389c8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:03.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.480+0000 7f8a389c8640 1 -- 172.21.15.46:0/1331577402 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a2c031600 con 0x7f8a3413be90 2024-09-16T04:39:03.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.480+0000 7f8a3bc2e640 5 monclient: authenticate success, global_id 4338 2024-09-16T04:39:03.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.480+0000 7f8a3bc2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:03.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.480+0000 7f8a389c8640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:03.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.480+0000 7f8a389c8640 1 -- 172.21.15.46:0/1331577402 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c031a10 con 0x7f8a3413be90 2024-09-16T04:39:03.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.480+0000 7f8a389c8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.480+0000 7f8a389c8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:03.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.480+0000 7f8a389c8640 10 monclient: dump: 2024-09-16T04:39:03.499 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.499 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.499 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.499 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.499 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.499 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.499 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.499 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.499 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.499 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.480+0000 7f8a3bc2e640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:03.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.480+0000 7f8a3bc2e640 4 set_mon_vals no callback set 2024-09-16T04:39:03.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.481+0000 7f8a3bc2e640 10 monclient: shutdown 2024-09-16T04:39:03.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.481+0000 7f8a3bc2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:03.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.481+0000 7f8a3bc2e640 1 -- 172.21.15.46:0/1331577402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a3413be90 msgr2=0x7f8a3413e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.481+0000 7f8a3bc2e640 1 --2- 172.21.15.46:0/1331577402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a3413be90 0x7f8a3413e280 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c009e50 tx=0x7f8a2c02a610 comp rx=0 tx=0).stop 2024-09-16T04:39:03.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.481+0000 7f8a3bc2e640 1 -- 172.21.15.46:0/1331577402 shutdown_connections 2024-09-16T04:39:03.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.481+0000 7f8a3bc2e640 1 --2- 172.21.15.46:0/1331577402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a3413be90 0x7f8a3413e280 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.481+0000 7f8a3bc2e640 1 --2- 172.21.15.46:0/1331577402 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8a340a3c10 0x7f8a3409bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.481+0000 7f8a3bc2e640 1 -- 172.21.15.46:0/1331577402 >> 172.21.15.46:0/1331577402 conn(0x7f8a340a5d10 msgr2=0x7f8a340a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:03.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.481+0000 7f8a3bc2e640 1 -- 172.21.15.46:0/1331577402 shutdown_connections 2024-09-16T04:39:03.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.481+0000 7f8a3bc2e640 1 -- 172.21.15.46:0/1331577402 wait complete. 2024-09-16T04:39:03.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.482+0000 7f8a3bc2e640 10 monclient: build_initial_monmap 2024-09-16T04:39:03.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.482+0000 7f8a3bc2e640 10 monclient: monmap: 2024-09-16T04:39:03.500 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:03.501 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:03.501 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:03.501 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:03.501 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:03.501 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.501 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:03.501 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:03.501 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:03.501 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.483+0000 7f8a3bc2e640 1 Processor -- start 2024-09-16T04:39:03.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.483+0000 7f8a3bc2e640 1 -- start start 2024-09-16T04:39:03.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.483+0000 7f8a3bc2e640 10 monclient: init 2024-09-16T04:39:03.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a3bc2e640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:03.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a3bc2e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:39:03.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a3bc2e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a340a3c10 0x7f8a341cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a3bc2e640 10 monclient(hunting): picked mon.noname-b con 0x7f8a340a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:03.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a3bc2e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8a340a3ff0 0x7f8a34136e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a3bc2e640 10 monclient(hunting): picked mon.noname-c con 0x7f8a340a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:03.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a3bc2e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a3413be90 0x7f8a3409ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a391c9640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8a340a3ff0 0x7f8a34136e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a399ca640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a340a3c10 0x7f8a341cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a391c9640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8a340a3ff0 0x7f8a34136e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:32952/0 (socket says 172.21.15.46:32952) 2024-09-16T04:39:03.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a399ca640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a340a3c10 0x7f8a341cb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.46:38434/0 (socket says 172.21.15.46:38434) 2024-09-16T04:39:03.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a3a1cb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a3413be90 0x7f8a3409ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a3a1cb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a3413be90 0x7f8a3409ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59444/0 (socket says 172.21.15.46:59444) 2024-09-16T04:39:03.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a3bc2e640 10 monclient(hunting): picked mon.noname-a con 0x7f8a3413be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:03.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a3bc2e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a3bc2e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3406e550 con 0x7f8a3413be90 2024-09-16T04:39:03.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a3bc2e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a391c9640 1 -- 172.21.15.46:0/1141936145 learned_addr learned my addr 172.21.15.46:0/1141936145 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:03.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a3bc2e640 1 -- 172.21.15.46:0/1141936145 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3406e3d0 con 0x7f8a340a3c10 2024-09-16T04:39:03.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a3bc2e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a3bc2e640 1 -- 172.21.15.46:0/1141936145 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f8a3413e7c0 con 0x7f8a340a3ff0 2024-09-16T04:39:03.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a3bc2e640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:03.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a3bc2e640 10 monclient(hunting): authenticate will time out at 741.899928s 2024-09-16T04:39:03.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a391c9640 10 monclient(hunting): get_auth_request con 0x7f8a340a3ff0 auth_method 0 2024-09-16T04:39:03.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a391c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a391c9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a391c9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a399ca640 10 monclient(hunting): get_auth_request con 0x7f8a340a3c10 auth_method 0 2024-09-16T04:39:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a399ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a399ca640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.484+0000 7f8a399ca640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a3a1cb640 10 monclient(hunting): get_auth_request con 0x7f8a3413be90 auth_method 0 2024-09-16T04:39:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a3a1cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a3a1cb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a3a1cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a391c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a391c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a391c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a399ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a399ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a399ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a3a1cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a3a1cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a3a1cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a391c9640 10 monclient(hunting): handle_auth_done global_id 4298 payload 824 2024-09-16T04:39:03.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a391c9640 1 -- 172.21.15.46:0/1141936145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a340a3c10 msgr2=0x7f8a341cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a391c9640 1 --2- 172.21.15.46:0/1141936145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a340a3c10 0x7f8a341cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a391c9640 1 -- 172.21.15.46:0/1141936145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a3413be90 msgr2=0x7f8a3409ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a391c9640 1 --2- 172.21.15.46:0/1141936145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a3413be90 0x7f8a3409ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a391c9640 10 monclient: _finish_hunting 0 2024-09-16T04:39:03.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a391c9640 1 monclient: found mon.noname-c 2024-09-16T04:39:03.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a391c9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:03.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a391c9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a391c9640 1 -- 172.21.15.46:0/1141936145 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a341cc880 con 0x7f8a340a3ff0 2024-09-16T04:39:03.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.485+0000 7f8a3a1cb640 1 --2- 172.21.15.46:0/1141936145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a3413be90 0x7f8a3409ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:03.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.486+0000 7f8a391c9640 1 --2- 172.21.15.46:0/1141936145 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8a340a3ff0 0x7f8a34136e40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8a24001b00 tx=0x7f8a24014420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:03.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.487+0000 7f8a2a7fc640 1 -- 172.21.15.46:0/1141936145 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a24012670 con 0x7f8a340a3ff0 2024-09-16T04:39:03.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.487+0000 7f8a2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.487+0000 7f8a2a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:39:03.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.487+0000 7f8a2a7fc640 10 monclient: dump: 2024-09-16T04:39:03.506 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.506 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.506 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.506 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.506 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.506 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.507 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.507 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.507 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.507 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.487+0000 7f8a2a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:39:03.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.487+0000 7f8a2a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:39:03.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.487+0000 7f8a2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:03.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.487+0000 7f8a2a7fc640 1 -- 172.21.15.46:0/1141936145 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a24001d10 con 0x7f8a340a3ff0 2024-09-16T04:39:03.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.487+0000 7f8a2a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:03.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.487+0000 7f8a2a7fc640 1 -- 172.21.15.46:0/1141936145 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a24014d50 con 0x7f8a340a3ff0 2024-09-16T04:39:03.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.487+0000 7f8a2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.487+0000 7f8a2a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:03.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.487+0000 7f8a2a7fc640 10 monclient: dump: 2024-09-16T04:39:03.507 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.508 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.508 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.508 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.508 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.508 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.508 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.508 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.508 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.508 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.487+0000 7f8a3bc2e640 5 monclient: authenticate success, global_id 4298 2024-09-16T04:39:03.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.487+0000 7f8a3bc2e640 10 monclient: _renew_subs 2024-09-16T04:39:03.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.487+0000 7f8a2bfff640 4 set_mon_vals no callback set 2024-09-16T04:39:03.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.487+0000 7f8a3bc2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.487+0000 7f8a3bc2e640 1 -- 172.21.15.46:0/1141936145 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a340a0400 con 0x7f8a340a3ff0 2024-09-16T04:39:03.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.487+0000 7f8a3bc2e640 10 monclient: _renew_subs 2024-09-16T04:39:03.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.487+0000 7f8a3bc2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.487+0000 7f8a3bc2e640 1 -- 172.21.15.46:0/1141936145 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a34120260 con 0x7f8a340a3ff0 2024-09-16T04:39:03.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.489+0000 7f8a2a7fc640 1 -- 172.21.15.46:0/1141936145 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f8a240050d0 con 0x7f8a340a3ff0 2024-09-16T04:39:03.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.489+0000 7f8a2a7fc640 1 --2- 172.21.15.46:0/1141936145 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8a180383b0 0x7f8a1803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.489+0000 7f8a2a7fc640 1 -- 172.21.15.46:0/1141936145 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f8a2404b530 con 0x7f8a340a3ff0 2024-09-16T04:39:03.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.489+0000 7f8a3bc2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:03.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.489+0000 7f8a3bc2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:03.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.489+0000 7f8a3bc2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.489+0000 7f8a3bc2e640 1 -- 172.21.15.46:0/1141936145 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a340a3000 con 0x7f8a340a3ff0 2024-09-16T04:39:03.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.490+0000 7f8a399ca640 1 --2- 172.21.15.46:0/1141936145 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8a180383b0 0x7f8a1803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.490+0000 7f8a399ca640 10 monclient: get_auth_request con 0x7f8a180383b0 auth_method 0 2024-09-16T04:39:03.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.491+0000 7f8a399ca640 1 --2- 172.21.15.46:0/1141936145 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8a180383b0 0x7f8a1803a870 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8a30009560 tx=0x7f8a30009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:03.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.497+0000 7fd18cbbd640 10 monclient: get_monmap_and_config 2024-09-16T04:39:03.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.497+0000 7fd18cbbd640 10 monclient: build_initial_monmap 2024-09-16T04:39:03.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.497+0000 7fd18cbbd640 10 monclient: monmap: 2024-09-16T04:39:03.512 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:03.512 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.512 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:03.497914+0000 2024-09-16T04:39:03.512 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:03.497914+0000 2024-09-16T04:39:03.512 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:03.512 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.512 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:03.512 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:03.513 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:03.513 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:03.513 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:03.514 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.498+0000 7fd18cbbd640 1 Processor -- start 2024-09-16T04:39:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.498+0000 7fd18cbbd640 1 -- start start 2024-09-16T04:39:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.498+0000 7fd18cbbd640 10 monclient: init 2024-09-16T04:39:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.498+0000 7fd18cbbd640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.498+0000 7fd18cbbd640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-16T04:39:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.498+0000 7fd18cbbd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd1880a3c10 0x7fd18809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.498+0000 7fd18cbbd640 10 monclient(hunting): picked mon.noname-b con 0x7fd1880a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18cbbd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd1880a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18cbbd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd18813be90 0x7fd18813e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd185d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd1880a3ff0 0x7fd1880a43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53068/0 (socket says 172.21.15.46:53068) 2024-09-16T04:39:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18cbbd640 10 monclient(hunting): picked mon.noname-c con 0x7fd18813be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd185d9b640 1 -- 172.21.15.46:0/498171261 learned_addr learned my addr 172.21.15.46:0/498171261 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18cbbd640 1 -- 172.21.15.46:0/498171261 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd18809ab70 con 0x7fd1880a3ff0 2024-09-16T04:39:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18cbbd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18cbbd640 1 -- 172.21.15.46:0/498171261 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fd1880a4d60 con 0x7fd18813be90 2024-09-16T04:39:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18cbbd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18cbbd640 1 -- 172.21.15.46:0/498171261 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd18809b030 con 0x7fd1880a3c10 2024-09-16T04:39:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18cbbd640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18cbbd640 10 monclient(hunting): authenticate will time out at 741.914306s 2024-09-16T04:39:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7f8a2a7fc640 1 -- 172.21.15.46:0/1141936145 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8a340a3000 con 0x7f8a340a3ff0 2024-09-16T04:39:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18659c640 1 --2- 172.21.15.46:0/498171261 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd1880a3c10 0x7fd18809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7f8a2a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7f8a2a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:03.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18659c640 10 monclient(hunting): get_auth_request con 0x7fd1880a3c10 auth_method 0 2024-09-16T04:39:03.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18559a640 1 -- 172.21.15.46:0/498171261 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 771351618 0 0) 0x7fd18809ab70 con 0x7fd1880a3ff0 2024-09-16T04:39:03.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18559a640 10 monclient(hunting): _negotiate 2024-09-16T04:39:03.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18559a640 10 monclient(hunting): my global_id is 4344 2024-09-16T04:39:03.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18559a640 1 -- 172.21.15.46:0/498171261 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd170003520 con 0x7fd1880a3ff0 2024-09-16T04:39:03.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.499+0000 7fd18659c640 10 monclient(hunting): handle_auth_done global_id 4301 payload 274 2024-09-16T04:39:03.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.500+0000 7fd18659c640 1 -- 172.21.15.46:0/498171261 >> v2:172.21.15.114:3300/0 conn(0x7fd18813be90 msgr2=0x7fd18813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.500+0000 7fd186d9d640 1 --2- 172.21.15.46:0/498171261 >> v2:172.21.15.114:3300/0 conn(0x7fd18813be90 0x7fd18813e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.500+0000 7fd18659c640 1 --2- 172.21.15.46:0/498171261 >> v2:172.21.15.114:3300/0 conn(0x7fd18813be90 0x7fd18813e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.500+0000 7fd18659c640 1 -- 172.21.15.46:0/498171261 >> v1:172.21.15.46:6789/0 conn(0x7fd1880a3ff0 legacy=0x7fd1880a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.500+0000 7fd18659c640 10 monclient: _finish_hunting 0 2024-09-16T04:39:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.500+0000 7fd18659c640 1 monclient: found mon.noname-b 2024-09-16T04:39:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.500+0000 7fd18659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.500+0000 7fd18659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.500+0000 7fd18659c640 1 -- 172.21.15.46:0/498171261 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd18809b310 con 0x7fd1880a3c10 2024-09-16T04:39:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.501+0000 7fd18659c640 1 --2- 172.21.15.46:0/498171261 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd1880a3c10 0x7fd18809bcf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd1780097a0 tx=0x7fd17802a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b1a09690858c6fc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.501+0000 7fd18559a640 1 -- 172.21.15.46:0/498171261 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd178037070 con 0x7fd1880a3c10 2024-09-16T04:39:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.501+0000 7fd18559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.501+0000 7fd18559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.501+0000 7fd18559a640 10 monclient: dump: 2024-09-16T04:39:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.525 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.501+0000 7fd18559a640 10 monclient: _finish_auth 0 2024-09-16T04:39:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.501+0000 7fd18559a640 10 monclient: _check_auth_tickets 2024-09-16T04:39:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.501+0000 7fd18559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.501+0000 7fd18559a640 1 -- 172.21.15.46:0/498171261 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd17802add0 con 0x7fd1880a3c10 2024-09-16T04:39:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.501+0000 7fd18559a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.501+0000 7fd18559a640 1 -- 172.21.15.46:0/498171261 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd178032660 con 0x7fd1880a3c10 2024-09-16T04:39:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.501+0000 7fd18cbbd640 5 monclient: authenticate success, global_id 4301 2024-09-16T04:39:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.501+0000 7fd18cbbd640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.501+0000 7fd18cbbd640 4 set_mon_vals no callback set 2024-09-16T04:39:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.501+0000 7fd18559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.501+0000 7fd18559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.501+0000 7fd18559a640 10 monclient: dump: 2024-09-16T04:39:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.527 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.527 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.527 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.502+0000 7fd18cbbd640 10 monclient: shutdown 2024-09-16T04:39:03.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.502+0000 7fd18cbbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:03.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.502+0000 7fd18cbbd640 1 -- 172.21.15.46:0/498171261 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd1880a3c10 msgr2=0x7fd18809bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.502+0000 7fd18cbbd640 1 --2- 172.21.15.46:0/498171261 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd1880a3c10 0x7fd18809bcf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd1780097a0 tx=0x7fd17802a080 comp rx=0 tx=0).stop 2024-09-16T04:39:03.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.502+0000 7fd18cbbd640 1 -- 172.21.15.46:0/498171261 shutdown_connections 2024-09-16T04:39:03.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.502+0000 7fd18cbbd640 1 --2- 172.21.15.46:0/498171261 >> v2:172.21.15.114:3300/0 conn(0x7fd18813be90 0x7fd18813e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.502+0000 7fd18cbbd640 1 --2- 172.21.15.46:0/498171261 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd1880a3c10 0x7fd18809bcf0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.502+0000 7fd18cbbd640 1 -- 172.21.15.46:0/498171261 >> 172.21.15.46:0/498171261 conn(0x7fd1880a5d10 msgr2=0x7fd1880a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:03.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.502+0000 7fd18cbbd640 1 -- 172.21.15.46:0/498171261 shutdown_connections 2024-09-16T04:39:03.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.502+0000 7fd18cbbd640 1 -- 172.21.15.46:0/498171261 wait complete. 2024-09-16T04:39:03.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.503+0000 7fd18cbbd640 10 monclient: build_initial_monmap 2024-09-16T04:39:03.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.503+0000 7fd18cbbd640 10 monclient: monmap: 2024-09-16T04:39:03.528 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:03.528 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:03.528 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:03.528 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:03.528 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:03.528 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.528 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:03.528 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:03.528 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:03.528 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.503+0000 7fd18cbbd640 1 Processor -- start 2024-09-16T04:39:03.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.503+0000 7fd18cbbd640 1 -- start start 2024-09-16T04:39:03.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.503+0000 7fd18cbbd640 10 monclient: init 2024-09-16T04:39:03.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd18cbbd640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:03.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd18cbbd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:39:03.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd18cbbd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1880a3c10 0x7fd188133a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd18cbbd640 10 monclient(hunting): picked mon.noname-b con 0x7fd1880a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:03.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd18cbbd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd1880a3ff0 0x7fd188133f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd18cbbd640 10 monclient(hunting): picked mon.noname-c con 0x7fd1880a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:03.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd18cbbd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18813be90 0x7fd1881344b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd185d9b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd1880a3ff0 0x7fd188133f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd185d9b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd1880a3ff0 0x7fd188133f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:32972/0 (socket says 172.21.15.46:32972) 2024-09-16T04:39:03.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd185d9b640 1 -- 172.21.15.46:0/3431112973 learned_addr learned my addr 172.21.15.46:0/3431112973 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:03.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd18cbbd640 10 monclient(hunting): picked mon.noname-a con 0x7fd18813be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:03.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd18cbbd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd18cbbd640 1 -- 172.21.15.46:0/3431112973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd18806e550 con 0x7fd18813be90 2024-09-16T04:39:03.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd18cbbd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd18cbbd640 1 -- 172.21.15.46:0/3431112973 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd18806e3d0 con 0x7fd1880a3c10 2024-09-16T04:39:03.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd18cbbd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd18cbbd640 1 -- 172.21.15.46:0/3431112973 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd18813e7c0 con 0x7fd1880a3ff0 2024-09-16T04:39:03.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd18cbbd640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:03.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd18cbbd640 10 monclient(hunting): authenticate will time out at 741.919576s 2024-09-16T04:39:03.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd185d9b640 10 monclient(hunting): get_auth_request con 0x7fd1880a3ff0 auth_method 0 2024-09-16T04:39:03.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd185d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd185d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd185d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd186d9d640 1 --2- 172.21.15.46:0/3431112973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18813be90 0x7fd1881344b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd186d9d640 10 monclient(hunting): get_auth_request con 0x7fd18813be90 auth_method 0 2024-09-16T04:39:03.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd186d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd186d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd186d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd185d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd185d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd185d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd186d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd186d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.504+0000 7fd186d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd185d9b640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-09-16T04:39:03.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd185d9b640 1 -- 172.21.15.46:0/3431112973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1880a3c10 msgr2=0x7fd188133a30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T04:39:03.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd185d9b640 1 --2- 172.21.15.46:0/3431112973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1880a3c10 0x7fd188133a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd185d9b640 1 -- 172.21.15.46:0/3431112973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18813be90 msgr2=0x7fd1881344b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd185d9b640 1 --2- 172.21.15.46:0/3431112973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18813be90 0x7fd1881344b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd185d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:39:03.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd185d9b640 1 monclient: found mon.noname-c 2024-09-16T04:39:03.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd185d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:03.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd185d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd185d9b640 1 -- 172.21.15.46:0/3431112973 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd188139ec0 con 0x7fd1880a3ff0 2024-09-16T04:39:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd186d9d640 1 --2- 172.21.15.46:0/3431112973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18813be90 0x7fd1881344b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd185d9b640 1 --2- 172.21.15.46:0/3431112973 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd1880a3ff0 0x7fd188133f70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd17c00c500 tx=0x7fd17c00c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd176ffd640 1 -- 172.21.15.46:0/3431112973 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd17c018070 con 0x7fd1880a3ff0 2024-09-16T04:39:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd176ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd176ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:39:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd176ffd640 10 monclient: dump: 2024-09-16T04:39:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.533 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd176ffd640 10 monclient: _finish_auth 0 2024-09-16T04:39:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd176ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:39:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd176ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd18cbbd640 5 monclient: authenticate success, global_id 4304 2024-09-16T04:39:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd18cbbd640 10 monclient: _renew_subs 2024-09-16T04:39:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd18cbbd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd18cbbd640 1 -- 172.21.15.46:0/3431112973 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd18813c9c0 con 0x7fd1880a3ff0 2024-09-16T04:39:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd176ffd640 1 -- 172.21.15.46:0/3431112973 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd17c007560 con 0x7fd1880a3ff0 2024-09-16T04:39:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd18cbbd640 10 monclient: _renew_subs 2024-09-16T04:39:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd18cbbd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd18cbbd640 1 -- 172.21.15.46:0/3431112973 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd188139be0 con 0x7fd1880a3ff0 2024-09-16T04:39:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd176ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd176ffd640 1 -- 172.21.15.46:0/3431112973 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd17c004a20 con 0x7fd1880a3ff0 2024-09-16T04:39:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd176ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd176ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd176ffd640 10 monclient: dump: 2024-09-16T04:39:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.534 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.505+0000 7fd184d99640 4 set_mon_vals no callback set 2024-09-16T04:39:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.507+0000 7fd18cbbd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.507+0000 7fd18cbbd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.507+0000 7fd18cbbd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.507+0000 7fd176ffd640 1 -- 172.21.15.46:0/3431112973 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fd17c004bc0 con 0x7fd1880a3ff0 2024-09-16T04:39:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.507+0000 7fd18cbbd640 1 -- 172.21.15.46:0/3431112973 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd16c005080 con 0x7fd1880a3ff0 2024-09-16T04:39:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.507+0000 7fd176ffd640 1 --2- 172.21.15.46:0/3431112973 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd1680383b0 0x7fd16803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.507+0000 7fd176ffd640 1 -- 172.21.15.46:0/3431112973 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fd17c04a9c0 con 0x7fd1880a3ff0 2024-09-16T04:39:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.507+0000 7fd18659c640 1 --2- 172.21.15.46:0/3431112973 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd1680383b0 0x7fd16803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.507+0000 7fd18659c640 10 monclient: get_auth_request con 0x7fd1680383b0 auth_method 0 2024-09-16T04:39:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.508+0000 7fd18659c640 1 --2- 172.21.15.46:0/3431112973 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd1680383b0 0x7fd16803a870 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd178034000 tx=0x7fd17802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.516+0000 7fd176ffd640 1 -- 172.21.15.46:0/3431112973 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd17c012da0 con 0x7fd1880a3ff0 2024-09-16T04:39:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.516+0000 7fd176ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.516+0000 7fd176ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:03.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.558+0000 7fbc5d9f1640 10 monclient: get_monmap_and_config 2024-09-16T04:39:03.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.558+0000 7fbc5d9f1640 10 monclient: build_initial_monmap 2024-09-16T04:39:03.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.558+0000 7fbc5d9f1640 10 monclient: monmap: 2024-09-16T04:39:03.565 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:03.565 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.565 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:03.558982+0000 2024-09-16T04:39:03.565 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:03.558982+0000 2024-09-16T04:39:03.565 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:03.566 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.566 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:03.566 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:03.566 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:03.566 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:03.566 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:03.566 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.559+0000 7fbc5d9f1640 1 Processor -- start 2024-09-16T04:39:03.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.559+0000 7fbc5d9f1640 1 -- start start 2024-09-16T04:39:03.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.559+0000 7fbc5d9f1640 10 monclient: init 2024-09-16T04:39:03.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.559+0000 7fbc5d9f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:03.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.559+0000 7fbc5d9f1640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-16T04:39:03.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.559+0000 7fbc5d9f1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbc580a3c10 0x7fbc5809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.559+0000 7fbc5d9f1640 10 monclient(hunting): picked mon.noname-a con 0x7fbc580a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:03.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc5d9f1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc580a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:03.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc5d9f1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc5813be90 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:03.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc5d9f1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc5809ab70 con 0x7fbc580a3ff0 2024-09-16T04:39:03.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc5d9f1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc580a4d60 con 0x7fbc5813be90 2024-09-16T04:39:03.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc5d9f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc5d9f1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fbc5809b030 con 0x7fbc580a3c10 2024-09-16T04:39:03.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc5d9f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:03.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc5d9f1640 10 monclient(hunting): authenticate will time out at 741.975181s 2024-09-16T04:39:03.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc56ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbc580a3c10 0x7fbc5809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc567fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbc580a3ff0 0x7fbc580a4400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53080/0 (socket says 172.21.15.46:53080) 2024-09-16T04:39:03.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc56ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbc580a3c10 0x7fbc5809bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59464/0 (socket says 172.21.15.46:59464) 2024-09-16T04:39:03.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc567fc640 1 -- 172.21.15.46:0/4257030744 learned_addr learned my addr 172.21.15.46:0/4257030744 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:03.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc56ffd640 10 monclient(hunting): get_auth_request con 0x7fbc580a3c10 auth_method 0 2024-09-16T04:39:03.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc55ffb640 1 -- 172.21.15.46:0/4257030744 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1323334095 0 0) 0x7fbc5809ab70 con 0x7fbc580a3ff0 2024-09-16T04:39:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc55ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:39:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc55ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc55ffb640 10 monclient(hunting): my global_id is 4353 2024-09-16T04:39:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc55ffb640 1 -- 172.21.15.46:0/4257030744 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc40003520 con 0x7fbc580a3ff0 2024-09-16T04:39:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc56ffd640 10 monclient(hunting): handle_auth_done global_id 4350 payload 274 2024-09-16T04:39:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc55ffb640 1 -- 172.21.15.46:0/4257030744 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 919466318 0 0) 0x7fbc580a4d60 con 0x7fbc5813be90 2024-09-16T04:39:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc56ffd640 1 -- 172.21.15.46:0/4257030744 >> v1:172.21.15.114:6789/0 conn(0x7fbc5813be90 legacy=0x7fbc5813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc56ffd640 1 -- 172.21.15.46:0/4257030744 >> v1:172.21.15.46:6789/0 conn(0x7fbc580a3ff0 legacy=0x7fbc580a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc56ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:39:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc56ffd640 1 monclient: found mon.noname-a 2024-09-16T04:39:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.560+0000 7fbc56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:03.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.561+0000 7fbc56ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.561+0000 7fbc56ffd640 1 -- 172.21.15.46:0/4257030744 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc5809b310 con 0x7fbc580a3c10 2024-09-16T04:39:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.561+0000 7fbc55ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:39:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.562+0000 7fbc56ffd640 1 --2- 172.21.15.46:0/4257030744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc580a3c10 0x7fbc5809bcf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbc480097a0 tx=0x7fbc4802a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86db8ccad484a7e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.562+0000 7fbc55ffb640 1 -- 172.21.15.46:0/4257030744 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc48037070 con 0x7fbc580a3c10 2024-09-16T04:39:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.562+0000 7fbc55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.562+0000 7fbc55ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.562+0000 7fbc55ffb640 10 monclient: dump: 2024-09-16T04:39:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.570 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.571 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.571 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.571 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.562+0000 7fbc55ffb640 10 monclient: _finish_auth 0 2024-09-16T04:39:03.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.562+0000 7fbc55ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:39:03.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.562+0000 7fbc55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:03.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.562+0000 7fbc55ffb640 1 -- 172.21.15.46:0/4257030744 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc4802adc0 con 0x7fbc580a3c10 2024-09-16T04:39:03.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.562+0000 7fbc55ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:03.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.562+0000 7fbc5d9f1640 5 monclient: authenticate success, global_id 4350 2024-09-16T04:39:03.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.562+0000 7fbc5d9f1640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:03.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.562+0000 7fbc5d9f1640 4 set_mon_vals no callback set 2024-09-16T04:39:03.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.562+0000 7fbc55ffb640 1 -- 172.21.15.46:0/4257030744 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc48032660 con 0x7fbc580a3c10 2024-09-16T04:39:03.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.562+0000 7fbc55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.562+0000 7fbc55ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.562+0000 7fbc55ffb640 10 monclient: dump: 2024-09-16T04:39:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.572 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.563+0000 7fbc5d9f1640 10 monclient: shutdown 2024-09-16T04:39:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.563+0000 7fbc5d9f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:03.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.563+0000 7fbc5d9f1640 1 -- 172.21.15.46:0/4257030744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc580a3c10 msgr2=0x7fbc5809bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.563+0000 7fbc5d9f1640 1 --2- 172.21.15.46:0/4257030744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc580a3c10 0x7fbc5809bcf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbc480097a0 tx=0x7fbc4802a0b0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.563+0000 7fbc5d9f1640 1 -- 172.21.15.46:0/4257030744 shutdown_connections 2024-09-16T04:39:03.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.563+0000 7fbc5d9f1640 1 --2- 172.21.15.46:0/4257030744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc580a3c10 0x7fbc5809bcf0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.563+0000 7fbc5d9f1640 1 -- 172.21.15.46:0/4257030744 >> 172.21.15.46:0/4257030744 conn(0x7fbc580a5d10 msgr2=0x7fbc580a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:03.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.563+0000 7fbc5d9f1640 1 -- 172.21.15.46:0/4257030744 shutdown_connections 2024-09-16T04:39:03.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.563+0000 7fbc5d9f1640 1 -- 172.21.15.46:0/4257030744 wait complete. 2024-09-16T04:39:03.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.563+0000 7fbc5d9f1640 10 monclient: build_initial_monmap 2024-09-16T04:39:03.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.563+0000 7fbc5d9f1640 10 monclient: monmap: 2024-09-16T04:39:03.573 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:03.573 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:03.573 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:03.573 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:03.573 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:03.573 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.574 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:03.574 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:03.574 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:03.574 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.564+0000 7fbc5d9f1640 1 Processor -- start 2024-09-16T04:39:03.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.564+0000 7fbc5d9f1640 1 -- start start 2024-09-16T04:39:03.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.564+0000 7fbc5d9f1640 10 monclient: init 2024-09-16T04:39:03.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc5d9f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:03.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc5d9f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:39:03.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc5d9f1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc580a3ff0 0x7fbc58137650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc5d9f1640 10 monclient(hunting): picked mon.noname-b con 0x7fbc580a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:03.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc5d9f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc5813be90 0x7fbc58138ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc5d9f1640 10 monclient(hunting): picked mon.noname-a con 0x7fbc5813be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc5d9f1640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fbc58139400 0x7fbc581398b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc5d9f1640 10 monclient(hunting): picked mon.noname-c con 0x7fbc58139400 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc5d9f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc5d9f1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5806e550 con 0x7fbc5813be90 2024-09-16T04:39:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc5d9f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc5d9f1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5806e3d0 con 0x7fbc580a3ff0 2024-09-16T04:39:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc5d9f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc5d9f1640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fbc5813f9a0 con 0x7fbc58139400 2024-09-16T04:39:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc5d9f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc5d9f1640 10 monclient(hunting): authenticate will time out at 741.980204s 2024-09-16T04:39:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc567fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc5813be90 0x7fbc58138ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc567fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc5813be90 0x7fbc58138ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59470/0 (socket says 172.21.15.46:59470) 2024-09-16T04:39:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc567fc640 1 -- 172.21.15.46:0/1993900832 learned_addr learned my addr 172.21.15.46:0/1993900832 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc567fc640 10 monclient(hunting): get_auth_request con 0x7fbc5813be90 auth_method 0 2024-09-16T04:39:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc567fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc577fe640 1 --2- 172.21.15.46:0/1993900832 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fbc58139400 0x7fbc581398b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc577fe640 10 monclient(hunting): get_auth_request con 0x7fbc58139400 auth_method 0 2024-09-16T04:39:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc567fc640 10 monclient(hunting): handle_auth_done global_id 4356 payload 824 2024-09-16T04:39:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc567fc640 1 -- 172.21.15.46:0/1993900832 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fbc58139400 msgr2=0x7fbc581398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc567fc640 1 --2- 172.21.15.46:0/1993900832 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fbc58139400 0x7fbc581398b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc567fc640 1 -- 172.21.15.46:0/1993900832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc580a3ff0 msgr2=0x7fbc58137650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc567fc640 1 --2- 172.21.15.46:0/1993900832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc580a3ff0 0x7fbc58137650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc567fc640 10 monclient: _finish_hunting 0 2024-09-16T04:39:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc567fc640 1 monclient: found mon.noname-a 2024-09-16T04:39:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc567fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.565+0000 7fbc567fc640 1 -- 172.21.15.46:0/1993900832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc5806ef30 con 0x7fbc5813be90 2024-09-16T04:39:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.566+0000 7fbc567fc640 1 --2- 172.21.15.46:0/1993900832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc5813be90 0x7fbc58138ec0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c00c500 tx=0x7fbc4c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.566+0000 7fbc3ffff640 1 -- 172.21.15.46:0/1993900832 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c018070 con 0x7fbc5813be90 2024-09-16T04:39:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.566+0000 7fbc3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.566+0000 7fbc3ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:03.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.566+0000 7fbc3ffff640 10 monclient: dump: 2024-09-16T04:39:03.584 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.584 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.584 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.584 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.584 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.584 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.584 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.584 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.584 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.584 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.566+0000 7fbc3ffff640 10 monclient: _finish_auth 0 2024-09-16T04:39:03.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.566+0000 7fbc3ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:39:03.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.566+0000 7fbc3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:03.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.566+0000 7fbc3ffff640 1 -- 172.21.15.46:0/1993900832 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc4c007560 con 0x7fbc5813be90 2024-09-16T04:39:03.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.566+0000 7fbc3ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:03.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.566+0000 7fbc3ffff640 1 -- 172.21.15.46:0/1993900832 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c0049e0 con 0x7fbc5813be90 2024-09-16T04:39:03.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.566+0000 7fbc5d9f1640 5 monclient: authenticate success, global_id 4356 2024-09-16T04:39:03.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.566+0000 7fbc3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.566+0000 7fbc3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:03.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.566+0000 7fbc3ffff640 10 monclient: dump: 2024-09-16T04:39:03.585 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.585 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.585 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.585 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.585 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.586 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.566+0000 7fbc5d9f1640 10 monclient: _renew_subs 2024-09-16T04:39:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.566+0000 7fbc5d9f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.566+0000 7fbc5d9f1640 1 -- 172.21.15.46:0/1993900832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc5813a8e0 con 0x7fbc5813be90 2024-09-16T04:39:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.566+0000 7fbc557fa640 4 set_mon_vals no callback set 2024-09-16T04:39:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.567+0000 7fbc5d9f1640 10 monclient: _renew_subs 2024-09-16T04:39:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.567+0000 7fbc5d9f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.567+0000 7fbc5d9f1640 1 -- 172.21.15.46:0/1993900832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc5806fc70 con 0x7fbc5813be90 2024-09-16T04:39:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.567+0000 7fbc3ffff640 1 -- 172.21.15.46:0/1993900832 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fbc4c004b80 con 0x7fbc5813be90 2024-09-16T04:39:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.568+0000 7fbc3ffff640 1 --2- 172.21.15.46:0/1993900832 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fbc440383e0 0x7fbc4403a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.568+0000 7fbc3ffff640 1 -- 172.21.15.46:0/1993900832 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fbc4c01bde0 con 0x7fbc5813be90 2024-09-16T04:39:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.568+0000 7fbc56ffd640 1 --2- 172.21.15.46:0/1993900832 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fbc440383e0 0x7fbc4403a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.568+0000 7fbc56ffd640 10 monclient: get_auth_request con 0x7fbc440383e0 auth_method 0 2024-09-16T04:39:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.568+0000 7fbc5d9f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.568+0000 7fbc5d9f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.568+0000 7fbc5d9f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.568+0000 7fbc5d9f1640 1 -- 172.21.15.46:0/1993900832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc580a8b00 con 0x7fbc5813be90 2024-09-16T04:39:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.569+0000 7fbc56ffd640 1 --2- 172.21.15.46:0/1993900832 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fbc440383e0 0x7fbc4403a8a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbc48034000 tx=0x7fbc4802c000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.573+0000 7fbc3ffff640 1 -- 172.21.15.46:0/1993900832 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fbc580a8b00 con 0x7fbc5813be90 2024-09-16T04:39:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.573+0000 7fbc3ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.573+0000 7fbc3ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:03.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.594+0000 7fd8f872c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T04:39:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.594+0000 7fd8f872c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T04:39:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.594+0000 7fd8f872c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.594+0000 7fd8f872c640 1 -- 172.21.15.46:0/3448657720 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fd8f0067e40 con 0x7fd8f00a3ff0 2024-09-16T04:39:03.596 INFO:teuthology.orchestra.run.smithi046.stdout:47244640262 2024-09-16T04:39:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.595+0000 7fd8e6ffd640 1 -- 172.21.15.46:0/3448657720 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd8f013cdb0 con 0x7fd8f00a3ff0 2024-09-16T04:39:03.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.595+0000 7fd8e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T04:39:03.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.595+0000 7fd8e6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:03.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.597+0000 7fd8f872c640 1 -- 172.21.15.46:0/3448657720 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd8d40383b0 msgr2=0x7fd8d403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.597+0000 7fd8f872c640 1 --2- 172.21.15.46:0/3448657720 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd8d40383b0 0x7fd8d403a870 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec001e70 tx=0x7fd8ec009040 comp rx=0 tx=0).stop 2024-09-16T04:39:03.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.597+0000 7fd8f872c640 10 monclient: shutdown 2024-09-16T04:39:03.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.597+0000 7fd8f872c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:03.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.597+0000 7fd8f872c640 1 -- 172.21.15.46:0/3448657720 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd8f00a3ff0 msgr2=0x7fd8f0136ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.597+0000 7fd8f872c640 1 --2- 172.21.15.46:0/3448657720 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd8f00a3ff0 0x7fd8f0136ff0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000ebe0 tx=0x7fd8e000c520 comp rx=0 tx=0).stop 2024-09-16T04:39:03.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.597+0000 7fd8f872c640 1 -- 172.21.15.46:0/3448657720 shutdown_connections 2024-09-16T04:39:03.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.597+0000 7fd8f872c640 1 --2- 172.21.15.46:0/3448657720 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd8d40383b0 0x7fd8d403a870 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.597+0000 7fd8f872c640 1 --2- 172.21.15.46:0/3448657720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f013be90 0x7fd8f00a1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.597+0000 7fd8f872c640 1 --2- 172.21.15.46:0/3448657720 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd8f00a3ff0 0x7fd8f0136ff0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.597+0000 7fd8f872c640 1 --2- 172.21.15.46:0/3448657720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8f00a3c10 0x7fd8f01cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.597+0000 7fd8f872c640 1 -- 172.21.15.46:0/3448657720 >> 172.21.15.46:0/3448657720 conn(0x7fd8f00a5d10 msgr2=0x7fd8f013d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:03.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.598+0000 7fd8f872c640 1 -- 172.21.15.46:0/3448657720 shutdown_connections 2024-09-16T04:39:03.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.598+0000 7fd8f872c640 1 -- 172.21.15.46:0/3448657720 wait complete. 2024-09-16T04:39:03.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.601+0000 7f79c3b66640 10 monclient: get_monmap_and_config 2024-09-16T04:39:03.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.601+0000 7f79c3b66640 10 monclient: build_initial_monmap 2024-09-16T04:39:03.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.601+0000 7f79c3b66640 10 monclient: monmap: 2024-09-16T04:39:03.602 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:03.602 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.602 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:03.601917+0000 2024-09-16T04:39:03.603 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:03.601917+0000 2024-09-16T04:39:03.603 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:03.603 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.603 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:03.603 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:03.603 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:03.603 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:03.603 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:03.603 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.602+0000 7f79c3b66640 1 Processor -- start 2024-09-16T04:39:03.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.602+0000 7f79c3b66640 1 -- start start 2024-09-16T04:39:03.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.602+0000 7f79c3b66640 10 monclient: init 2024-09-16T04:39:03.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c3b66640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:03.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c3b66640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-16T04:39:03.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c3b66640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f79bc0a3c10 0x7f79bc09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c3b66640 10 monclient(hunting): picked mon.noname-a con 0x7f79bc0a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:03.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c3b66640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f79bc0a3ff0 0x7f79bc0a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c3b66640 10 monclient(hunting): picked mon.noname-b con 0x7f79bc0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:03.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c3b66640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f79bc1369d0 0x7f79bc136e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c3b66640 10 monclient(hunting): picked mon.noname-c con 0x7f79bc1369d0 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:03.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c3b66640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c3b66640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f79bc09ab70 con 0x7f79bc0a3c10 2024-09-16T04:39:03.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c3b66640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c3b66640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f79bc0a4d60 con 0x7f79bc1369d0 2024-09-16T04:39:03.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c3b66640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c3b66640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f79bc09b030 con 0x7f79bc0a3ff0 2024-09-16T04:39:03.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c3b66640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:03.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c3b66640 10 monclient(hunting): authenticate will time out at 742.018616s 2024-09-16T04:39:03.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c1101640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f79bc0a3ff0 0x7f79bc0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c1902640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f79bc0a3c10 0x7f79bc09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c1902640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f79bc0a3c10 0x7f79bc09bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59484/0 (socket says 172.21.15.46:59484) 2024-09-16T04:39:03.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c1902640 1 -- 172.21.15.46:0/176863593 learned_addr learned my addr 172.21.15.46:0/176863593 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:03.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c1902640 10 monclient(hunting): get_auth_request con 0x7f79bc0a3c10 auth_method 0 2024-09-16T04:39:03.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c1902640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c1902640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c1902640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c2103640 1 --2- 172.21.15.46:0/176863593 >> v2:172.21.15.114:3300/0 conn(0x7f79bc1369d0 0x7f79bc136e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c2103640 10 monclient(hunting): get_auth_request con 0x7f79bc1369d0 auth_method 0 2024-09-16T04:39:03.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c2103640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c2103640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.603+0000 7f79c2103640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c2103640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c2103640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c2103640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c1902640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c1902640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c1902640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c1101640 10 monclient(hunting): get_auth_request con 0x7f79bc0a3ff0 auth_method 0 2024-09-16T04:39:03.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c1101640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c1101640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c1101640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c2103640 10 monclient(hunting): handle_auth_done global_id 4348 payload 274 2024-09-16T04:39:03.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c2103640 1 -- 172.21.15.46:0/176863593 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f79bc0a3ff0 msgr2=0x7f79bc0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:03.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c2103640 1 --2- 172.21.15.46:0/176863593 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f79bc0a3ff0 0x7f79bc0a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c2103640 1 -- 172.21.15.46:0/176863593 >> v2:172.21.15.46:3300/0 conn(0x7f79bc0a3c10 msgr2=0x7f79bc09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:03.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c2103640 1 --2- 172.21.15.46:0/176863593 >> v2:172.21.15.46:3300/0 conn(0x7f79bc0a3c10 0x7f79bc09bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c2103640 10 monclient: _finish_hunting 0 2024-09-16T04:39:03.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c2103640 1 monclient: found mon.noname-c 2024-09-16T04:39:03.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c2103640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:03.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c2103640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:39:03.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c2103640 1 -- 172.21.15.46:0/176863593 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79bc09b310 con 0x7f79bc1369d0 2024-09-16T04:39:03.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c1902640 1 --2- 172.21.15.46:0/176863593 >> v2:172.21.15.46:3300/0 conn(0x7f79bc0a3c10 0x7f79bc09bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:03.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.604+0000 7f79c1101640 1 --2- 172.21.15.46:0/176863593 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f79bc0a3ff0 0x7f79bc0a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:39:03.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.605+0000 7f79c2103640 1 --2- 172.21.15.46:0/176863593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc1369d0 0x7f79bc136e20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f79bc09d0c0 tx=0x7f79b402a540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc4ece62c267a35 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:03.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.606+0000 7f79c0900640 1 -- 172.21.15.46:0/176863593 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b4007740 con 0x7f79bc1369d0 2024-09-16T04:39:03.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.606+0000 7f79c0900640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.606+0000 7f79c0900640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:39:03.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.606+0000 7f79c0900640 10 monclient: dump: 2024-09-16T04:39:03.614 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.614 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.614 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.614 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.614 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.614 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.614 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.614 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.615 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.615 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.606+0000 7f79c0900640 10 monclient: _finish_auth 0 2024-09-16T04:39:03.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.606+0000 7f79c0900640 10 monclient: _check_auth_tickets 2024-09-16T04:39:03.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.606+0000 7f79c0900640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:03.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.606+0000 7f79c0900640 1 -- 172.21.15.46:0/176863593 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79b40078e0 con 0x7f79bc1369d0 2024-09-16T04:39:03.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.606+0000 7f79c3b66640 5 monclient: authenticate success, global_id 4348 2024-09-16T04:39:03.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.606+0000 7f79c0900640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:03.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.606+0000 7f79c0900640 1 -- 172.21.15.46:0/176863593 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b4007cb0 con 0x7f79bc1369d0 2024-09-16T04:39:03.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.606+0000 7f79c0900640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.606+0000 7f79c0900640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:03.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.606+0000 7f79c0900640 10 monclient: dump: 2024-09-16T04:39:03.615 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.615 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.615 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.616 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.616 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.616 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.616 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.616 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.616 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.616 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.606+0000 7f79c3b66640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:03.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.606+0000 7f79c3b66640 4 set_mon_vals no callback set 2024-09-16T04:39:03.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.607+0000 7f79c3b66640 10 monclient: shutdown 2024-09-16T04:39:03.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.607+0000 7f79c3b66640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:03.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.607+0000 7f79c3b66640 1 -- 172.21.15.46:0/176863593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc1369d0 msgr2=0x7f79bc136e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.607+0000 7f79c3b66640 1 --2- 172.21.15.46:0/176863593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc1369d0 0x7f79bc136e20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f79bc09d0c0 tx=0x7f79b402a540 comp rx=0 tx=0).stop 2024-09-16T04:39:03.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.607+0000 7f105607b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T04:39:03.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.607+0000 7f105607b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T04:39:03.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.607+0000 7f105607b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.607+0000 7f105607b640 1 -- 172.21.15.46:0/3878965616 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f10501203b0 con 0x7f10500a3ff0 2024-09-16T04:39:03.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.607+0000 7f79c3b66640 1 -- 172.21.15.46:0/176863593 shutdown_connections 2024-09-16T04:39:03.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.607+0000 7f79c3b66640 1 --2- 172.21.15.46:0/176863593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc1369d0 0x7f79bc136e20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.607+0000 7f79c3b66640 1 --2- 172.21.15.46:0/176863593 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f79bc0a3ff0 0x7f79bc0a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.607+0000 7f79c3b66640 1 --2- 172.21.15.46:0/176863593 >> v2:172.21.15.46:3300/0 conn(0x7f79bc0a3c10 0x7f79bc09bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.607+0000 7f79c3b66640 1 -- 172.21.15.46:0/176863593 >> 172.21.15.46:0/176863593 conn(0x7f79bc0a5d10 msgr2=0x7f79bc0a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:03.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.607+0000 7f79c3b66640 1 -- 172.21.15.46:0/176863593 shutdown_connections 2024-09-16T04:39:03.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.607+0000 7f79c3b66640 1 -- 172.21.15.46:0/176863593 wait complete. 2024-09-16T04:39:03.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.607+0000 7f1037fff640 1 -- 172.21.15.46:0/3878965616 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f104401a760 con 0x7f10500a3ff0 2024-09-16T04:39:03.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.607+0000 7f1037fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T04:39:03.619 INFO:teuthology.orchestra.run.smithi046.stdout:51539607558 2024-09-16T04:39:03.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.607+0000 7f1037fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:03.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.608+0000 7f79c3b66640 10 monclient: build_initial_monmap 2024-09-16T04:39:03.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.608+0000 7f79c3b66640 10 monclient: monmap: 2024-09-16T04:39:03.621 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:03.621 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:03.621 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:03.621 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:03.621 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:03.621 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.621 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:03.622 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:03.622 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:03.622 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.609+0000 7f79c3b66640 1 Processor -- start 2024-09-16T04:39:03.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.609+0000 7f79c3b66640 1 -- start start 2024-09-16T04:39:03.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.609+0000 7f79c3b66640 10 monclient: init 2024-09-16T04:39:03.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.610+0000 7f79c3b66640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:03.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.610+0000 7f79c3b66640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:39:03.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.610+0000 7f79c3b66640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc0a3c10 0x7f79bc1cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.610+0000 7f79c3b66640 10 monclient(hunting): picked mon.noname-b con 0x7f79bc0a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:03.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.610+0000 7f79c3b66640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79bc0a3ff0 0x7f79bc1cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.610+0000 7f79c3b66640 10 monclient(hunting): picked mon.noname-a con 0x7f79bc0a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:03.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f79c1101640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79bc0a3ff0 0x7f79bc1cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f105607b640 1 -- 172.21.15.46:0/3878965616 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f103c0383b0 msgr2=0x7f103c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f79c1902640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc0a3c10 0x7f79bc1cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f105607b640 1 --2- 172.21.15.46:0/3878965616 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f103c0383b0 0x7f103c03a870 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f104002a150 tx=0x7f104002e000 comp rx=0 tx=0).stop 2024-09-16T04:39:03.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f105607b640 10 monclient: shutdown 2024-09-16T04:39:03.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f105607b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:03.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f105607b640 1 -- 172.21.15.46:0/3878965616 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f10500a3ff0 msgr2=0x7f10501369a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f105607b640 1 --2- 172.21.15.46:0/3878965616 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f10500a3ff0 0x7f10501369a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f104400c390 tx=0x7f104400c850 comp rx=0 tx=0).stop 2024-09-16T04:39:03.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f79c1902640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc0a3c10 0x7f79bc1cbc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.46:38462/0 (socket says 172.21.15.46:38462) 2024-09-16T04:39:03.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f105607b640 1 -- 172.21.15.46:0/3878965616 shutdown_connections 2024-09-16T04:39:03.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f105607b640 1 --2- 172.21.15.46:0/3878965616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f105013be90 0x7f1050138b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f105607b640 1 --2- 172.21.15.46:0/3878965616 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f10500a3ff0 0x7f10501369a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f105607b640 1 --2- 172.21.15.46:0/3878965616 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f103c0383b0 0x7f103c03a870 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f104002a150 tx=0x7f104002e000 comp rx=0 tx=0).stop 2024-09-16T04:39:03.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f105607b640 1 --2- 172.21.15.46:0/3878965616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10500a3c10 0x7f1050136460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f105607b640 1 -- 172.21.15.46:0/3878965616 >> 172.21.15.46:0/3878965616 conn(0x7f10500a5d10 msgr2=0x7f10500a2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:03.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f79c1902640 1 -- 172.21.15.46:0/1325605161 learned_addr learned my addr 172.21.15.46:0/1325605161 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:03.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f79c3b66640 1 --2- 172.21.15.46:0/1325605161 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f79bc1369d0 0x7f79bc1cc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f79c3b66640 10 monclient(hunting): picked mon.noname-c con 0x7f79bc1369d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:03.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f79c3b66640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f79c2103640 1 --2- 172.21.15.46:0/1325605161 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f79bc1369d0 0x7f79bc1cc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f79c3b66640 1 -- 172.21.15.46:0/1325605161 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f79bc0a4910 con 0x7f79bc0a3ff0 2024-09-16T04:39:03.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f79c3b66640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.627 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640262 for osd.5 2024-09-16T04:39:03.628 DEBUG:teuthology.parallel:result is None 2024-09-16T04:39:03.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.611+0000 7f79c3b66640 1 -- 172.21.15.46:0/1325605161 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f79bc06e3d0 con 0x7f79bc0a3c10 2024-09-16T04:39:03.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c3b66640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c3b66640 1 -- 172.21.15.46:0/1325605161 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f79bc125830 con 0x7f79bc1369d0 2024-09-16T04:39:03.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f105607b640 1 -- 172.21.15.46:0/3878965616 shutdown_connections 2024-09-16T04:39:03.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c3b66640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:03.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c3b66640 10 monclient(hunting): authenticate will time out at 742.027284s 2024-09-16T04:39:03.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1902640 10 monclient(hunting): get_auth_request con 0x7f79bc0a3c10 auth_method 0 2024-09-16T04:39:03.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1902640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1902640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1902640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1101640 10 monclient(hunting): get_auth_request con 0x7f79bc0a3ff0 auth_method 0 2024-09-16T04:39:03.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1101640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1101640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1101640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c2103640 10 monclient(hunting): get_auth_request con 0x7f79bc1369d0 auth_method 0 2024-09-16T04:39:03.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c2103640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c2103640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c2103640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1902640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1902640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1902640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1101640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1101640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1101640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f105607b640 1 -- 172.21.15.46:0/3878965616 wait complete. 2024-09-16T04:39:03.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c2103640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c2103640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c2103640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1902640 10 monclient(hunting): handle_auth_done global_id 4351 payload 824 2024-09-16T04:39:03.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1902640 1 -- 172.21.15.46:0/1325605161 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f79bc1369d0 msgr2=0x7f79bc1cc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1902640 1 --2- 172.21.15.46:0/1325605161 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f79bc1369d0 0x7f79bc1cc6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1902640 1 -- 172.21.15.46:0/1325605161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79bc0a3ff0 msgr2=0x7f79bc1cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1902640 1 --2- 172.21.15.46:0/1325605161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79bc0a3ff0 0x7f79bc1cc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1902640 10 monclient: _finish_hunting 0 2024-09-16T04:39:03.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1902640 1 monclient: found mon.noname-b 2024-09-16T04:39:03.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1902640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:03.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1902640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3300/0 2024-09-16T04:39:03.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1902640 1 -- 172.21.15.46:0/1325605161 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79bc0a8c50 con 0x7f79bc0a3c10 2024-09-16T04:39:03.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c2103640 1 --2- 172.21.15.46:0/1325605161 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f79bc1369d0 0x7f79bc1cc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:03.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.612+0000 7f79c1101640 1 --2- 172.21.15.46:0/1325605161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79bc0a3ff0 0x7f79bc1cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:03.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79c1902640 1 --2- 172.21.15.46:0/1325605161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc0a3c10 0x7f79bc1cbc70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f79b800e9d0 tx=0x7f79b800ee90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79b27fc640 1 -- 172.21.15.46:0/1325605161 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b8017070 con 0x7f79bc0a3c10 2024-09-16T04:39:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79b27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79b27fc640 10 monclient: dump: 2024-09-16T04:39:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.639 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79b27fc640 10 monclient: _finish_auth 0 2024-09-16T04:39:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79b27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:39:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79b27fc640 1 -- 172.21.15.46:0/1325605161 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79b800ca60 con 0x7f79bc0a3c10 2024-09-16T04:39:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79b27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79b27fc640 1 -- 172.21.15.46:0/1325605161 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b800ce10 con 0x7f79bc0a3c10 2024-09-16T04:39:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79b27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79b27fc640 10 monclient: dump: 2024-09-16T04:39:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.640 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79c3b66640 5 monclient: authenticate success, global_id 4351 2024-09-16T04:39:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79c3b66640 10 monclient: _renew_subs 2024-09-16T04:39:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79c3b66640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79c3b66640 1 -- 172.21.15.46:0/1325605161 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79bc0a8700 con 0x7f79bc0a3c10 2024-09-16T04:39:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79b3fff640 4 set_mon_vals no callback set 2024-09-16T04:39:03.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79c3b66640 10 monclient: _renew_subs 2024-09-16T04:39:03.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79c3b66640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:03.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.613+0000 7f79c3b66640 1 -- 172.21.15.46:0/1325605161 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79bc1267e0 con 0x7f79bc0a3c10 2024-09-16T04:39:03.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.614+0000 7f79b27fc640 1 -- 172.21.15.46:0/1325605161 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f79b801a070 con 0x7f79bc0a3c10 2024-09-16T04:39:03.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.614+0000 7f79b27fc640 1 --2- 172.21.15.46:0/1325605161 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f79a00383b0 0x7f79a003a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.614+0000 7f79b27fc640 1 -- 172.21.15.46:0/1325605161 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f79b804b5a0 con 0x7f79bc0a3c10 2024-09-16T04:39:03.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.614+0000 7f79c1101640 1 --2- 172.21.15.46:0/1325605161 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f79a00383b0 0x7f79a003a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.614+0000 7f79c1101640 10 monclient: get_auth_request con 0x7f79a00383b0 auth_method 0 2024-09-16T04:39:03.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.615+0000 7f79c1101640 1 --2- 172.21.15.46:0/1325605161 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f79a00383b0 0x7f79a003a870 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f79bc126030 tx=0x7f79ac008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:03.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.615+0000 7f79c3b66640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:03.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.615+0000 7f79c3b66640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:03.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.615+0000 7f79c3b66640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:03.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.615+0000 7f79c3b66640 1 -- 172.21.15.46:0/1325605161 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79bc1265e0 con 0x7f79bc0a3c10 2024-09-16T04:39:03.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.620+0000 7f79b27fc640 1 -- 172.21.15.46:0/1325605161 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f79b8011350 con 0x7f79bc0a3c10 2024-09-16T04:39:03.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.620+0000 7f79b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:03.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.620+0000 7f79b27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:03.642 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607558 for osd.0 2024-09-16T04:39:03.643 DEBUG:teuthology.parallel:result is None 2024-09-16T04:39:03.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.763+0000 7fbc5d9f1640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T04:39:03.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.763+0000 7fbc5d9f1640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T04:39:03.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.763+0000 7fbc5d9f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:03.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.763+0000 7fbc5d9f1640 1 -- 172.21.15.46:0/1993900832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fbc580a8700 con 0x7fbc5813be90 2024-09-16T04:39:03.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.763+0000 7fbc3ffff640 1 -- 172.21.15.46:0/1993900832 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbc580a8700 con 0x7fbc5813be90 2024-09-16T04:39:03.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.763+0000 7fbc3ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T04:39:03.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.763+0000 7fbc3ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:03.765 INFO:teuthology.orchestra.run.smithi046.stdout:47244640261 2024-09-16T04:39:03.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.765+0000 7fbc5d9f1640 1 -- 172.21.15.46:0/1993900832 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fbc440383e0 msgr2=0x7fbc4403a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.765+0000 7fbc5d9f1640 1 --2- 172.21.15.46:0/1993900832 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fbc440383e0 0x7fbc4403a8a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbc48034000 tx=0x7fbc4802c000 comp rx=0 tx=0).stop 2024-09-16T04:39:03.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.765+0000 7fbc5d9f1640 10 monclient: shutdown 2024-09-16T04:39:03.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.765+0000 7fbc5d9f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:03.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.765+0000 7fbc5d9f1640 1 -- 172.21.15.46:0/1993900832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc5813be90 msgr2=0x7fbc58138ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.765+0000 7fbc5d9f1640 1 --2- 172.21.15.46:0/1993900832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc5813be90 0x7fbc58138ec0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c00c500 tx=0x7fbc4c00c9c0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.765+0000 7fbc5d9f1640 1 -- 172.21.15.46:0/1993900832 shutdown_connections 2024-09-16T04:39:03.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.765+0000 7fbc5d9f1640 1 --2- 172.21.15.46:0/1993900832 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fbc58139400 0x7fbc581398b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.765+0000 7fbc5d9f1640 1 --2- 172.21.15.46:0/1993900832 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fbc440383e0 0x7fbc4403a8a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.765+0000 7fbc5d9f1640 1 --2- 172.21.15.46:0/1993900832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc5813be90 0x7fbc58138ec0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.765+0000 7fbc5d9f1640 1 --2- 172.21.15.46:0/1993900832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc580a3ff0 0x7fbc58137650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.765+0000 7fbc5d9f1640 1 -- 172.21.15.46:0/1993900832 >> 172.21.15.46:0/1993900832 conn(0x7fbc580a5d10 msgr2=0x7fbc580a7060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:03.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.766+0000 7fbc5d9f1640 1 -- 172.21.15.46:0/1993900832 shutdown_connections 2024-09-16T04:39:03.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.766+0000 7fbc5d9f1640 1 -- 172.21.15.46:0/1993900832 wait complete. 2024-09-16T04:39:03.778 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.1 2024-09-16T04:39:03.778 DEBUG:teuthology.parallel:result is None 2024-09-16T04:39:03.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.785+0000 7f8a3bc2e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T04:39:03.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.785+0000 7f8a3bc2e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T04:39:03.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.785+0000 7f8a3bc2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.785+0000 7f8a3bc2e640 1 -- 172.21.15.46:0/1141936145 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f8a34121890 con 0x7f8a340a3ff0 2024-09-16T04:39:03.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.786+0000 7f8a2a7fc640 1 -- 172.21.15.46:0/1141936145 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8a340a0400 con 0x7f8a340a3ff0 2024-09-16T04:39:03.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.786+0000 7f8a2a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T04:39:03.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.786+0000 7f8a2a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:03.787 INFO:teuthology.orchestra.run.smithi046.stdout:47244640261 2024-09-16T04:39:03.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.788+0000 7f8a3bc2e640 1 -- 172.21.15.46:0/1141936145 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8a180383b0 msgr2=0x7f8a1803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.788+0000 7f8a3bc2e640 1 --2- 172.21.15.46:0/1141936145 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8a180383b0 0x7f8a1803a870 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8a30009560 tx=0x7f8a30009190 comp rx=0 tx=0).stop 2024-09-16T04:39:03.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.788+0000 7f8a3bc2e640 10 monclient: shutdown 2024-09-16T04:39:03.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.788+0000 7f8a3bc2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:03.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.788+0000 7f8a3bc2e640 1 -- 172.21.15.46:0/1141936145 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8a340a3ff0 msgr2=0x7f8a34136e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.788+0000 7f8a3bc2e640 1 --2- 172.21.15.46:0/1141936145 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8a340a3ff0 0x7f8a34136e40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8a24001b00 tx=0x7f8a24014420 comp rx=0 tx=0).stop 2024-09-16T04:39:03.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.788+0000 7f8a3bc2e640 1 -- 172.21.15.46:0/1141936145 shutdown_connections 2024-09-16T04:39:03.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.788+0000 7f8a3bc2e640 1 --2- 172.21.15.46:0/1141936145 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8a180383b0 0x7f8a1803a870 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.788+0000 7f8a3bc2e640 1 --2- 172.21.15.46:0/1141936145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a3413be90 0x7f8a3409ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.788+0000 7f8a3bc2e640 1 --2- 172.21.15.46:0/1141936145 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8a340a3ff0 0x7f8a34136e40 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.788+0000 7f8a3bc2e640 1 --2- 172.21.15.46:0/1141936145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a340a3c10 0x7f8a341cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.788+0000 7f8a3bc2e640 1 -- 172.21.15.46:0/1141936145 >> 172.21.15.46:0/1141936145 conn(0x7f8a340a5d10 msgr2=0x7f8a340a28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:03.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.788+0000 7f8a3bc2e640 1 -- 172.21.15.46:0/1141936145 shutdown_connections 2024-09-16T04:39:03.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.788+0000 7f8a3bc2e640 1 -- 172.21.15.46:0/1141936145 wait complete. 2024-09-16T04:39:03.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.796+0000 7fd18cbbd640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T04:39:03.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.796+0000 7fd18cbbd640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T04:39:03.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.796+0000 7fd18cbbd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.796+0000 7fd18cbbd640 1 -- 172.21.15.46:0/3431112973 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fd16c005640 con 0x7fd1880a3ff0 2024-09-16T04:39:03.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.796+0000 7fd176ffd640 1 -- 172.21.15.46:0/3431112973 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd17c0104c0 con 0x7fd1880a3ff0 2024-09-16T04:39:03.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.796+0000 7fd176ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T04:39:03.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.796+0000 7fd176ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:03.798 INFO:teuthology.orchestra.run.smithi046.stdout:51539607558 2024-09-16T04:39:03.800 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.2 2024-09-16T04:39:03.800 DEBUG:teuthology.parallel:result is None 2024-09-16T04:39:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.799+0000 7fd18cbbd640 1 -- 172.21.15.46:0/3431112973 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd1680383b0 msgr2=0x7fd16803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.799+0000 7fd18cbbd640 1 --2- 172.21.15.46:0/3431112973 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd1680383b0 0x7fd16803a870 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd178034000 tx=0x7fd17802e000 comp rx=0 tx=0).stop 2024-09-16T04:39:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.799+0000 7fd18cbbd640 10 monclient: shutdown 2024-09-16T04:39:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.799+0000 7fd18cbbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.799+0000 7fd18cbbd640 1 -- 172.21.15.46:0/3431112973 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd1880a3ff0 msgr2=0x7fd188133f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.799+0000 7fd18cbbd640 1 --2- 172.21.15.46:0/3431112973 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd1880a3ff0 0x7fd188133f70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd17c00c500 tx=0x7fd17c00c9c0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.799+0000 7fd18cbbd640 1 -- 172.21.15.46:0/3431112973 shutdown_connections 2024-09-16T04:39:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.799+0000 7fd18cbbd640 1 --2- 172.21.15.46:0/3431112973 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd1680383b0 0x7fd16803a870 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.799+0000 7fd18cbbd640 1 --2- 172.21.15.46:0/3431112973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18813be90 0x7fd1881344b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.799+0000 7fd18cbbd640 1 --2- 172.21.15.46:0/3431112973 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd1880a3ff0 0x7fd188133f70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.799+0000 7fd18cbbd640 1 --2- 172.21.15.46:0/3431112973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1880a3c10 0x7fd188133a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.799+0000 7fd18cbbd640 1 -- 172.21.15.46:0/3431112973 >> 172.21.15.46:0/3431112973 conn(0x7fd1880a5d10 msgr2=0x7fd18813b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.799+0000 7fd18cbbd640 1 -- 172.21.15.46:0/3431112973 shutdown_connections 2024-09-16T04:39:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.799+0000 7fd18cbbd640 1 -- 172.21.15.46:0/3431112973 wait complete. 2024-09-16T04:39:03.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.802+0000 7f79c3b66640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T04:39:03.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.802+0000 7f79c3b66640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T04:39:03.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.802+0000 7f79c3b66640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:03.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.802+0000 7f79c3b66640 1 -- 172.21.15.46:0/1325605161 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f79bc069150 con 0x7f79bc0a3c10 2024-09-16T04:39:03.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.802+0000 7f79b27fc640 1 -- 172.21.15.46:0/1325605161 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f79b804b270 con 0x7f79bc0a3c10 2024-09-16T04:39:03.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.802+0000 7f79b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T04:39:03.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.802+0000 7f79b27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:03.804 INFO:teuthology.orchestra.run.smithi046.stdout:51539607558 2024-09-16T04:39:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.805+0000 7f79c3b66640 1 -- 172.21.15.46:0/1325605161 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f79a00383b0 msgr2=0x7f79a003a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.805+0000 7f79c3b66640 1 --2- 172.21.15.46:0/1325605161 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f79a00383b0 0x7f79a003a870 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f79bc126030 tx=0x7f79ac008040 comp rx=0 tx=0).stop 2024-09-16T04:39:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.805+0000 7f79c3b66640 10 monclient: shutdown 2024-09-16T04:39:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.805+0000 7f79c3b66640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:03.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.805+0000 7f79c3b66640 1 -- 172.21.15.46:0/1325605161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc0a3c10 msgr2=0x7f79bc1cbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.805+0000 7f79c3b66640 1 --2- 172.21.15.46:0/1325605161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc0a3c10 0x7f79bc1cbc70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f79b800e9d0 tx=0x7f79b800ee90 comp rx=0 tx=0).stop 2024-09-16T04:39:03.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.805+0000 7f79c3b66640 1 -- 172.21.15.46:0/1325605161 shutdown_connections 2024-09-16T04:39:03.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.805+0000 7f79c3b66640 1 --2- 172.21.15.46:0/1325605161 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f79bc1369d0 0x7f79bc1cc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.805+0000 7f79c3b66640 1 --2- 172.21.15.46:0/1325605161 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f79a00383b0 0x7f79a003a870 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.805+0000 7f79c3b66640 1 --2- 172.21.15.46:0/1325605161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79bc0a3ff0 0x7f79bc1cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.805+0000 7f79c3b66640 1 --2- 172.21.15.46:0/1325605161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc0a3c10 0x7f79bc1cbc70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.805+0000 7f79c3b66640 1 -- 172.21.15.46:0/1325605161 >> 172.21.15.46:0/1325605161 conn(0x7f79bc0a5d10 msgr2=0x7f79bc0a0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:03.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.805+0000 7f79c3b66640 1 -- 172.21.15.46:0/1325605161 shutdown_connections 2024-09-16T04:39:03.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.805+0000 7f79c3b66640 1 -- 172.21.15.46:0/1325605161 wait complete. 2024-09-16T04:39:03.811 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607558 for osd.3 2024-09-16T04:39:03.811 DEBUG:teuthology.parallel:result is None 2024-09-16T04:39:03.817 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607558 for osd.4 2024-09-16T04:39:03.817 DEBUG:teuthology.parallel:result is None 2024-09-16T04:39:03.817 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T04:39:03.817 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:39:03.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.957+0000 7facdea9b640 10 monclient: get_monmap_and_config 2024-09-16T04:39:03.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.957+0000 7facdea9b640 10 monclient: build_initial_monmap 2024-09-16T04:39:03.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.957+0000 7facdea9b640 10 monclient: monmap: 2024-09-16T04:39:03.959 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:03.959 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.959 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:03.957948+0000 2024-09-16T04:39:03.959 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:03.957948+0000 2024-09-16T04:39:03.959 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:03.959 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.959 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:03.959 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:03.959 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:03.959 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:03.959 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:03.959 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7facdea9b640 1 Processor -- start 2024-09-16T04:39:03.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7facdea9b640 1 -- start start 2024-09-16T04:39:03.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7facdea9b640 10 monclient: init 2024-09-16T04:39:03.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7facdea9b640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:03.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7facdea9b640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T04:39:03.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7facdea9b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7facd80a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:03.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7facdea9b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facd80a3ff0 0x7facd80a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7facdea9b640 10 monclient(hunting): picked mon.noname-b con 0x7facd80a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:03.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7facdea9b640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7facd813be90 0x7facd813e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7facdea9b640 10 monclient(hunting): picked mon.noname-c con 0x7facd813be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:03.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7facdea9b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facd809b940 con 0x7facd80a3c10 2024-09-16T04:39:03.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7facdea9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7facdea9b640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7facd809ab00 con 0x7facd813be90 2024-09-16T04:39:03.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7facdea9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7facdea9b640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7facd809b030 con 0x7facd80a3ff0 2024-09-16T04:39:03.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7facdea9b640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:03.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7facdea9b640 10 monclient(hunting): authenticate will time out at 742.373788s 2024-09-16T04:39:03.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7faccffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facd80a3ff0 0x7facd80a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7faccffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facd80a3ff0 0x7facd80a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:33014/0 (socket says 172.21.15.46:33014) 2024-09-16T04:39:03.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7faccffff640 1 -- 172.21.15.46:0/861818515 learned_addr learned my addr 172.21.15.46:0/861818515 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:03.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7faccffff640 10 monclient(hunting): get_auth_request con 0x7facd80a3ff0 auth_method 0 2024-09-16T04:39:03.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7faccffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7faccffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.958+0000 7faccffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.959+0000 7faccffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.959+0000 7faccffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.959+0000 7faccffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.959+0000 7faccffff640 10 monclient(hunting): handle_auth_done global_id 4316 payload 274 2024-09-16T04:39:03.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.959+0000 7facdd038640 1 --2- 172.21.15.46:0/861818515 >> v2:172.21.15.114:3300/0 conn(0x7facd813be90 0x7facd813e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.959+0000 7faccffff640 1 -- 172.21.15.46:0/861818515 >> v2:172.21.15.114:3300/0 conn(0x7facd813be90 msgr2=0x7facd813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:03.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.959+0000 7faccffff640 1 --2- 172.21.15.46:0/861818515 >> v2:172.21.15.114:3300/0 conn(0x7facd813be90 0x7facd813e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.959+0000 7faccffff640 1 -- 172.21.15.46:0/861818515 >> v1:172.21.15.114:6789/0 conn(0x7facd80a3c10 legacy=0x7facd809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:03.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.959+0000 7faccffff640 10 monclient: _finish_hunting 0 2024-09-16T04:39:03.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.959+0000 7faccffff640 1 monclient: found mon.noname-b 2024-09-16T04:39:03.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.959+0000 7faccffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:03.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.959+0000 7faccffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.959+0000 7faccffff640 1 -- 172.21.15.46:0/861818515 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facd809b310 con 0x7facd80a3ff0 2024-09-16T04:39:03.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.959+0000 7faccffff640 1 --2- 172.21.15.46:0/861818515 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facd80a3ff0 0x7facd80a43d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7facc8009870 tx=0x7facc802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a05c40ff8ff03a09 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:03.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7faccf7fe640 1 -- 172.21.15.46:0/861818515 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc8037070 con 0x7facd80a3ff0 2024-09-16T04:39:03.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7faccf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7faccf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:03.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7faccf7fe640 10 monclient: dump: 2024-09-16T04:39:03.970 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.970 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.970 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.970 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.970 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.970 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.970 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.970 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.970 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.970 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7faccf7fe640 10 monclient: _finish_auth 0 2024-09-16T04:39:03.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7faccf7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:39:03.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7faccf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:03.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7faccf7fe640 1 -- 172.21.15.46:0/861818515 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facc8032440 con 0x7facd80a3ff0 2024-09-16T04:39:03.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7facdea9b640 5 monclient: authenticate success, global_id 4316 2024-09-16T04:39:03.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7facdea9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:03.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7faccf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:03.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7facdea9b640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:03.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7faccf7fe640 1 -- 172.21.15.46:0/861818515 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc8032810 con 0x7facd80a3ff0 2024-09-16T04:39:03.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7facdea9b640 4 set_mon_vals no callback set 2024-09-16T04:39:03.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7faccf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7faccf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:03.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7faccf7fe640 10 monclient: dump: 2024-09-16T04:39:03.971 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.971 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.972 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.972 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.972 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.972 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.972 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.972 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.972 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.972 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7facdea9b640 10 monclient: shutdown 2024-09-16T04:39:03.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7facdea9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:03.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7facdea9b640 1 -- 172.21.15.46:0/861818515 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facd80a3ff0 msgr2=0x7facd80a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7facdea9b640 1 --2- 172.21.15.46:0/861818515 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facd80a3ff0 0x7facd80a43d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7facc8009870 tx=0x7facc802a180 comp rx=0 tx=0).stop 2024-09-16T04:39:03.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7facdea9b640 1 -- 172.21.15.46:0/861818515 shutdown_connections 2024-09-16T04:39:03.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7facdea9b640 1 --2- 172.21.15.46:0/861818515 >> v2:172.21.15.114:3300/0 conn(0x7facd813be90 0x7facd813e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7facdea9b640 1 --2- 172.21.15.46:0/861818515 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facd80a3ff0 0x7facd80a43d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7facdea9b640 1 -- 172.21.15.46:0/861818515 >> 172.21.15.46:0/861818515 conn(0x7facd80a6340 msgr2=0x7facd806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:03.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7facdea9b640 1 -- 172.21.15.46:0/861818515 shutdown_connections 2024-09-16T04:39:03.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.960+0000 7facdea9b640 1 -- 172.21.15.46:0/861818515 wait complete. 2024-09-16T04:39:03.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.961+0000 7facdea9b640 10 monclient: build_initial_monmap 2024-09-16T04:39:03.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.961+0000 7facdea9b640 10 monclient: monmap: 2024-09-16T04:39:03.973 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:03.973 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:03.973 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:03.973 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:03.973 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:03.973 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.973 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:03.974 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:03.974 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:03.974 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.961+0000 7facdea9b640 1 Processor -- start 2024-09-16T04:39:03.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.961+0000 7facdea9b640 1 -- start start 2024-09-16T04:39:03.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.961+0000 7facdea9b640 10 monclient: init 2024-09-16T04:39:03.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdea9b640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:03.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdea9b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:39:03.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdea9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd80a3c10 0x7facd8133d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdea9b640 10 monclient(hunting): picked mon.noname-b con 0x7facd80a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:03.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdea9b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facd80a3ff0 0x7facd81342a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdea9b640 10 monclient(hunting): picked mon.noname-c con 0x7facd80a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:03.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdea9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facd813be90 0x7facd81cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdea9b640 10 monclient(hunting): picked mon.noname-a con 0x7facd813be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:03.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdea9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdea9b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7facd80a6b40 con 0x7facd813be90 2024-09-16T04:39:03.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdea9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdea9b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7facd80a69c0 con 0x7facd80a3c10 2024-09-16T04:39:03.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdea9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:03.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdea9b640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7facd813e7c0 con 0x7facd80a3ff0 2024-09-16T04:39:03.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdea9b640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:03.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdea9b640 10 monclient(hunting): authenticate will time out at 742.377378s 2024-09-16T04:39:03.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7faccffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facd80a3ff0 0x7facd81342a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7faccffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facd80a3ff0 0x7facd81342a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:33022/0 (socket says 172.21.15.46:33022) 2024-09-16T04:39:03.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7faccffff640 1 -- 172.21.15.46:0/4031580337 learned_addr learned my addr 172.21.15.46:0/4031580337 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:03.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7faccffff640 10 monclient(hunting): get_auth_request con 0x7facd80a3ff0 auth_method 0 2024-09-16T04:39:03.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7faccffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7faccffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7faccffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdd038640 1 --2- 172.21.15.46:0/4031580337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facd813be90 0x7facd81cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdd038640 10 monclient(hunting): get_auth_request con 0x7facd813be90 auth_method 0 2024-09-16T04:39:03.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdd038640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdd038640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdd038640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7faccffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7faccffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7faccffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdd038640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:03.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdd038640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:03.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdd038640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:03.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdc837640 1 --2- 172.21.15.46:0/4031580337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd80a3c10 0x7facd8133d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdc837640 10 monclient(hunting): get_auth_request con 0x7facd80a3c10 auth_method 0 2024-09-16T04:39:03.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdc837640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:03.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdc837640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:03.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdc837640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:03.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7faccffff640 10 monclient(hunting): handle_auth_done global_id 4319 payload 824 2024-09-16T04:39:03.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7faccffff640 1 -- 172.21.15.46:0/4031580337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd80a3c10 msgr2=0x7facd8133d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7faccffff640 1 --2- 172.21.15.46:0/4031580337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd80a3c10 0x7facd8133d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7faccffff640 1 -- 172.21.15.46:0/4031580337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facd813be90 msgr2=0x7facd81cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:03.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7faccffff640 1 --2- 172.21.15.46:0/4031580337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facd813be90 0x7facd81cbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:03.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7faccffff640 10 monclient: _finish_hunting 0 2024-09-16T04:39:03.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7faccffff640 1 monclient: found mon.noname-c 2024-09-16T04:39:03.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7faccffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:03.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7faccffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7faccffff640 1 -- 172.21.15.46:0/4031580337 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facd813c3b0 con 0x7facd80a3ff0 2024-09-16T04:39:03.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7facdc837640 1 --2- 172.21.15.46:0/4031580337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd80a3c10 0x7facd8133d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:39:03.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.962+0000 7faccffff640 1 --2- 172.21.15.46:0/4031580337 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facd80a3ff0 0x7facd81342a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7facc8000c00 tx=0x7facc80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:03.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.963+0000 7faccd7fa640 1 -- 172.21.15.46:0/4031580337 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc8037040 con 0x7facd80a3ff0 2024-09-16T04:39:03.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.963+0000 7faccd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.963+0000 7faccd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:39:03.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.963+0000 7faccd7fa640 10 monclient: dump: 2024-09-16T04:39:03.978 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.978 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.978 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.979 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.979 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.979 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.979 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.979 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.979 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.979 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.963+0000 7faccd7fa640 10 monclient: _finish_auth 0 2024-09-16T04:39:03.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.963+0000 7faccd7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:39:03.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.963+0000 7faccd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:03.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.963+0000 7faccd7fa640 1 -- 172.21.15.46:0/4031580337 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facc8042070 con 0x7facd80a3ff0 2024-09-16T04:39:03.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.963+0000 7facdea9b640 5 monclient: authenticate success, global_id 4319 2024-09-16T04:39:03.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.963+0000 7facdea9b640 10 monclient: _renew_subs 2024-09-16T04:39:03.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.963+0000 7facdea9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.963+0000 7facdea9b640 1 -- 172.21.15.46:0/4031580337 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facd809eee0 con 0x7facd80a3ff0 2024-09-16T04:39:03.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.963+0000 7faccd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:03.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.963+0000 7faccd7fa640 1 -- 172.21.15.46:0/4031580337 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc8030bb0 con 0x7facd80a3ff0 2024-09-16T04:39:03.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.963+0000 7faccd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:03.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.963+0000 7faccd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:03.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.963+0000 7faccd7fa640 10 monclient: dump: 2024-09-16T04:39:03.980 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:03.980 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:03.980 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.980 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:03.980 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:03.980 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:03.981 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:03.981 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:03.981 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:03.981 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:03.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.963+0000 7facdea9b640 10 monclient: _renew_subs 2024-09-16T04:39:03.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.963+0000 7facdea9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.963+0000 7facdea9b640 1 -- 172.21.15.46:0/4031580337 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7facd80a3500 con 0x7facd80a3ff0 2024-09-16T04:39:03.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.963+0000 7facceffd640 4 set_mon_vals no callback set 2024-09-16T04:39:03.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.964+0000 7faccd7fa640 1 -- 172.21.15.46:0/4031580337 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7facc8047460 con 0x7facd80a3ff0 2024-09-16T04:39:03.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.964+0000 7facdea9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:03.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.964+0000 7facdea9b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:03.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.964+0000 7facdea9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:03.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.964+0000 7faccd7fa640 1 --2- 172.21.15.46:0/4031580337 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7facc40383b0 0x7facc403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:03.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.964+0000 7facdea9b640 1 -- 172.21.15.46:0/4031580337 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facd80a43d0 con 0x7facd80a3ff0 2024-09-16T04:39:03.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.964+0000 7faccd7fa640 1 -- 172.21.15.46:0/4031580337 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7facc806c260 con 0x7facd80a3ff0 2024-09-16T04:39:03.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.964+0000 7facdc837640 1 --2- 172.21.15.46:0/4031580337 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7facc40383b0 0x7facc403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:03.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.964+0000 7facdc837640 10 monclient: get_auth_request con 0x7facc40383b0 auth_method 0 2024-09-16T04:39:03.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.965+0000 7facdc837640 1 --2- 172.21.15.46:0/4031580337 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7facc40383b0 0x7facc403a870 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7facd4006fd0 tx=0x7facd4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:03.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.968+0000 7faccd7fa640 1 -- 172.21.15.46:0/4031580337 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7facc8030d50 con 0x7facd80a3ff0 2024-09-16T04:39:03.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.969+0000 7faccd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:03.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:03.969+0000 7faccd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:04.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.143+0000 7facdea9b640 1 -- 172.21.15.46:0/4031580337 --> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7facd80a0400 con 0x7facc40383b0 2024-09-16T04:39:04.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.145+0000 7faccd7fa640 1 -- 172.21.15.46:0/4031580337 <== mgr.4099 v2:172.21.15.114:6800/3521677055 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34116 (secure 0 0 0) 0x7facd80a0400 con 0x7facc40383b0 2024-09-16T04:39:04.146 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:39:04.147 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-09-16T04:39:04.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.147+0000 7facdea9b640 1 -- 172.21.15.46:0/4031580337 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7facc40383b0 msgr2=0x7facc403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:04.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.147+0000 7facdea9b640 1 --2- 172.21.15.46:0/4031580337 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7facc40383b0 0x7facc403a870 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7facd4006fd0 tx=0x7facd4008040 comp rx=0 tx=0).stop 2024-09-16T04:39:04.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.147+0000 7facdea9b640 10 monclient: shutdown 2024-09-16T04:39:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.147+0000 7facdea9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.147+0000 7facdea9b640 1 -- 172.21.15.46:0/4031580337 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facd80a3ff0 msgr2=0x7facd81342a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.147+0000 7facdea9b640 1 --2- 172.21.15.46:0/4031580337 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facd80a3ff0 0x7facd81342a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7facc8000c00 tx=0x7facc80046c0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.147+0000 7facdea9b640 1 -- 172.21.15.46:0/4031580337 shutdown_connections 2024-09-16T04:39:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.147+0000 7facdea9b640 1 --2- 172.21.15.46:0/4031580337 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7facc40383b0 0x7facc403a870 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.147+0000 7facdea9b640 1 --2- 172.21.15.46:0/4031580337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facd813be90 0x7facd81cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.147+0000 7facdea9b640 1 --2- 172.21.15.46:0/4031580337 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facd80a3ff0 0x7facd81342a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.147+0000 7facdea9b640 1 --2- 172.21.15.46:0/4031580337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd80a3c10 0x7facd8133d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.147+0000 7facdea9b640 1 -- 172.21.15.46:0/4031580337 >> 172.21.15.46:0/4031580337 conn(0x7facd80a6340 msgr2=0x7facd813b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:04.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.147+0000 7facdea9b640 1 -- 172.21.15.46:0/4031580337 shutdown_connections 2024-09-16T04:39:04.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.147+0000 7facdea9b640 1 -- 172.21.15.46:0/4031580337 wait complete. 2024-09-16T04:39:04.159 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-16T04:39:04.070098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":172616,"kb_used_data":2104,"kb_used_omap":9,"kb_used_meta":170358,"kb_avail":566058424,"statfs":{"total":579820584960,"available":579643826176,"internally_reserved":0,"allocated":2154496,"data_stored":1420330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":174447264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.552973"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.524013+0000","last_change":"2024-09-16T04:39:00.524013+0000","last_active":"2024-09-16T04:39:00.524013+0000","last_peered":"2024-09-16T04:39:00.524013+0000","last_clean":"2024-09-16T04:39:00.524013+0000","last_became_active":"2024-09-16T04:38:58.756778+0000","last_became_peered":"2024-09-16T04:38:58.756778+0000","last_unstale":"2024-09-16T04:39:00.524013+0000","last_undegraded":"2024-09-16T04:39:00.524013+0000","last_fullsized":"2024-09-16T04:39:00.524013+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:24:15.429766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054536700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.531971+0000","last_change":"2024-09-16T04:39:00.532294+0000","last_active":"2024-09-16T04:39:00.531971+0000","last_peered":"2024-09-16T04:39:00.531971+0000","last_clean":"2024-09-16T04:39:00.531971+0000","last_became_active":"2024-09-16T04:38:58.520473+0000","last_became_peered":"2024-09-16T04:38:58.520473+0000","last_unstale":"2024-09-16T04:39:00.531971+0000","last_undegraded":"2024-09-16T04:39:00.531971+0000","last_fullsized":"2024-09-16T04:39:00.531971+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:15:41.548834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053532500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.523783+0000","last_change":"2024-09-16T04:39:00.523783+0000","last_active":"2024-09-16T04:39:00.523783+0000","last_peered":"2024-09-16T04:39:00.523783+0000","last_clean":"2024-09-16T04:39:00.523783+0000","last_became_active":"2024-09-16T04:38:58.521000+0000","last_became_peered":"2024-09-16T04:38:58.521000+0000","last_unstale":"2024-09-16T04:39:00.523783+0000","last_undegraded":"2024-09-16T04:39:00.523783+0000","last_fullsized":"2024-09-16T04:39:00.523783+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:42.837022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035484999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.532031+0000","last_change":"2024-09-16T04:39:00.532400+0000","last_active":"2024-09-16T04:39:00.532031+0000","last_peered":"2024-09-16T04:39:00.532031+0000","last_clean":"2024-09-16T04:39:00.532031+0000","last_became_active":"2024-09-16T04:38:58.521627+0000","last_became_peered":"2024-09-16T04:38:58.521627+0000","last_unstale":"2024-09-16T04:39:00.532031+0000","last_undegraded":"2024-09-16T04:39:00.532031+0000","last_fullsized":"2024-09-16T04:39:00.532031+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:41.804897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062256500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.530578+0000","last_change":"2024-09-16T04:39:00.530578+0000","last_active":"2024-09-16T04:39:00.530578+0000","last_peered":"2024-09-16T04:39:00.530578+0000","last_clean":"2024-09-16T04:39:00.530578+0000","last_became_active":"2024-09-16T04:38:58.522744+0000","last_became_peered":"2024-09-16T04:38:58.522744+0000","last_unstale":"2024-09-16T04:39:00.530578+0000","last_undegraded":"2024-09-16T04:39:00.530578+0000","last_fullsized":"2024-09-16T04:39:00.530578+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:04:47.450053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000966258,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.523413+0000","last_change":"2024-09-16T04:39:00.523661+0000","last_active":"2024-09-16T04:39:00.523413+0000","last_peered":"2024-09-16T04:39:00.523413+0000","last_clean":"2024-09-16T04:39:00.523413+0000","last_became_active":"2024-09-16T04:38:58.524372+0000","last_became_peered":"2024-09-16T04:38:58.524372+0000","last_unstale":"2024-09-16T04:39:00.523413+0000","last_undegraded":"2024-09-16T04:39:00.523413+0000","last_fullsized":"2024-09-16T04:39:00.523413+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:46.534727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047533599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.523838+0000","last_change":"2024-09-16T04:39:00.523838+0000","last_active":"2024-09-16T04:39:00.523838+0000","last_peered":"2024-09-16T04:39:00.523838+0000","last_clean":"2024-09-16T04:39:00.523838+0000","last_became_active":"2024-09-16T04:38:58.522142+0000","last_became_peered":"2024-09-16T04:38:58.522142+0000","last_unstale":"2024-09-16T04:39:00.523838+0000","last_undegraded":"2024-09-16T04:39:00.523838+0000","last_fullsized":"2024-09-16T04:39:00.523838+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:35.411495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039704200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.529810+0000","last_change":"2024-09-16T04:39:00.530064+0000","last_active":"2024-09-16T04:39:00.529810+0000","last_peered":"2024-09-16T04:39:00.529810+0000","last_clean":"2024-09-16T04:39:00.529810+0000","last_became_active":"2024-09-16T04:38:58.523268+0000","last_became_peered":"2024-09-16T04:38:58.523268+0000","last_unstale":"2024-09-16T04:39:00.529810+0000","last_undegraded":"2024-09-16T04:39:00.529810+0000","last_fullsized":"2024-09-16T04:39:00.529810+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:12:33.958554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043992499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":170,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.523350+0000","last_change":"2024-09-16T04:38:52.501054+0000","last_active":"2024-09-16T04:39:00.523350+0000","last_peered":"2024-09-16T04:39:00.523350+0000","last_clean":"2024-09-16T04:39:00.523350+0000","last_became_active":"2024-09-16T04:38:52.500796+0000","last_became_peered":"2024-09-16T04:38:52.500796+0000","last_unstale":"2024-09-16T04:39:00.523350+0000","last_undegraded":"2024-09-16T04:39:00.523350+0000","last_fullsized":"2024-09-16T04:39:00.523350+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:51.492404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:51.492404+0000","last_clean_scrub_stamp":"2024-09-16T04:38:51.492404+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:36.122039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":180224,"data_stored":60406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":712704,"data_stored":589334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31884,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339956,"statfs":{"total":96636764160,"available":96604114944,"internally_reserved":0,"allocated":712704,"data_stored":589334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":184320,"data_stored":60425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":180224,"data_stored":60406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31376,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340464,"statfs":{"total":96636764160,"available":96604635136,"internally_reserved":0,"allocated":184320,"data_stored":60425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:39:04.160 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:39:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.259+0000 7faac216d640 10 monclient: get_monmap_and_config 2024-09-16T04:39:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.259+0000 7faac216d640 10 monclient: build_initial_monmap 2024-09-16T04:39:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.259+0000 7faac216d640 10 monclient: monmap: 2024-09-16T04:39:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:04.259716+0000 2024-09-16T04:39:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:04.259716+0000 2024-09-16T04:39:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:04.261 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.259+0000 7faac216d640 1 Processor -- start 2024-09-16T04:39:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faac216d640 1 -- start start 2024-09-16T04:39:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faac216d640 10 monclient: init 2024-09-16T04:39:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faac216d640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faac216d640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-16T04:39:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faac216d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7faabc0a3c10 0x7faabc09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faac216d640 10 monclient(hunting): picked mon.noname-c con 0x7faabc0a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faac216d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faabc0a3ff0 0x7faabc0a4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faac216d640 10 monclient(hunting): picked mon.noname-a con 0x7faabc0a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faac216d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faabc1369d0 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faac216d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faabc09ab70 con 0x7faabc1369d0 2024-09-16T04:39:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faac216d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faac216d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7faabc0a4d60 con 0x7faabc0a3ff0 2024-09-16T04:39:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faac216d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faac216d640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7faabc09b030 con 0x7faabc0a3c10 2024-09-16T04:39:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faac216d640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faac216d640 10 monclient(hunting): authenticate will time out at 742.675604s 2024-09-16T04:39:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faabaffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faabc0a3ff0 0x7faabc0a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faabaffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faabc0a3ff0 0x7faabc0a4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59498/0 (socket says 172.21.15.46:59498) 2024-09-16T04:39:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faabaffd640 1 -- 172.21.15.46:0/483520697 learned_addr learned my addr 172.21.15.46:0/483520697 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faabaffd640 10 monclient(hunting): get_auth_request con 0x7faabc0a3ff0 auth_method 0 2024-09-16T04:39:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faabaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faabaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faabaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:04.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faabaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:04.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faabaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:04.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faabaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:04.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faabb7fe640 1 --2- 172.21.15.46:0/483520697 >> v2:172.21.15.114:3300/0 conn(0x7faabc0a3c10 0x7faabc09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:04.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faabb7fe640 10 monclient(hunting): get_auth_request con 0x7faabc0a3c10 auth_method 0 2024-09-16T04:39:04.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faabb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:04.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faabb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:04.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faabb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:04.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faabaffd640 10 monclient(hunting): handle_auth_done global_id 4368 payload 274 2024-09-16T04:39:04.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faabaffd640 1 -- 172.21.15.46:0/483520697 >> v2:172.21.15.114:3300/0 conn(0x7faabc0a3c10 msgr2=0x7faabc09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:04.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faabaffd640 1 --2- 172.21.15.46:0/483520697 >> v2:172.21.15.114:3300/0 conn(0x7faabc0a3c10 0x7faabc09bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.260+0000 7faabaffd640 1 -- 172.21.15.46:0/483520697 >> v1:172.21.15.114:6789/0 conn(0x7faabc1369d0 legacy=0x7faabc136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:04.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faabaffd640 10 monclient: _finish_hunting 0 2024-09-16T04:39:04.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faabaffd640 1 monclient: found mon.noname-a 2024-09-16T04:39:04.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faabaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:04.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faabaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:04.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faabaffd640 1 -- 172.21.15.46:0/483520697 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faabc09b310 con 0x7faabc0a3ff0 2024-09-16T04:39:04.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faabaffd640 1 --2- 172.21.15.46:0/483520697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faabc0a3ff0 0x7faabc0a4400 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faabc06be80 tx=0x7faab002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79cd3bf03b012a34 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:04.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faaba7fc640 1 -- 172.21.15.46:0/483520697 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab0037070 con 0x7faabc0a3ff0 2024-09-16T04:39:04.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faaba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:04.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faaba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:04.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faaba7fc640 10 monclient: dump: 2024-09-16T04:39:04.272 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:04.272 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:04.272 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.272 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.272 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:04.272 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:04.273 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:04.273 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:04.273 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:04.273 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:04.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faaba7fc640 10 monclient: _finish_auth 0 2024-09-16T04:39:04.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faaba7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:39:04.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faaba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:04.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faac216d640 5 monclient: authenticate success, global_id 4368 2024-09-16T04:39:04.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faac216d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:04.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faaba7fc640 1 -- 172.21.15.46:0/483520697 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faab002ae20 con 0x7faabc0a3ff0 2024-09-16T04:39:04.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faaba7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:04.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faac216d640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:04.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faac216d640 4 set_mon_vals no callback set 2024-09-16T04:39:04.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faaba7fc640 1 -- 172.21.15.46:0/483520697 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab0032660 con 0x7faabc0a3ff0 2024-09-16T04:39:04.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faaba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:04.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faaba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:04.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.261+0000 7faaba7fc640 10 monclient: dump: 2024-09-16T04:39:04.274 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:04.274 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:04.274 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.274 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.274 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:04.274 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:04.274 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:04.274 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:04.274 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:04.274 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:04.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.262+0000 7faac216d640 10 monclient: shutdown 2024-09-16T04:39:04.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.262+0000 7faac216d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:04.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.262+0000 7faac216d640 1 -- 172.21.15.46:0/483520697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faabc0a3ff0 msgr2=0x7faabc0a4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:04.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.262+0000 7faac216d640 1 --2- 172.21.15.46:0/483520697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faabc0a3ff0 0x7faabc0a4400 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faabc06be80 tx=0x7faab002a110 comp rx=0 tx=0).stop 2024-09-16T04:39:04.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.262+0000 7faac216d640 1 -- 172.21.15.46:0/483520697 shutdown_connections 2024-09-16T04:39:04.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.262+0000 7faac216d640 1 --2- 172.21.15.46:0/483520697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faabc0a3ff0 0x7faabc0a4400 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.262+0000 7faac216d640 1 --2- 172.21.15.46:0/483520697 >> v2:172.21.15.114:3300/0 conn(0x7faabc0a3c10 0x7faabc09bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.262+0000 7faac216d640 1 -- 172.21.15.46:0/483520697 >> 172.21.15.46:0/483520697 conn(0x7faabc0a5d10 msgr2=0x7faabc0a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:04.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.262+0000 7faac216d640 1 -- 172.21.15.46:0/483520697 shutdown_connections 2024-09-16T04:39:04.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.262+0000 7faac216d640 1 -- 172.21.15.46:0/483520697 wait complete. 2024-09-16T04:39:04.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.263+0000 7faac216d640 10 monclient: build_initial_monmap 2024-09-16T04:39:04.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.263+0000 7faac216d640 10 monclient: monmap: 2024-09-16T04:39:04.275 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:04.275 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:04.275 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:04.276 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.263+0000 7faac216d640 1 Processor -- start 2024-09-16T04:39:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.263+0000 7faac216d640 1 -- start start 2024-09-16T04:39:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.263+0000 7faac216d640 10 monclient: init 2024-09-16T04:39:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faac216d640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faac216d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:39:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faac216d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faabc0a3c10 0x7faabc1cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faac216d640 10 monclient(hunting): picked mon.noname-b con 0x7faabc0a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:04.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faac216d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faabc0a3ff0 0x7faabc1cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:04.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faac216d640 10 monclient(hunting): picked mon.noname-a con 0x7faabc0a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:04.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faac216d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faabc1369d0 0x7faabc123010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:04.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faac216d640 10 monclient(hunting): picked mon.noname-c con 0x7faabc1369d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:04.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faac216d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:04.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faac216d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faabc06e3d0 con 0x7faabc0a3ff0 2024-09-16T04:39:04.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faac216d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:04.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faac216d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7faabc0a4940 con 0x7faabc0a3c10 2024-09-16T04:39:04.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faac216d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:04.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faac216d640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7faabc06e550 con 0x7faabc1369d0 2024-09-16T04:39:04.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faac216d640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:04.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faac216d640 10 monclient(hunting): authenticate will time out at 742.679136s 2024-09-16T04:39:04.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faabc0a3ff0 0x7faabc1cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:04.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faabc0a3ff0 0x7faabc1cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59512/0 (socket says 172.21.15.46:59512) 2024-09-16T04:39:04.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabaffd640 1 -- 172.21.15.46:0/3281433624 learned_addr learned my addr 172.21.15.46:0/3281433624 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:04.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabaffd640 10 monclient(hunting): get_auth_request con 0x7faabc0a3ff0 auth_method 0 2024-09-16T04:39:04.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:04.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:04.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:04.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabbfff640 1 --2- 172.21.15.46:0/3281433624 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faabc1369d0 0x7faabc123010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:04.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:04.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:04.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabb7fe640 1 --2- 172.21.15.46:0/3281433624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faabc0a3c10 0x7faabc1cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:04.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:04.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabbfff640 10 monclient(hunting): get_auth_request con 0x7faabc1369d0 auth_method 0 2024-09-16T04:39:04.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:04.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:04.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:04.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabb7fe640 10 monclient(hunting): get_auth_request con 0x7faabc0a3c10 auth_method 0 2024-09-16T04:39:04.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:04.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:04.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabaffd640 10 monclient(hunting): handle_auth_done global_id 4371 payload 824 2024-09-16T04:39:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabaffd640 1 -- 172.21.15.46:0/3281433624 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faabc1369d0 msgr2=0x7faabc123010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabaffd640 1 --2- 172.21.15.46:0/3281433624 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faabc1369d0 0x7faabc123010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabaffd640 1 -- 172.21.15.46:0/3281433624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faabc0a3c10 msgr2=0x7faabc1cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabaffd640 1 --2- 172.21.15.46:0/3281433624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faabc0a3c10 0x7faabc1cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabaffd640 10 monclient: _finish_hunting 0 2024-09-16T04:39:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabaffd640 1 monclient: found mon.noname-a 2024-09-16T04:39:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabaffd640 1 -- 172.21.15.46:0/3281433624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faabc1269e0 con 0x7faabc0a3ff0 2024-09-16T04:39:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faabaffd640 1 --2- 172.21.15.46:0/3281433624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faabc0a3ff0 0x7faabc1cc420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7faab0000c00 tx=0x7faab00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faaa3fff640 1 -- 172.21.15.46:0/3281433624 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab0041070 con 0x7faabc0a3ff0 2024-09-16T04:39:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faaa3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faaa3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faaa3fff640 10 monclient: dump: 2024-09-16T04:39:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:04.282 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:04.282 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:04.282 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:04.282 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:04.282 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:04.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faaa3fff640 10 monclient: _finish_auth 0 2024-09-16T04:39:04.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faaa3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:39:04.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faaa3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:04.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faaa3fff640 1 -- 172.21.15.46:0/3281433624 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faab003c670 con 0x7faabc0a3ff0 2024-09-16T04:39:04.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faaa3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:04.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faaa3fff640 1 -- 172.21.15.46:0/3281433624 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab0037040 con 0x7faabc0a3ff0 2024-09-16T04:39:04.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faac216d640 5 monclient: authenticate success, global_id 4371 2024-09-16T04:39:04.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faaa3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:04.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faaa3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:04.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faaa3fff640 10 monclient: dump: 2024-09-16T04:39:04.283 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:04.283 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:04.283 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.283 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.283 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:04.283 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:04.283 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:04.283 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:04.283 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:04.283 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:04.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faac216d640 10 monclient: _renew_subs 2024-09-16T04:39:04.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faac216d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:04.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faac216d640 1 -- 172.21.15.46:0/3281433624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faabc125150 con 0x7faabc0a3ff0 2024-09-16T04:39:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.264+0000 7faab9ffb640 4 set_mon_vals no callback set 2024-09-16T04:39:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.265+0000 7faac216d640 10 monclient: _renew_subs 2024-09-16T04:39:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.265+0000 7faac216d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.265+0000 7faac216d640 1 -- 172.21.15.46:0/3281433624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faabc126cc0 con 0x7faabc0a3ff0 2024-09-16T04:39:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.265+0000 7faaa3fff640 1 -- 172.21.15.46:0/3281433624 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7faab0044070 con 0x7faabc0a3ff0 2024-09-16T04:39:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.265+0000 7faaa3fff640 1 --2- 172.21.15.46:0/3281433624 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7faaa80383b0 0x7faaa803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.265+0000 7faaa3fff640 1 -- 172.21.15.46:0/3281433624 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7faab006bc80 con 0x7faabc0a3ff0 2024-09-16T04:39:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.266+0000 7faac216d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.266+0000 7faac216d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.266+0000 7faac216d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.266+0000 7faac216d640 1 -- 172.21.15.46:0/3281433624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faabc09bcf0 con 0x7faabc0a3ff0 2024-09-16T04:39:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.266+0000 7faabb7fe640 1 --2- 172.21.15.46:0/3281433624 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7faaa80383b0 0x7faaa803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.266+0000 7faabb7fe640 10 monclient: get_auth_request con 0x7faaa80383b0 auth_method 0 2024-09-16T04:39:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.266+0000 7faabb7fe640 1 --2- 172.21.15.46:0/3281433624 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7faaa80383b0 0x7faaa803a870 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faaac006fd0 tx=0x7faaac008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:04.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.270+0000 7faaa3fff640 1 -- 172.21.15.46:0/3281433624 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7faab002ae20 con 0x7faabc0a3ff0 2024-09-16T04:39:04.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.270+0000 7faaa3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:04.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.270+0000 7faaa3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:04.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.445+0000 7faac216d640 1 -- 172.21.15.46:0/3281433624 --> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faabc069150 con 0x7faaa80383b0 2024-09-16T04:39:04.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.447+0000 7faaa3fff640 1 -- 172.21.15.46:0/3281433624 <== mgr.4099 v2:172.21.15.114:6800/3521677055 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34116 (secure 0 0 0) 0x7faabc069150 con 0x7faaa80383b0 2024-09-16T04:39:04.449 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:39:04.449 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-09-16T04:39:04.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.449+0000 7faac216d640 1 -- 172.21.15.46:0/3281433624 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7faaa80383b0 msgr2=0x7faaa803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:04.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.449+0000 7faac216d640 1 --2- 172.21.15.46:0/3281433624 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7faaa80383b0 0x7faaa803a870 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faaac006fd0 tx=0x7faaac008040 comp rx=0 tx=0).stop 2024-09-16T04:39:04.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.449+0000 7faac216d640 10 monclient: shutdown 2024-09-16T04:39:04.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.449+0000 7faac216d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:04.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.449+0000 7faac216d640 1 -- 172.21.15.46:0/3281433624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faabc0a3ff0 msgr2=0x7faabc1cc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:04.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.449+0000 7faac216d640 1 --2- 172.21.15.46:0/3281433624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faabc0a3ff0 0x7faabc1cc420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7faab0000c00 tx=0x7faab00046c0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.449+0000 7faac216d640 1 -- 172.21.15.46:0/3281433624 shutdown_connections 2024-09-16T04:39:04.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.449+0000 7faac216d640 1 --2- 172.21.15.46:0/3281433624 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faabc1369d0 0x7faabc123010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.449+0000 7faac216d640 1 --2- 172.21.15.46:0/3281433624 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7faaa80383b0 0x7faaa803a870 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.449+0000 7faac216d640 1 --2- 172.21.15.46:0/3281433624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faabc0a3ff0 0x7faabc1cc420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.449+0000 7faac216d640 1 --2- 172.21.15.46:0/3281433624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faabc0a3c10 0x7faabc1cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.449+0000 7faac216d640 1 -- 172.21.15.46:0/3281433624 >> 172.21.15.46:0/3281433624 conn(0x7faabc0a5d10 msgr2=0x7faabc0a70c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:04.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.449+0000 7faac216d640 1 -- 172.21.15.46:0/3281433624 shutdown_connections 2024-09-16T04:39:04.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.449+0000 7faac216d640 1 -- 172.21.15.46:0/3281433624 wait complete. 2024-09-16T04:39:04.461 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-16T04:39:04.070098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":172616,"kb_used_data":2104,"kb_used_omap":9,"kb_used_meta":170358,"kb_avail":566058424,"statfs":{"total":579820584960,"available":579643826176,"internally_reserved":0,"allocated":2154496,"data_stored":1420330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":174447264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.552973"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.524013+0000","last_change":"2024-09-16T04:39:00.524013+0000","last_active":"2024-09-16T04:39:00.524013+0000","last_peered":"2024-09-16T04:39:00.524013+0000","last_clean":"2024-09-16T04:39:00.524013+0000","last_became_active":"2024-09-16T04:38:58.756778+0000","last_became_peered":"2024-09-16T04:38:58.756778+0000","last_unstale":"2024-09-16T04:39:00.524013+0000","last_undegraded":"2024-09-16T04:39:00.524013+0000","last_fullsized":"2024-09-16T04:39:00.524013+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:24:15.429766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054536700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.531971+0000","last_change":"2024-09-16T04:39:00.532294+0000","last_active":"2024-09-16T04:39:00.531971+0000","last_peered":"2024-09-16T04:39:00.531971+0000","last_clean":"2024-09-16T04:39:00.531971+0000","last_became_active":"2024-09-16T04:38:58.520473+0000","last_became_peered":"2024-09-16T04:38:58.520473+0000","last_unstale":"2024-09-16T04:39:00.531971+0000","last_undegraded":"2024-09-16T04:39:00.531971+0000","last_fullsized":"2024-09-16T04:39:00.531971+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:15:41.548834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053532500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.523783+0000","last_change":"2024-09-16T04:39:00.523783+0000","last_active":"2024-09-16T04:39:00.523783+0000","last_peered":"2024-09-16T04:39:00.523783+0000","last_clean":"2024-09-16T04:39:00.523783+0000","last_became_active":"2024-09-16T04:38:58.521000+0000","last_became_peered":"2024-09-16T04:38:58.521000+0000","last_unstale":"2024-09-16T04:39:00.523783+0000","last_undegraded":"2024-09-16T04:39:00.523783+0000","last_fullsized":"2024-09-16T04:39:00.523783+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:42.837022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035484999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.532031+0000","last_change":"2024-09-16T04:39:00.532400+0000","last_active":"2024-09-16T04:39:00.532031+0000","last_peered":"2024-09-16T04:39:00.532031+0000","last_clean":"2024-09-16T04:39:00.532031+0000","last_became_active":"2024-09-16T04:38:58.521627+0000","last_became_peered":"2024-09-16T04:38:58.521627+0000","last_unstale":"2024-09-16T04:39:00.532031+0000","last_undegraded":"2024-09-16T04:39:00.532031+0000","last_fullsized":"2024-09-16T04:39:00.532031+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:41.804897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062256500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.530578+0000","last_change":"2024-09-16T04:39:00.530578+0000","last_active":"2024-09-16T04:39:00.530578+0000","last_peered":"2024-09-16T04:39:00.530578+0000","last_clean":"2024-09-16T04:39:00.530578+0000","last_became_active":"2024-09-16T04:38:58.522744+0000","last_became_peered":"2024-09-16T04:38:58.522744+0000","last_unstale":"2024-09-16T04:39:00.530578+0000","last_undegraded":"2024-09-16T04:39:00.530578+0000","last_fullsized":"2024-09-16T04:39:00.530578+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:04:47.450053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000966258,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.523413+0000","last_change":"2024-09-16T04:39:00.523661+0000","last_active":"2024-09-16T04:39:00.523413+0000","last_peered":"2024-09-16T04:39:00.523413+0000","last_clean":"2024-09-16T04:39:00.523413+0000","last_became_active":"2024-09-16T04:38:58.524372+0000","last_became_peered":"2024-09-16T04:38:58.524372+0000","last_unstale":"2024-09-16T04:39:00.523413+0000","last_undegraded":"2024-09-16T04:39:00.523413+0000","last_fullsized":"2024-09-16T04:39:00.523413+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:46.534727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047533599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.523838+0000","last_change":"2024-09-16T04:39:00.523838+0000","last_active":"2024-09-16T04:39:00.523838+0000","last_peered":"2024-09-16T04:39:00.523838+0000","last_clean":"2024-09-16T04:39:00.523838+0000","last_became_active":"2024-09-16T04:38:58.522142+0000","last_became_peered":"2024-09-16T04:38:58.522142+0000","last_unstale":"2024-09-16T04:39:00.523838+0000","last_undegraded":"2024-09-16T04:39:00.523838+0000","last_fullsized":"2024-09-16T04:39:00.523838+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:35.411495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039704200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.529810+0000","last_change":"2024-09-16T04:39:00.530064+0000","last_active":"2024-09-16T04:39:00.529810+0000","last_peered":"2024-09-16T04:39:00.529810+0000","last_clean":"2024-09-16T04:39:00.529810+0000","last_became_active":"2024-09-16T04:38:58.523268+0000","last_became_peered":"2024-09-16T04:38:58.523268+0000","last_unstale":"2024-09-16T04:39:00.529810+0000","last_undegraded":"2024-09-16T04:39:00.529810+0000","last_fullsized":"2024-09-16T04:39:00.529810+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:12:33.958554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043992499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":170,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.523350+0000","last_change":"2024-09-16T04:38:52.501054+0000","last_active":"2024-09-16T04:39:00.523350+0000","last_peered":"2024-09-16T04:39:00.523350+0000","last_clean":"2024-09-16T04:39:00.523350+0000","last_became_active":"2024-09-16T04:38:52.500796+0000","last_became_peered":"2024-09-16T04:38:52.500796+0000","last_unstale":"2024-09-16T04:39:00.523350+0000","last_undegraded":"2024-09-16T04:39:00.523350+0000","last_fullsized":"2024-09-16T04:39:00.523350+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:51.492404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:51.492404+0000","last_clean_scrub_stamp":"2024-09-16T04:38:51.492404+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:36.122039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":180224,"data_stored":60406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":712704,"data_stored":589334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31884,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339956,"statfs":{"total":96636764160,"available":96604114944,"internally_reserved":0,"allocated":712704,"data_stored":589334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":184320,"data_stored":60425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":180224,"data_stored":60406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31376,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340464,"statfs":{"total":96636764160,"available":96604635136,"internally_reserved":0,"allocated":184320,"data_stored":60425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:39:04.462 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T04:39:04.462 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T04:39:04.462 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T04:39:04.462 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T04:39:04.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.560+0000 7f064deb7640 10 monclient: get_monmap_and_config 2024-09-16T04:39:04.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.560+0000 7f064deb7640 10 monclient: build_initial_monmap 2024-09-16T04:39:04.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.561+0000 7f064deb7640 10 monclient: monmap: 2024-09-16T04:39:04.562 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:04.562 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:04.562 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:04.561466+0000 2024-09-16T04:39:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:04.561466+0000 2024-09-16T04:39:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:04.563 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:04.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.561+0000 7f064deb7640 1 Processor -- start 2024-09-16T04:39:04.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.561+0000 7f064deb7640 1 -- start start 2024-09-16T04:39:04.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.561+0000 7f064deb7640 10 monclient: init 2024-09-16T04:39:04.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f064deb7640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:04.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f064deb7640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-16T04:39:04.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f064deb7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f06480a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:04.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f064deb7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f06480a3ff0 0x7f06480a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:04.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f064deb7640 10 monclient(hunting): picked mon.noname-a con 0x7f06480a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:04.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f064deb7640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f064813be90 0x7f064813e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:04.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f064deb7640 10 monclient(hunting): picked mon.noname-b con 0x7f064813be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:04.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f064deb7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f064809b940 con 0x7f06480a3c10 2024-09-16T04:39:04.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f064deb7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:04.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f064deb7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f064809ab00 con 0x7f06480a3ff0 2024-09-16T04:39:04.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f064deb7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:04.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f064deb7640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f064809b030 con 0x7f064813be90 2024-09-16T04:39:04.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f064deb7640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:04.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f064deb7640 10 monclient(hunting): authenticate will time out at 742.977285s 2024-09-16T04:39:04.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0646ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f06480a3ff0 0x7f06480a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:04.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0647fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f064813be90 0x7f064813e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:04.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0646ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f06480a3ff0 0x7f06480a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34834/0 (socket says 172.21.15.46:34834) 2024-09-16T04:39:04.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0647fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f064813be90 0x7f064813e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:36394/0 (socket says 172.21.15.46:36394) 2024-09-16T04:39:04.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0646ffd640 1 -- 172.21.15.46:0/1396712181 learned_addr learned my addr 172.21.15.46:0/1396712181 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:04.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0646ffd640 10 monclient(hunting): get_auth_request con 0x7f06480a3ff0 auth_method 0 2024-09-16T04:39:04.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0646ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:04.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0646ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:04.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:04.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f06477fe640 1 --1- 172.21.15.46:0/1396712181 >> v1:172.21.15.46:6789/0 conn(0x7f06480a3c10 0x7f064809bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35296/0 (socket says 172.21.15.46:35296) 2024-09-16T04:39:04.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0647fff640 10 monclient(hunting): get_auth_request con 0x7f064813be90 auth_method 0 2024-09-16T04:39:04.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0647fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:04.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0647fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:04.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0647fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:04.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0646ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:04.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0646ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:04.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0646ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:04.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0647fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:04.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0647fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:04.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0647fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:04.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0646ffd640 10 monclient(hunting): handle_auth_done global_id 4374 payload 274 2024-09-16T04:39:04.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0646ffd640 1 -- 172.21.15.46:0/1396712181 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f064813be90 msgr2=0x7f064813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:04.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0646ffd640 1 --2- 172.21.15.46:0/1396712181 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f064813be90 0x7f064813e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0646ffd640 1 -- 172.21.15.46:0/1396712181 >> v1:172.21.15.46:6789/0 conn(0x7f06480a3c10 legacy=0x7f064809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:04.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0646ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:39:04.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0646ffd640 1 monclient: found mon.noname-a 2024-09-16T04:39:04.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0646ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:04.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0646ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:04.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f0646ffd640 1 -- 172.21.15.46:0/1396712181 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f064809b310 con 0x7f06480a3ff0 2024-09-16T04:39:04.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f06467fc640 1 -- 172.21.15.46:0/1396712181 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4141863754 0 0) 0x7f064809b940 con 0x7f06480a3c10 2024-09-16T04:39:04.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.562+0000 7f06467fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:39:04.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.563+0000 7f0646ffd640 1 --2- 172.21.15.46:0/1396712181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06480a3ff0 0x7f06480a43d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f063c009870 tx=0x7f063c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7838ad2dc12e36c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:04.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.563+0000 7f06467fc640 1 -- 172.21.15.46:0/1396712181 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063c037070 con 0x7f06480a3ff0 2024-09-16T04:39:04.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.563+0000 7f06467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:04.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.563+0000 7f06467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:04.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.563+0000 7f06467fc640 10 monclient: dump: 2024-09-16T04:39:04.574 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:04.575 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:04.575 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.575 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.575 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:04.575 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:04.575 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:04.575 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:04.575 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:04.575 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:04.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.563+0000 7f06467fc640 10 monclient: _finish_auth 0 2024-09-16T04:39:04.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.563+0000 7f06467fc640 10 monclient: _check_auth_tickets 2024-09-16T04:39:04.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.563+0000 7f06467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:04.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.563+0000 7f064deb7640 5 monclient: authenticate success, global_id 4374 2024-09-16T04:39:04.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.563+0000 7f06467fc640 1 -- 172.21.15.46:0/1396712181 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f063c032440 con 0x7f06480a3ff0 2024-09-16T04:39:04.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.563+0000 7f064deb7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:04.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.563+0000 7f06467fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:04.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.563+0000 7f064deb7640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:04.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.563+0000 7f06467fc640 1 -- 172.21.15.46:0/1396712181 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063c032810 con 0x7f06480a3ff0 2024-09-16T04:39:04.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.563+0000 7f064deb7640 4 set_mon_vals no callback set 2024-09-16T04:39:04.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.563+0000 7f06467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:04.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.563+0000 7f06467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:04.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.563+0000 7f06467fc640 10 monclient: dump: 2024-09-16T04:39:04.576 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:04.576 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:04.576 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.576 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.576 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:04.576 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:04.577 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:04.577 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:04.577 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:04.577 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:04.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.564+0000 7f064deb7640 10 monclient: shutdown 2024-09-16T04:39:04.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.564+0000 7f064deb7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:04.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.564+0000 7f064deb7640 1 -- 172.21.15.46:0/1396712181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06480a3ff0 msgr2=0x7f06480a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:04.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.564+0000 7f064deb7640 1 --2- 172.21.15.46:0/1396712181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06480a3ff0 0x7f06480a43d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f063c009870 tx=0x7f063c02a1c0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.564+0000 7f064deb7640 1 -- 172.21.15.46:0/1396712181 shutdown_connections 2024-09-16T04:39:04.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.564+0000 7f064deb7640 1 --2- 172.21.15.46:0/1396712181 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f064813be90 0x7f064813e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.564+0000 7f064deb7640 1 --2- 172.21.15.46:0/1396712181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06480a3ff0 0x7f06480a43d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.564+0000 7f064deb7640 1 -- 172.21.15.46:0/1396712181 >> 172.21.15.46:0/1396712181 conn(0x7f06480a6340 msgr2=0x7f064806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:04.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.564+0000 7f064deb7640 1 -- 172.21.15.46:0/1396712181 shutdown_connections 2024-09-16T04:39:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.564+0000 7f064deb7640 1 -- 172.21.15.46:0/1396712181 wait complete. 2024-09-16T04:39:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.564+0000 7f064deb7640 10 monclient: build_initial_monmap 2024-09-16T04:39:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.564+0000 7f064deb7640 10 monclient: monmap: 2024-09-16T04:39:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:04.578 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.565+0000 7f064deb7640 1 Processor -- start 2024-09-16T04:39:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.565+0000 7f064deb7640 1 -- start start 2024-09-16T04:39:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.565+0000 7f064deb7640 10 monclient: init 2024-09-16T04:39:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.565+0000 7f064deb7640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.565+0000 7f064deb7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:39:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.565+0000 7f064deb7640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f06480a3c10 0x7f06481cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.565+0000 7f064deb7640 10 monclient(hunting): picked mon.noname-c con 0x7f06480a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.565+0000 7f064deb7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06480a3ff0 0x7f0648136ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.565+0000 7f064deb7640 10 monclient(hunting): picked mon.noname-a con 0x7f06480a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.565+0000 7f064deb7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f064813be90 0x7f064809e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.565+0000 7f064deb7640 10 monclient(hunting): picked mon.noname-b con 0x7f064813be90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.565+0000 7f064deb7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.565+0000 7f064deb7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f06480a6b40 con 0x7f06480a3ff0 2024-09-16T04:39:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.565+0000 7f064deb7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.565+0000 7f064deb7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f06480a69c0 con 0x7f064813be90 2024-09-16T04:39:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.565+0000 7f064deb7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.565+0000 7f064deb7640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f064813e7c0 con 0x7f06480a3c10 2024-09-16T04:39:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.565+0000 7f064deb7640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.565+0000 7f064deb7640 10 monclient(hunting): authenticate will time out at 742.980933s 2024-09-16T04:39:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.565+0000 7f06477fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f06480a3c10 0x7f06481cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f06477fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f06480a3c10 0x7f06481cb740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:36400/0 (socket says 172.21.15.46:36400) 2024-09-16T04:39:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f06477fe640 1 -- 172.21.15.46:0/438414670 learned_addr learned my addr 172.21.15.46:0/438414670 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f06477fe640 10 monclient(hunting): get_auth_request con 0x7f06480a3c10 auth_method 0 2024-09-16T04:39:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f06477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f06477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f06477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f0646ffd640 1 --2- 172.21.15.46:0/438414670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06480a3ff0 0x7f0648136ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f0646ffd640 10 monclient(hunting): get_auth_request con 0x7f06480a3ff0 auth_method 0 2024-09-16T04:39:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f0646ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:04.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f0646ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:04.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f0646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:04.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f06477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:04.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f06477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:04.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f06477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:04.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f0646ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:04.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f0646ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:04.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f0646ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:04.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f06477fe640 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2024-09-16T04:39:04.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f0647fff640 1 --2- 172.21.15.46:0/438414670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f064813be90 0x7f064809e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:04.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f06477fe640 1 -- 172.21.15.46:0/438414670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f064813be90 msgr2=0x7f064809e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:04.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f06477fe640 1 --2- 172.21.15.46:0/438414670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f064813be90 0x7f064809e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f06477fe640 1 -- 172.21.15.46:0/438414670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06480a3ff0 msgr2=0x7f0648136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:04.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f06477fe640 1 --2- 172.21.15.46:0/438414670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06480a3ff0 0x7f0648136ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f06477fe640 10 monclient: _finish_hunting 0 2024-09-16T04:39:04.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f06477fe640 1 monclient: found mon.noname-c 2024-09-16T04:39:04.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f06477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:04.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f06477fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:39:04.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f06477fe640 1 -- 172.21.15.46:0/438414670 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f064813c400 con 0x7f06480a3c10 2024-09-16T04:39:04.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f0646ffd640 1 --2- 172.21.15.46:0/438414670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06480a3ff0 0x7f0648136ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:04.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f06477fe640 1 --2- 172.21.15.46:0/438414670 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f06480a3c10 0x7f06481cb740 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f063800c500 tx=0x7f063800c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:04.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f062ffff640 1 -- 172.21.15.46:0/438414670 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0638018070 con 0x7f06480a3c10 2024-09-16T04:39:04.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f062ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:04.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f062ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:39:04.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f062ffff640 10 monclient: dump: 2024-09-16T04:39:04.582 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:04.582 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:04.583 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.583 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.583 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:04.583 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:04.583 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:04.583 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:04.583 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:04.583 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:04.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f062ffff640 10 monclient: _finish_auth 0 2024-09-16T04:39:04.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f062ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:39:04.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f062ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:04.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f062ffff640 1 -- 172.21.15.46:0/438414670 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0638007580 con 0x7f06480a3c10 2024-09-16T04:39:04.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f062ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:04.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f062ffff640 1 -- 172.21.15.46:0/438414670 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0638004a20 con 0x7f06480a3c10 2024-09-16T04:39:04.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f064deb7640 5 monclient: authenticate success, global_id 4328 2024-09-16T04:39:04.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f064deb7640 10 monclient: _renew_subs 2024-09-16T04:39:04.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f064deb7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:04.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f064deb7640 1 -- 172.21.15.46:0/438414670 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06480a8be0 con 0x7f06480a3c10 2024-09-16T04:39:04.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f062ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:04.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f062ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:04.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f062ffff640 10 monclient: dump: 2024-09-16T04:39:04.584 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:04.584 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:04.584 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.584 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.584 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:04.584 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:04.584 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:04.585 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:04.585 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:04.585 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:04.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f0645ffb640 4 set_mon_vals no callback set 2024-09-16T04:39:04.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f064deb7640 10 monclient: _renew_subs 2024-09-16T04:39:04.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f064deb7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:04.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.566+0000 7f064deb7640 1 -- 172.21.15.46:0/438414670 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f064809fda0 con 0x7f06480a3c10 2024-09-16T04:39:04.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.567+0000 7f062ffff640 1 -- 172.21.15.46:0/438414670 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f063801b070 con 0x7f06480a3c10 2024-09-16T04:39:04.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.567+0000 7f064deb7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:04.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.567+0000 7f064deb7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:04.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.567+0000 7f064deb7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:04.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.567+0000 7f064deb7640 1 -- 172.21.15.46:0/438414670 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06480a43d0 con 0x7f06480a3c10 2024-09-16T04:39:04.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.567+0000 7f062ffff640 1 --2- 172.21.15.46:0/438414670 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f06340383b0 0x7f063403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:04.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.567+0000 7f062ffff640 1 -- 172.21.15.46:0/438414670 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f063804a9e0 con 0x7f06480a3c10 2024-09-16T04:39:04.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.568+0000 7f0646ffd640 1 --2- 172.21.15.46:0/438414670 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f06340383b0 0x7f063403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:04.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.568+0000 7f0646ffd640 10 monclient: get_auth_request con 0x7f06340383b0 auth_method 0 2024-09-16T04:39:04.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.568+0000 7f0646ffd640 1 --2- 172.21.15.46:0/438414670 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f06340383b0 0x7f063403a870 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f063c034000 tx=0x7f063c02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:04.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.572+0000 7f062ffff640 1 -- 172.21.15.46:0/438414670 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f063801ae10 con 0x7f06480a3c10 2024-09-16T04:39:04.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.572+0000 7f062ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:04.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.572+0000 7f062ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:04.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.785+0000 7f064deb7640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-16T04:39:04.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.785+0000 7f064deb7640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-16T04:39:04.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.785+0000 7f064deb7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:04.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.785+0000 7f064deb7640 1 -- 172.21.15.46:0/438414670 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f06480008d0 con 0x7f06480a3c10 2024-09-16T04:39:04.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.785+0000 7f062ffff640 1 -- 172.21.15.46:0/438414670 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f0638010900 con 0x7f06480a3c10 2024-09-16T04:39:04.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.785+0000 7f062ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-16T04:39:04.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.785+0000 7f062ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:04.787 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:39:04.787 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T04:39:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.787+0000 7f064deb7640 1 -- 172.21.15.46:0/438414670 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f06340383b0 msgr2=0x7f063403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.787+0000 7f064deb7640 1 --2- 172.21.15.46:0/438414670 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f06340383b0 0x7f063403a870 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f063c034000 tx=0x7f063c02e000 comp rx=0 tx=0).stop 2024-09-16T04:39:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.787+0000 7f064deb7640 10 monclient: shutdown 2024-09-16T04:39:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.787+0000 7f064deb7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.787+0000 7f064deb7640 1 -- 172.21.15.46:0/438414670 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f06480a3c10 msgr2=0x7f06481cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.787+0000 7f064deb7640 1 --2- 172.21.15.46:0/438414670 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f06480a3c10 0x7f06481cb740 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f063800c500 tx=0x7f063800c9c0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.787+0000 7f064deb7640 1 -- 172.21.15.46:0/438414670 shutdown_connections 2024-09-16T04:39:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.787+0000 7f064deb7640 1 --2- 172.21.15.46:0/438414670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f064813be90 0x7f064809e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.787+0000 7f064deb7640 1 --2- 172.21.15.46:0/438414670 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f06340383b0 0x7f063403a870 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.787+0000 7f064deb7640 1 --2- 172.21.15.46:0/438414670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06480a3ff0 0x7f0648136ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.787+0000 7f064deb7640 1 --2- 172.21.15.46:0/438414670 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f06480a3c10 0x7f06481cb740 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.787+0000 7f064deb7640 1 -- 172.21.15.46:0/438414670 >> 172.21.15.46:0/438414670 conn(0x7f06480a6340 msgr2=0x7f06480a2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:04.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.787+0000 7f064deb7640 1 -- 172.21.15.46:0/438414670 shutdown_connections 2024-09-16T04:39:04.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.787+0000 7f064deb7640 1 -- 172.21.15.46:0/438414670 wait complete. 2024-09-16T04:39:04.799 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T04:39:04.799 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-16T04:39:04.813 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-16T04:39:04.813 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:39:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.913+0000 7f0925952640 10 monclient: get_monmap_and_config 2024-09-16T04:39:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.913+0000 7f0925952640 10 monclient: build_initial_monmap 2024-09-16T04:39:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.913+0000 7f0925952640 10 monclient: monmap: 2024-09-16T04:39:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:04.913749+0000 2024-09-16T04:39:04.915 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:04.913749+0000 2024-09-16T04:39:04.915 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:04.915 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:04.915 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:04.915 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:04.915 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:04.915 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:04.915 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:04.915 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:04.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.913+0000 7f0925952640 1 Processor -- start 2024-09-16T04:39:04.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.913+0000 7f0925952640 1 -- start start 2024-09-16T04:39:04.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.913+0000 7f0925952640 10 monclient: init 2024-09-16T04:39:04.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f0925952640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f0925952640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T04:39:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f0925952640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09200a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f0925952640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f09200a3ff0 0x7f09200a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f0925952640 10 monclient(hunting): picked mon.noname-b con 0x7f09200a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f0925952640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f092013be90 0x7f092013e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f0925952640 10 monclient(hunting): picked mon.noname-c con 0x7f092013be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f0925952640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f092009ab70 con 0x7f09200a3c10 2024-09-16T04:39:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f0925952640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f0925952640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f09200a4d60 con 0x7f092013be90 2024-09-16T04:39:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f0925952640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f0925952640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f092009b030 con 0x7f09200a3ff0 2024-09-16T04:39:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f0925952640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f0925952640 10 monclient(hunting): authenticate will time out at 743.329449s 2024-09-16T04:39:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091e7fc640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f09200a3ff0 0x7f09200a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091effd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f09200a3c10 0x7f092009bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35306/0 (socket says 172.21.15.46:35306) 2024-09-16T04:39:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091effd640 1 -- 172.21.15.46:0/3800215511 learned_addr learned my addr 172.21.15.46:0/3800215511 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091e7fc640 10 monclient(hunting): get_auth_request con 0x7f09200a3ff0 auth_method 0 2024-09-16T04:39:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091e7fc640 10 monclient(hunting): handle_auth_done global_id 4331 payload 274 2024-09-16T04:39:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091f7fe640 1 --2- 172.21.15.46:0/3800215511 >> v2:172.21.15.114:3300/0 conn(0x7f092013be90 0x7f092013e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091e7fc640 1 -- 172.21.15.46:0/3800215511 >> v2:172.21.15.114:3300/0 conn(0x7f092013be90 msgr2=0x7f092013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091dffb640 1 -- 172.21.15.46:0/3800215511 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3665867652 0 0) 0x7f092009ab70 con 0x7f09200a3c10 2024-09-16T04:39:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091e7fc640 1 --2- 172.21.15.46:0/3800215511 >> v2:172.21.15.114:3300/0 conn(0x7f092013be90 0x7f092013e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091e7fc640 1 -- 172.21.15.46:0/3800215511 >> v1:172.21.15.46:6789/0 conn(0x7f09200a3c10 legacy=0x7f092009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:39:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091e7fc640 1 monclient: found mon.noname-b 2024-09-16T04:39:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:39:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091e7fc640 1 -- 172.21.15.46:0/3800215511 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f092009b310 con 0x7f09200a3ff0 2024-09-16T04:39:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.914+0000 7f091dffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:39:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.915+0000 7f091e7fc640 1 --2- 172.21.15.46:0/3800215511 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f09200a3ff0 0x7f09200a43d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f09140097a0 tx=0x7f091402a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebd4efc8a244ad7f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.915+0000 7f091dffb640 1 -- 172.21.15.46:0/3800215511 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0914037070 con 0x7f09200a3ff0 2024-09-16T04:39:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.915+0000 7f091dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.915+0000 7f091dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.915+0000 7f091dffb640 10 monclient: dump: 2024-09-16T04:39:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:04.927 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.915+0000 7f091dffb640 10 monclient: _finish_auth 0 2024-09-16T04:39:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.915+0000 7f091dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:39:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.915+0000 7f091dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.915+0000 7f0925952640 5 monclient: authenticate success, global_id 4331 2024-09-16T04:39:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.915+0000 7f0925952640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.915+0000 7f091dffb640 1 -- 172.21.15.46:0/3800215511 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f091402adb0 con 0x7f09200a3ff0 2024-09-16T04:39:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.915+0000 7f091dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.915+0000 7f091dffb640 1 -- 172.21.15.46:0/3800215511 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0914032660 con 0x7f09200a3ff0 2024-09-16T04:39:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.915+0000 7f0925952640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.915+0000 7f0925952640 4 set_mon_vals no callback set 2024-09-16T04:39:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.915+0000 7f091dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.915+0000 7f091dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.915+0000 7f091dffb640 10 monclient: dump: 2024-09-16T04:39:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:04.928 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.916+0000 7f0925952640 10 monclient: shutdown 2024-09-16T04:39:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.916+0000 7f0925952640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.916+0000 7f0925952640 1 -- 172.21.15.46:0/3800215511 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f09200a3ff0 msgr2=0x7f09200a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.916+0000 7f0925952640 1 --2- 172.21.15.46:0/3800215511 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f09200a3ff0 0x7f09200a43d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f09140097a0 tx=0x7f091402a060 comp rx=0 tx=0).stop 2024-09-16T04:39:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.916+0000 7f0925952640 1 -- 172.21.15.46:0/3800215511 shutdown_connections 2024-09-16T04:39:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.916+0000 7f0925952640 1 --2- 172.21.15.46:0/3800215511 >> v2:172.21.15.114:3300/0 conn(0x7f092013be90 0x7f092013e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.916+0000 7f0925952640 1 --2- 172.21.15.46:0/3800215511 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f09200a3ff0 0x7f09200a43d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.916+0000 7f0925952640 1 -- 172.21.15.46:0/3800215511 >> 172.21.15.46:0/3800215511 conn(0x7f09200a5d10 msgr2=0x7f09200a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.916+0000 7f0925952640 1 -- 172.21.15.46:0/3800215511 shutdown_connections 2024-09-16T04:39:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.916+0000 7f0925952640 1 -- 172.21.15.46:0/3800215511 wait complete. 2024-09-16T04:39:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.916+0000 7f0925952640 10 monclient: build_initial_monmap 2024-09-16T04:39:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.916+0000 7f0925952640 10 monclient: monmap: 2024-09-16T04:39:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:04.930 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.917+0000 7f0925952640 1 Processor -- start 2024-09-16T04:39:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.917+0000 7f0925952640 1 -- start start 2024-09-16T04:39:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.917+0000 7f0925952640 10 monclient: init 2024-09-16T04:39:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.917+0000 7f0925952640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.917+0000 7f0925952640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:39:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.917+0000 7f0925952640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09200a3c10 0x7f09201cb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.917+0000 7f0925952640 10 monclient(hunting): picked mon.noname-a con 0x7f09200a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.917+0000 7f0925952640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09200a3ff0 0x7f09201cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.917+0000 7f0925952640 10 monclient(hunting): picked mon.noname-b con 0x7f09200a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.917+0000 7f0925952640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f092013be90 0x7f09201cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.917+0000 7f0925952640 10 monclient(hunting): picked mon.noname-c con 0x7f092013be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.917+0000 7f0925952640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.917+0000 7f0925952640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f092006e550 con 0x7f09200a3c10 2024-09-16T04:39:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.917+0000 7f0925952640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.917+0000 7f0925952640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f092006e3d0 con 0x7f09200a3ff0 2024-09-16T04:39:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0925952640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0925952640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f092013e7c0 con 0x7f092013be90 2024-09-16T04:39:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0925952640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0925952640 10 monclient(hunting): authenticate will time out at 743.333012s 2024-09-16T04:39:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09200a3c10 0x7f09201cb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091f7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f092013be90 0x7f09201cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09200a3c10 0x7f09201cb840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34850/0 (socket says 172.21.15.46:34850) 2024-09-16T04:39:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091f7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f092013be90 0x7f09201cc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:36422/0 (socket says 172.21.15.46:36422) 2024-09-16T04:39:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091effd640 1 -- 172.21.15.46:0/1143436197 learned_addr learned my addr 172.21.15.46:0/1143436197 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091effd640 10 monclient(hunting): get_auth_request con 0x7f09200a3c10 auth_method 0 2024-09-16T04:39:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091f7fe640 10 monclient(hunting): get_auth_request con 0x7f092013be90 auth_method 0 2024-09-16T04:39:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091e7fc640 1 --2- 172.21.15.46:0/1143436197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09200a3ff0 0x7f09201cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091e7fc640 10 monclient(hunting): get_auth_request con 0x7f09200a3ff0 auth_method 0 2024-09-16T04:39:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091effd640 10 monclient(hunting): handle_auth_done global_id 4386 payload 824 2024-09-16T04:39:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091effd640 1 -- 172.21.15.46:0/1143436197 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f092013be90 msgr2=0x7f09201cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091effd640 1 --2- 172.21.15.46:0/1143436197 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f092013be90 0x7f09201cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091effd640 1 -- 172.21.15.46:0/1143436197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09200a3ff0 msgr2=0x7f09201cbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091effd640 1 --2- 172.21.15.46:0/1143436197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09200a3ff0 0x7f09201cbd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091effd640 10 monclient: _finish_hunting 0 2024-09-16T04:39:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091effd640 1 monclient: found mon.noname-a 2024-09-16T04:39:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091effd640 1 -- 172.21.15.46:0/1143436197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0920139800 con 0x7f09200a3c10 2024-09-16T04:39:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091f7fe640 1 --2- 172.21.15.46:0/1143436197 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f092013be90 0x7f09201cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091effd640 1 --2- 172.21.15.46:0/1143436197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09200a3c10 0x7f09201cb840 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f091000c890 tx=0x7f091000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0907fff640 1 -- 172.21.15.46:0/1143436197 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0910018070 con 0x7f09200a3c10 2024-09-16T04:39:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0907fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0907fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0907fff640 10 monclient: dump: 2024-09-16T04:39:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:04.936 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0907fff640 10 monclient: _finish_auth 0 2024-09-16T04:39:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0907fff640 10 monclient: _check_auth_tickets 2024-09-16T04:39:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0907fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0925952640 5 monclient: authenticate success, global_id 4386 2024-09-16T04:39:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0907fff640 1 -- 172.21.15.46:0/1143436197 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f091000a640 con 0x7f09200a3c10 2024-09-16T04:39:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0925952640 10 monclient: _renew_subs 2024-09-16T04:39:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0925952640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0925952640 1 -- 172.21.15.46:0/1143436197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09201385c0 con 0x7f09200a3c10 2024-09-16T04:39:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0907fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0907fff640 1 -- 172.21.15.46:0/1143436197 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091000aa30 con 0x7f09200a3c10 2024-09-16T04:39:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0907fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0907fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0907fff640 10 monclient: dump: 2024-09-16T04:39:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:04.937 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0925952640 10 monclient: _renew_subs 2024-09-16T04:39:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f091d7fa640 4 set_mon_vals no callback set 2024-09-16T04:39:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0925952640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.918+0000 7f0925952640 1 -- 172.21.15.46:0/1143436197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f092013c7c0 con 0x7f09200a3c10 2024-09-16T04:39:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.919+0000 7f0925952640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.919+0000 7f0925952640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.919+0000 7f0925952640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.919+0000 7f0907fff640 1 -- 172.21.15.46:0/1143436197 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f091001b070 con 0x7f09200a3c10 2024-09-16T04:39:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.919+0000 7f0925952640 1 -- 172.21.15.46:0/1143436197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09200a43d0 con 0x7f09200a3c10 2024-09-16T04:39:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.920+0000 7f0907fff640 1 --2- 172.21.15.46:0/1143436197 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f090c0383b0 0x7f090c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.920+0000 7f0907fff640 1 -- 172.21.15.46:0/1143436197 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f091004ac90 con 0x7f09200a3c10 2024-09-16T04:39:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.920+0000 7f091e7fc640 1 --2- 172.21.15.46:0/1143436197 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f090c0383b0 0x7f090c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.920+0000 7f091e7fc640 10 monclient: get_auth_request con 0x7f090c0383b0 auth_method 0 2024-09-16T04:39:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.920+0000 7f091e7fc640 1 --2- 172.21.15.46:0/1143436197 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f090c0383b0 0x7f090c03a870 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0914034000 tx=0x7f091402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:04.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.924+0000 7f0907fff640 1 -- 172.21.15.46:0/1143436197 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0910010300 con 0x7f09200a3c10 2024-09-16T04:39:04.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.924+0000 7f0907fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:04.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:04.924+0000 7f0907fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:05.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.100+0000 7f0925952640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:39:05.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.100+0000 7f0925952640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:39:05.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.100+0000 7f0925952640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:05.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.100+0000 7f0925952640 1 -- 172.21.15.46:0/1143436197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0920067e40 con 0x7f09200a3c10 2024-09-16T04:39:05.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.101+0000 7f0907fff640 1 -- 172.21.15.46:0/1143436197 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+12334 (secure 0 0 0) 0x7f0910010e20 con 0x7f09200a3c10 2024-09-16T04:39:05.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.101+0000 7f0907fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:39:05.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.101+0000 7f0907fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:05.102 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:39:05.102 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","created":"2024-09-16T04:38:39.286708+0000","modified":"2024-09-16T04:39:04.067933+0000","last_up_change":"2024-09-16T04:38:50.487347+0000","last_in_change":"2024-09-16T04:38:42.117316+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:38:51.392177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:38:57.324878+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ff88780-0630-4bb1-9f5a-5331d7724d17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3454452571},{"type":"v1","addr":"172.21.15.46:6809","nonce":3454452571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3454452571},{"type":"v1","addr":"172.21.15.46:6811","nonce":3454452571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3454452571},{"type":"v1","addr":"172.21.15.46:6815","nonce":3454452571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3454452571},{"type":"v1","addr":"172.21.15.46:6813","nonce":3454452571}]},"public_addr":"172.21.15.46:6809/3454452571","cluster_addr":"172.21.15.46:6811/3454452571","heartbeat_back_addr":"172.21.15.46:6815/3454452571","heartbeat_front_addr":"172.21.15.46:6813/3454452571","state":["exists","up"]},{"osd":1,"uuid":"37be2e88-32d0-4a30-8979-899b34f3be17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6817","nonce":4217009477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6819","nonce":4217009477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6823","nonce":4217009477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6821","nonce":4217009477}]},"public_addr":"172.21.15.46:6817/4217009477","cluster_addr":"172.21.15.46:6819/4217009477","heartbeat_back_addr":"172.21.15.46:6823/4217009477","heartbeat_front_addr":"172.21.15.46:6821/4217009477","state":["exists","up"]},{"osd":2,"uuid":"73b9a838-944e-48cd-afab-5260608d1c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6801","nonce":3101080311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6803","nonce":3101080311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6807","nonce":3101080311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6805","nonce":3101080311}]},"public_addr":"172.21.15.46:6801/3101080311","cluster_addr":"172.21.15.46:6803/3101080311","heartbeat_back_addr":"172.21.15.46:6807/3101080311","heartbeat_front_addr":"172.21.15.46:6805/3101080311","state":["exists","up"]},{"osd":3,"uuid":"d0c22455-5685-4ade-b540-3d5428a7cb46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":890769734},{"type":"v1","addr":"172.21.15.114:6811","nonce":890769734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":890769734},{"type":"v1","addr":"172.21.15.114:6813","nonce":890769734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":890769734},{"type":"v1","addr":"172.21.15.114:6817","nonce":890769734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":890769734},{"type":"v1","addr":"172.21.15.114:6815","nonce":890769734}]},"public_addr":"172.21.15.114:6811/890769734","cluster_addr":"172.21.15.114:6813/890769734","heartbeat_back_addr":"172.21.15.114:6817/890769734","heartbeat_front_addr":"172.21.15.114:6815/890769734","state":["exists","up"]},{"osd":4,"uuid":"f37724e4-2796-43c8-975b-4458986c4ea6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":936316557},{"type":"v1","addr":"172.21.15.114:6803","nonce":936316557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":936316557},{"type":"v1","addr":"172.21.15.114:6805","nonce":936316557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":936316557},{"type":"v1","addr":"172.21.15.114:6809","nonce":936316557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":936316557},{"type":"v1","addr":"172.21.15.114:6807","nonce":936316557}]},"public_addr":"172.21.15.114:6803/936316557","cluster_addr":"172.21.15.114:6805/936316557","heartbeat_back_addr":"172.21.15.114:6809/936316557","heartbeat_front_addr":"172.21.15.114:6807/936316557","state":["exists","up"]},{"osd":5,"uuid":"979362db-e173-47a0-93d1-187742938a85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6819","nonce":938868100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6821","nonce":938868100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6825","nonce":938868100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6823","nonce":938868100}]},"public_addr":"172.21.15.114:6819/938868100","cluster_addr":"172.21.15.114:6821/938868100","heartbeat_back_addr":"172.21.15.114:6825/938868100","heartbeat_front_addr":"172.21.15.114:6823/938868100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.647737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.394775+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.463096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.655270+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.652725+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.431803+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:39:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.103+0000 7f0925952640 1 -- 172.21.15.46:0/1143436197 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f090c0383b0 msgr2=0x7f090c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.103+0000 7f0925952640 1 --2- 172.21.15.46:0/1143436197 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f090c0383b0 0x7f090c03a870 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0914034000 tx=0x7f091402e000 comp rx=0 tx=0).stop 2024-09-16T04:39:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.103+0000 7f0925952640 10 monclient: shutdown 2024-09-16T04:39:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.103+0000 7f0925952640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.103+0000 7f0925952640 1 -- 172.21.15.46:0/1143436197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09200a3c10 msgr2=0x7f09201cb840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.103+0000 7f0925952640 1 --2- 172.21.15.46:0/1143436197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09200a3c10 0x7f09201cb840 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f091000c890 tx=0x7f091000cd50 comp rx=0 tx=0).stop 2024-09-16T04:39:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.103+0000 7f0925952640 1 -- 172.21.15.46:0/1143436197 shutdown_connections 2024-09-16T04:39:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.103+0000 7f0925952640 1 --2- 172.21.15.46:0/1143436197 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f092013be90 0x7f09201cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.103+0000 7f0925952640 1 --2- 172.21.15.46:0/1143436197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09200a3ff0 0x7f09201cbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.103+0000 7f0925952640 1 --2- 172.21.15.46:0/1143436197 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f090c0383b0 0x7f090c03a870 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.103+0000 7f0925952640 1 --2- 172.21.15.46:0/1143436197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09200a3c10 0x7f09201cb840 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.103+0000 7f0925952640 1 -- 172.21.15.46:0/1143436197 >> 172.21.15.46:0/1143436197 conn(0x7f09200a5d10 msgr2=0x7f092013b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:05.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.103+0000 7f0925952640 1 -- 172.21.15.46:0/1143436197 shutdown_connections 2024-09-16T04:39:05.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.103+0000 7f0925952640 1 -- 172.21.15.46:0/1143436197 wait complete. 2024-09-16T04:39:05.115 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T04:38:51.392177+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T04:38:57.324878+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T04:39:05.116 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T04:39:05.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.218+0000 7f0030477640 10 monclient: get_monmap_and_config 2024-09-16T04:39:05.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.218+0000 7f0030477640 10 monclient: build_initial_monmap 2024-09-16T04:39:05.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.218+0000 7f0030477640 10 monclient: monmap: 2024-09-16T04:39:05.220 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:05.220 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:05.220 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:05.219088+0000 2024-09-16T04:39:05.220 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:05.219088+0000 2024-09-16T04:39:05.220 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:05.220 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:05.220 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:05.220 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:05.220 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:05.220 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:05.220 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:05.220 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:05.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.219+0000 7f0030477640 1 Processor -- start 2024-09-16T04:39:05.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.219+0000 7f0030477640 1 -- start start 2024-09-16T04:39:05.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.219+0000 7f0030477640 10 monclient: init 2024-09-16T04:39:05.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.219+0000 7f0030477640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:05.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.219+0000 7f0030477640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-16T04:39:05.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.219+0000 7f0030477640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f00280a3c10 0x7f002809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.219+0000 7f0030477640 10 monclient(hunting): picked mon.noname-c con 0x7f00280a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:05.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.219+0000 7f0030477640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f00280a3ff0 0x7f00280a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.219+0000 7f0030477640 10 monclient(hunting): picked mon.noname-b con 0x7f00280a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:05.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.219+0000 7f0030477640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f00281369d0 0x7f0028136e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.219+0000 7f0030477640 10 monclient(hunting): picked mon.noname-a con 0x7f00281369d0 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:05.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.219+0000 7f0030477640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:05.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.219+0000 7f0030477640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f002809b940 con 0x7f00281369d0 2024-09-16T04:39:05.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.219+0000 7f0030477640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:05.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.219+0000 7f0030477640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f002809ab00 con 0x7f00280a3c10 2024-09-16T04:39:05.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.219+0000 7f0030477640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:05.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.219+0000 7f0030477640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f002809b030 con 0x7f00280a3ff0 2024-09-16T04:39:05.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.219+0000 7f0030477640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:05.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.219+0000 7f0030477640 10 monclient(hunting): authenticate will time out at 743.634855s 2024-09-16T04:39:05.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.219+0000 7f002da12640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f00280a3ff0 0x7f00280a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002da12640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f00280a3ff0 0x7f00280a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:36434/0 (socket says 172.21.15.46:36434) 2024-09-16T04:39:05.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002da12640 1 -- 172.21.15.46:0/3706789131 learned_addr learned my addr 172.21.15.46:0/3706789131 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:05.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002da12640 10 monclient(hunting): get_auth_request con 0x7f00280a3ff0 auth_method 0 2024-09-16T04:39:05.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002da12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:05.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002da12640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002da12640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:05.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002ea14640 1 --2- 172.21.15.46:0/3706789131 >> v2:172.21.15.46:3300/0 conn(0x7f00281369d0 0x7f0028136e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002ea14640 10 monclient(hunting): get_auth_request con 0x7f00281369d0 auth_method 0 2024-09-16T04:39:05.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002ea14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:05.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002ea14640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002ea14640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:05.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002da12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:05.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002da12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:05.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002da12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:05.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002ea14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:05.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002ea14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:05.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002e213640 1 --2- 172.21.15.46:0/3706789131 >> v2:172.21.15.114:3300/0 conn(0x7f00280a3c10 0x7f002809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002ea14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:05.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002e213640 10 monclient(hunting): get_auth_request con 0x7f00280a3c10 auth_method 0 2024-09-16T04:39:05.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002e213640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:05.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002e213640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002e213640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:05.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002da12640 10 monclient(hunting): handle_auth_done global_id 4337 payload 274 2024-09-16T04:39:05.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002da12640 1 -- 172.21.15.46:0/3706789131 >> v2:172.21.15.114:3300/0 conn(0x7f00280a3c10 msgr2=0x7f002809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:05.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002da12640 1 --2- 172.21.15.46:0/3706789131 >> v2:172.21.15.114:3300/0 conn(0x7f00280a3c10 0x7f002809bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002da12640 1 -- 172.21.15.46:0/3706789131 >> v2:172.21.15.46:3300/0 conn(0x7f00281369d0 msgr2=0x7f0028136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:05.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002da12640 1 --2- 172.21.15.46:0/3706789131 >> v2:172.21.15.46:3300/0 conn(0x7f00281369d0 0x7f0028136e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002da12640 10 monclient: _finish_hunting 0 2024-09-16T04:39:05.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002da12640 1 monclient: found mon.noname-b 2024-09-16T04:39:05.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002da12640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:05.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002da12640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:39:05.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.220+0000 7f002da12640 1 -- 172.21.15.46:0/3706789131 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f002809b310 con 0x7f00280a3ff0 2024-09-16T04:39:05.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f002da12640 1 --2- 172.21.15.46:0/3706789131 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f00280a3ff0 0x7f00280a43d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0028068240 tx=0x7f001802a0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71a5448aebdaadff server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:05.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f002d211640 1 -- 172.21.15.46:0/3706789131 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0018037070 con 0x7f00280a3ff0 2024-09-16T04:39:05.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f002d211640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:05.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f002d211640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:05.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f002d211640 10 monclient: dump: 2024-09-16T04:39:05.232 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:05.232 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:05.232 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.232 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.232 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:05.232 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:05.232 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:05.232 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:05.232 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:05.232 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:05.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f002d211640 10 monclient: _finish_auth 0 2024-09-16T04:39:05.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f002d211640 10 monclient: _check_auth_tickets 2024-09-16T04:39:05.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f002d211640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:05.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f0030477640 5 monclient: authenticate success, global_id 4337 2024-09-16T04:39:05.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f002d211640 1 -- 172.21.15.46:0/3706789131 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f001802ae00 con 0x7f00280a3ff0 2024-09-16T04:39:05.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f0030477640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:05.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f002d211640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:05.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f0030477640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:05.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f0030477640 4 set_mon_vals no callback set 2024-09-16T04:39:05.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f002d211640 1 -- 172.21.15.46:0/3706789131 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0018032660 con 0x7f00280a3ff0 2024-09-16T04:39:05.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f002d211640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:05.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f002d211640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:05.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f002d211640 10 monclient: dump: 2024-09-16T04:39:05.233 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:05.233 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:05.233 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.234 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.234 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:05.234 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:05.234 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:05.234 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:05.234 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:05.234 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:05.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f0030477640 10 monclient: shutdown 2024-09-16T04:39:05.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f0030477640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:05.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f0030477640 1 -- 172.21.15.46:0/3706789131 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f00280a3ff0 msgr2=0x7f00280a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:05.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f0030477640 1 --2- 172.21.15.46:0/3706789131 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f00280a3ff0 0x7f00280a43d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0028068240 tx=0x7f001802a0b0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f0030477640 1 -- 172.21.15.46:0/3706789131 shutdown_connections 2024-09-16T04:39:05.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f0030477640 1 --2- 172.21.15.46:0/3706789131 >> v2:172.21.15.46:3300/0 conn(0x7f00281369d0 0x7f0028136e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f0030477640 1 --2- 172.21.15.46:0/3706789131 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f00280a3ff0 0x7f00280a43d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f0030477640 1 --2- 172.21.15.46:0/3706789131 >> v2:172.21.15.114:3300/0 conn(0x7f00280a3c10 0x7f002809bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f0030477640 1 -- 172.21.15.46:0/3706789131 >> 172.21.15.46:0/3706789131 conn(0x7f00280a6340 msgr2=0x7f002806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f0030477640 1 -- 172.21.15.46:0/3706789131 shutdown_connections 2024-09-16T04:39:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.221+0000 7f0030477640 1 -- 172.21.15.46:0/3706789131 wait complete. 2024-09-16T04:39:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.222+0000 7f0030477640 10 monclient: build_initial_monmap 2024-09-16T04:39:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.222+0000 7f0030477640 10 monclient: monmap: 2024-09-16T04:39:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:05.236 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:05.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.222+0000 7f0030477640 1 Processor -- start 2024-09-16T04:39:05.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.222+0000 7f0030477640 1 -- start start 2024-09-16T04:39:05.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.222+0000 7f0030477640 10 monclient: init 2024-09-16T04:39:05.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f0030477640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:05.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f0030477640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:39:05.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f0030477640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f00280a3c10 0x7f00281cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f0030477640 10 monclient(hunting): picked mon.noname-c con 0x7f00280a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:05.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f0030477640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00280a3ff0 0x7f00281cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f0030477640 10 monclient(hunting): picked mon.noname-b con 0x7f00280a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:05.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f0030477640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00281369d0 0x7f0028123210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f0030477640 10 monclient(hunting): picked mon.noname-a con 0x7f00281369d0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:05.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f0030477640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:05.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f0030477640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f00280a4910 con 0x7f00281369d0 2024-09-16T04:39:05.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f0030477640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:05.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f0030477640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f00280a69c0 con 0x7f00280a3ff0 2024-09-16T04:39:05.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f0030477640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:05.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f0030477640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f00281265d0 con 0x7f00280a3c10 2024-09-16T04:39:05.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f0030477640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:05.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f0030477640 10 monclient(hunting): authenticate will time out at 743.638339s 2024-09-16T04:39:05.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002e213640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f00280a3c10 0x7f00281cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002e213640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f00280a3c10 0x7f00281cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:36450/0 (socket says 172.21.15.46:36450) 2024-09-16T04:39:05.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002e213640 1 -- 172.21.15.46:0/2798621478 learned_addr learned my addr 172.21.15.46:0/2798621478 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:05.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002e213640 10 monclient(hunting): get_auth_request con 0x7f00280a3c10 auth_method 0 2024-09-16T04:39:05.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002ea14640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00281369d0 0x7f0028123210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002e213640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:05.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002e213640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002e213640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:05.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002ea14640 10 monclient(hunting): get_auth_request con 0x7f00281369d0 auth_method 0 2024-09-16T04:39:05.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002ea14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:05.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002ea14640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002ea14640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:05.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002e213640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:05.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002e213640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:05.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002e213640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:05.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002ea14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:05.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002ea14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:05.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002ea14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:05.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002da12640 1 --2- 172.21.15.46:0/2798621478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00280a3ff0 0x7f00281cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002da12640 10 monclient(hunting): get_auth_request con 0x7f00280a3ff0 auth_method 0 2024-09-16T04:39:05.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002da12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:05.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002da12640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002da12640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002da12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002da12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002da12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002ea14640 10 monclient(hunting): handle_auth_done global_id 4392 payload 824 2024-09-16T04:39:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002ea14640 1 -- 172.21.15.46:0/2798621478 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f00280a3c10 msgr2=0x7f00281cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002ea14640 1 --2- 172.21.15.46:0/2798621478 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f00280a3c10 0x7f00281cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002ea14640 1 -- 172.21.15.46:0/2798621478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00280a3ff0 msgr2=0x7f00281cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002ea14640 1 --2- 172.21.15.46:0/2798621478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00280a3ff0 0x7f00281cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002ea14640 10 monclient: _finish_hunting 0 2024-09-16T04:39:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002ea14640 1 monclient: found mon.noname-a 2024-09-16T04:39:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002ea14640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002ea14640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002ea14640 1 -- 172.21.15.46:0/2798621478 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0028125a10 con 0x7f00281369d0 2024-09-16T04:39:05.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002e213640 1 --2- 172.21.15.46:0/2798621478 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f00280a3c10 0x7f00281cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:05.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.223+0000 7f002ea14640 1 --2- 172.21.15.46:0/2798621478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00281369d0 0x7f0028123210 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f002000efc0 tx=0x7f002000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:05.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f001effd640 1 -- 172.21.15.46:0/2798621478 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020018020 con 0x7f00281369d0 2024-09-16T04:39:05.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f001effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:05.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f001effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:05.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f001effd640 10 monclient: dump: 2024-09-16T04:39:05.240 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:05.240 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:05.240 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.240 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.240 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:05.240 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:05.240 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:05.240 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:05.241 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f001effd640 10 monclient: _finish_auth 0 2024-09-16T04:39:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f001effd640 10 monclient: _check_auth_tickets 2024-09-16T04:39:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f001effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f0030477640 5 monclient: authenticate success, global_id 4392 2024-09-16T04:39:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f001effd640 1 -- 172.21.15.46:0/2798621478 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0020014480 con 0x7f00281369d0 2024-09-16T04:39:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f0030477640 10 monclient: _renew_subs 2024-09-16T04:39:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f0030477640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f0030477640 1 -- 172.21.15.46:0/2798621478 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00281271a0 con 0x7f00281369d0 2024-09-16T04:39:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f001effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f001effd640 1 -- 172.21.15.46:0/2798621478 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020009b10 con 0x7f00281369d0 2024-09-16T04:39:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f001effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f001effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f001effd640 10 monclient: dump: 2024-09-16T04:39:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:05.242 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f002ca10640 4 set_mon_vals no callback set 2024-09-16T04:39:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f0030477640 10 monclient: _renew_subs 2024-09-16T04:39:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f0030477640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:05.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f0030477640 1 -- 172.21.15.46:0/2798621478 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0028126be0 con 0x7f00281369d0 2024-09-16T04:39:05.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.224+0000 7f001effd640 1 -- 172.21.15.46:0/2798621478 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f0020005d10 con 0x7f00281369d0 2024-09-16T04:39:05.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.225+0000 7f0030477640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:05.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.225+0000 7f0030477640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:05.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.225+0000 7f0030477640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:05.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.225+0000 7f001effd640 1 --2- 172.21.15.46:0/2798621478 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f000c0383e0 0x7f000c03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.225+0000 7f0030477640 1 -- 172.21.15.46:0/2798621478 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00280a5730 con 0x7f00281369d0 2024-09-16T04:39:05.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.225+0000 7f001effd640 1 -- 172.21.15.46:0/2798621478 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f002004c270 con 0x7f00281369d0 2024-09-16T04:39:05.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.225+0000 7f002e213640 1 --2- 172.21.15.46:0/2798621478 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f000c0383e0 0x7f000c03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.225+0000 7f002e213640 10 monclient: get_auth_request con 0x7f000c0383e0 auth_method 0 2024-09-16T04:39:05.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.225+0000 7f002e213640 1 --2- 172.21.15.46:0/2798621478 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f000c0383e0 0x7f000c03a8a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0024006fd0 tx=0x7f0024008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:05.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.229+0000 7f001effd640 1 -- 172.21.15.46:0/2798621478 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f00280a5730 con 0x7f00281369d0 2024-09-16T04:39:05.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.229+0000 7f001effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:05.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.229+0000 7f001effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:05.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.405+0000 7f0030477640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T04:39:05.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.405+0000 7f0030477640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T04:39:05.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.405+0000 7f0030477640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:05.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.405+0000 7f0030477640 1 -- 172.21.15.46:0/2798621478 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f00280a43d0 con 0x7f00281369d0 2024-09-16T04:39:05.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.405+0000 7f001effd640 1 -- 172.21.15.46:0/2798621478 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f0020022030 con 0x7f00281369d0 2024-09-16T04:39:05.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.405+0000 7f001effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T04:39:05.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.405+0000 7f001effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:05.407 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-09-16T04:39:05.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.407+0000 7f0030477640 1 -- 172.21.15.46:0/2798621478 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f000c0383e0 msgr2=0x7f000c03a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:05.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.407+0000 7f0030477640 1 --2- 172.21.15.46:0/2798621478 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f000c0383e0 0x7f000c03a8a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0024006fd0 tx=0x7f0024008040 comp rx=0 tx=0).stop 2024-09-16T04:39:05.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.407+0000 7f0030477640 10 monclient: shutdown 2024-09-16T04:39:05.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.407+0000 7f0030477640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:05.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.407+0000 7f0030477640 1 -- 172.21.15.46:0/2798621478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00281369d0 msgr2=0x7f0028123210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:05.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.407+0000 7f0030477640 1 --2- 172.21.15.46:0/2798621478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00281369d0 0x7f0028123210 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f002000efc0 tx=0x7f002000c490 comp rx=0 tx=0).stop 2024-09-16T04:39:05.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.407+0000 7f0030477640 1 -- 172.21.15.46:0/2798621478 shutdown_connections 2024-09-16T04:39:05.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.407+0000 7f0030477640 1 --2- 172.21.15.46:0/2798621478 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f000c0383e0 0x7f000c03a8a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.407+0000 7f0030477640 1 --2- 172.21.15.46:0/2798621478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00281369d0 0x7f0028123210 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.407+0000 7f0030477640 1 --2- 172.21.15.46:0/2798621478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00280a3ff0 0x7f00281cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.407+0000 7f0030477640 1 --2- 172.21.15.46:0/2798621478 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f00280a3c10 0x7f00281cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.407+0000 7f0030477640 1 -- 172.21.15.46:0/2798621478 >> 172.21.15.46:0/2798621478 conn(0x7f00280a6340 msgr2=0x7f00280a1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:05.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.407+0000 7f0030477640 1 -- 172.21.15.46:0/2798621478 shutdown_connections 2024-09-16T04:39:05.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.407+0000 7f0030477640 1 -- 172.21.15.46:0/2798621478 wait complete. 2024-09-16T04:39:05.419 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-16T04:39:05.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.522+0000 7f09d6c77640 10 monclient: get_monmap_and_config 2024-09-16T04:39:05.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.522+0000 7f09d6c77640 10 monclient: build_initial_monmap 2024-09-16T04:39:05.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.522+0000 7f09d6c77640 10 monclient: monmap: 2024-09-16T04:39:05.524 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:05.524 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:05.524 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:05.523275+0000 2024-09-16T04:39:05.524 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:05.523275+0000 2024-09-16T04:39:05.524 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:05.524 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:05.524 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:05.524 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:05.524 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:05.525 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:05.525 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:05.525 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:05.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.523+0000 7f09d6c77640 1 Processor -- start 2024-09-16T04:39:05.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.523+0000 7f09d6c77640 1 -- start start 2024-09-16T04:39:05.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.523+0000 7f09d6c77640 10 monclient: init 2024-09-16T04:39:05.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09d6c77640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:05.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09d6c77640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-16T04:39:05.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09d6c77640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09d00a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09d6c77640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f09d00a3ff0 0x7f09d00a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09d6c77640 10 monclient(hunting): picked mon.noname-c con 0x7f09d00a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09d6c77640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09d013be90 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09d6c77640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09d009b940 con 0x7f09d013be90 2024-09-16T04:39:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09d6c77640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09d009ab00 con 0x7f09d00a3c10 2024-09-16T04:39:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09d6c77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09d6c77640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f09d009b030 con 0x7f09d00a3ff0 2024-09-16T04:39:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09d6c77640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09d6c77640 10 monclient(hunting): authenticate will time out at 743.939153s 2024-09-16T04:39:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09d5214640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f09d013be90 0x7f09d013e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35312/0 (socket says 172.21.15.46:35312) 2024-09-16T04:39:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09d5214640 1 -- 172.21.15.46:0/3290521461 learned_addr learned my addr 172.21.15.46:0/3290521461 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cffff640 1 --2- 172.21.15.46:0/3290521461 >> v2:172.21.15.114:3300/0 conn(0x7f09d00a3ff0 0x7f09d00a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cffff640 10 monclient(hunting): get_auth_request con 0x7f09d00a3ff0 auth_method 0 2024-09-16T04:39:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 1 -- 172.21.15.46:0/3290521461 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3671378350 0 0) 0x7f09d009b940 con 0x7f09d013be90 2024-09-16T04:39:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:39:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 10 monclient(hunting): my global_id is 4395 2024-09-16T04:39:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 1 -- 172.21.15.46:0/3290521461 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09b8003520 con 0x7f09d013be90 2024-09-16T04:39:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:05.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 1 -- 172.21.15.46:0/3290521461 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2236709984 0 0) 0x7f09b8003520 con 0x7f09d013be90 2024-09-16T04:39:05.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 1 -- 172.21.15.46:0/3290521461 >> v2:172.21.15.114:3300/0 conn(0x7f09d00a3ff0 msgr2=0x7f09d00a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:05.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 1 --2- 172.21.15.46:0/3290521461 >> v2:172.21.15.114:3300/0 conn(0x7f09d00a3ff0 0x7f09d00a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 1 -- 172.21.15.46:0/3290521461 >> v1:172.21.15.114:6789/0 conn(0x7f09d00a3c10 legacy=0x7f09d009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:05.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:39:05.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:39:05.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:05.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:39:05.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 1 -- 172.21.15.46:0/3290521461 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09d009b310 con 0x7f09d013be90 2024-09-16T04:39:05.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 10 monclient: _finish_auth 0 2024-09-16T04:39:05.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:39:05.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:05.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 1 -- 172.21.15.46:0/3290521461 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f09c80019f0 con 0x7f09d013be90 2024-09-16T04:39:05.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09d6c77640 5 monclient: authenticate success, global_id 4395 2024-09-16T04:39:05.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09d6c77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:05.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:05.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:39:05.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 10 monclient: dump: 2024-09-16T04:39:05.540 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:05.540 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:05.540 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.540 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.540 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:05.540 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:05.540 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:05.540 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:05.540 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:05.540 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:05.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:39:05.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:05.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 1 -- 172.21.15.46:0/3290521461 >> v1:172.21.15.46:6789/0 conn(0x7f09d013be90 legacy=0x7f09d013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:05.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:39:05.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 1 --2- 172.21.15.46:0/3290521461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09b80039b0 0x7f09b8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 10 monclient(hunting): picked mon.a con 0x7f09b80039b0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:05.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 1 --2- 172.21.15.46:0/3290521461 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f09b8004800 0x7f09b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 10 monclient(hunting): picked mon.c con 0x7f09b8004800 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:05.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 1 --2- 172.21.15.46:0/3290521461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09b80051c0 0x7f09b800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 10 monclient(hunting): picked mon.b con 0x7f09b80051c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:05.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:05.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 1 -- 172.21.15.46:0/3290521461 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f09d009b030 con 0x7f09b80039b0 2024-09-16T04:39:05.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:05.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 1 -- 172.21.15.46:0/3290521461 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f09b800c160 con 0x7f09b80051c0 2024-09-16T04:39:05.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:05.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 1 -- 172.21.15.46:0/3290521461 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f09b800c360 con 0x7f09b8004800 2024-09-16T04:39:05.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:05.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09cf7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:39:05.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09d6c77640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:05.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.524+0000 7f09d4a13640 1 --2- 172.21.15.46:0/3290521461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09b80039b0 0x7f09b8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d4a13640 10 monclient(hunting): get_auth_request con 0x7f09b80039b0 auth_method 0 2024-09-16T04:39:05.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d4a13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:05.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d4a13640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d4a13640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:05.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09cffff640 1 --2- 172.21.15.46:0/3290521461 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f09b8004800 0x7f09b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09cffff640 10 monclient(hunting): get_auth_request con 0x7f09b8004800 auth_method 0 2024-09-16T04:39:05.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:05.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09cffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:05.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d5214640 1 --2- 172.21.15.46:0/3290521461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09b80051c0 0x7f09b800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d4a13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:05.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d4a13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:05.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d4a13640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:05.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d5214640 10 monclient(hunting): get_auth_request con 0x7f09b80051c0 auth_method 0 2024-09-16T04:39:05.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d5214640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:05.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d5214640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d5214640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:05.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:05.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:05.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09cffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:05.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d5214640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:05.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d5214640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:05.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d5214640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:05.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d4a13640 10 monclient(hunting): handle_auth_done global_id 4395 payload 293 2024-09-16T04:39:05.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d4a13640 1 -- 172.21.15.46:0/3290521461 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f09b8004800 msgr2=0x7f09b8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:05.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d4a13640 1 --2- 172.21.15.46:0/3290521461 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f09b8004800 0x7f09b8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d4a13640 1 -- 172.21.15.46:0/3290521461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09b80051c0 msgr2=0x7f09b800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:05.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d4a13640 1 --2- 172.21.15.46:0/3290521461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09b80051c0 0x7f09b800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d4a13640 10 monclient: _finish_hunting 0 2024-09-16T04:39:05.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d4a13640 1 monclient: found mon.a 2024-09-16T04:39:05.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d4a13640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:05.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d5214640 1 -- 172.21.15.46:0/3290521461 reap_dead start 2024-09-16T04:39:05.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d4a13640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:05.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d4a13640 1 -- 172.21.15.46:0/3290521461 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09b8003520 con 0x7f09b80039b0 2024-09-16T04:39:05.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d4a13640 10 monclient: _finish_auth 0 2024-09-16T04:39:05.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d4a13640 10 monclient: _check_auth_tickets 2024-09-16T04:39:05.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09d4a13640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:05.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.525+0000 7f09cffff640 1 --2- 172.21.15.46:0/3290521461 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f09b8004800 0x7f09b8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:05.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09d4a13640 1 --2- 172.21.15.46:0/3290521461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09b80039b0 0x7f09b8003dd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f09c0001750 tx=0x7f09b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9b6ba12d00ca03e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:05.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09cf7fe640 1 -- 172.21.15.46:0/3290521461 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c0001370 con 0x7f09b80039b0 2024-09-16T04:39:05.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:05.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:05.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09cf7fe640 10 monclient: dump: 2024-09-16T04:39:05.545 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:05.545 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:05.546 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.546 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.546 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:05.546 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:05.546 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:05.546 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:05.546 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:05.546 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:05.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09d6c77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:05.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09cf7fe640 1 -- 172.21.15.46:0/3290521461 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09c00052d0 con 0x7f09b80039b0 2024-09-16T04:39:05.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09cf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:05.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09d6c77640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:05.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09d6c77640 4 set_mon_vals no callback set 2024-09-16T04:39:05.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09cf7fe640 1 -- 172.21.15.46:0/3290521461 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c002abc0 con 0x7f09b80039b0 2024-09-16T04:39:05.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:05.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:05.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09cf7fe640 10 monclient: dump: 2024-09-16T04:39:05.547 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:05.547 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:05.547 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.547 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.547 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:05.547 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:05.548 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:05.548 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:05.548 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:05.548 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:05.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09d6c77640 10 monclient: shutdown 2024-09-16T04:39:05.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09d6c77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:05.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09d6c77640 1 -- 172.21.15.46:0/3290521461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09b80039b0 msgr2=0x7f09b8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:05.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09d6c77640 1 --2- 172.21.15.46:0/3290521461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09b80039b0 0x7f09b8003dd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f09c0001750 tx=0x7f09b8005830 comp rx=0 tx=0).stop 2024-09-16T04:39:05.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09d6c77640 1 -- 172.21.15.46:0/3290521461 shutdown_connections 2024-09-16T04:39:05.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09d6c77640 1 --2- 172.21.15.46:0/3290521461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09b80039b0 0x7f09b8003dd0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09d6c77640 1 -- 172.21.15.46:0/3290521461 >> 172.21.15.46:0/3290521461 conn(0x7f09d00a6340 msgr2=0x7f09d006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:05.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09d6c77640 1 -- 172.21.15.46:0/3290521461 shutdown_connections 2024-09-16T04:39:05.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.526+0000 7f09d6c77640 1 -- 172.21.15.46:0/3290521461 wait complete. 2024-09-16T04:39:05.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.527+0000 7f09d6c77640 10 monclient: build_initial_monmap 2024-09-16T04:39:05.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.527+0000 7f09d6c77640 10 monclient: monmap: 2024-09-16T04:39:05.549 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:05.549 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:05.549 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:05.549 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:05.549 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:05.549 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:05.549 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:05.549 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:05.549 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:05.549 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:05.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.527+0000 7f09d6c77640 1 Processor -- start 2024-09-16T04:39:05.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.527+0000 7f09d6c77640 1 -- start start 2024-09-16T04:39:05.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.527+0000 7f09d6c77640 10 monclient: init 2024-09-16T04:39:05.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d6c77640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:05.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d6c77640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:39:05.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d6c77640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f09b80039b0 0x7f09d01cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d6c77640 10 monclient(hunting): picked mon.noname-c con 0x7f09b80039b0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:05.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d6c77640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09d01cc7e0 0x7f09d011ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d6c77640 10 monclient(hunting): picked mon.noname-a con 0x7f09d01cc7e0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:05.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d6c77640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09d01ccc40 0x7f09d01204d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d6c77640 10 monclient(hunting): picked mon.noname-b con 0x7f09d01ccc40 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:05.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d6c77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:05.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d6c77640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f09d009be70 con 0x7f09d01cc7e0 2024-09-16T04:39:05.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d6c77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:05.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d6c77640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f09d009bcf0 con 0x7f09d01ccc40 2024-09-16T04:39:05.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d6c77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:05.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d6c77640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f09d009bff0 con 0x7f09b80039b0 2024-09-16T04:39:05.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d6c77640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:05.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d6c77640 10 monclient(hunting): authenticate will time out at 743.943438s 2024-09-16T04:39:05.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d4a13640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f09b80039b0 0x7f09d01cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09cffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09d01cc7e0 0x7f09d011ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d4a13640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f09b80039b0 0x7f09d01cc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:36480/0 (socket says 172.21.15.46:36480) 2024-09-16T04:39:05.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d4a13640 1 -- 172.21.15.46:0/294508760 learned_addr learned my addr 172.21.15.46:0/294508760 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:05.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d4a13640 10 monclient(hunting): get_auth_request con 0x7f09b80039b0 auth_method 0 2024-09-16T04:39:05.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d4a13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:05.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d4a13640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d4a13640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:05.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09cffff640 10 monclient(hunting): get_auth_request con 0x7f09d01cc7e0 auth_method 0 2024-09-16T04:39:05.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:05.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:05.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d5214640 1 --2- 172.21.15.46:0/294508760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09d01ccc40 0x7f09d01204d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d5214640 10 monclient(hunting): get_auth_request con 0x7f09d01ccc40 auth_method 0 2024-09-16T04:39:05.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d5214640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:05.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d5214640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d5214640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:05.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:05.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:05.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:05.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d4a13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:05.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d4a13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:05.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d4a13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:05.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d5214640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:05.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d5214640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:05.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d5214640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:05.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09cffff640 10 monclient(hunting): handle_auth_done global_id 4398 payload 824 2024-09-16T04:39:05.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09cffff640 1 -- 172.21.15.46:0/294508760 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f09b80039b0 msgr2=0x7f09d01cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:05.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09cffff640 1 --2- 172.21.15.46:0/294508760 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f09b80039b0 0x7f09d01cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09cffff640 1 -- 172.21.15.46:0/294508760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09d01ccc40 msgr2=0x7f09d01204d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:05.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09cffff640 1 --2- 172.21.15.46:0/294508760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09d01ccc40 0x7f09d01204d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09cffff640 10 monclient: _finish_hunting 0 2024-09-16T04:39:05.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09cffff640 1 monclient: found mon.noname-a 2024-09-16T04:39:05.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:05.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09cffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:05.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09cffff640 1 -- 172.21.15.46:0/294508760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09d0122470 con 0x7f09d01cc7e0 2024-09-16T04:39:05.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.528+0000 7f09d4a13640 1 --2- 172.21.15.46:0/294508760 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f09b80039b0 0x7f09d01cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:05.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09cffff640 1 --2- 172.21.15.46:0/294508760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09d01cc7e0 0x7f09d011ff90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f09d0074ef0 tx=0x7f09c4011e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:05.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09cd7fa640 1 -- 172.21.15.46:0/294508760 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c4019070 con 0x7f09d01cc7e0 2024-09-16T04:39:05.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:05.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09cd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:05.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09cd7fa640 10 monclient: dump: 2024-09-16T04:39:05.554 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:05.554 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:05.554 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.554 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.554 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:05.554 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:05.554 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:05.554 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:05.555 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:05.555 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:05.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09cd7fa640 10 monclient: _finish_auth 0 2024-09-16T04:39:05.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09cd7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:39:05.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:05.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09cd7fa640 1 -- 172.21.15.46:0/294508760 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09c4004240 con 0x7f09d01cc7e0 2024-09-16T04:39:05.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09d6c77640 5 monclient: authenticate success, global_id 4398 2024-09-16T04:39:05.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09d6c77640 10 monclient: _renew_subs 2024-09-16T04:39:05.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09d6c77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:05.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09d6c77640 1 -- 172.21.15.46:0/294508760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09d0123e80 con 0x7f09d01cc7e0 2024-09-16T04:39:05.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09cd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:05.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09cd7fa640 1 -- 172.21.15.46:0/294508760 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c4009690 con 0x7f09d01cc7e0 2024-09-16T04:39:05.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:05.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09cd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:05.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09cd7fa640 10 monclient: dump: 2024-09-16T04:39:05.556 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:05.556 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:05.556 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.556 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.556 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:05.556 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:05.556 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:05.556 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:05.556 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:05.556 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:05.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09d6c77640 10 monclient: _renew_subs 2024-09-16T04:39:05.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09d6c77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:05.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09d6c77640 1 -- 172.21.15.46:0/294508760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09d00a0c70 con 0x7f09d01cc7e0 2024-09-16T04:39:05.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.529+0000 7f09ceffd640 4 set_mon_vals no callback set 2024-09-16T04:39:05.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.530+0000 7f09cd7fa640 1 -- 172.21.15.46:0/294508760 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f09c401c070 con 0x7f09d01cc7e0 2024-09-16T04:39:05.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.530+0000 7f09d6c77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:05.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.530+0000 7f09d6c77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:05.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.530+0000 7f09d6c77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:05.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.530+0000 7f09d6c77640 1 -- 172.21.15.46:0/294508760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09d00a9030 con 0x7f09d01cc7e0 2024-09-16T04:39:05.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.530+0000 7f09cd7fa640 1 --2- 172.21.15.46:0/294508760 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f09bc0383b0 0x7f09bc03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.530+0000 7f09cd7fa640 1 -- 172.21.15.46:0/294508760 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f09c404b850 con 0x7f09d01cc7e0 2024-09-16T04:39:05.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.530+0000 7f09d4a13640 1 --2- 172.21.15.46:0/294508760 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f09bc0383b0 0x7f09bc03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.530+0000 7f09d4a13640 10 monclient: get_auth_request con 0x7f09bc0383b0 auth_method 0 2024-09-16T04:39:05.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.530+0000 7f09d4a13640 1 --2- 172.21.15.46:0/294508760 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f09bc0383b0 0x7f09bc03a870 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f09c00015c0 tx=0x7f09c002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:05.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.535+0000 7f09cd7fa640 1 -- 172.21.15.46:0/294508760 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f09d00a9030 con 0x7f09d01cc7e0 2024-09-16T04:39:05.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.535+0000 7f09cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:05.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.535+0000 7f09cd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:05.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.710+0000 7f09d6c77640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T04:39:05.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.710+0000 7f09d6c77640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T04:39:05.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.710+0000 7f09d6c77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:05.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.710+0000 7f09d6c77640 1 -- 172.21.15.46:0/294508760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7f09d00a43d0 con 0x7f09d01cc7e0 2024-09-16T04:39:05.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.710+0000 7f09cd7fa640 1 -- 172.21.15.46:0/294508760 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) ==== 92+0+10 (secure 0 0 0) 0x7f09c400f8e0 con 0x7f09d01cc7e0 2024-09-16T04:39:05.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.710+0000 7f09cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T04:39:05.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.710+0000 7f09cd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:05.712 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 8 2024-09-16T04:39:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.712+0000 7f09d6c77640 1 -- 172.21.15.46:0/294508760 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f09bc0383b0 msgr2=0x7f09bc03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.712+0000 7f09d6c77640 1 --2- 172.21.15.46:0/294508760 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f09bc0383b0 0x7f09bc03a870 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f09c00015c0 tx=0x7f09c002e000 comp rx=0 tx=0).stop 2024-09-16T04:39:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.712+0000 7f09d6c77640 10 monclient: shutdown 2024-09-16T04:39:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.712+0000 7f09d6c77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.712+0000 7f09d6c77640 1 -- 172.21.15.46:0/294508760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09d01cc7e0 msgr2=0x7f09d011ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.712+0000 7f09d6c77640 1 --2- 172.21.15.46:0/294508760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09d01cc7e0 0x7f09d011ff90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f09d0074ef0 tx=0x7f09c4011e10 comp rx=0 tx=0).stop 2024-09-16T04:39:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.712+0000 7f09d6c77640 1 -- 172.21.15.46:0/294508760 shutdown_connections 2024-09-16T04:39:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.712+0000 7f09d6c77640 1 --2- 172.21.15.46:0/294508760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09d01ccc40 0x7f09d01204d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.712+0000 7f09d6c77640 1 --2- 172.21.15.46:0/294508760 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f09bc0383b0 0x7f09bc03a870 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.712+0000 7f09d6c77640 1 --2- 172.21.15.46:0/294508760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09d01cc7e0 0x7f09d011ff90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.712+0000 7f09d6c77640 1 --2- 172.21.15.46:0/294508760 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f09b80039b0 0x7f09d01cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.712+0000 7f09d6c77640 1 -- 172.21.15.46:0/294508760 >> 172.21.15.46:0/294508760 conn(0x7f09d00a6340 msgr2=0x7f09d013b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.712+0000 7f09d6c77640 1 -- 172.21.15.46:0/294508760 shutdown_connections 2024-09-16T04:39:05.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.712+0000 7f09d6c77640 1 -- 172.21.15.46:0/294508760 wait complete. 2024-09-16T04:39:05.725 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-16T04:39:05.725 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:39:05.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.824+0000 7f1d95806640 10 monclient: get_monmap_and_config 2024-09-16T04:39:05.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.824+0000 7f1d95806640 10 monclient: build_initial_monmap 2024-09-16T04:39:05.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.824+0000 7f1d95806640 10 monclient: monmap: 2024-09-16T04:39:05.826 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:05.826 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:05.826 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:05.825434+0000 2024-09-16T04:39:05.826 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:05.825434+0000 2024-09-16T04:39:05.826 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:05.826 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:05.826 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:05.827 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:05.827 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:05.827 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:05.827 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:05.827 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:05.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.825+0000 7f1d95806640 1 Processor -- start 2024-09-16T04:39:05.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.825+0000 7f1d95806640 1 -- start start 2024-09-16T04:39:05.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.825+0000 7f1d95806640 10 monclient: init 2024-09-16T04:39:05.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d95806640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:05.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d95806640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-16T04:39:05.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d95806640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1d9006e160 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:05.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d95806640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1d90139c50 0x7f1d9013c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d95806640 10 monclient(hunting): picked mon.noname-c con 0x7f1d90139c50 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:05.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d95806640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1d9013c580 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:05.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d95806640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d900a3fd0 con 0x7f1d9006e160 2024-09-16T04:39:05.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d95806640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d900a0bd0 con 0x7f1d9013c580 2024-09-16T04:39:05.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d95806640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:05.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d95806640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f1d9006e540 con 0x7f1d90139c50 2024-09-16T04:39:05.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d95806640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:05.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d95806640 10 monclient(hunting): authenticate will time out at 744.241137s 2024-09-16T04:39:05.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8effd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1d9006e160 0x7f1d900a29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35320/0 (socket says 172.21.15.46:35320) 2024-09-16T04:39:05.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8effd640 1 -- 172.21.15.46:0/4236615577 learned_addr learned my addr 172.21.15.46:0/4236615577 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:05.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 1 -- 172.21.15.46:0/4236615577 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3399974852 0 0) 0x7f1d80001110 con 0x7f1d9006e160 2024-09-16T04:39:05.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:39:05.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:05.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 10 monclient(hunting): my global_id is 4401 2024-09-16T04:39:05.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 1 -- 172.21.15.46:0/4236615577 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d78003520 con 0x7f1d9006e160 2024-09-16T04:39:05.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8e7fc640 1 --2- 172.21.15.46:0/4236615577 >> v2:172.21.15.114:3300/0 conn(0x7f1d90139c50 0x7f1d9013c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8e7fc640 10 monclient(hunting): get_auth_request con 0x7f1d90139c50 auth_method 0 2024-09-16T04:39:05.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:05.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:05.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:05.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:05.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:05.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 1 -- 172.21.15.46:0/4236615577 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1436624841 0 0) 0x7f1d78003520 con 0x7f1d9006e160 2024-09-16T04:39:05.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 1 -- 172.21.15.46:0/4236615577 >> v2:172.21.15.114:3300/0 conn(0x7f1d90139c50 msgr2=0x7f1d9013c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:05.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 1 --2- 172.21.15.46:0/4236615577 >> v2:172.21.15.114:3300/0 conn(0x7f1d90139c50 0x7f1d9013c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 1 -- 172.21.15.46:0/4236615577 >> v1:172.21.15.114:6789/0 conn(0x7f1d9013c580 legacy=0x7f1d9013e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:05.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:39:05.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:39:05.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:05.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:39:05.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 1 -- 172.21.15.46:0/4236615577 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d900a0eb0 con 0x7f1d9006e160 2024-09-16T04:39:05.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 10 monclient: _finish_auth 0 2024-09-16T04:39:05.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:39:05.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:05.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 1 -- 172.21.15.46:0/4236615577 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f1d80001840 con 0x7f1d9006e160 2024-09-16T04:39:05.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:05.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:39:05.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 10 monclient: dump: 2024-09-16T04:39:05.841 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:05.841 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:05.841 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.841 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.842 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:05.842 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:05.842 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:05.842 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:05.842 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:05.842 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:05.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:39:05.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:05.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 1 -- 172.21.15.46:0/4236615577 >> v1:172.21.15.46:6789/0 conn(0x7f1d9006e160 legacy=0x7f1d900a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:05.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:39:05.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 1 --2- 172.21.15.46:0/4236615577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d780039b0 0x7f1d78003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 10 monclient(hunting): picked mon.a con 0x7f1d780039b0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:05.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 1 --2- 172.21.15.46:0/4236615577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d78004800 0x7f1d78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 10 monclient(hunting): picked mon.b con 0x7f1d78004800 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:05.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 1 --2- 172.21.15.46:0/4236615577 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d780052e0 0x7f1d7800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 10 monclient(hunting): picked mon.c con 0x7f1d780052e0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:05.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:05.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 1 -- 172.21.15.46:0/4236615577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9006e540 con 0x7f1d780039b0 2024-09-16T04:39:05.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:05.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 1 -- 172.21.15.46:0/4236615577 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7800c1b0 con 0x7f1d78004800 2024-09-16T04:39:05.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:05.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 1 -- 172.21.15.46:0/4236615577 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f1d7800c3b0 con 0x7f1d780052e0 2024-09-16T04:39:05.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:05.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:39:05.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.826+0000 7f1d8effd640 1 --2- 172.21.15.46:0/4236615577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d780039b0 0x7f1d78003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8effd640 10 monclient(hunting): get_auth_request con 0x7f1d780039b0 auth_method 0 2024-09-16T04:39:05.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:05.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:05.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8e7fc640 1 --2- 172.21.15.46:0/4236615577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d78004800 0x7f1d78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8e7fc640 10 monclient(hunting): get_auth_request con 0x7f1d78004800 auth_method 0 2024-09-16T04:39:05.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:05.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:05.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:05.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:05.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:05.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8f7fe640 1 --2- 172.21.15.46:0/4236615577 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d780052e0 0x7f1d7800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8f7fe640 10 monclient(hunting): get_auth_request con 0x7f1d780052e0 auth_method 0 2024-09-16T04:39:05.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:05.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:05.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:05.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:05.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:05.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8effd640 10 monclient(hunting): handle_auth_done global_id 4401 payload 293 2024-09-16T04:39:05.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8effd640 1 -- 172.21.15.46:0/4236615577 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d780052e0 msgr2=0x7f1d7800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:05.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8effd640 1 --2- 172.21.15.46:0/4236615577 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d780052e0 0x7f1d7800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8effd640 1 -- 172.21.15.46:0/4236615577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d78004800 msgr2=0x7f1d78004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:05.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8effd640 1 --2- 172.21.15.46:0/4236615577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d78004800 0x7f1d78004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8effd640 10 monclient: _finish_hunting 0 2024-09-16T04:39:05.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8effd640 1 monclient: found mon.a 2024-09-16T04:39:05.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:05.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8effd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:05.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8effd640 1 -- 172.21.15.46:0/4236615577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d78003520 con 0x7f1d780039b0 2024-09-16T04:39:05.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8effd640 10 monclient: _finish_auth 0 2024-09-16T04:39:05.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8effd640 10 monclient: _check_auth_tickets 2024-09-16T04:39:05.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:05.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d95806640 5 monclient: authenticate success, global_id 4401 2024-09-16T04:39:05.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d95806640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:05.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8f7fe640 1 --2- 172.21.15.46:0/4236615577 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d780052e0 0x7f1d7800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:39:05.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8f7fe640 1 -- 172.21.15.46:0/4236615577 reap_dead start 2024-09-16T04:39:05.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.827+0000 7f1d8effd640 1 --2- 172.21.15.46:0/4236615577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d780039b0 0x7f1d78003dd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1d8000c490 tx=0x7f1d78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea85c1ec3ce994ea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:05.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d8dffb640 1 -- 172.21.15.46:0/4236615577 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d8000c0e0 con 0x7f1d780039b0 2024-09-16T04:39:05.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d8dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:05.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d8dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:05.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d8dffb640 10 monclient: dump: 2024-09-16T04:39:05.847 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:05.847 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:05.847 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.847 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.847 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:05.847 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:05.847 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:05.847 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:05.847 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:05.848 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:05.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d95806640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:05.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d8dffb640 1 -- 172.21.15.46:0/4236615577 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d80034070 con 0x7f1d780039b0 2024-09-16T04:39:05.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d8dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:05.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d95806640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:05.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d8dffb640 1 -- 172.21.15.46:0/4236615577 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d80005d70 con 0x7f1d780039b0 2024-09-16T04:39:05.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d95806640 4 set_mon_vals no callback set 2024-09-16T04:39:05.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d8dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:05.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d8dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:05.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d8dffb640 10 monclient: dump: 2024-09-16T04:39:05.848 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:05.848 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:05.849 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.849 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.849 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:05.849 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:05.849 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:05.849 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:05.849 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:05.849 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:05.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d95806640 10 monclient: shutdown 2024-09-16T04:39:05.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d95806640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:05.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d95806640 1 -- 172.21.15.46:0/4236615577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d780039b0 msgr2=0x7f1d78003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:05.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d95806640 1 --2- 172.21.15.46:0/4236615577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d780039b0 0x7f1d78003dd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1d8000c490 tx=0x7f1d78005830 comp rx=0 tx=0).stop 2024-09-16T04:39:05.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d95806640 1 -- 172.21.15.46:0/4236615577 shutdown_connections 2024-09-16T04:39:05.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d95806640 1 --2- 172.21.15.46:0/4236615577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d780039b0 0x7f1d78003dd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d95806640 1 -- 172.21.15.46:0/4236615577 >> 172.21.15.46:0/4236615577 conn(0x7f1d900a5c10 msgr2=0x7f1d900a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:05.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d95806640 1 -- 172.21.15.46:0/4236615577 shutdown_connections 2024-09-16T04:39:05.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.828+0000 7f1d95806640 1 -- 172.21.15.46:0/4236615577 wait complete. 2024-09-16T04:39:05.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.829+0000 7f1d95806640 10 monclient: build_initial_monmap 2024-09-16T04:39:05.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.829+0000 7f1d95806640 10 monclient: monmap: 2024-09-16T04:39:05.850 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:05.850 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:05.850 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:05.850 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:05.850 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:05.850 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:05.850 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:05.850 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:05.850 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:05.851 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:05.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.829+0000 7f1d95806640 1 Processor -- start 2024-09-16T04:39:05.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.829+0000 7f1d95806640 1 -- start start 2024-09-16T04:39:05.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.829+0000 7f1d95806640 10 monclient: init 2024-09-16T04:39:05.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d95806640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:05.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d95806640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:39:05.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d95806640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d780039b0 0x7f1d90136cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d95806640 10 monclient(hunting): picked mon.noname-c con 0x7f1d780039b0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:05.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d95806640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d901cbc00 0x7f1d90137210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d95806640 10 monclient(hunting): picked mon.noname-a con 0x7f1d901cbc00 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:05.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d95806640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d9011f760 0x7f1d9011fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d95806640 10 monclient(hunting): picked mon.noname-b con 0x7f1d9011f760 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:05.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d95806640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:05.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d95806640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1d900a34a0 con 0x7f1d901cbc00 2024-09-16T04:39:05.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d95806640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:05.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d95806640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1d900a1250 con 0x7f1d9011f760 2024-09-16T04:39:05.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d95806640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:05.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d95806640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f1d900a84b0 con 0x7f1d780039b0 2024-09-16T04:39:05.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d95806640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:05.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d95806640 10 monclient(hunting): authenticate will time out at 744.245418s 2024-09-16T04:39:05.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8effd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d780039b0 0x7f1d90136cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8effd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d780039b0 0x7f1d90136cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:36490/0 (socket says 172.21.15.46:36490) 2024-09-16T04:39:05.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8effd640 1 -- 172.21.15.46:0/3739846582 learned_addr learned my addr 172.21.15.46:0/3739846582 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:05.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8effd640 10 monclient(hunting): get_auth_request con 0x7f1d780039b0 auth_method 0 2024-09-16T04:39:05.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:05.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8e7fc640 1 --2- 172.21.15.46:0/3739846582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d901cbc00 0x7f1d90137210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:05.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8e7fc640 10 monclient(hunting): get_auth_request con 0x7f1d901cbc00 auth_method 0 2024-09-16T04:39:05.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:05.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:05.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:05.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:05.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:05.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8f7fe640 1 --2- 172.21.15.46:0/3739846582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d9011f760 0x7f1d9011fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:05.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:05.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:05.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8f7fe640 10 monclient(hunting): get_auth_request con 0x7f1d9011f760 auth_method 0 2024-09-16T04:39:05.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:05.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:05.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:05.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:05.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:05.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:05.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8e7fc640 10 monclient(hunting): handle_auth_done global_id 4404 payload 824 2024-09-16T04:39:05.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8e7fc640 1 -- 172.21.15.46:0/3739846582 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d780039b0 msgr2=0x7f1d90136cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:05.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8e7fc640 1 --2- 172.21.15.46:0/3739846582 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d780039b0 0x7f1d90136cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8e7fc640 1 -- 172.21.15.46:0/3739846582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d9011f760 msgr2=0x7f1d9011fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:05.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8e7fc640 1 --2- 172.21.15.46:0/3739846582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d9011f760 0x7f1d9011fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:05.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:39:05.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8e7fc640 1 monclient: found mon.noname-a 2024-09-16T04:39:05.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:05.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:05.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.830+0000 7f1d8e7fc640 1 -- 172.21.15.46:0/3739846582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d901233c0 con 0x7f1d901cbc00 2024-09-16T04:39:05.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d8e7fc640 1 --2- 172.21.15.46:0/3739846582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d901cbc00 0x7f1d90137210 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1d84011910 tx=0x7f1d84011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:05.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d77fff640 1 -- 172.21.15.46:0/3739846582 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d84019070 con 0x7f1d901cbc00 2024-09-16T04:39:05.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:05.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d77fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:05.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d77fff640 10 monclient: dump: 2024-09-16T04:39:05.855 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:05.855 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:05.855 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.855 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.856 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:05.856 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:05.856 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:05.856 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:05.856 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:05.856 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:05.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d77fff640 10 monclient: _finish_auth 0 2024-09-16T04:39:05.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d77fff640 10 monclient: _check_auth_tickets 2024-09-16T04:39:05.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:05.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d77fff640 1 -- 172.21.15.46:0/3739846582 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d84004240 con 0x7f1d901cbc00 2024-09-16T04:39:05.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d77fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:05.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d77fff640 1 -- 172.21.15.46:0/3739846582 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d8400f820 con 0x7f1d901cbc00 2024-09-16T04:39:05.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d95806640 5 monclient: authenticate success, global_id 4404 2024-09-16T04:39:05.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d95806640 10 monclient: _renew_subs 2024-09-16T04:39:05.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d95806640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:05.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d95806640 1 -- 172.21.15.46:0/3739846582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d901236a0 con 0x7f1d901cbc00 2024-09-16T04:39:05.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:05.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d77fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:05.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d77fff640 10 monclient: dump: 2024-09-16T04:39:05.857 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:05.857 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:05.857 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.857 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:05.857 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:05.857 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:05.857 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:05.857 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:05.857 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:05.858 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:05.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d8d7fa640 4 set_mon_vals no callback set 2024-09-16T04:39:05.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d95806640 10 monclient: _renew_subs 2024-09-16T04:39:05.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d95806640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:05.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d95806640 1 -- 172.21.15.46:0/3739846582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d901230e0 con 0x7f1d901cbc00 2024-09-16T04:39:05.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.831+0000 7f1d77fff640 1 -- 172.21.15.46:0/3739846582 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f1d8401c070 con 0x7f1d901cbc00 2024-09-16T04:39:05.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.832+0000 7f1d95806640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:05.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.832+0000 7f1d95806640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:05.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.832+0000 7f1d95806640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:05.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.832+0000 7f1d77fff640 1 --2- 172.21.15.46:0/3739846582 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1d7c0383b0 0x7f1d7c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:05.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.832+0000 7f1d95806640 1 -- 172.21.15.46:0/3739846582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d900a29a0 con 0x7f1d901cbc00 2024-09-16T04:39:05.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.832+0000 7f1d77fff640 1 -- 172.21.15.46:0/3739846582 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f1d8404b960 con 0x7f1d901cbc00 2024-09-16T04:39:05.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.832+0000 7f1d8effd640 1 --2- 172.21.15.46:0/3739846582 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1d7c0383b0 0x7f1d7c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:05.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.832+0000 7f1d8effd640 10 monclient: get_auth_request con 0x7f1d7c0383b0 auth_method 0 2024-09-16T04:39:05.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.832+0000 7f1d8effd640 1 --2- 172.21.15.46:0/3739846582 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1d7c0383b0 0x7f1d7c03a870 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1d80031040 tx=0x7f1d8002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:05.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.836+0000 7f1d77fff640 1 -- 172.21.15.46:0/3739846582 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1d840139e0 con 0x7f1d901cbc00 2024-09-16T04:39:05.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.837+0000 7f1d77fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:05.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:05.837+0000 7f1d77fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:06.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.010+0000 7f1d95806640 1 -- 172.21.15.46:0/3739846582 --> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1d900a7ff0 con 0x7f1d7c0383b0 2024-09-16T04:39:06.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.012+0000 7f1d77fff640 1 -- 172.21.15.46:0/3739846582 <== mgr.4099 v2:172.21.15.114:6800/3521677055 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34116 (secure 0 0 0) 0x7f1d900a7ff0 con 0x7f1d7c0383b0 2024-09-16T04:39:06.013 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:39:06.014 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-09-16T04:39:06.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.014+0000 7f1d95806640 1 -- 172.21.15.46:0/3739846582 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1d7c0383b0 msgr2=0x7f1d7c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.014+0000 7f1d95806640 1 --2- 172.21.15.46:0/3739846582 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1d7c0383b0 0x7f1d7c03a870 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1d80031040 tx=0x7f1d8002e000 comp rx=0 tx=0).stop 2024-09-16T04:39:06.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.014+0000 7f1d95806640 10 monclient: shutdown 2024-09-16T04:39:06.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.014+0000 7f1d95806640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:06.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.014+0000 7f1d95806640 1 -- 172.21.15.46:0/3739846582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d901cbc00 msgr2=0x7f1d90137210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.014+0000 7f1d95806640 1 --2- 172.21.15.46:0/3739846582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d901cbc00 0x7f1d90137210 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1d84011910 tx=0x7f1d84011d10 comp rx=0 tx=0).stop 2024-09-16T04:39:06.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.014+0000 7f1d95806640 1 -- 172.21.15.46:0/3739846582 shutdown_connections 2024-09-16T04:39:06.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.014+0000 7f1d95806640 1 --2- 172.21.15.46:0/3739846582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d9011f760 0x7f1d9011fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.014+0000 7f1d95806640 1 --2- 172.21.15.46:0/3739846582 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1d7c0383b0 0x7f1d7c03a870 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.014+0000 7f1d95806640 1 --2- 172.21.15.46:0/3739846582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d901cbc00 0x7f1d90137210 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.014+0000 7f1d95806640 1 --2- 172.21.15.46:0/3739846582 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d780039b0 0x7f1d90136cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.014+0000 7f1d95806640 1 -- 172.21.15.46:0/3739846582 >> 172.21.15.46:0/3739846582 conn(0x7f1d900a5c10 msgr2=0x7f1d901392b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:06.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.014+0000 7f1d95806640 1 -- 172.21.15.46:0/3739846582 shutdown_connections 2024-09-16T04:39:06.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.014+0000 7f1d95806640 1 -- 172.21.15.46:0/3739846582 wait complete. 2024-09-16T04:39:06.027 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-16T04:39:05.348788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":172616,"kb_used_data":2104,"kb_used_omap":9,"kb_used_meta":170358,"kb_avail":566058424,"statfs":{"total":579820584960,"available":579643826176,"internally_reserved":0,"allocated":2154496,"data_stored":1420330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":174447264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001066"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.524013+0000","last_change":"2024-09-16T04:39:00.524013+0000","last_active":"2024-09-16T04:39:00.524013+0000","last_peered":"2024-09-16T04:39:00.524013+0000","last_clean":"2024-09-16T04:39:00.524013+0000","last_became_active":"2024-09-16T04:38:58.756778+0000","last_became_peered":"2024-09-16T04:38:58.756778+0000","last_unstale":"2024-09-16T04:39:00.524013+0000","last_undegraded":"2024-09-16T04:39:00.524013+0000","last_fullsized":"2024-09-16T04:39:00.524013+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:24:15.429766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054536700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.531971+0000","last_change":"2024-09-16T04:39:00.532294+0000","last_active":"2024-09-16T04:39:00.531971+0000","last_peered":"2024-09-16T04:39:00.531971+0000","last_clean":"2024-09-16T04:39:00.531971+0000","last_became_active":"2024-09-16T04:38:58.520473+0000","last_became_peered":"2024-09-16T04:38:58.520473+0000","last_unstale":"2024-09-16T04:39:00.531971+0000","last_undegraded":"2024-09-16T04:39:00.531971+0000","last_fullsized":"2024-09-16T04:39:00.531971+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:15:41.548834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053532500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.523783+0000","last_change":"2024-09-16T04:39:00.523783+0000","last_active":"2024-09-16T04:39:00.523783+0000","last_peered":"2024-09-16T04:39:00.523783+0000","last_clean":"2024-09-16T04:39:00.523783+0000","last_became_active":"2024-09-16T04:38:58.521000+0000","last_became_peered":"2024-09-16T04:38:58.521000+0000","last_unstale":"2024-09-16T04:39:00.523783+0000","last_undegraded":"2024-09-16T04:39:00.523783+0000","last_fullsized":"2024-09-16T04:39:00.523783+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:42.837022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035484999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.532031+0000","last_change":"2024-09-16T04:39:00.532400+0000","last_active":"2024-09-16T04:39:00.532031+0000","last_peered":"2024-09-16T04:39:00.532031+0000","last_clean":"2024-09-16T04:39:00.532031+0000","last_became_active":"2024-09-16T04:38:58.521627+0000","last_became_peered":"2024-09-16T04:38:58.521627+0000","last_unstale":"2024-09-16T04:39:00.532031+0000","last_undegraded":"2024-09-16T04:39:00.532031+0000","last_fullsized":"2024-09-16T04:39:00.532031+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:41.804897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062256500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.530578+0000","last_change":"2024-09-16T04:39:00.530578+0000","last_active":"2024-09-16T04:39:00.530578+0000","last_peered":"2024-09-16T04:39:00.530578+0000","last_clean":"2024-09-16T04:39:00.530578+0000","last_became_active":"2024-09-16T04:38:58.522744+0000","last_became_peered":"2024-09-16T04:38:58.522744+0000","last_unstale":"2024-09-16T04:39:00.530578+0000","last_undegraded":"2024-09-16T04:39:00.530578+0000","last_fullsized":"2024-09-16T04:39:00.530578+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:04:47.450053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000966258,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.523413+0000","last_change":"2024-09-16T04:39:00.523661+0000","last_active":"2024-09-16T04:39:00.523413+0000","last_peered":"2024-09-16T04:39:00.523413+0000","last_clean":"2024-09-16T04:39:00.523413+0000","last_became_active":"2024-09-16T04:38:58.524372+0000","last_became_peered":"2024-09-16T04:38:58.524372+0000","last_unstale":"2024-09-16T04:39:00.523413+0000","last_undegraded":"2024-09-16T04:39:00.523413+0000","last_fullsized":"2024-09-16T04:39:00.523413+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:46.534727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047533599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.523838+0000","last_change":"2024-09-16T04:39:00.523838+0000","last_active":"2024-09-16T04:39:00.523838+0000","last_peered":"2024-09-16T04:39:00.523838+0000","last_clean":"2024-09-16T04:39:00.523838+0000","last_became_active":"2024-09-16T04:38:58.522142+0000","last_became_peered":"2024-09-16T04:38:58.522142+0000","last_unstale":"2024-09-16T04:39:00.523838+0000","last_undegraded":"2024-09-16T04:39:00.523838+0000","last_fullsized":"2024-09-16T04:39:00.523838+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:35.411495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039704200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.529810+0000","last_change":"2024-09-16T04:39:00.530064+0000","last_active":"2024-09-16T04:39:00.529810+0000","last_peered":"2024-09-16T04:39:00.529810+0000","last_clean":"2024-09-16T04:39:00.529810+0000","last_became_active":"2024-09-16T04:38:58.523268+0000","last_became_peered":"2024-09-16T04:38:58.523268+0000","last_unstale":"2024-09-16T04:39:00.529810+0000","last_undegraded":"2024-09-16T04:39:00.529810+0000","last_fullsized":"2024-09-16T04:39:00.529810+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:12:33.958554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043992499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":170,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.523350+0000","last_change":"2024-09-16T04:38:52.501054+0000","last_active":"2024-09-16T04:39:00.523350+0000","last_peered":"2024-09-16T04:39:00.523350+0000","last_clean":"2024-09-16T04:39:00.523350+0000","last_became_active":"2024-09-16T04:38:52.500796+0000","last_became_peered":"2024-09-16T04:38:52.500796+0000","last_unstale":"2024-09-16T04:39:00.523350+0000","last_undegraded":"2024-09-16T04:39:00.523350+0000","last_fullsized":"2024-09-16T04:39:00.523350+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:51.492404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:51.492404+0000","last_clean_scrub_stamp":"2024-09-16T04:38:51.492404+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:36.122039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":180224,"data_stored":60406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":712704,"data_stored":589334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31884,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339956,"statfs":{"total":96636764160,"available":96604114944,"internally_reserved":0,"allocated":712704,"data_stored":589334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":184320,"data_stored":60425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":180224,"data_stored":60406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31376,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340464,"statfs":{"total":96636764160,"available":96604635136,"internally_reserved":0,"allocated":184320,"data_stored":60425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:39:06.028 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:39:06.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.127+0000 7f312b2e4640 10 monclient: get_monmap_and_config 2024-09-16T04:39:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.127+0000 7f312b2e4640 10 monclient: build_initial_monmap 2024-09-16T04:39:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.127+0000 7f312b2e4640 10 monclient: monmap: 2024-09-16T04:39:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:06.128090+0000 2024-09-16T04:39:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:06.128090+0000 2024-09-16T04:39:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:06.130 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.128+0000 7f312b2e4640 1 Processor -- start 2024-09-16T04:39:06.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.128+0000 7f312b2e4640 1 -- start start 2024-09-16T04:39:06.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.128+0000 7f312b2e4640 10 monclient: init 2024-09-16T04:39:06.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.128+0000 7f312b2e4640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:06.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.128+0000 7f312b2e4640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-16T04:39:06.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.128+0000 7f312b2e4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f31240a3c10 0x7f312409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.128+0000 7f312b2e4640 10 monclient(hunting): picked mon.noname-a con 0x7f31240a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:06.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.128+0000 7f312b2e4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f31240a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:06.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.128+0000 7f312b2e4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f312413be90 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:06.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.128+0000 7f312b2e4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f312409b940 con 0x7f312413be90 2024-09-16T04:39:06.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.128+0000 7f312b2e4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f312409ab00 con 0x7f31240a3ff0 2024-09-16T04:39:06.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.128+0000 7f312b2e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:06.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.128+0000 7f312b2e4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f312409b030 con 0x7f31240a3c10 2024-09-16T04:39:06.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.128+0000 7f312b2e4640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:06.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.128+0000 7f312b2e4640 10 monclient(hunting): authenticate will time out at 744.543941s 2024-09-16T04:39:06.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.129+0000 7f3129080640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f31240a3c10 0x7f312409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.129+0000 7f3129881640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f312413be90 0x7f312413e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35332/0 (socket says 172.21.15.46:35332) 2024-09-16T04:39:06.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.129+0000 7f3129080640 10 monclient(hunting): get_auth_request con 0x7f31240a3c10 auth_method 0 2024-09-16T04:39:06.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.129+0000 7f3129881640 1 -- 172.21.15.46:0/2563353504 learned_addr learned my addr 172.21.15.46:0/2563353504 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:06.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.129+0000 7f3129080640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:06.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.129+0000 7f3129080640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.129+0000 7f3129080640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.129+0000 7f3129080640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:06.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.129+0000 7f3129080640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:06.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.129+0000 7f3129080640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:06.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.129+0000 7f3129080640 10 monclient(hunting): handle_auth_done global_id 4407 payload 274 2024-09-16T04:39:06.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.129+0000 7f3129080640 1 -- 172.21.15.46:0/2563353504 >> v1:172.21.15.114:6789/0 conn(0x7f31240a3ff0 legacy=0x7f31240a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:06.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.129+0000 7f3129080640 1 -- 172.21.15.46:0/2563353504 >> v1:172.21.15.46:6789/0 conn(0x7f312413be90 legacy=0x7f312413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.129+0000 7f3129080640 10 monclient: _finish_hunting 0 2024-09-16T04:39:06.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.129+0000 7f3129080640 1 monclient: found mon.noname-a 2024-09-16T04:39:06.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.129+0000 7f3129080640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:06.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.129+0000 7f311bfff640 1 -- 172.21.15.46:0/2563353504 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3267814711 0 0) 0x7f312409b940 con 0x7f312413be90 2024-09-16T04:39:06.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.129+0000 7f3129080640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:06.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.129+0000 7f3129080640 1 -- 172.21.15.46:0/2563353504 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f312409b310 con 0x7f31240a3c10 2024-09-16T04:39:06.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.129+0000 7f311bfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:39:06.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f3129080640 1 --2- 172.21.15.46:0/2563353504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31240a3c10 0x7f312409bcf0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f31200097a0 tx=0x7f312002a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b71d5e4cf5d7801 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:06.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f311bfff640 1 -- 172.21.15.46:0/2563353504 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3120037070 con 0x7f31240a3c10 2024-09-16T04:39:06.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f311bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:06.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f311bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:06.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f311bfff640 10 monclient: dump: 2024-09-16T04:39:06.141 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:06.141 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.141 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.141 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.141 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:06.141 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.141 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:06.141 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:06.141 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:06.141 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f311bfff640 10 monclient: _finish_auth 0 2024-09-16T04:39:06.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f311bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:39:06.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f311bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:06.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f311bfff640 1 -- 172.21.15.46:0/2563353504 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f312002adc0 con 0x7f31240a3c10 2024-09-16T04:39:06.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f312b2e4640 5 monclient: authenticate success, global_id 4407 2024-09-16T04:39:06.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f312b2e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:06.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f311bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:06.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f312b2e4640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:06.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f311bfff640 1 -- 172.21.15.46:0/2563353504 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3120032660 con 0x7f31240a3c10 2024-09-16T04:39:06.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f311bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:06.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f312b2e4640 4 set_mon_vals no callback set 2024-09-16T04:39:06.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f311bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:06.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f311bfff640 10 monclient: dump: 2024-09-16T04:39:06.142 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:06.142 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.142 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.142 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.143 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:06.143 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.143 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:06.143 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:06.143 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:06.143 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f312b2e4640 10 monclient: shutdown 2024-09-16T04:39:06.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f312b2e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:06.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f312b2e4640 1 -- 172.21.15.46:0/2563353504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31240a3c10 msgr2=0x7f312409bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f312b2e4640 1 --2- 172.21.15.46:0/2563353504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31240a3c10 0x7f312409bcf0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f31200097a0 tx=0x7f312002a0b0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f312b2e4640 1 -- 172.21.15.46:0/2563353504 shutdown_connections 2024-09-16T04:39:06.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f312b2e4640 1 --2- 172.21.15.46:0/2563353504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31240a3c10 0x7f312409bcf0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f312b2e4640 1 -- 172.21.15.46:0/2563353504 >> 172.21.15.46:0/2563353504 conn(0x7f31240a6340 msgr2=0x7f312406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:06.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f312b2e4640 1 -- 172.21.15.46:0/2563353504 shutdown_connections 2024-09-16T04:39:06.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.130+0000 7f312b2e4640 1 -- 172.21.15.46:0/2563353504 wait complete. 2024-09-16T04:39:06.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.131+0000 7f312b2e4640 10 monclient: build_initial_monmap 2024-09-16T04:39:06.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.131+0000 7f312b2e4640 10 monclient: monmap: 2024-09-16T04:39:06.144 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:06.144 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:06.144 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:06.144 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:06.144 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:06.144 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.144 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:06.144 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:06.144 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:06.144 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.131+0000 7f312b2e4640 1 Processor -- start 2024-09-16T04:39:06.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.131+0000 7f312b2e4640 1 -- start start 2024-09-16T04:39:06.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.131+0000 7f312b2e4640 10 monclient: init 2024-09-16T04:39:06.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312b2e4640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:06.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312b2e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:39:06.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312b2e4640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f31240a3c10 0x7f31241cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312b2e4640 10 monclient(hunting): picked mon.noname-c con 0x7f31240a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:06.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312b2e4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31240a3ff0 0x7f3124136dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312b2e4640 10 monclient(hunting): picked mon.noname-b con 0x7f31240a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:06.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312b2e4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f312413be90 0x7f3124138ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312b2e4640 10 monclient(hunting): picked mon.noname-a con 0x7f312413be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:06.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312b2e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:06.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312b2e4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f31240a6b40 con 0x7f312413be90 2024-09-16T04:39:06.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312b2e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:06.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312b2e4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f31240a69c0 con 0x7f31240a3ff0 2024-09-16T04:39:06.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312b2e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:06.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312b2e4640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f312413f9a0 con 0x7f31240a3c10 2024-09-16T04:39:06.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312b2e4640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:06.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312b2e4640 10 monclient(hunting): authenticate will time out at 744.547384s 2024-09-16T04:39:06.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129080640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f31240a3c10 0x7f31241cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129881640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f312413be90 0x7f3124138ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129080640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f31240a3c10 0x7f31241cbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:36506/0 (socket says 172.21.15.46:36506) 2024-09-16T04:39:06.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129080640 1 -- 172.21.15.46:0/3674801695 learned_addr learned my addr 172.21.15.46:0/3674801695 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:06.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129080640 10 monclient(hunting): get_auth_request con 0x7f31240a3c10 auth_method 0 2024-09-16T04:39:06.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129080640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:06.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129080640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129080640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129881640 10 monclient(hunting): get_auth_request con 0x7f312413be90 auth_method 0 2024-09-16T04:39:06.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129881640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:06.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129881640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129881640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312887f640 1 --2- 172.21.15.46:0/3674801695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31240a3ff0 0x7f3124136dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129881640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:06.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129881640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:06.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129881640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:06.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129080640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:06.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129080640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:06.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129080640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:06.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312887f640 10 monclient(hunting): get_auth_request con 0x7f31240a3ff0 auth_method 0 2024-09-16T04:39:06.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312887f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:06.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312887f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312887f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129881640 10 monclient(hunting): handle_auth_done global_id 4413 payload 824 2024-09-16T04:39:06.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129881640 1 -- 172.21.15.46:0/3674801695 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f31240a3c10 msgr2=0x7f31241cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129881640 1 --2- 172.21.15.46:0/3674801695 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f31240a3c10 0x7f31241cbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129881640 1 -- 172.21.15.46:0/3674801695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31240a3ff0 msgr2=0x7f3124136dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129881640 1 --2- 172.21.15.46:0/3674801695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31240a3ff0 0x7f3124136dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129881640 10 monclient: _finish_hunting 0 2024-09-16T04:39:06.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129881640 1 monclient: found mon.noname-a 2024-09-16T04:39:06.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129881640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:06.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129881640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:06.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129881640 1 -- 172.21.15.46:0/3674801695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3124120460 con 0x7f312413be90 2024-09-16T04:39:06.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f312887f640 1 --2- 172.21.15.46:0/3674801695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31240a3ff0 0x7f3124136dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:39:06.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.132+0000 7f3129881640 1 --2- 172.21.15.46:0/3674801695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f312413be90 0x7f3124138ea0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f311c00c750 tx=0x7f311c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:06.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f3119ffb640 1 -- 172.21.15.46:0/3674801695 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311c018070 con 0x7f312413be90 2024-09-16T04:39:06.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f3119ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:06.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f3119ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:06.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f3119ffb640 10 monclient: dump: 2024-09-16T04:39:06.149 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:06.149 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.149 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.149 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.149 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:06.149 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.149 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:06.149 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:06.149 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:06.149 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f3119ffb640 10 monclient: _finish_auth 0 2024-09-16T04:39:06.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f3119ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:39:06.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f3119ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:06.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f3119ffb640 1 -- 172.21.15.46:0/3674801695 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f311c004d70 con 0x7f312413be90 2024-09-16T04:39:06.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f312b2e4640 5 monclient: authenticate success, global_id 4413 2024-09-16T04:39:06.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f3119ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:06.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f312b2e4640 10 monclient: _renew_subs 2024-09-16T04:39:06.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f3119ffb640 1 -- 172.21.15.46:0/3674801695 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311c00a880 con 0x7f312413be90 2024-09-16T04:39:06.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f312b2e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:06.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f312b2e4640 1 -- 172.21.15.46:0/3674801695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3124120740 con 0x7f312413be90 2024-09-16T04:39:06.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f3119ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:06.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f311b7fe640 4 set_mon_vals no callback set 2024-09-16T04:39:06.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f3119ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:06.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f3119ffb640 10 monclient: dump: 2024-09-16T04:39:06.150 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:06.151 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.151 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.151 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.151 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:06.151 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.151 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:06.151 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:06.151 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:06.151 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f312b2e4640 10 monclient: _renew_subs 2024-09-16T04:39:06.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f312b2e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:06.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f312b2e4640 1 -- 172.21.15.46:0/3674801695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3124120180 con 0x7f312413be90 2024-09-16T04:39:06.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.133+0000 7f3119ffb640 1 -- 172.21.15.46:0/3674801695 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f311c01b070 con 0x7f312413be90 2024-09-16T04:39:06.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.134+0000 7f3119ffb640 1 --2- 172.21.15.46:0/3674801695 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f31080383b0 0x7f310803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.134+0000 7f3119ffb640 1 -- 172.21.15.46:0/3674801695 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f311c04aa60 con 0x7f312413be90 2024-09-16T04:39:06.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.134+0000 7f312b2e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:06.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.134+0000 7f312b2e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:06.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.134+0000 7f312b2e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:06.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.134+0000 7f312b2e4640 1 -- 172.21.15.46:0/3674801695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f312413aab0 con 0x7f312413be90 2024-09-16T04:39:06.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.134+0000 7f3129080640 1 --2- 172.21.15.46:0/3674801695 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f31080383b0 0x7f310803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.134+0000 7f3129080640 10 monclient: get_auth_request con 0x7f31080383b0 auth_method 0 2024-09-16T04:39:06.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.134+0000 7f3129080640 1 --2- 172.21.15.46:0/3674801695 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f31080383b0 0x7f310803a870 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3120000c00 tx=0x7f312002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:06.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.139+0000 7f3119ffb640 1 -- 172.21.15.46:0/3674801695 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f311c011360 con 0x7f312413be90 2024-09-16T04:39:06.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.139+0000 7f3119ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:06.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.139+0000 7f3119ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:06.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.312+0000 7f312b2e4640 1 -- 172.21.15.46:0/3674801695 --> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f312409bd70 con 0x7f31080383b0 2024-09-16T04:39:06.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.313+0000 7f3119ffb640 1 -- 172.21.15.46:0/3674801695 <== mgr.4099 v2:172.21.15.114:6800/3521677055 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34116 (secure 0 0 0) 0x7f312409bd70 con 0x7f31080383b0 2024-09-16T04:39:06.315 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:39:06.315 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-09-16T04:39:06.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.315+0000 7f312b2e4640 1 -- 172.21.15.46:0/3674801695 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f31080383b0 msgr2=0x7f310803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.315+0000 7f312b2e4640 1 --2- 172.21.15.46:0/3674801695 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f31080383b0 0x7f310803a870 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3120000c00 tx=0x7f312002e000 comp rx=0 tx=0).stop 2024-09-16T04:39:06.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.315+0000 7f312b2e4640 10 monclient: shutdown 2024-09-16T04:39:06.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.315+0000 7f312b2e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:06.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.315+0000 7f312b2e4640 1 -- 172.21.15.46:0/3674801695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f312413be90 msgr2=0x7f3124138ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.315+0000 7f312b2e4640 1 --2- 172.21.15.46:0/3674801695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f312413be90 0x7f3124138ea0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f311c00c750 tx=0x7f311c00cc10 comp rx=0 tx=0).stop 2024-09-16T04:39:06.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.315+0000 7f312b2e4640 1 -- 172.21.15.46:0/3674801695 shutdown_connections 2024-09-16T04:39:06.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.315+0000 7f312b2e4640 1 --2- 172.21.15.46:0/3674801695 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f31080383b0 0x7f310803a870 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.315+0000 7f312b2e4640 1 --2- 172.21.15.46:0/3674801695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f312413be90 0x7f3124138ea0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.315+0000 7f312b2e4640 1 --2- 172.21.15.46:0/3674801695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31240a3ff0 0x7f3124136dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.315+0000 7f312b2e4640 1 --2- 172.21.15.46:0/3674801695 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f31240a3c10 0x7f31241cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.315+0000 7f312b2e4640 1 -- 172.21.15.46:0/3674801695 >> 172.21.15.46:0/3674801695 conn(0x7f31240a6340 msgr2=0x7f31240a2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:06.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.315+0000 7f312b2e4640 1 -- 172.21.15.46:0/3674801695 shutdown_connections 2024-09-16T04:39:06.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.315+0000 7f312b2e4640 1 -- 172.21.15.46:0/3674801695 wait complete. 2024-09-16T04:39:06.327 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-16T04:39:05.348788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":172616,"kb_used_data":2104,"kb_used_omap":9,"kb_used_meta":170358,"kb_avail":566058424,"statfs":{"total":579820584960,"available":579643826176,"internally_reserved":0,"allocated":2154496,"data_stored":1420330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":174447264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001066"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.524013+0000","last_change":"2024-09-16T04:39:00.524013+0000","last_active":"2024-09-16T04:39:00.524013+0000","last_peered":"2024-09-16T04:39:00.524013+0000","last_clean":"2024-09-16T04:39:00.524013+0000","last_became_active":"2024-09-16T04:38:58.756778+0000","last_became_peered":"2024-09-16T04:38:58.756778+0000","last_unstale":"2024-09-16T04:39:00.524013+0000","last_undegraded":"2024-09-16T04:39:00.524013+0000","last_fullsized":"2024-09-16T04:39:00.524013+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:24:15.429766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054536700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.531971+0000","last_change":"2024-09-16T04:39:00.532294+0000","last_active":"2024-09-16T04:39:00.531971+0000","last_peered":"2024-09-16T04:39:00.531971+0000","last_clean":"2024-09-16T04:39:00.531971+0000","last_became_active":"2024-09-16T04:38:58.520473+0000","last_became_peered":"2024-09-16T04:38:58.520473+0000","last_unstale":"2024-09-16T04:39:00.531971+0000","last_undegraded":"2024-09-16T04:39:00.531971+0000","last_fullsized":"2024-09-16T04:39:00.531971+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:15:41.548834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053532500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.523783+0000","last_change":"2024-09-16T04:39:00.523783+0000","last_active":"2024-09-16T04:39:00.523783+0000","last_peered":"2024-09-16T04:39:00.523783+0000","last_clean":"2024-09-16T04:39:00.523783+0000","last_became_active":"2024-09-16T04:38:58.521000+0000","last_became_peered":"2024-09-16T04:38:58.521000+0000","last_unstale":"2024-09-16T04:39:00.523783+0000","last_undegraded":"2024-09-16T04:39:00.523783+0000","last_fullsized":"2024-09-16T04:39:00.523783+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:42.837022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035484999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.532031+0000","last_change":"2024-09-16T04:39:00.532400+0000","last_active":"2024-09-16T04:39:00.532031+0000","last_peered":"2024-09-16T04:39:00.532031+0000","last_clean":"2024-09-16T04:39:00.532031+0000","last_became_active":"2024-09-16T04:38:58.521627+0000","last_became_peered":"2024-09-16T04:38:58.521627+0000","last_unstale":"2024-09-16T04:39:00.532031+0000","last_undegraded":"2024-09-16T04:39:00.532031+0000","last_fullsized":"2024-09-16T04:39:00.532031+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:41.804897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062256500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.530578+0000","last_change":"2024-09-16T04:39:00.530578+0000","last_active":"2024-09-16T04:39:00.530578+0000","last_peered":"2024-09-16T04:39:00.530578+0000","last_clean":"2024-09-16T04:39:00.530578+0000","last_became_active":"2024-09-16T04:38:58.522744+0000","last_became_peered":"2024-09-16T04:38:58.522744+0000","last_unstale":"2024-09-16T04:39:00.530578+0000","last_undegraded":"2024-09-16T04:39:00.530578+0000","last_fullsized":"2024-09-16T04:39:00.530578+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:04:47.450053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000966258,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.523413+0000","last_change":"2024-09-16T04:39:00.523661+0000","last_active":"2024-09-16T04:39:00.523413+0000","last_peered":"2024-09-16T04:39:00.523413+0000","last_clean":"2024-09-16T04:39:00.523413+0000","last_became_active":"2024-09-16T04:38:58.524372+0000","last_became_peered":"2024-09-16T04:38:58.524372+0000","last_unstale":"2024-09-16T04:39:00.523413+0000","last_undegraded":"2024-09-16T04:39:00.523413+0000","last_fullsized":"2024-09-16T04:39:00.523413+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:46.534727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047533599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.523838+0000","last_change":"2024-09-16T04:39:00.523838+0000","last_active":"2024-09-16T04:39:00.523838+0000","last_peered":"2024-09-16T04:39:00.523838+0000","last_clean":"2024-09-16T04:39:00.523838+0000","last_became_active":"2024-09-16T04:38:58.522142+0000","last_became_peered":"2024-09-16T04:38:58.522142+0000","last_unstale":"2024-09-16T04:39:00.523838+0000","last_undegraded":"2024-09-16T04:39:00.523838+0000","last_fullsized":"2024-09-16T04:39:00.523838+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:35.411495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039704200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.529810+0000","last_change":"2024-09-16T04:39:00.530064+0000","last_active":"2024-09-16T04:39:00.529810+0000","last_peered":"2024-09-16T04:39:00.529810+0000","last_clean":"2024-09-16T04:39:00.529810+0000","last_became_active":"2024-09-16T04:38:58.523268+0000","last_became_peered":"2024-09-16T04:38:58.523268+0000","last_unstale":"2024-09-16T04:39:00.529810+0000","last_undegraded":"2024-09-16T04:39:00.529810+0000","last_fullsized":"2024-09-16T04:39:00.529810+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:12:33.958554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043992499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":170,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T04:39:00.523350+0000","last_change":"2024-09-16T04:38:52.501054+0000","last_active":"2024-09-16T04:39:00.523350+0000","last_peered":"2024-09-16T04:39:00.523350+0000","last_clean":"2024-09-16T04:39:00.523350+0000","last_became_active":"2024-09-16T04:38:52.500796+0000","last_became_peered":"2024-09-16T04:38:52.500796+0000","last_unstale":"2024-09-16T04:39:00.523350+0000","last_undegraded":"2024-09-16T04:39:00.523350+0000","last_fullsized":"2024-09-16T04:39:00.523350+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:51.492404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:51.492404+0000","last_clean_scrub_stamp":"2024-09-16T04:38:51.492404+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:36.122039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":180224,"data_stored":60406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":712704,"data_stored":589334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31884,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339956,"statfs":{"total":96636764160,"available":96604114944,"internally_reserved":0,"allocated":712704,"data_stored":589334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":184320,"data_stored":60425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":180224,"data_stored":60406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31376,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340464,"statfs":{"total":96636764160,"available":96604635136,"internally_reserved":0,"allocated":184320,"data_stored":60425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:39:06.328 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-16T04:39:06.328 DEBUG:tasks.mon_thrash:Yielding 2024-09-16T04:39:06.328 INFO:teuthology.run_tasks:Running task exec... 2024-09-16T04:39:06.338 INFO:teuthology.task.exec:Executing custom commands... 2024-09-16T04:39:06.338 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T04:39:06.338 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:39:06.339 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-16T04:39:06.339 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726461546, revive delay: 20.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 20.0 2024-09-16T04:39:06.339 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:39:06.339 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:39:06.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.438+0000 7f352615f640 10 monclient: get_monmap_and_config 2024-09-16T04:39:06.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.438+0000 7f352615f640 10 monclient: build_initial_monmap 2024-09-16T04:39:06.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.438+0000 7f352615f640 10 monclient: monmap: 2024-09-16T04:39:06.440 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:06.440 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.440 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:06.439396+0000 2024-09-16T04:39:06.440 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:06.439396+0000 2024-09-16T04:39:06.440 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:06.440 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.441 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:06.441 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:06.441 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:06.441 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:06.441 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:06.441 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.439+0000 7f352615f640 1 Processor -- start 2024-09-16T04:39:06.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.439+0000 7f352615f640 1 -- start start 2024-09-16T04:39:06.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.439+0000 7f352615f640 10 monclient: init 2024-09-16T04:39:06.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f352615f640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:06.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f352615f640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-16T04:39:06.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f352615f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f35200a3c10 0x7f352009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f352615f640 10 monclient(hunting): picked mon.noname-a con 0x7f35200a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:06.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f352615f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35200a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:06.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f352615f640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f352013be90 0x7f352013e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f352615f640 10 monclient(hunting): picked mon.noname-c con 0x7f352013be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:06.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f352615f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f352009b940 con 0x7f35200a3ff0 2024-09-16T04:39:06.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f352615f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:06.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f352615f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f352009ab00 con 0x7f35200a3c10 2024-09-16T04:39:06.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f352615f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:06.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f352615f640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f352009b030 con 0x7f352013be90 2024-09-16T04:39:06.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f352615f640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:06.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f352615f640 10 monclient(hunting): authenticate will time out at 744.855268s 2024-09-16T04:39:06.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f35200a3c10 0x7f352009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351effd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f35200a3ff0 0x7f35200a43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35348/0 (socket says 172.21.15.46:35348) 2024-09-16T04:39:06.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351effd640 1 -- 172.21.15.46:0/815146920 learned_addr learned my addr 172.21.15.46:0/815146920 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:06.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351f7fe640 10 monclient(hunting): get_auth_request con 0x7f35200a3c10 auth_method 0 2024-09-16T04:39:06.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:06.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:06.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:06.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:06.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351ffff640 1 --2- 172.21.15.46:0/815146920 >> v2:172.21.15.114:3300/0 conn(0x7f352013be90 0x7f352013e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351ffff640 10 monclient(hunting): get_auth_request con 0x7f352013be90 auth_method 0 2024-09-16T04:39:06.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:06.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351f7fe640 10 monclient(hunting): handle_auth_done global_id 4428 payload 274 2024-09-16T04:39:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351f7fe640 1 -- 172.21.15.46:0/815146920 >> v2:172.21.15.114:3300/0 conn(0x7f352013be90 msgr2=0x7f352013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351f7fe640 1 --2- 172.21.15.46:0/815146920 >> v2:172.21.15.114:3300/0 conn(0x7f352013be90 0x7f352013e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351f7fe640 1 -- 172.21.15.46:0/815146920 >> v1:172.21.15.46:6789/0 conn(0x7f35200a3ff0 legacy=0x7f35200a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:39:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351e7fc640 1 -- 172.21.15.46:0/815146920 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3456284120 0 0) 0x7f352009b940 con 0x7f35200a3ff0 2024-09-16T04:39:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351f7fe640 1 monclient: found mon.noname-a 2024-09-16T04:39:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351f7fe640 1 -- 172.21.15.46:0/815146920 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f352009b310 con 0x7f35200a3c10 2024-09-16T04:39:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.440+0000 7f351e7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:39:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.441+0000 7f351f7fe640 1 --2- 172.21.15.46:0/815146920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35200a3c10 0x7f352009bcf0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f352006ee70 tx=0x7f351002a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a8b605c5ea12e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.441+0000 7f351e7fc640 1 -- 172.21.15.46:0/815146920 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3510037070 con 0x7f35200a3c10 2024-09-16T04:39:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.441+0000 7f351e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.441+0000 7f351e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.441+0000 7f351e7fc640 10 monclient: dump: 2024-09-16T04:39:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:06.453 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.441+0000 7f351e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:39:06.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.441+0000 7f351e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:39:06.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.441+0000 7f351e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:06.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.441+0000 7f351e7fc640 1 -- 172.21.15.46:0/815146920 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f351002add0 con 0x7f35200a3c10 2024-09-16T04:39:06.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.441+0000 7f351e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:06.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.441+0000 7f351e7fc640 1 -- 172.21.15.46:0/815146920 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3510032660 con 0x7f35200a3c10 2024-09-16T04:39:06.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.441+0000 7f352615f640 5 monclient: authenticate success, global_id 4428 2024-09-16T04:39:06.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.441+0000 7f352615f640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:06.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.441+0000 7f351e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:06.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.441+0000 7f352615f640 4 set_mon_vals no callback set 2024-09-16T04:39:06.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.441+0000 7f351e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:06.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.441+0000 7f351e7fc640 10 monclient: dump: 2024-09-16T04:39:06.454 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:06.454 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.454 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:06.455 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.442+0000 7f352615f640 10 monclient: shutdown 2024-09-16T04:39:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.442+0000 7f352615f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.442+0000 7f352615f640 1 -- 172.21.15.46:0/815146920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35200a3c10 msgr2=0x7f352009bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.442+0000 7f352615f640 1 --2- 172.21.15.46:0/815146920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35200a3c10 0x7f352009bcf0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f352006ee70 tx=0x7f351002a080 comp rx=0 tx=0).stop 2024-09-16T04:39:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.442+0000 7f352615f640 1 -- 172.21.15.46:0/815146920 shutdown_connections 2024-09-16T04:39:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.442+0000 7f352615f640 1 --2- 172.21.15.46:0/815146920 >> v2:172.21.15.114:3300/0 conn(0x7f352013be90 0x7f352013e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.442+0000 7f352615f640 1 --2- 172.21.15.46:0/815146920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35200a3c10 0x7f352009bcf0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.442+0000 7f352615f640 1 -- 172.21.15.46:0/815146920 >> 172.21.15.46:0/815146920 conn(0x7f35200a6340 msgr2=0x7f352006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.442+0000 7f352615f640 1 -- 172.21.15.46:0/815146920 shutdown_connections 2024-09-16T04:39:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.442+0000 7f352615f640 1 -- 172.21.15.46:0/815146920 wait complete. 2024-09-16T04:39:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.442+0000 7f352615f640 10 monclient: build_initial_monmap 2024-09-16T04:39:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.442+0000 7f352615f640 10 monclient: monmap: 2024-09-16T04:39:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:06.457 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.443+0000 7f352615f640 1 Processor -- start 2024-09-16T04:39:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.443+0000 7f352615f640 1 -- start start 2024-09-16T04:39:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.443+0000 7f352615f640 10 monclient: init 2024-09-16T04:39:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.443+0000 7f352615f640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.443+0000 7f352615f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:39:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.443+0000 7f352615f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35200a3c10 0x7f35201cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.443+0000 7f352615f640 10 monclient(hunting): picked mon.noname-b con 0x7f35200a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.443+0000 7f352615f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35200a3ff0 0x7f3520136f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.443+0000 7f352615f640 10 monclient(hunting): picked mon.noname-a con 0x7f35200a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.443+0000 7f352615f640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f352013be90 0x7f3520139f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.443+0000 7f352615f640 10 monclient(hunting): picked mon.noname-c con 0x7f352013be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.443+0000 7f352615f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.443+0000 7f352615f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f35200a6b40 con 0x7f35200a3ff0 2024-09-16T04:39:06.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.443+0000 7f352615f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:06.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.443+0000 7f352615f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f35200a69c0 con 0x7f35200a3c10 2024-09-16T04:39:06.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.443+0000 7f352615f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:06.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.443+0000 7f352615f640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f352013e7c0 con 0x7f352013be90 2024-09-16T04:39:06.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.443+0000 7f352615f640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:06.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.443+0000 7f352615f640 10 monclient(hunting): authenticate will time out at 744.858913s 2024-09-16T04:39:06.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.443+0000 7f351effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35200a3ff0 0x7f3520136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35200a3ff0 0x7f3520136f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34952/0 (socket says 172.21.15.46:34952) 2024-09-16T04:39:06.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351effd640 1 -- 172.21.15.46:0/2936219296 learned_addr learned my addr 172.21.15.46:0/2936219296 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:06.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351effd640 10 monclient(hunting): get_auth_request con 0x7f35200a3ff0 auth_method 0 2024-09-16T04:39:06.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:06.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351ffff640 1 --2- 172.21.15.46:0/2936219296 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f352013be90 0x7f3520139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351ffff640 10 monclient(hunting): get_auth_request con 0x7f352013be90 auth_method 0 2024-09-16T04:39:06.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:06.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:06.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:06.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:06.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:06.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:06.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:06.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351f7fe640 1 --2- 172.21.15.46:0/2936219296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35200a3c10 0x7f35201cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351f7fe640 10 monclient(hunting): get_auth_request con 0x7f35200a3c10 auth_method 0 2024-09-16T04:39:06.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:06.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351effd640 10 monclient(hunting): handle_auth_done global_id 4434 payload 824 2024-09-16T04:39:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351effd640 1 -- 172.21.15.46:0/2936219296 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f352013be90 msgr2=0x7f3520139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351effd640 1 --2- 172.21.15.46:0/2936219296 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f352013be90 0x7f3520139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351effd640 1 -- 172.21.15.46:0/2936219296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35200a3c10 msgr2=0x7f35201cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351effd640 1 --2- 172.21.15.46:0/2936219296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35200a3c10 0x7f35201cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351effd640 10 monclient: _finish_hunting 0 2024-09-16T04:39:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351effd640 1 monclient: found mon.noname-a 2024-09-16T04:39:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351effd640 1 -- 172.21.15.46:0/2936219296 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35201cc900 con 0x7f35200a3ff0 2024-09-16T04:39:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351effd640 1 --2- 172.21.15.46:0/2936219296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35200a3ff0 0x7f3520136f40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f351400c4d0 tx=0x7f351400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f3507fff640 1 -- 172.21.15.46:0/2936219296 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3514018070 con 0x7f35200a3ff0 2024-09-16T04:39:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f3507fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f3507fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f3507fff640 10 monclient: dump: 2024-09-16T04:39:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:06.461 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f3507fff640 10 monclient: _finish_auth 0 2024-09-16T04:39:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f3507fff640 10 monclient: _check_auth_tickets 2024-09-16T04:39:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f3507fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f352615f640 5 monclient: authenticate success, global_id 4434 2024-09-16T04:39:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f3507fff640 1 -- 172.21.15.46:0/2936219296 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3514007500 con 0x7f35200a3ff0 2024-09-16T04:39:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f3507fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f3507fff640 1 -- 172.21.15.46:0/2936219296 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3514004a20 con 0x7f35200a3ff0 2024-09-16T04:39:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f352615f640 10 monclient: _renew_subs 2024-09-16T04:39:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f352615f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f352615f640 1 -- 172.21.15.46:0/2936219296 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f352011fa60 con 0x7f35200a3ff0 2024-09-16T04:39:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f3507fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f3507fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f3507fff640 10 monclient: dump: 2024-09-16T04:39:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.463 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.463 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:06.463 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.463 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:06.463 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:06.463 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:06.463 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f351d7fa640 4 set_mon_vals no callback set 2024-09-16T04:39:06.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f352615f640 10 monclient: _renew_subs 2024-09-16T04:39:06.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f352615f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:06.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.444+0000 7f352615f640 1 -- 172.21.15.46:0/2936219296 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f352013bbe0 con 0x7f35200a3ff0 2024-09-16T04:39:06.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.445+0000 7f352615f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:06.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.445+0000 7f352615f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:06.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.445+0000 7f352615f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:06.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.445+0000 7f352615f640 1 -- 172.21.15.46:0/2936219296 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f352009bcf0 con 0x7f35200a3ff0 2024-09-16T04:39:06.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.445+0000 7f3507fff640 1 -- 172.21.15.46:0/2936219296 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f351401b070 con 0x7f35200a3ff0 2024-09-16T04:39:06.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.445+0000 7f3507fff640 1 --2- 172.21.15.46:0/2936219296 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f350c0383b0 0x7f350c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.445+0000 7f3507fff640 1 -- 172.21.15.46:0/2936219296 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f351404aa20 con 0x7f35200a3ff0 2024-09-16T04:39:06.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.446+0000 7f351f7fe640 1 --2- 172.21.15.46:0/2936219296 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f350c0383b0 0x7f350c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.446+0000 7f351f7fe640 10 monclient: get_auth_request con 0x7f350c0383b0 auth_method 0 2024-09-16T04:39:06.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.446+0000 7f351f7fe640 1 --2- 172.21.15.46:0/2936219296 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f350c0383b0 0x7f350c03a870 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3510034000 tx=0x7f351002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:06.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.450+0000 7f3507fff640 1 -- 172.21.15.46:0/2936219296 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f351401ae10 con 0x7f35200a3ff0 2024-09-16T04:39:06.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.450+0000 7f3507fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:06.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.450+0000 7f3507fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.704+0000 7f352615f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:39:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.704+0000 7f352615f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:39:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.704+0000 7f352615f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:06.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.704+0000 7f352615f640 1 -- 172.21.15.46:0/2936219296 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3520121410 con 0x7f35200a3ff0 2024-09-16T04:39:06.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.704+0000 7f3507fff640 1 -- 172.21.15.46:0/2936219296 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f3514010900 con 0x7f35200a3ff0 2024-09-16T04:39:06.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.704+0000 7f3507fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:39:06.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.704+0000 7f3507fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:06.706 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":27,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:39:06.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.706+0000 7f352615f640 1 -- 172.21.15.46:0/2936219296 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f350c0383b0 msgr2=0x7f350c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.706+0000 7f352615f640 1 --2- 172.21.15.46:0/2936219296 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f350c0383b0 0x7f350c03a870 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3510034000 tx=0x7f351002e000 comp rx=0 tx=0).stop 2024-09-16T04:39:06.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.706+0000 7f352615f640 10 monclient: shutdown 2024-09-16T04:39:06.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.706+0000 7f352615f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:06.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.706+0000 7f352615f640 1 -- 172.21.15.46:0/2936219296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35200a3ff0 msgr2=0x7f3520136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.706+0000 7f352615f640 1 --2- 172.21.15.46:0/2936219296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35200a3ff0 0x7f3520136f40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f351400c4d0 tx=0x7f351400c990 comp rx=0 tx=0).stop 2024-09-16T04:39:06.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.707+0000 7f352615f640 1 -- 172.21.15.46:0/2936219296 shutdown_connections 2024-09-16T04:39:06.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.707+0000 7f352615f640 1 --2- 172.21.15.46:0/2936219296 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f352013be90 0x7f3520139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.707+0000 7f352615f640 1 --2- 172.21.15.46:0/2936219296 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f350c0383b0 0x7f350c03a870 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.707+0000 7f352615f640 1 --2- 172.21.15.46:0/2936219296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35200a3ff0 0x7f3520136f40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.707+0000 7f352615f640 1 --2- 172.21.15.46:0/2936219296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35200a3c10 0x7f35201cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.707+0000 7f352615f640 1 -- 172.21.15.46:0/2936219296 >> 172.21.15.46:0/2936219296 conn(0x7f35200a6340 msgr2=0x7f35200a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:06.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.707+0000 7f352615f640 1 -- 172.21.15.46:0/2936219296 shutdown_connections 2024-09-16T04:39:06.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.707+0000 7f352615f640 1 -- 172.21.15.46:0/2936219296 wait complete. 2024-09-16T04:39:06.718 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:39:06.718 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:39:06.718 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:39:06.719 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:39:06.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.819+0000 7f49ddf6d640 10 monclient: get_monmap_and_config 2024-09-16T04:39:06.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.819+0000 7f49ddf6d640 10 monclient: build_initial_monmap 2024-09-16T04:39:06.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.819+0000 7f49ddf6d640 10 monclient: monmap: 2024-09-16T04:39:06.821 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:06.821 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.821 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:06.819822+0000 2024-09-16T04:39:06.821 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:06.819822+0000 2024-09-16T04:39:06.821 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:06.821 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.821 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:06.822 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:06.822 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:06.822 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:06.822 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:06.822 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49ddf6d640 1 Processor -- start 2024-09-16T04:39:06.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49ddf6d640 1 -- start start 2024-09-16T04:39:06.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49ddf6d640 10 monclient: init 2024-09-16T04:39:06.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49ddf6d640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49ddf6d640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-16T04:39:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49ddf6d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f49d80a3c10 0x7f49d809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49ddf6d640 10 monclient(hunting): picked mon.noname-c con 0x7f49d80a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49ddf6d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f49d80a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49ddf6d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d813be90 0x7f49d813e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49ddf6d640 10 monclient(hunting): picked mon.noname-b con 0x7f49d813be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49ddf6d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49d809b940 con 0x7f49d80a3ff0 2024-09-16T04:39:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49ddf6d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49ddf6d640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f49d809ab00 con 0x7f49d80a3c10 2024-09-16T04:39:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49ddf6d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49ddf6d640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f49d809b030 con 0x7f49d813be90 2024-09-16T04:39:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49ddf6d640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49ddf6d640 10 monclient(hunting): authenticate will time out at 745.235669s 2024-09-16T04:39:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49d7fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d813be90 0x7f49d813e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49d6ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f49d80a3ff0 0x7f49d80a43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35360/0 (socket says 172.21.15.46:35360) 2024-09-16T04:39:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49d6ffd640 1 -- 172.21.15.46:0/433123328 learned_addr learned my addr 172.21.15.46:0/433123328 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49d7fff640 10 monclient(hunting): get_auth_request con 0x7f49d813be90 auth_method 0 2024-09-16T04:39:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49d77fe640 1 --2- 172.21.15.46:0/433123328 >> v2:172.21.15.114:3300/0 conn(0x7f49d80a3c10 0x7f49d809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49d77fe640 10 monclient(hunting): get_auth_request con 0x7f49d80a3c10 auth_method 0 2024-09-16T04:39:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.820+0000 7f49d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.821+0000 7f49d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:06.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.821+0000 7f49d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:06.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.821+0000 7f49d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:06.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.821+0000 7f49d7fff640 10 monclient(hunting): handle_auth_done global_id 4361 payload 274 2024-09-16T04:39:06.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.821+0000 7f49d67fc640 1 -- 172.21.15.46:0/433123328 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1433847566 0 0) 0x7f49d809b940 con 0x7f49d80a3ff0 2024-09-16T04:39:06.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.821+0000 7f49d7fff640 1 -- 172.21.15.46:0/433123328 >> v2:172.21.15.114:3300/0 conn(0x7f49d80a3c10 msgr2=0x7f49d809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:06.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.821+0000 7f49d7fff640 1 --2- 172.21.15.46:0/433123328 >> v2:172.21.15.114:3300/0 conn(0x7f49d80a3c10 0x7f49d809bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.821+0000 7f49d7fff640 1 -- 172.21.15.46:0/433123328 >> v1:172.21.15.46:6789/0 conn(0x7f49d80a3ff0 legacy=0x7f49d80a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.821+0000 7f49d7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:39:06.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.821+0000 7f49d7fff640 1 monclient: found mon.noname-b 2024-09-16T04:39:06.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.821+0000 7f49d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:06.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.821+0000 7f49d7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:39:06.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.821+0000 7f49d7fff640 1 -- 172.21.15.46:0/433123328 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d809b310 con 0x7f49d813be90 2024-09-16T04:39:06.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.821+0000 7f49d67fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:39:06.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.821+0000 7f49d7fff640 1 --2- 172.21.15.46:0/433123328 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d813be90 0x7f49d813e280 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f49d0009d90 tx=0x7f49d002a530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16f22d4c0f2f5528 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:06.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49d67fc640 1 -- 172.21.15.46:0/433123328 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d0007740 con 0x7f49d813be90 2024-09-16T04:39:06.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:06.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49d67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:06.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49d67fc640 10 monclient: dump: 2024-09-16T04:39:06.837 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:06.837 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.837 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.837 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.837 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:06.837 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.837 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:06.837 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:06.838 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49d67fc640 10 monclient: _finish_auth 0 2024-09-16T04:39:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49d67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:39:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49ddf6d640 5 monclient: authenticate success, global_id 4361 2024-09-16T04:39:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49ddf6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49d67fc640 1 -- 172.21.15.46:0/433123328 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49d00078e0 con 0x7f49d813be90 2024-09-16T04:39:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49d67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49ddf6d640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49ddf6d640 4 set_mon_vals no callback set 2024-09-16T04:39:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49d67fc640 1 -- 172.21.15.46:0/433123328 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d0007cb0 con 0x7f49d813be90 2024-09-16T04:39:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49d67fc640 10 monclient: dump: 2024-09-16T04:39:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:06.839 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49ddf6d640 10 monclient: shutdown 2024-09-16T04:39:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49ddf6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49ddf6d640 1 -- 172.21.15.46:0/433123328 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d813be90 msgr2=0x7f49d813e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49ddf6d640 1 --2- 172.21.15.46:0/433123328 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d813be90 0x7f49d813e280 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f49d0009d90 tx=0x7f49d002a530 comp rx=0 tx=0).stop 2024-09-16T04:39:06.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49ddf6d640 1 -- 172.21.15.46:0/433123328 shutdown_connections 2024-09-16T04:39:06.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49ddf6d640 1 --2- 172.21.15.46:0/433123328 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d813be90 0x7f49d813e280 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49ddf6d640 1 --2- 172.21.15.46:0/433123328 >> v2:172.21.15.114:3300/0 conn(0x7f49d80a3c10 0x7f49d809bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49ddf6d640 1 -- 172.21.15.46:0/433123328 >> 172.21.15.46:0/433123328 conn(0x7f49d80a6340 msgr2=0x7f49d806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:06.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49ddf6d640 1 -- 172.21.15.46:0/433123328 shutdown_connections 2024-09-16T04:39:06.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.822+0000 7f49ddf6d640 1 -- 172.21.15.46:0/433123328 wait complete. 2024-09-16T04:39:06.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.823+0000 7f49ddf6d640 10 monclient: build_initial_monmap 2024-09-16T04:39:06.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.823+0000 7f49ddf6d640 10 monclient: monmap: 2024-09-16T04:39:06.840 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:06.840 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:06.840 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:06.840 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:06.840 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:06.840 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.841 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:06.841 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:06.841 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:06.841 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.823+0000 7f49ddf6d640 1 Processor -- start 2024-09-16T04:39:06.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.823+0000 7f49ddf6d640 1 -- start start 2024-09-16T04:39:06.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.823+0000 7f49ddf6d640 10 monclient: init 2024-09-16T04:39:06.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49ddf6d640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:06.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49ddf6d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:39:06.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49ddf6d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d80a3c10 0x7f49d80a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49ddf6d640 10 monclient(hunting): picked mon.noname-b con 0x7f49d80a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:06.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49ddf6d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d80a3ff0 0x7f49d80a1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49ddf6d640 10 monclient(hunting): picked mon.noname-a con 0x7f49d80a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:06.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49ddf6d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d813be90 0x7f49d80a2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49ddf6d640 10 monclient(hunting): picked mon.noname-c con 0x7f49d813be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:06.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49ddf6d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:06.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49ddf6d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f49d80a6b40 con 0x7f49d80a3ff0 2024-09-16T04:39:06.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49ddf6d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:06.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49ddf6d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f49d80a69c0 con 0x7f49d80a3c10 2024-09-16T04:39:06.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49ddf6d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:06.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49ddf6d640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f49d813e7c0 con 0x7f49d813be90 2024-09-16T04:39:06.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49ddf6d640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:06.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49ddf6d640 10 monclient(hunting): authenticate will time out at 745.239245s 2024-09-16T04:39:06.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d80a3ff0 0x7f49d80a1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d7fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d813be90 0x7f49d80a2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d80a3ff0 0x7f49d80a1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34956/0 (socket says 172.21.15.46:34956) 2024-09-16T04:39:06.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d6ffd640 1 -- 172.21.15.46:0/3413752686 learned_addr learned my addr 172.21.15.46:0/3413752686 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:06.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d77fe640 1 --2- 172.21.15.46:0/3413752686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d80a3c10 0x7f49d80a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d7fff640 10 monclient(hunting): get_auth_request con 0x7f49d813be90 auth_method 0 2024-09-16T04:39:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d77fe640 10 monclient(hunting): get_auth_request con 0x7f49d80a3c10 auth_method 0 2024-09-16T04:39:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d6ffd640 10 monclient(hunting): get_auth_request con 0x7f49d80a3ff0 auth_method 0 2024-09-16T04:39:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:06.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:06.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:06.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:06.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:06.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:06.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:06.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:06.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d77fe640 10 monclient(hunting): handle_auth_done global_id 4420 payload 824 2024-09-16T04:39:06.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d77fe640 1 -- 172.21.15.46:0/3413752686 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d813be90 msgr2=0x7f49d80a2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d77fe640 1 --2- 172.21.15.46:0/3413752686 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d813be90 0x7f49d80a2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d77fe640 1 -- 172.21.15.46:0/3413752686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d80a3ff0 msgr2=0x7f49d80a1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d77fe640 1 --2- 172.21.15.46:0/3413752686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d80a3ff0 0x7f49d80a1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:39:06.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d77fe640 1 monclient: found mon.noname-b 2024-09-16T04:39:06.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:06.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3300/0 2024-09-16T04:39:06.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d77fe640 1 -- 172.21.15.46:0/3413752686 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d8139550 con 0x7f49d80a3c10 2024-09-16T04:39:06.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d7fff640 1 --2- 172.21.15.46:0/3413752686 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d813be90 0x7f49d80a2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:06.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.824+0000 7f49d77fe640 1 --2- 172.21.15.46:0/3413752686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d80a3c10 0x7f49d80a16a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f49c800e850 tx=0x7f49c800ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:06.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.825+0000 7f49bffff640 1 -- 172.21.15.46:0/3413752686 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c8017070 con 0x7f49d80a3c10 2024-09-16T04:39:06.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.825+0000 7f49bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:06.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.825+0000 7f49bffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:06.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.825+0000 7f49bffff640 10 monclient: dump: 2024-09-16T04:39:06.845 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:06.845 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.845 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.845 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.845 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:06.846 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.846 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:06.846 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:06.846 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:06.846 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.825+0000 7f49bffff640 10 monclient: _finish_auth 0 2024-09-16T04:39:06.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.825+0000 7f49bffff640 10 monclient: _check_auth_tickets 2024-09-16T04:39:06.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.825+0000 7f49bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:06.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.825+0000 7f49bffff640 1 -- 172.21.15.46:0/3413752686 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49c800c940 con 0x7f49d80a3c10 2024-09-16T04:39:06.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.825+0000 7f49ddf6d640 5 monclient: authenticate success, global_id 4420 2024-09-16T04:39:06.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.825+0000 7f49ddf6d640 10 monclient: _renew_subs 2024-09-16T04:39:06.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.825+0000 7f49ddf6d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:06.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.825+0000 7f49ddf6d640 1 -- 172.21.15.46:0/3413752686 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49d8139b10 con 0x7f49d80a3c10 2024-09-16T04:39:06.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.825+0000 7f49bffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:06.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.825+0000 7f49bffff640 1 -- 172.21.15.46:0/3413752686 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c800cd10 con 0x7f49d80a3c10 2024-09-16T04:39:06.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.825+0000 7f49bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:06.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.825+0000 7f49bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:06.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.825+0000 7f49bffff640 10 monclient: dump: 2024-09-16T04:39:06.847 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:06.847 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.847 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.847 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.847 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:06.847 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.847 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:06.847 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:06.847 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:06.847 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.825+0000 7f49ddf6d640 10 monclient: _renew_subs 2024-09-16T04:39:06.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.825+0000 7f49d5ffb640 4 set_mon_vals no callback set 2024-09-16T04:39:06.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.825+0000 7f49ddf6d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:06.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.825+0000 7f49ddf6d640 1 -- 172.21.15.46:0/3413752686 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49d813a2a0 con 0x7f49d80a3c10 2024-09-16T04:39:06.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49bffff640 1 -- 172.21.15.46:0/3413752686 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f49c801a070 con 0x7f49d80a3c10 2024-09-16T04:39:06.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49ddf6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:06.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49ddf6d640 1 --2- 172.21.15.46:0/3413752686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d81339a0 0x7f49d8133d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49ddf6d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:06.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49ddf6d640 1 -- 172.21.15.46:0/3413752686 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f49d8139270 con 0x7f49d81339a0 2024-09-16T04:39:06.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49bffff640 1 --2- 172.21.15.46:0/3413752686 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f49c40383b0 0x7f49c403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49d6ffd640 1 --2- 172.21.15.46:0/3413752686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d81339a0 0x7f49d8133d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49bffff640 1 -- 172.21.15.46:0/3413752686 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4738+0+0 (secure 0 0 0) 0x7f49c804b780 con 0x7f49d80a3c10 2024-09-16T04:39:06.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49d6ffd640 10 monclient: get_auth_request con 0x7f49d81339a0 auth_method 0 2024-09-16T04:39:06.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:06.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:06.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:06.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:06.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49d7fff640 1 --2- 172.21.15.46:0/3413752686 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f49c40383b0 0x7f49c403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49d7fff640 10 monclient: get_auth_request con 0x7f49c40383b0 auth_method 0 2024-09-16T04:39:06.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49d6ffd640 10 monclient(hunting): handle_auth_done global_id 4443 payload 824 2024-09-16T04:39:06.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49d6ffd640 1 -- 172.21.15.46:0/3413752686 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f49d80a3620 con 0x7f49d81339a0 2024-09-16T04:39:06.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49d6ffd640 1 --2- 172.21.15.46:0/3413752686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d81339a0 0x7f49d8133d80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f49cc0011e0 tx=0x7f49cc001a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:06.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49bffff640 1 -- 172.21.15.46:0/3413752686 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49cc00f820 con 0x7f49d81339a0 2024-09-16T04:39:06.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:06.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:06.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.826+0000 7f49bffff640 10 monclient: dump: 2024-09-16T04:39:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:06.850 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.827+0000 7f49d7fff640 1 --2- 172.21.15.46:0/3413752686 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f49c40383b0 0x7f49c403a870 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f49d0034000 tx=0x7f49d002f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.827+0000 7f49bffff640 1 -- 172.21.15.46:0/3413752686 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f49cc018070 con 0x7f49d81339a0 2024-09-16T04:39:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.827+0000 7f49bffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.827+0000 7f49bffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.827+0000 7f49bffff640 1 -- 172.21.15.46:0/3413752686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d81339a0 msgr2=0x7f49d8133d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.827+0000 7f49bffff640 1 --2- 172.21.15.46:0/3413752686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d81339a0 0x7f49d8133d80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f49cc0011e0 tx=0x7f49cc001a50 comp rx=0 tx=0).stop 2024-09-16T04:39:06.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.827+0000 7f49bffff640 1 -- 172.21.15.46:0/3413752686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d81339a0 msgr2=0x7f49d8133d80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:06.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.827+0000 7f49bffff640 1 --2- 172.21.15.46:0/3413752686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d81339a0 0x7f49d8133d80 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f49cc0011e0 tx=0x7f49cc001a50 comp rx=0 tx=0).stop 2024-09-16T04:39:06.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.833+0000 7f49ddf6d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:39:06.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.833+0000 7f49ddf6d640 1 --2- 172.21.15.46:0/3413752686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d81342c0 0x7f49d809d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.833+0000 7f49ddf6d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:06.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.833+0000 7f49ddf6d640 1 -- 172.21.15.46:0/3413752686 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f49d809bcf0 con 0x7f49d81342c0 2024-09-16T04:39:06.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.833+0000 7f49d6ffd640 1 --2- 172.21.15.46:0/3413752686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d81342c0 0x7f49d809d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.833+0000 7f49d6ffd640 10 monclient: get_auth_request con 0x7f49d81342c0 auth_method 0 2024-09-16T04:39:06.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.833+0000 7f49d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:06.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.833+0000 7f49d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.833+0000 7f49d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.833+0000 7f49d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:06.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.833+0000 7f49d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:06.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.833+0000 7f49d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:06.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.833+0000 7f49d6ffd640 10 monclient(hunting): handle_auth_done global_id 4446 payload 824 2024-09-16T04:39:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.833+0000 7f49d6ffd640 1 -- 172.21.15.46:0/3413752686 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f49d80a0400 con 0x7f49d81342c0 2024-09-16T04:39:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.833+0000 7f49d6ffd640 1 --2- 172.21.15.46:0/3413752686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d81342c0 0x7f49d809d110 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f49cc015040 tx=0x7f49cc001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49bffff640 1 -- 172.21.15.46:0/3413752686 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49cc01ee10 con 0x7f49d81342c0 2024-09-16T04:39:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49bffff640 10 monclient: dump: 2024-09-16T04:39:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:06.855 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:06.855 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49bffff640 1 -- 172.21.15.46:0/3413752686 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f49cc01ee10 con 0x7f49d81342c0 2024-09-16T04:39:06.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49bffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:39:06.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49bffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:06.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49bffff640 1 -- 172.21.15.46:0/3413752686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d81342c0 msgr2=0x7f49d809d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49bffff640 1 --2- 172.21.15.46:0/3413752686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d81342c0 0x7f49d809d110 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f49cc015040 tx=0x7f49cc001080 comp rx=0 tx=0).stop 2024-09-16T04:39:06.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49bffff640 1 -- 172.21.15.46:0/3413752686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d81342c0 msgr2=0x7f49d809d110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:06.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49bffff640 1 --2- 172.21.15.46:0/3413752686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d81342c0 0x7f49d809d110 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f49cc015040 tx=0x7f49cc001080 comp rx=0 tx=0).stop 2024-09-16T04:39:06.855 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:39:06.855 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:39:06.856 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:39:06.856 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:39:06.856 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 8, 2024-09-16T04:39:06.856 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 32772, 2024-09-16T04:39:06.856 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:39:06.856 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:39:06.856 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:39:06.856 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:39:06.856 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:06.856 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 27, 2024-09-16T04:39:06.856 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:06.856 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:39:06.856 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:39:06.857 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:06.857 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:06.857 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:06.857 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:06.857 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:06.857 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:06.857 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:06.857 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:06.857 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:06.857 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:06.857 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:06.857 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:06.857 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:39:06.857 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:39:06.858 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:06.858 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:06.858 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:06.858 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:06.858 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:06.858 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:06.858 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:06.858 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:06.858 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:06.858 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:06.858 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:06.858 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:06.858 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:06.859 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:39:06.859 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:39:06.859 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:39:06.859 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:39:06.859 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:39:06.859 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:39:06.859 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:06.859 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:06.859 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:39:06.859 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:39:06.859 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:39:06.859 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:39:06.860 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:39:06.860 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:39:06.860 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:39:06.860 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:06.860 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:39:06.860 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:06.860 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:06.860 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:06.860 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:06.861 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:06.861 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:06.861 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:06.861 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:06.861 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:06.861 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:06.861 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:06.861 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:06.861 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:39:06.861 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:06.861 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:39:06.861 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:06.861 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:39:06.861 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:39:06.862 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:06.862 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:06.862 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:06.862 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:06.862 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:39:06.862 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:06.862 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:06.862 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:06.862 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:06.862 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:39:06.862 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:06.862 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:06.862 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:06.862 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:06.863 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:39:06.863 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:39:06.863 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:06.863 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:06.863 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:06.863 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:06.863 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:06.863 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:39:06.863 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:39:06.863 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:06.863 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:06.863 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:06.863 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:06.863 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:39:06.863 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:06.864 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:06.864 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:06.864 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:06.864 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:39:06.864 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:06.864 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:06.864 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:06.864 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:06.864 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:39:06.864 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:39:06.864 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:06.864 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:06.864 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:06.864 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:06.865 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:06.865 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:39:06.865 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:39:06.865 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:06.865 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:06.865 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:06.865 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:06.865 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:39:06.865 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:06.865 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:06.865 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:06.865 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:06.865 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:39:06.865 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:06.866 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:06.866 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:06.866 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:06.866 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:39:06.866 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:39:06.866 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:06.866 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:06.866 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:06.866 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:06.866 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:06.866 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:06.866 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:39:06.866 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:39:06.867 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:06.867 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:06.867 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:06.867 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:39:06.867 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:06.867 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:06.867 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:39:06.867 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:06.867 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:06.867 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:06.867 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:39:06.867 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:06.868 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:06.868 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:39:06.868 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:06.868 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:06.868 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:06.868 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:39:06.868 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:06.868 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:06.868 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:06.868 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:39:06.868 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:39:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49ddf6d640 1 -- 172.21.15.46:0/3413752686 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f49c40383b0 msgr2=0x7f49c403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49ddf6d640 1 --2- 172.21.15.46:0/3413752686 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f49c40383b0 0x7f49c403a870 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f49d0034000 tx=0x7f49d002f000 comp rx=0 tx=0).stop 2024-09-16T04:39:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49ddf6d640 10 monclient: shutdown 2024-09-16T04:39:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49ddf6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49ddf6d640 1 -- 172.21.15.46:0/3413752686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d80a3c10 msgr2=0x7f49d80a16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49ddf6d640 1 --2- 172.21.15.46:0/3413752686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d80a3c10 0x7f49d80a16a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f49c800e850 tx=0x7f49c800ed10 comp rx=0 tx=0).stop 2024-09-16T04:39:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49d7fff640 1 -- 172.21.15.46:0/3413752686 reap_dead start 2024-09-16T04:39:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49d7fff640 1 -- 172.21.15.46:0/3413752686 reap_dead start 2024-09-16T04:39:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49ddf6d640 1 -- 172.21.15.46:0/3413752686 shutdown_connections 2024-09-16T04:39:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49ddf6d640 1 -- 172.21.15.46:0/3413752686 >> 172.21.15.46:0/3413752686 conn(0x7f49d80a6340 msgr2=0x7f49d813c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49ddf6d640 1 -- 172.21.15.46:0/3413752686 shutdown_connections 2024-09-16T04:39:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.834+0000 7f49ddf6d640 1 -- 172.21.15.46:0/3413752686 wait complete. 2024-09-16T04:39:06.871 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:39:06.871 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:39:06.871 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:39:06.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.972+0000 7f56db59e640 10 monclient: get_monmap_and_config 2024-09-16T04:39:06.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.972+0000 7f56db59e640 10 monclient: build_initial_monmap 2024-09-16T04:39:06.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.972+0000 7f56db59e640 10 monclient: monmap: 2024-09-16T04:39:06.973 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:06.973 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.973 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:06.972504+0000 2024-09-16T04:39:06.973 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:06.972504+0000 2024-09-16T04:39:06.973 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:06.974 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.974 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:06.974 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:06.974 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:06.974 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:06.974 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:06.974 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.972+0000 7f56db59e640 1 Processor -- start 2024-09-16T04:39:06.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.972+0000 7f56db59e640 1 -- start start 2024-09-16T04:39:06.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.972+0000 7f56db59e640 10 monclient: init 2024-09-16T04:39:06.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56db59e640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:06.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56db59e640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-16T04:39:06.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56db59e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f56dc136960 0x7f56dc136d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56db59e640 10 monclient(hunting): picked mon.noname-c con 0x7f56dc136960 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:06.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56db59e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f56dc0a32b0 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:06.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56db59e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f56dc137280 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:06.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56db59e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56dc0a1480 con 0x7f56dc0a32b0 2024-09-16T04:39:06.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56db59e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56dc1345e0 con 0x7f56dc137280 2024-09-16T04:39:06.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56db59e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:06.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56db59e640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f56dc0a4e40 con 0x7f56dc136960 2024-09-16T04:39:06.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56db59e640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:06.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56db59e640 10 monclient(hunting): authenticate will time out at 745.388314s 2024-09-16T04:39:06.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56d9d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f56dc0a32b0 0x7f56dc1337c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35374/0 (socket says 172.21.15.46:35374) 2024-09-16T04:39:06.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56d9d9b640 1 -- 172.21.15.46:0/2004151287 learned_addr learned my addr 172.21.15.46:0/2004151287 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:06.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56da59c640 1 --2- 172.21.15.46:0/2004151287 >> v2:172.21.15.114:3300/0 conn(0x7f56dc136960 0x7f56dc136d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56da59c640 10 monclient(hunting): get_auth_request con 0x7f56dc136960 auth_method 0 2024-09-16T04:39:06.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:06.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56da59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56da59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:06.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:06.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:06.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56d959a640 1 -- 172.21.15.46:0/2004151287 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1370608706 0 0) 0x7f56dc0a1480 con 0x7f56dc0a32b0 2024-09-16T04:39:06.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56d959a640 10 monclient(hunting): _negotiate 2024-09-16T04:39:06.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56d959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56d959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56d959a640 10 monclient(hunting): my global_id is 4449 2024-09-16T04:39:06.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56d959a640 1 -- 172.21.15.46:0/2004151287 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56c8003520 con 0x7f56dc0a32b0 2024-09-16T04:39:06.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56d959a640 1 -- 172.21.15.46:0/2004151287 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1106222345 0 0) 0x7f56dc1345e0 con 0x7f56dc137280 2024-09-16T04:39:06.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56d959a640 10 monclient(hunting): _negotiate 2024-09-16T04:39:06.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56d959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56d959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56d959a640 10 monclient(hunting): my global_id is 4426 2024-09-16T04:39:06.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56d959a640 1 -- 172.21.15.46:0/2004151287 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56dc0a1480 con 0x7f56dc137280 2024-09-16T04:39:06.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56da59c640 10 monclient(hunting): handle_auth_done global_id 4423 payload 274 2024-09-16T04:39:06.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56da59c640 1 -- 172.21.15.46:0/2004151287 >> v1:172.21.15.114:6789/0 conn(0x7f56dc137280 legacy=0x7f56dc133ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56da59c640 1 -- 172.21.15.46:0/2004151287 >> v1:172.21.15.46:6789/0 conn(0x7f56dc0a32b0 legacy=0x7f56dc1337c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56da59c640 10 monclient: _finish_hunting 0 2024-09-16T04:39:06.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56da59c640 1 monclient: found mon.noname-c 2024-09-16T04:39:06.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56da59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:06.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56da59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:39:06.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.973+0000 7f56da59c640 1 -- 172.21.15.46:0/2004151287 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56dc0a0b70 con 0x7f56dc136960 2024-09-16T04:39:06.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.974+0000 7f56da59c640 1 --2- 172.21.15.46:0/2004151287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56dc136960 0x7f56dc136d40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009d90 tx=0x7f56d402a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51a3033303c36014 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:06.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.974+0000 7f56d959a640 1 -- 172.21.15.46:0/2004151287 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d4031600 con 0x7f56dc136960 2024-09-16T04:39:06.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.974+0000 7f56d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:06.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.974+0000 7f56d959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:39:06.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.974+0000 7f56d959a640 10 monclient: dump: 2024-09-16T04:39:06.987 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:06.987 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.987 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.988 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.988 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:06.988 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.988 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:06.988 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:06.988 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:06.988 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.974+0000 7f56d959a640 10 monclient: _finish_auth 0 2024-09-16T04:39:06.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.974+0000 7f56d959a640 10 monclient: _check_auth_tickets 2024-09-16T04:39:06.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.974+0000 7f56d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:06.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.974+0000 7f56d959a640 1 -- 172.21.15.46:0/2004151287 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56d40317a0 con 0x7f56dc136960 2024-09-16T04:39:06.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.974+0000 7f56d959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:06.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.974+0000 7f56db59e640 5 monclient: authenticate success, global_id 4423 2024-09-16T04:39:06.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.974+0000 7f56d959a640 1 -- 172.21.15.46:0/2004151287 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d4031b50 con 0x7f56dc136960 2024-09-16T04:39:06.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.974+0000 7f56db59e640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:06.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.974+0000 7f56db59e640 4 set_mon_vals no callback set 2024-09-16T04:39:06.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.974+0000 7f56d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:06.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.974+0000 7f56d959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:06.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.974+0000 7f56d959a640 10 monclient: dump: 2024-09-16T04:39:06.989 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:06.989 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.989 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.989 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.989 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:06.989 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.989 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:06.989 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:06.989 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:06.990 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.975+0000 7f56db59e640 10 monclient: shutdown 2024-09-16T04:39:06.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.975+0000 7f56db59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:06.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.975+0000 7f56db59e640 1 -- 172.21.15.46:0/2004151287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56dc136960 msgr2=0x7f56dc136d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.975+0000 7f56db59e640 1 --2- 172.21.15.46:0/2004151287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56dc136960 0x7f56dc136d40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009d90 tx=0x7f56d402a5d0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.975+0000 7f56db59e640 1 -- 172.21.15.46:0/2004151287 shutdown_connections 2024-09-16T04:39:06.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.975+0000 7f56db59e640 1 --2- 172.21.15.46:0/2004151287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56dc136960 0x7f56dc136d40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.975+0000 7f56db59e640 1 -- 172.21.15.46:0/2004151287 >> 172.21.15.46:0/2004151287 conn(0x7f56dc0a9b80 msgr2=0x7f56dc0a5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:06.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.975+0000 7f56db59e640 1 -- 172.21.15.46:0/2004151287 shutdown_connections 2024-09-16T04:39:06.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.975+0000 7f56db59e640 1 -- 172.21.15.46:0/2004151287 wait complete. 2024-09-16T04:39:06.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.976+0000 7f56db59e640 10 monclient: build_initial_monmap 2024-09-16T04:39:06.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.976+0000 7f56db59e640 10 monclient: monmap: 2024-09-16T04:39:06.990 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:06.990 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:06.991 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:06.991 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:06.991 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:06.991 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.991 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:06.991 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:06.991 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:06.991 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.976+0000 7f56db59e640 1 Processor -- start 2024-09-16T04:39:06.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.976+0000 7f56db59e640 1 -- start start 2024-09-16T04:39:06.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.976+0000 7f56db59e640 10 monclient: init 2024-09-16T04:39:06.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.976+0000 7f56db59e640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:06.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.976+0000 7f56db59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:39:06.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.976+0000 7f56db59e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f56dc0a32b0 0x7f56dc1cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.976+0000 7f56db59e640 10 monclient(hunting): picked mon.noname-c con 0x7f56dc0a32b0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:06.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.976+0000 7f56db59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56dc136960 0x7f56dc1cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.976+0000 7f56db59e640 10 monclient(hunting): picked mon.noname-a con 0x7f56dc136960 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:06.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.976+0000 7f56db59e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56dc137280 0x7f56dc120de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:06.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.976+0000 7f56db59e640 10 monclient(hunting): picked mon.noname-b con 0x7f56dc137280 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:06.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.976+0000 7f56db59e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:06.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.976+0000 7f56db59e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f56dc06e240 con 0x7f56dc136960 2024-09-16T04:39:06.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56db59e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:06.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56db59e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f56dc137730 con 0x7f56dc137280 2024-09-16T04:39:06.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56db59e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:06.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56db59e640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f56dc06e3c0 con 0x7f56dc0a32b0 2024-09-16T04:39:06.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56db59e640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:06.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56db59e640 10 monclient(hunting): authenticate will time out at 745.392025s 2024-09-16T04:39:06.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56dc136960 0x7f56dc1cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56da59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f56dc0a32b0 0x7f56dc1cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56dc136960 0x7f56dc1cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34984/0 (socket says 172.21.15.46:34984) 2024-09-16T04:39:06.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d9d9b640 1 -- 172.21.15.46:0/2926485487 learned_addr learned my addr 172.21.15.46:0/2926485487 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:06.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56da59c640 1 --2- 172.21.15.46:0/2926485487 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f56dc0a32b0 0x7f56dc1cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:36550/0 (socket says 172.21.15.46:36550) 2024-09-16T04:39:06.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d9d9b640 10 monclient(hunting): get_auth_request con 0x7f56dc136960 auth_method 0 2024-09-16T04:39:06.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:06.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56da59c640 10 monclient(hunting): get_auth_request con 0x7f56dc0a32b0 auth_method 0 2024-09-16T04:39:06.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:06.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56da59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56da59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56dad9d640 1 --2- 172.21.15.46:0/2926485487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56dc137280 0x7f56dc120de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:06.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56dad9d640 10 monclient(hunting): get_auth_request con 0x7f56dc137280 auth_method 0 2024-09-16T04:39:06.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:06.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56dad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:06.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56dad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:06.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:06.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:06.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:06.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:06.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:06.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:06.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56dad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:06.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56dad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:06.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56dad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:06.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d9d9b640 10 monclient(hunting): handle_auth_done global_id 4452 payload 824 2024-09-16T04:39:06.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d9d9b640 1 -- 172.21.15.46:0/2926485487 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f56dc0a32b0 msgr2=0x7f56dc1cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d9d9b640 1 --2- 172.21.15.46:0/2926485487 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f56dc0a32b0 0x7f56dc1cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d9d9b640 1 -- 172.21.15.46:0/2926485487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56dc137280 msgr2=0x7f56dc120de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:06.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d9d9b640 1 --2- 172.21.15.46:0/2926485487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56dc137280 0x7f56dc120de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:06.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d9d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:39:06.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d9d9b640 1 monclient: found mon.noname-a 2024-09-16T04:39:06.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:06.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:06.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d9d9b640 1 -- 172.21.15.46:0/2926485487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56dc122ed0 con 0x7f56dc136960 2024-09-16T04:39:06.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56da59c640 1 --2- 172.21.15.46:0/2926485487 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f56dc0a32b0 0x7f56dc1cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:06.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d9d9b640 1 --2- 172.21.15.46:0/2926485487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56dc136960 0x7f56dc1cc420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f56c400c500 tx=0x7f56c400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:06.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d2ffd640 1 -- 172.21.15.46:0/2926485487 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c4017070 con 0x7f56dc136960 2024-09-16T04:39:06.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:06.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:06.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d2ffd640 10 monclient: dump: 2024-09-16T04:39:06.996 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:06.996 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.996 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.996 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.996 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:06.996 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.996 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:06.996 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:06.996 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:06.996 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:39:06.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:39:06.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:06.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d2ffd640 1 -- 172.21.15.46:0/2926485487 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56c4005ce0 con 0x7f56dc136960 2024-09-16T04:39:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56db59e640 5 monclient: authenticate success, global_id 4452 2024-09-16T04:39:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56db59e640 10 monclient: _renew_subs 2024-09-16T04:39:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56db59e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56db59e640 1 -- 172.21.15.46:0/2926485487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56dc124a40 con 0x7f56dc136960 2024-09-16T04:39:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d2ffd640 1 -- 172.21.15.46:0/2926485487 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c40072f0 con 0x7f56dc136960 2024-09-16T04:39:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d2ffd640 10 monclient: dump: 2024-09-16T04:39:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:06.998 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:06.998 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:06.998 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:06.998 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:06.998 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:06.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56d8d99640 4 set_mon_vals no callback set 2024-09-16T04:39:06.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56db59e640 10 monclient: _renew_subs 2024-09-16T04:39:06.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56db59e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:06.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.977+0000 7f56db59e640 1 -- 172.21.15.46:0/2926485487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56dc0a75e0 con 0x7f56dc136960 2024-09-16T04:39:06.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.978+0000 7f56d2ffd640 1 -- 172.21.15.46:0/2926485487 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f56c401a070 con 0x7f56dc136960 2024-09-16T04:39:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.978+0000 7f56db59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.978+0000 7f56db59e640 1 --2- 172.21.15.46:0/2926485487 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f56dc124480 0x7f56dc121320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.978+0000 7f56db59e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.978+0000 7f56db59e640 1 -- 172.21.15.46:0/2926485487 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f56dc123590 con 0x7f56dc124480 2024-09-16T04:39:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.978+0000 7f56da59c640 1 --2- 172.21.15.46:0/2926485487 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f56dc124480 0x7f56dc121320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.978+0000 7f56da59c640 10 monclient: get_auth_request con 0x7f56dc124480 auth_method 0 2024-09-16T04:39:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.978+0000 7f56da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.978+0000 7f56da59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.978+0000 7f56da59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.978+0000 7f56da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.978+0000 7f56da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.978+0000 7f56da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.978+0000 7f56d2ffd640 1 --2- 172.21.15.46:0/2926485487 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f56bc0383b0 0x7f56bc03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.979+0000 7f56d2ffd640 1 -- 172.21.15.46:0/2926485487 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4738+0+0 (secure 0 0 0) 0x7f56c404ac00 con 0x7f56dc136960 2024-09-16T04:39:07.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.979+0000 7f56da59c640 10 monclient(hunting): handle_auth_done global_id 4370 payload 824 2024-09-16T04:39:07.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.979+0000 7f56da59c640 1 -- 172.21.15.46:0/2926485487 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f56dc124860 con 0x7f56dc124480 2024-09-16T04:39:07.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.979+0000 7f56dad9d640 1 --2- 172.21.15.46:0/2926485487 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f56bc0383b0 0x7f56bc03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:07.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.979+0000 7f56dad9d640 10 monclient: get_auth_request con 0x7f56bc0383b0 auth_method 0 2024-09-16T04:39:07.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.979+0000 7f56da59c640 1 --2- 172.21.15.46:0/2926485487 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f56dc124480 0x7f56dc121320 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f56dc06e840 tx=0x7f56d4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:07.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.979+0000 7f56d2ffd640 1 -- 172.21.15.46:0/2926485487 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d4009450 con 0x7f56dc124480 2024-09-16T04:39:07.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.979+0000 7f56d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:07.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.979+0000 7f56d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:07.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.979+0000 7f56d2ffd640 10 monclient: dump: 2024-09-16T04:39:07.001 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:07.001 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:07.001 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.001 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.001 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:07.002 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:07.002 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:07.002 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:07.002 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:07.002 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:07.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.979+0000 7f56dad9d640 1 --2- 172.21.15.46:0/2926485487 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f56bc0383b0 0x7f56bc03a870 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f56cc007fd0 tx=0x7f56cc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:07.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.979+0000 7f56d2ffd640 1 -- 172.21.15.46:0/2926485487 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f56d403dc10 con 0x7f56dc124480 2024-09-16T04:39:07.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.979+0000 7f56d2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:07.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.979+0000 7f56d2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:07.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.979+0000 7f56d2ffd640 1 -- 172.21.15.46:0/2926485487 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f56dc124480 msgr2=0x7f56dc121320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:07.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.979+0000 7f56d2ffd640 1 --2- 172.21.15.46:0/2926485487 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f56dc124480 0x7f56dc121320 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f56dc06e840 tx=0x7f56d4004a40 comp rx=0 tx=0).stop 2024-09-16T04:39:07.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.979+0000 7f56d2ffd640 1 -- 172.21.15.46:0/2926485487 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f56dc124480 msgr2=0x7f56dc121320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:07.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.979+0000 7f56d2ffd640 1 --2- 172.21.15.46:0/2926485487 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f56dc124480 0x7f56dc121320 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:07.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.985+0000 7f56db59e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:39:07.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.985+0000 7f56db59e640 1 --2- 172.21.15.46:0/2926485487 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f56dc121860 0x7f56dc121c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:07.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.985+0000 7f56db59e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:07.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.985+0000 7f56db59e640 1 -- 172.21.15.46:0/2926485487 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f56dc1221b0 con 0x7f56dc121860 2024-09-16T04:39:07.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.985+0000 7f56da59c640 1 --2- 172.21.15.46:0/2926485487 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f56dc121860 0x7f56dc121c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:07.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.985+0000 7f56da59c640 10 monclient: get_auth_request con 0x7f56dc121860 auth_method 0 2024-09-16T04:39:07.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.985+0000 7f56da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:07.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.985+0000 7f56da59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:07.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.985+0000 7f56da59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.985+0000 7f56da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.985+0000 7f56da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.985+0000 7f56da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.986+0000 7f56da59c640 10 monclient(hunting): handle_auth_done global_id 4373 payload 824 2024-09-16T04:39:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.986+0000 7f56da59c640 1 -- 172.21.15.46:0/2926485487 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f56dc067c60 con 0x7f56dc121860 2024-09-16T04:39:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.986+0000 7f56da59c640 1 --2- 172.21.15.46:0/2926485487 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f56dc121860 0x7f56dc121c70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f56dc06dd80 tx=0x7f56d4030000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.986+0000 7f56d2ffd640 1 -- 172.21.15.46:0/2926485487 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d4009450 con 0x7f56dc121860 2024-09-16T04:39:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.986+0000 7f56d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.986+0000 7f56d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.986+0000 7f56d2ffd640 10 monclient: dump: 2024-09-16T04:39:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:07.006 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.986+0000 7f56d2ffd640 1 -- 172.21.15.46:0/2926485487 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f56d404c2b0 con 0x7f56dc121860 2024-09-16T04:39:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.986+0000 7f56d2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:39:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.986+0000 7f56d2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.986+0000 7f56d2ffd640 1 -- 172.21.15.46:0/2926485487 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f56dc121860 msgr2=0x7f56dc121c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.986+0000 7f56d2ffd640 1 --2- 172.21.15.46:0/2926485487 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f56dc121860 0x7f56dc121c70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f56dc06dd80 tx=0x7f56d4030000 comp rx=0 tx=0).stop 2024-09-16T04:39:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.986+0000 7f56d2ffd640 1 -- 172.21.15.46:0/2926485487 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f56dc121860 msgr2=0x7f56dc121c70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.986+0000 7f56d2ffd640 1 --2- 172.21.15.46:0/2926485487 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f56dc121860 0x7f56dc121c70 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f56dc06dd80 tx=0x7f56d4030000 comp rx=0 tx=0).stop 2024-09-16T04:39:07.007 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:39:07.007 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:39:07.007 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:39:07.007 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:39:07.007 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 8, 2024-09-16T04:39:07.007 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 32924, 2024-09-16T04:39:07.007 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:39:07.007 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:39:07.007 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:39:07.007 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:39:07.007 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:07.007 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 27, 2024-09-16T04:39:07.008 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:07.008 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:39:07.008 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:39:07.008 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:07.008 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:07.008 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:07.008 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:07.008 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:07.008 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:07.008 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:07.008 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:07.008 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:07.008 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:07.008 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:07.009 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:07.009 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:39:07.009 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:39:07.009 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:07.009 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:07.009 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:07.009 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:07.009 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:07.009 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:07.009 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:07.009 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:07.009 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:07.009 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:07.010 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:07.010 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:07.010 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.010 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:39:07.010 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:39:07.010 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:39:07.010 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:39:07.010 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:39:07.010 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:39:07.010 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:07.010 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:07.010 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:39:07.010 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:39:07.010 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:39:07.011 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:39:07.011 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:39:07.011 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:39:07.011 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:39:07.011 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:07.011 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:39:07.011 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:07.011 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:07.011 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:07.011 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:07.011 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:07.011 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:07.011 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:07.011 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:07.012 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:07.012 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:07.012 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:07.012 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:07.012 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:39:07.012 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.012 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:39:07.012 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.012 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:39:07.012 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:39:07.012 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:07.012 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:07.012 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.013 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:07.013 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:39:07.013 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:07.013 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.013 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.013 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:07.013 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:39:07.013 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:07.013 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:07.013 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:07.013 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.014 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:39:07.014 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:39:07.014 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:07.014 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:07.014 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:07.014 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.014 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.014 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:39:07.014 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:39:07.014 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:07.014 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:07.014 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.014 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:07.014 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:39:07.015 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:07.015 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.015 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.015 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:07.015 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:39:07.015 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:07.015 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:07.015 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:07.015 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.015 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:39:07.015 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:39:07.015 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:07.015 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:07.016 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:07.016 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.016 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.016 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:39:07.016 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:39:07.016 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:07.016 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:07.016 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.016 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:07.016 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:39:07.016 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:07.016 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.016 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.017 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:07.017 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:39:07.017 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:07.017 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:07.017 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:07.017 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.017 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:39:07.017 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:39:07.017 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:07.017 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:07.017 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:07.017 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:07.017 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:07.018 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.018 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:39:07.018 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:39:07.018 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.018 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:07.018 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:07.018 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:39:07.018 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:07.018 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:07.018 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:39:07.018 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.018 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:07.018 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:07.018 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:39:07.019 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:07.019 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:07.019 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:39:07.019 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.019 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:07.019 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:07.019 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:39:07.019 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:07.019 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:07.019 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.019 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:39:07.019 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:39:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.986+0000 7f56db59e640 1 -- 172.21.15.46:0/2926485487 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f56bc0383b0 msgr2=0x7f56bc03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.986+0000 7f56db59e640 1 --2- 172.21.15.46:0/2926485487 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f56bc0383b0 0x7f56bc03a870 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f56cc007fd0 tx=0x7f56cc009040 comp rx=0 tx=0).stop 2024-09-16T04:39:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.986+0000 7f56db59e640 10 monclient: shutdown 2024-09-16T04:39:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.986+0000 7f56db59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.986+0000 7f56db59e640 1 -- 172.21.15.46:0/2926485487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56dc136960 msgr2=0x7f56dc1cc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.986+0000 7f56db59e640 1 --2- 172.21.15.46:0/2926485487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56dc136960 0x7f56dc1cc420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f56c400c500 tx=0x7f56c400c9c0 comp rx=0 tx=0).stop 2024-09-16T04:39:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.986+0000 7f56dad9d640 1 -- 172.21.15.46:0/2926485487 reap_dead start 2024-09-16T04:39:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.987+0000 7f56dad9d640 1 -- 172.21.15.46:0/2926485487 reap_dead start 2024-09-16T04:39:07.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.987+0000 7f56db59e640 1 -- 172.21.15.46:0/2926485487 shutdown_connections 2024-09-16T04:39:07.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.987+0000 7f56db59e640 1 -- 172.21.15.46:0/2926485487 >> 172.21.15.46:0/2926485487 conn(0x7f56dc0a9b80 msgr2=0x7f56dc0a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:07.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.987+0000 7f56db59e640 1 -- 172.21.15.46:0/2926485487 shutdown_connections 2024-09-16T04:39:07.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:06.987+0000 7f56db59e640 1 -- 172.21.15.46:0/2926485487 wait complete. 2024-09-16T04:39:07.022 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:39:07.022 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:39:07.022 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:39:07.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.123+0000 7f2b761b7640 10 monclient: get_monmap_and_config 2024-09-16T04:39:07.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.123+0000 7f2b761b7640 10 monclient: build_initial_monmap 2024-09-16T04:39:07.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.123+0000 7f2b761b7640 10 monclient: monmap: 2024-09-16T04:39:07.125 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:07.125 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:07.125 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:07.124092+0000 2024-09-16T04:39:07.125 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:07.124092+0000 2024-09-16T04:39:07.125 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:07.125 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:07.125 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:07.125 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:07.125 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:07.125 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:07.125 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:07.126 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:07.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.124+0000 7f2b761b7640 1 Processor -- start 2024-09-16T04:39:07.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.124+0000 7f2b761b7640 1 -- start start 2024-09-16T04:39:07.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.124+0000 7f2b761b7640 10 monclient: init 2024-09-16T04:39:07.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.124+0000 7f2b761b7640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:07.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.124+0000 7f2b761b7640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-16T04:39:07.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.124+0000 7f2b761b7640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2b700a3c10 0x7f2b7009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:07.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.124+0000 7f2b761b7640 10 monclient(hunting): picked mon.noname-c con 0x7f2b700a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:07.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.124+0000 7f2b761b7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2b700a3ff0 0x7f2b700a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:07.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.124+0000 7f2b761b7640 10 monclient(hunting): picked mon.noname-a con 0x7f2b700a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:07.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.124+0000 7f2b761b7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2b701369d0 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:07.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.124+0000 7f2b761b7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b7009b940 con 0x7f2b701369d0 2024-09-16T04:39:07.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.124+0000 7f2b761b7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:07.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.124+0000 7f2b761b7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2b7009ab00 con 0x7f2b700a3ff0 2024-09-16T04:39:07.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.124+0000 7f2b761b7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:07.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.124+0000 7f2b761b7640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f2b7009b030 con 0x7f2b700a3c10 2024-09-16T04:39:07.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.124+0000 7f2b761b7640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:07.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.124+0000 7f2b761b7640 10 monclient(hunting): authenticate will time out at 745.539962s 2024-09-16T04:39:07.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2b700a3ff0 0x7f2b700a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:07.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2b701369d0 0x7f2b70136e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35390/0 (socket says 172.21.15.46:35390) 2024-09-16T04:39:07.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6ffff640 1 -- 172.21.15.46:0/2880922182 learned_addr learned my addr 172.21.15.46:0/2880922182 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:07.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6effd640 10 monclient(hunting): get_auth_request con 0x7f2b700a3ff0 auth_method 0 2024-09-16T04:39:07.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:07.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:07.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:07.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:07.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:07.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:07.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6f7fe640 1 --2- 172.21.15.46:0/2880922182 >> v2:172.21.15.114:3300/0 conn(0x7f2b700a3c10 0x7f2b7009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:07.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6f7fe640 10 monclient(hunting): get_auth_request con 0x7f2b700a3c10 auth_method 0 2024-09-16T04:39:07.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:07.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:07.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:07.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6effd640 10 monclient(hunting): handle_auth_done global_id 4455 payload 274 2024-09-16T04:39:07.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6effd640 1 -- 172.21.15.46:0/2880922182 >> v2:172.21.15.114:3300/0 conn(0x7f2b700a3c10 msgr2=0x7f2b7009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:07.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6effd640 1 --2- 172.21.15.46:0/2880922182 >> v2:172.21.15.114:3300/0 conn(0x7f2b700a3c10 0x7f2b7009bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:07.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6effd640 1 -- 172.21.15.46:0/2880922182 >> v1:172.21.15.46:6789/0 conn(0x7f2b701369d0 legacy=0x7f2b70136e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:07.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6effd640 10 monclient: _finish_hunting 0 2024-09-16T04:39:07.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6effd640 1 monclient: found mon.noname-a 2024-09-16T04:39:07.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:07.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:07.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6effd640 1 -- 172.21.15.46:0/2880922182 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b7009b310 con 0x7f2b700a3ff0 2024-09-16T04:39:07.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6e7fc640 1 -- 172.21.15.46:0/2880922182 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3421630577 0 0) 0x7f2b7009b940 con 0x7f2b701369d0 2024-09-16T04:39:07.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6e7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:39:07.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.125+0000 7f2b6f7fe640 1 --2- 172.21.15.46:0/2880922182 >> v2:172.21.15.114:3300/0 conn(0x7f2b700a3c10 0x7f2b7009bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:39:07.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b6effd640 1 --2- 172.21.15.46:0/2880922182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b700a3ff0 0x7f2b700a43d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2b70068240 tx=0x7f2b5802a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f6f9581afc52caa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:07.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b6e7fc640 1 -- 172.21.15.46:0/2880922182 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b58037070 con 0x7f2b700a3ff0 2024-09-16T04:39:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b6e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b6e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b6e7fc640 10 monclient: dump: 2024-09-16T04:39:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:07.139 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b6e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:39:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b6e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:39:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b761b7640 5 monclient: authenticate success, global_id 4455 2024-09-16T04:39:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b6e7fc640 1 -- 172.21.15.46:0/2880922182 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b5802add0 con 0x7f2b700a3ff0 2024-09-16T04:39:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b761b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b6e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b761b7640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b761b7640 4 set_mon_vals no callback set 2024-09-16T04:39:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b6e7fc640 1 -- 172.21.15.46:0/2880922182 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b58032660 con 0x7f2b700a3ff0 2024-09-16T04:39:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b6e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b6e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b6e7fc640 10 monclient: dump: 2024-09-16T04:39:07.141 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:07.141 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:07.141 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.141 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.141 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:07.141 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:07.141 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:07.141 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:07.141 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:07.141 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:07.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b761b7640 10 monclient: shutdown 2024-09-16T04:39:07.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b761b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:07.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b761b7640 1 -- 172.21.15.46:0/2880922182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b700a3ff0 msgr2=0x7f2b700a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b761b7640 1 --2- 172.21.15.46:0/2880922182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b700a3ff0 0x7f2b700a43d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2b70068240 tx=0x7f2b5802a080 comp rx=0 tx=0).stop 2024-09-16T04:39:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b761b7640 1 -- 172.21.15.46:0/2880922182 shutdown_connections 2024-09-16T04:39:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b761b7640 1 --2- 172.21.15.46:0/2880922182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b700a3ff0 0x7f2b700a43d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b761b7640 1 --2- 172.21.15.46:0/2880922182 >> v2:172.21.15.114:3300/0 conn(0x7f2b700a3c10 0x7f2b7009bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b761b7640 1 -- 172.21.15.46:0/2880922182 >> 172.21.15.46:0/2880922182 conn(0x7f2b700a6340 msgr2=0x7f2b7006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b761b7640 1 -- 172.21.15.46:0/2880922182 shutdown_connections 2024-09-16T04:39:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.126+0000 7f2b761b7640 1 -- 172.21.15.46:0/2880922182 wait complete. 2024-09-16T04:39:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.127+0000 7f2b761b7640 10 monclient: build_initial_monmap 2024-09-16T04:39:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.127+0000 7f2b761b7640 10 monclient: monmap: 2024-09-16T04:39:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:07.143 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:07.143 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:07.143 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:07.143 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:07.143 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:07.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.127+0000 7f2b761b7640 1 Processor -- start 2024-09-16T04:39:07.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.127+0000 7f2b761b7640 1 -- start start 2024-09-16T04:39:07.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.127+0000 7f2b761b7640 10 monclient: init 2024-09-16T04:39:07.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b761b7640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:07.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b761b7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:39:07.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b761b7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b700a3c10 0x7f2b701cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:07.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b761b7640 10 monclient(hunting): picked mon.noname-a con 0x7f2b700a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:07.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b761b7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b700a3ff0 0x7f2b701cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:07.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b761b7640 10 monclient(hunting): picked mon.noname-b con 0x7f2b700a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:07.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b761b7640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2b701369d0 0x7f2b70123020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:07.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b761b7640 10 monclient(hunting): picked mon.noname-c con 0x7f2b701369d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:07.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b761b7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:07.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b761b7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2b700a4910 con 0x7f2b700a3c10 2024-09-16T04:39:07.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b761b7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:07.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b761b7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2b700a69c0 con 0x7f2b700a3ff0 2024-09-16T04:39:07.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b761b7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:07.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b761b7640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f2b70126f70 con 0x7f2b701369d0 2024-09-16T04:39:07.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b761b7640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:07.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b761b7640 10 monclient(hunting): authenticate will time out at 745.543401s 2024-09-16T04:39:07.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b700a3c10 0x7f2b701cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:07.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b700a3c10 0x7f2b701cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35006/0 (socket says 172.21.15.46:35006) 2024-09-16T04:39:07.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6ffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2b701369d0 0x7f2b70123020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:07.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6f7fe640 1 -- 172.21.15.46:0/3229999194 learned_addr learned my addr 172.21.15.46:0/3229999194 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:07.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6f7fe640 10 monclient(hunting): get_auth_request con 0x7f2b700a3c10 auth_method 0 2024-09-16T04:39:07.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:07.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:07.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:07.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6ffff640 10 monclient(hunting): get_auth_request con 0x7f2b701369d0 auth_method 0 2024-09-16T04:39:07.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6effd640 1 --2- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b700a3ff0 0x7f2b701cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:07.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:07.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:07.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:07.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6effd640 10 monclient(hunting): get_auth_request con 0x7f2b700a3ff0 auth_method 0 2024-09-16T04:39:07.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:07.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:07.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:07.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:07.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:07.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:07.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:07.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:07.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:07.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:07.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:07.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:07.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6f7fe640 10 monclient(hunting): handle_auth_done global_id 4461 payload 824 2024-09-16T04:39:07.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6f7fe640 1 -- 172.21.15.46:0/3229999194 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2b701369d0 msgr2=0x7f2b70123020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:07.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6f7fe640 1 --2- 172.21.15.46:0/3229999194 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2b701369d0 0x7f2b70123020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:07.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6f7fe640 1 -- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b700a3ff0 msgr2=0x7f2b701cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:07.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6f7fe640 1 --2- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b700a3ff0 0x7f2b701cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:07.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:39:07.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6f7fe640 1 monclient: found mon.noname-a 2024-09-16T04:39:07.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:07.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:07.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.128+0000 7f2b6f7fe640 1 -- 172.21.15.46:0/3229999194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b70126c90 con 0x7f2b700a3c10 2024-09-16T04:39:07.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b6f7fe640 1 --2- 172.21.15.46:0/3229999194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b700a3c10 0x7f2b701cbee0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000e760 tx=0x7f2b6000ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:07.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b67fff640 1 -- 172.21.15.46:0/3229999194 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6000f040 con 0x7f2b700a3c10 2024-09-16T04:39:07.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:07.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b67fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:07.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b67fff640 10 monclient: dump: 2024-09-16T04:39:07.147 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:07.147 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:07.147 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.147 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.147 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:07.148 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b67fff640 10 monclient: _finish_auth 0 2024-09-16T04:39:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b67fff640 10 monclient: _check_auth_tickets 2024-09-16T04:39:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b67fff640 1 -- 172.21.15.46:0/3229999194 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b60004230 con 0x7f2b700a3c10 2024-09-16T04:39:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b67fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b67fff640 1 -- 172.21.15.46:0/3229999194 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6000ee70 con 0x7f2b700a3c10 2024-09-16T04:39:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b67fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b67fff640 10 monclient: dump: 2024-09-16T04:39:07.149 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:07.149 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:07.149 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.149 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.149 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:07.149 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:07.149 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:07.149 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:07.149 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:07.149 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:07.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b761b7640 5 monclient: authenticate success, global_id 4461 2024-09-16T04:39:07.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b761b7640 10 monclient: _renew_subs 2024-09-16T04:39:07.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b761b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:07.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b761b7640 1 -- 172.21.15.46:0/3229999194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b700a2900 con 0x7f2b700a3c10 2024-09-16T04:39:07.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b6d7fa640 4 set_mon_vals no callback set 2024-09-16T04:39:07.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b761b7640 10 monclient: _renew_subs 2024-09-16T04:39:07.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b761b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:07.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.129+0000 7f2b761b7640 1 -- 172.21.15.46:0/3229999194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b701269b0 con 0x7f2b700a3c10 2024-09-16T04:39:07.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.130+0000 7f2b761b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:07.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.130+0000 7f2b761b7640 1 --2- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70123560 0x7f2b70123940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:07.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.130+0000 7f2b761b7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:07.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.130+0000 7f2b761b7640 1 -- 172.21.15.46:0/3229999194 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2b700a4a90 con 0x7f2b70123560 2024-09-16T04:39:07.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.130+0000 7f2b67fff640 1 -- 172.21.15.46:0/3229999194 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f2b6000c670 con 0x7f2b700a3c10 2024-09-16T04:39:07.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.130+0000 7f2b6effd640 1 --2- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70123560 0x7f2b70123940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:07.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.130+0000 7f2b6effd640 10 monclient: get_auth_request con 0x7f2b70123560 auth_method 0 2024-09-16T04:39:07.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.130+0000 7f2b6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:07.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.130+0000 7f2b6effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:07.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.130+0000 7f2b6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:07.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.130+0000 7f2b67fff640 1 --2- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2b500383b0 0x7f2b5003a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:07.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.130+0000 7f2b67fff640 1 -- 172.21.15.46:0/3229999194 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4738+0+0 (secure 0 0 0) 0x7f2b6004b460 con 0x7f2b700a3c10 2024-09-16T04:39:07.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.130+0000 7f2b6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:07.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.130+0000 7f2b6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:07.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.130+0000 7f2b6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:07.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.130+0000 7f2b6ffff640 1 --2- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2b500383b0 0x7f2b5003a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:07.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.130+0000 7f2b6ffff640 10 monclient: get_auth_request con 0x7f2b500383b0 auth_method 0 2024-09-16T04:39:07.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.130+0000 7f2b6effd640 10 monclient(hunting): handle_auth_done global_id 4438 payload 824 2024-09-16T04:39:07.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.130+0000 7f2b6effd640 1 -- 172.21.15.46:0/3229999194 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b701257e0 con 0x7f2b70123560 2024-09-16T04:39:07.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.130+0000 7f2b6effd640 1 --2- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70123560 0x7f2b70123940 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2b7006ee70 tx=0x7f2b580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:07.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.130+0000 7f2b6ffff640 1 --2- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2b500383b0 0x7f2b5003a870 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2b68001820 tx=0x7f2b680017b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:07.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.131+0000 7f2b67fff640 1 -- 172.21.15.46:0/3229999194 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b58043070 con 0x7f2b70123560 2024-09-16T04:39:07.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.131+0000 7f2b67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:07.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.131+0000 7f2b67fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:07.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.131+0000 7f2b67fff640 10 monclient: dump: 2024-09-16T04:39:07.152 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:07.152 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:07.152 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.152 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.152 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:07.152 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:07.152 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:07.152 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:07.152 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:07.152 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:07.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.131+0000 7f2b67fff640 1 -- 172.21.15.46:0/3229999194 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2b5803c440 con 0x7f2b70123560 2024-09-16T04:39:07.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.131+0000 7f2b67fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:07.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.131+0000 7f2b67fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:07.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.131+0000 7f2b67fff640 1 -- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70123560 msgr2=0x7f2b70123940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:07.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.131+0000 7f2b67fff640 1 --2- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70123560 0x7f2b70123940 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2b7006ee70 tx=0x7f2b580046c0 comp rx=0 tx=0).stop 2024-09-16T04:39:07.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.131+0000 7f2b67fff640 1 -- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70123560 msgr2=0x7f2b70123940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:07.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.131+0000 7f2b67fff640 1 --2- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70123560 0x7f2b70123940 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2b7006ee70 tx=0x7f2b580046c0 comp rx=0 tx=0).stop 2024-09-16T04:39:07.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.137+0000 7f2b761b7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:39:07.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.137+0000 7f2b761b7640 1 --2- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70123e80 0x7f2b7011f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:07.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.137+0000 7f2b761b7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:07.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.137+0000 7f2b761b7640 1 -- 172.21.15.46:0/3229999194 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7009bcf0 con 0x7f2b70123e80 2024-09-16T04:39:07.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.137+0000 7f2b6effd640 1 --2- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70123e80 0x7f2b7011f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:07.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.137+0000 7f2b6effd640 10 monclient: get_auth_request con 0x7f2b70123e80 auth_method 0 2024-09-16T04:39:07.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.137+0000 7f2b6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:07.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.137+0000 7f2b6effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:07.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.137+0000 7f2b6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:07.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.137+0000 7f2b6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:07.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.137+0000 7f2b6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:07.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.137+0000 7f2b6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:07.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.137+0000 7f2b6effd640 10 monclient(hunting): handle_auth_done global_id 4441 payload 824 2024-09-16T04:39:07.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.137+0000 7f2b6effd640 1 -- 172.21.15.46:0/3229999194 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2b701250e0 con 0x7f2b70123e80 2024-09-16T04:39:07.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b6effd640 1 --2- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70123e80 0x7f2b7011f860 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2b5802a0c0 tx=0x7f2b58008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:07.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b67fff640 1 -- 172.21.15.46:0/3229999194 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b58037040 con 0x7f2b70123e80 2024-09-16T04:39:07.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:07.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b67fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:07.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b67fff640 10 monclient: dump: 2024-09-16T04:39:07.156 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:07.156 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:07.156 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.156 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.156 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:07.157 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:07.157 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:07.157 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:07.157 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:07.157 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:07.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b67fff640 1 -- 172.21.15.46:0/3229999194 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7f2b58003810 con 0x7f2b70123e80 2024-09-16T04:39:07.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b67fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:39:07.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b67fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:07.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b67fff640 1 -- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70123e80 msgr2=0x7f2b7011f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:07.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b67fff640 1 --2- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70123e80 0x7f2b7011f860 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2b5802a0c0 tx=0x7f2b58008030 comp rx=0 tx=0).stop 2024-09-16T04:39:07.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b67fff640 1 -- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70123e80 msgr2=0x7f2b7011f860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:07.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b67fff640 1 --2- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70123e80 0x7f2b7011f860 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2b5802a0c0 tx=0x7f2b58008030 comp rx=0 tx=0).stop 2024-09-16T04:39:07.158 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:39:07.158 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:39:07.158 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:39:07.158 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:39:07.158 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 8, 2024-09-16T04:39:07.158 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 33040, 2024-09-16T04:39:07.158 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:39:07.158 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:39:07.158 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:39:07.158 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:39:07.158 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:07.158 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 27, 2024-09-16T04:39:07.159 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:07.159 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:39:07.159 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:39:07.159 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:07.159 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:07.159 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:07.159 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:07.159 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:07.159 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:07.159 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:07.159 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:07.159 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:07.159 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:07.159 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:07.160 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:07.160 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:39:07.160 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:39:07.160 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:07.160 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:07.160 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:07.160 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:07.160 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:07.160 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:07.160 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:07.160 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:07.160 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:07.160 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:07.160 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:07.161 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:07.161 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.161 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:39:07.161 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:39:07.161 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:39:07.161 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:39:07.161 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:39:07.161 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:39:07.161 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:07.161 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:07.162 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:39:07.162 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:39:07.162 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:39:07.162 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:39:07.162 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:39:07.162 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:39:07.162 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:39:07.162 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:07.162 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:39:07.162 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:07.162 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:07.162 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:07.162 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:07.163 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:07.163 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:07.163 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:07.163 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:07.163 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:07.163 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:07.163 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:07.163 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:07.163 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:39:07.163 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.163 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:39:07.163 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.163 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:39:07.163 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:39:07.164 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:07.164 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:07.164 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.164 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:07.164 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:39:07.164 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:07.164 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.164 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.164 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:07.164 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:39:07.164 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:07.164 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:07.164 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:07.164 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.165 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:39:07.165 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:39:07.165 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:07.165 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:07.165 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:07.165 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.165 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.165 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:39:07.165 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:39:07.165 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:07.165 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:07.165 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.165 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:07.166 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:39:07.166 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:07.166 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.166 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.166 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:07.166 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:39:07.166 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:07.166 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:07.166 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:07.166 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.166 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:39:07.166 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:39:07.166 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:07.166 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:07.166 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:07.167 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.167 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.167 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:39:07.167 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:39:07.167 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:07.167 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:07.167 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.167 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:07.167 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:39:07.167 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:07.167 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.167 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.167 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:07.167 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:39:07.168 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:07.168 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:07.168 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:07.168 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.168 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:39:07.168 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:39:07.168 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:07.168 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:07.168 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:07.168 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:07.168 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:07.168 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.168 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:39:07.168 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:39:07.169 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.169 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:07.169 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:07.169 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:39:07.169 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:07.169 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:07.169 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:39:07.169 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.169 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:07.169 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:07.169 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:39:07.169 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:07.169 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:07.169 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:39:07.170 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.170 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:07.170 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:07.170 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6 2024-09-16T04:39:07.170 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:07.170 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:07.170 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:39:07.170 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:07.170 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:07.170 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:07.170 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:39:07.170 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:07.170 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:07.170 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:07.171 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:39:07.171 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:39:07.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b761b7640 1 -- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2b500383b0 msgr2=0x7f2b5003a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:07.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b761b7640 1 --2- 172.21.15.46:0/3229999194 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2b500383b0 0x7f2b5003a870 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2b68001820 tx=0x7f2b680017b0 comp rx=0 tx=0).stop 2024-09-16T04:39:07.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b761b7640 10 monclient: shutdown 2024-09-16T04:39:07.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b761b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:07.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b761b7640 1 -- 172.21.15.46:0/3229999194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b700a3c10 msgr2=0x7f2b701cbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:07.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b761b7640 1 --2- 172.21.15.46:0/3229999194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b700a3c10 0x7f2b701cbee0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000e760 tx=0x7f2b6000ec20 comp rx=0 tx=0).stop 2024-09-16T04:39:07.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b6ffff640 1 -- 172.21.15.46:0/3229999194 reap_dead start 2024-09-16T04:39:07.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b6ffff640 1 -- 172.21.15.46:0/3229999194 reap_dead start 2024-09-16T04:39:07.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b761b7640 1 -- 172.21.15.46:0/3229999194 shutdown_connections 2024-09-16T04:39:07.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.138+0000 7f2b761b7640 1 -- 172.21.15.46:0/3229999194 >> 172.21.15.46:0/3229999194 conn(0x7f2b700a6340 msgr2=0x7f2b700a20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:07.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.139+0000 7f2b761b7640 1 -- 172.21.15.46:0/3229999194 shutdown_connections 2024-09-16T04:39:07.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.139+0000 7f2b761b7640 1 -- 172.21.15.46:0/3229999194 wait complete. 2024-09-16T04:39:07.173 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:39:07.173 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T04:39:07.173 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:39:07.174 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T04:39:07.174 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:39:07.174 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:39:07.174 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:39:07.184 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:39:07.184 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:39:07.184 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:39:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.324+0000 7fb305267640 10 monclient: get_monmap_and_config 2024-09-16T04:39:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.324+0000 7fb305267640 10 monclient: build_initial_monmap 2024-09-16T04:39:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.324+0000 7fb305267640 10 monclient: monmap: 2024-09-16T04:39:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:07.324781+0000 2024-09-16T04:39:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:07.324781+0000 2024-09-16T04:39:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:07.326 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.324+0000 7fb305267640 1 Processor -- start 2024-09-16T04:39:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb305267640 1 -- start start 2024-09-16T04:39:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb305267640 10 monclient: init 2024-09-16T04:39:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb305267640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb305267640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-16T04:39:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb305267640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb3000a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb305267640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb3000a3ff0 0x7fb3000a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb305267640 10 monclient(hunting): picked mon.noname-b con 0x7fb3000a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb305267640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb30013be90 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb305267640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb30009b940 con 0x7fb3000a3c10 2024-09-16T04:39:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb2fe7fc640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb3000a3ff0 msgr2=0x7fb3000a43d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:39:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb305267640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb30009ab00 con 0x7fb30013be90 2024-09-16T04:39:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb305267640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb305267640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fb30009b030 con 0x7fb3000a3ff0 2024-09-16T04:39:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb2fe7fc640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb3000a3ff0 0x7fb3000a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb305267640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb305267640 10 monclient(hunting): authenticate will time out at 745.740626s 2024-09-16T04:39:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb2feffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb3000a3c10 0x7fb30009bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35398/0 (socket says 172.21.15.46:35398) 2024-09-16T04:39:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb2feffd640 1 -- 172.21.15.46:0/479810834 learned_addr learned my addr 172.21.15.46:0/479810834 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb2fdffb640 1 -- 172.21.15.46:0/479810834 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 470286074 0 0) 0x7fb30009b940 con 0x7fb3000a3c10 2024-09-16T04:39:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb2fdffb640 10 monclient(hunting): _negotiate 2024-09-16T04:39:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb2fdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb2fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb2fdffb640 10 monclient(hunting): my global_id is 4464 2024-09-16T04:39:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.325+0000 7fb2fdffb640 1 -- 172.21.15.46:0/479810834 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2ec003520 con 0x7fb3000a3c10 2024-09-16T04:39:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 1 -- 172.21.15.46:0/479810834 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3192730331 0 0) 0x7fb30009ab00 con 0x7fb30013be90 2024-09-16T04:39:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 10 monclient(hunting): _negotiate 2024-09-16T04:39:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 10 monclient(hunting): my global_id is 4444 2024-09-16T04:39:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 1 -- 172.21.15.46:0/479810834 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb30009b940 con 0x7fb30013be90 2024-09-16T04:39:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 1 -- 172.21.15.46:0/479810834 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2782600639 0 0) 0x7fb2ec003520 con 0x7fb3000a3c10 2024-09-16T04:39:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 1 -- 172.21.15.46:0/479810834 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb3000a3ff0 msgr2=0x7fb3000a43d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:39:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 1 --2- 172.21.15.46:0/479810834 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb3000a3ff0 0x7fb3000a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 1 -- 172.21.15.46:0/479810834 >> v1:172.21.15.114:6789/0 conn(0x7fb30013be90 legacy=0x7fb30013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 10 monclient: _finish_hunting 0 2024-09-16T04:39:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:39:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:39:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 1 -- 172.21.15.46:0/479810834 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb30009b310 con 0x7fb3000a3c10 2024-09-16T04:39:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 10 monclient: _finish_auth 0 2024-09-16T04:39:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 10 monclient: _check_auth_tickets 2024-09-16T04:39:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 1 -- 172.21.15.46:0/479810834 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7fb2f0001670 con 0x7fb3000a3c10 2024-09-16T04:39:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:39:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 10 monclient: dump: 2024-09-16T04:39:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:07.340 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:39:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 1 -- 172.21.15.46:0/479810834 >> v1:172.21.15.46:6789/0 conn(0x7fb3000a3c10 legacy=0x7fb30009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:39:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 1 --2- 172.21.15.46:0/479810834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2ec003b70 0x7fb2ec003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 10 monclient(hunting): picked mon.b con 0x7fb2ec003b70 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 1 --2- 172.21.15.46:0/479810834 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb3000a3ff0 0x7fb2ec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 10 monclient(hunting): picked mon.c con 0x7fb3000a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 1 --2- 172.21.15.46:0/479810834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2ec004d90 0x7fb2ec0051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 10 monclient(hunting): picked mon.a con 0x7fb2ec004d90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 1 -- 172.21.15.46:0/479810834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb30009b030 con 0x7fb2ec004d90 2024-09-16T04:39:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 1 -- 172.21.15.46:0/479810834 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ec00bc20 con 0x7fb2ec003b70 2024-09-16T04:39:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 1 -- 172.21.15.46:0/479810834 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fb2ec00be20 con 0x7fb3000a3ff0 2024-09-16T04:39:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fdffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:39:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fe7fc640 1 -- 172.21.15.46:0/479810834 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb3000a3ff0 msgr2=0x7fb2ec004800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:39:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2fe7fc640 1 --2- 172.21.15.46:0/479810834 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb3000a3ff0 0x7fb2ec004800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2feffd640 1 --2- 172.21.15.46:0/479810834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2ec003b70 0x7fb2ec003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2ff7fe640 1 --2- 172.21.15.46:0/479810834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2ec004d90 0x7fb2ec0051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2feffd640 10 monclient(hunting): get_auth_request con 0x7fb2ec003b70 auth_method 0 2024-09-16T04:39:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2feffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2feffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2ff7fe640 10 monclient(hunting): get_auth_request con 0x7fb2ec004d90 auth_method 0 2024-09-16T04:39:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2ff7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2feffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2ff7fe640 10 monclient(hunting): handle_auth_done global_id 4464 payload 293 2024-09-16T04:39:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2ff7fe640 1 -- 172.21.15.46:0/479810834 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb3000a3ff0 msgr2=0x7fb2ec004800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:39:07.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2ff7fe640 1 --2- 172.21.15.46:0/479810834 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb3000a3ff0 0x7fb2ec004800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:07.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2ff7fe640 1 -- 172.21.15.46:0/479810834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2ec003b70 msgr2=0x7fb2ec003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:07.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2ff7fe640 1 --2- 172.21.15.46:0/479810834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2ec003b70 0x7fb2ec003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:07.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2ff7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:39:07.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2ff7fe640 1 monclient: found mon.a 2024-09-16T04:39:07.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:07.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2ff7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:07.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2ff7fe640 1 -- 172.21.15.46:0/479810834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2ec003520 con 0x7fb2ec004d90 2024-09-16T04:39:07.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2ff7fe640 10 monclient: _finish_auth 0 2024-09-16T04:39:07.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2ff7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:39:07.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb2ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:07.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb305267640 5 monclient: authenticate success, global_id 4464 2024-09-16T04:39:07.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.326+0000 7fb305267640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:07.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.327+0000 7fb2ff7fe640 1 --2- 172.21.15.46:0/479810834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2ec004d90 0x7fb2ec0051f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb30006eeb0 tx=0x7fb2ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85ba6d860df18f73 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.327+0000 7fb2fdffb640 1 -- 172.21.15.46:0/479810834 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8001bc0 con 0x7fb2ec004d90 2024-09-16T04:39:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.327+0000 7fb2fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.327+0000 7fb2fdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.327+0000 7fb2fdffb640 10 monclient: dump: 2024-09-16T04:39:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:07.345 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.327+0000 7fb2fdffb640 1 -- 172.21.15.46:0/479810834 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2f8003b70 con 0x7fb2ec004d90 2024-09-16T04:39:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.327+0000 7fb305267640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.327+0000 7fb2fdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.327+0000 7fb2fdffb640 1 -- 172.21.15.46:0/479810834 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f80050d0 con 0x7fb2ec004d90 2024-09-16T04:39:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.327+0000 7fb305267640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.327+0000 7fb305267640 4 set_mon_vals no callback set 2024-09-16T04:39:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.327+0000 7fb2fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.327+0000 7fb2fdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.327+0000 7fb2fdffb640 10 monclient: dump: 2024-09-16T04:39:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:07.346 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.327+0000 7fb305267640 10 monclient: shutdown 2024-09-16T04:39:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.327+0000 7fb305267640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.327+0000 7fb305267640 1 -- 172.21.15.46:0/479810834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2ec004d90 msgr2=0x7fb2ec0051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.327+0000 7fb305267640 1 --2- 172.21.15.46:0/479810834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2ec004d90 0x7fb2ec0051f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb30006eeb0 tx=0x7fb2ec005830 comp rx=0 tx=0).stop 2024-09-16T04:39:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.328+0000 7fb2ff7fe640 1 -- 172.21.15.46:0/479810834 reap_dead start 2024-09-16T04:39:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.328+0000 7fb305267640 1 -- 172.21.15.46:0/479810834 shutdown_connections 2024-09-16T04:39:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.328+0000 7fb305267640 1 -- 172.21.15.46:0/479810834 >> 172.21.15.46:0/479810834 conn(0x7fb3000a6340 msgr2=0x7fb30006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.328+0000 7fb305267640 1 -- 172.21.15.46:0/479810834 shutdown_connections 2024-09-16T04:39:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.328+0000 7fb305267640 1 -- 172.21.15.46:0/479810834 wait complete. 2024-09-16T04:39:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.328+0000 7fb305267640 10 monclient: build_initial_monmap 2024-09-16T04:39:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.328+0000 7fb305267640 10 monclient: monmap: 2024-09-16T04:39:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:07.348 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb305267640 1 Processor -- start 2024-09-16T04:39:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb305267640 1 -- start start 2024-09-16T04:39:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb305267640 10 monclient: init 2024-09-16T04:39:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb305267640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb305267640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:39:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb305267640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2ec004d90 0x7fb3001cc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb305267640 10 monclient(hunting): picked mon.noname-b con 0x7fb2ec004d90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb305267640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb3001cc750 0x7fb30011fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb305267640 10 monclient(hunting): picked mon.noname-c con 0x7fb3001cc750 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb305267640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3001ccbb0 0x7fb3001203d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb305267640 10 monclient(hunting): picked mon.noname-a con 0x7fb3001ccbb0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb305267640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb305267640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb3000a4550 con 0x7fb3001ccbb0 2024-09-16T04:39:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb305267640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb305267640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb3000a43d0 con 0x7fb2ec004d90 2024-09-16T04:39:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb305267640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb305267640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fb3000a46d0 con 0x7fb3001cc750 2024-09-16T04:39:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb305267640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb305267640 10 monclient(hunting): authenticate will time out at 745.744643s 2024-09-16T04:39:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2fe7fc640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb3001cc750 msgr2=0x7fb30011fe90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:39:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2fe7fc640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb3001cc750 0x7fb30011fe90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2ff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3001ccbb0 0x7fb3001203d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2ff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3001ccbb0 0x7fb3001203d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35034/0 (socket says 172.21.15.46:35034) 2024-09-16T04:39:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2ff7fe640 1 -- 172.21.15.46:0/2804311775 learned_addr learned my addr 172.21.15.46:0/2804311775 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2ff7fe640 10 monclient(hunting): get_auth_request con 0x7fb3001ccbb0 auth_method 0 2024-09-16T04:39:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2feffd640 1 --2- 172.21.15.46:0/2804311775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2ec004d90 0x7fb3001cc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2feffd640 10 monclient(hunting): get_auth_request con 0x7fb2ec004d90 auth_method 0 2024-09-16T04:39:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2feffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2ff7fe640 10 monclient(hunting): handle_auth_done global_id 4467 payload 824 2024-09-16T04:39:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2ff7fe640 1 -- 172.21.15.46:0/2804311775 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb3001cc750 msgr2=0x7fb30011fe90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:39:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2ff7fe640 1 --2- 172.21.15.46:0/2804311775 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb3001cc750 0x7fb30011fe90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2ff7fe640 1 -- 172.21.15.46:0/2804311775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2ec004d90 msgr2=0x7fb3001cc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2ff7fe640 1 --2- 172.21.15.46:0/2804311775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2ec004d90 0x7fb3001cc210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2ff7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:39:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2ff7fe640 1 monclient: found mon.noname-a 2024-09-16T04:39:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2ff7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.329+0000 7fb2ff7fe640 1 -- 172.21.15.46:0/2804311775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb300122410 con 0x7fb3001ccbb0 2024-09-16T04:39:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb2ff7fe640 1 --2- 172.21.15.46:0/2804311775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3001ccbb0 0x7fb3001203d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8005040 tx=0x7fb2f8001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb2f77fe640 1 -- 172.21.15.46:0/2804311775 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8041070 con 0x7fb3001ccbb0 2024-09-16T04:39:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb2f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb2f77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb2f77fe640 10 monclient: dump: 2024-09-16T04:39:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:07.353 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb2f77fe640 10 monclient: _finish_auth 0 2024-09-16T04:39:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb2f77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:39:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb2f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb305267640 5 monclient: authenticate success, global_id 4467 2024-09-16T04:39:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb2f77fe640 1 -- 172.21.15.46:0/2804311775 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2f803b430 con 0x7fb3001ccbb0 2024-09-16T04:39:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb305267640 10 monclient: _renew_subs 2024-09-16T04:39:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb305267640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb305267640 1 -- 172.21.15.46:0/2804311775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb300123e20 con 0x7fb3001ccbb0 2024-09-16T04:39:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb2f77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb2f77fe640 1 -- 172.21.15.46:0/2804311775 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f803b7e0 con 0x7fb3001ccbb0 2024-09-16T04:39:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb2f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb2f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb2f77fe640 10 monclient: dump: 2024-09-16T04:39:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:07.354 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb2fcff9640 4 set_mon_vals no callback set 2024-09-16T04:39:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb305267640 10 monclient: _renew_subs 2024-09-16T04:39:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb305267640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.330+0000 7fb305267640 1 -- 172.21.15.46:0/2804311775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3000a0de0 con 0x7fb3001ccbb0 2024-09-16T04:39:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.331+0000 7fb2f77fe640 1 -- 172.21.15.46:0/2804311775 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fb2f8044070 con 0x7fb3001ccbb0 2024-09-16T04:39:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.331+0000 7fb305267640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.331+0000 7fb305267640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.331+0000 7fb305267640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.331+0000 7fb305267640 1 -- 172.21.15.46:0/2804311775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3000a0400 con 0x7fb3001ccbb0 2024-09-16T04:39:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.331+0000 7fb2f77fe640 1 --2- 172.21.15.46:0/2804311775 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb2e0038360 0x7fb2e003a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.331+0000 7fb2f77fe640 1 -- 172.21.15.46:0/2804311775 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4738+0+0 (secure 0 0 0) 0x7fb2f806c1b0 con 0x7fb3001ccbb0 2024-09-16T04:39:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.331+0000 7fb2feffd640 1 --2- 172.21.15.46:0/2804311775 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb2e0038360 0x7fb2e003a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.331+0000 7fb2feffd640 10 monclient: get_auth_request con 0x7fb2e0038360 auth_method 0 2024-09-16T04:39:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.332+0000 7fb2feffd640 1 --2- 172.21.15.46:0/2804311775 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb2e0038360 0x7fb2e003a820 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0007d00 tx=0x7fb2f0000fe0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.336+0000 7fb2f77fe640 1 -- 172.21.15.46:0/2804311775 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb2f8030560 con 0x7fb3001ccbb0 2024-09-16T04:39:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.336+0000 7fb2f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.336+0000 7fb2f77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:07.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.578+0000 7fb305267640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:39:07.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.578+0000 7fb305267640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:39:07.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.578+0000 7fb305267640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:07.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.578+0000 7fb305267640 1 -- 172.21.15.46:0/2804311775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb300067e20 con 0x7fb3001ccbb0 2024-09-16T04:39:07.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.578+0000 7fb2f77fe640 1 -- 172.21.15.46:0/2804311775 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fb2f80307c0 con 0x7fb3001ccbb0 2024-09-16T04:39:07.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.578+0000 7fb2f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:39:07.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.578+0000 7fb2f77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:07.580 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":28,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:39:07.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.580+0000 7fb305267640 1 -- 172.21.15.46:0/2804311775 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb2e0038360 msgr2=0x7fb2e003a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:07.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.580+0000 7fb305267640 1 --2- 172.21.15.46:0/2804311775 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb2e0038360 0x7fb2e003a820 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0007d00 tx=0x7fb2f0000fe0 comp rx=0 tx=0).stop 2024-09-16T04:39:07.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.580+0000 7fb305267640 10 monclient: shutdown 2024-09-16T04:39:07.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.580+0000 7fb305267640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:07.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.580+0000 7fb305267640 1 -- 172.21.15.46:0/2804311775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3001ccbb0 msgr2=0x7fb3001203d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:07.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.580+0000 7fb305267640 1 --2- 172.21.15.46:0/2804311775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3001ccbb0 0x7fb3001203d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8005040 tx=0x7fb2f8001630 comp rx=0 tx=0).stop 2024-09-16T04:39:07.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.580+0000 7fb305267640 1 -- 172.21.15.46:0/2804311775 shutdown_connections 2024-09-16T04:39:07.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.580+0000 7fb305267640 1 --2- 172.21.15.46:0/2804311775 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb2e0038360 0x7fb2e003a820 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:07.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.580+0000 7fb305267640 1 --2- 172.21.15.46:0/2804311775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3001ccbb0 0x7fb3001203d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:07.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.580+0000 7fb305267640 1 --2- 172.21.15.46:0/2804311775 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb3001cc750 0x7fb30011fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:07.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.580+0000 7fb305267640 1 --2- 172.21.15.46:0/2804311775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2ec004d90 0x7fb3001cc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:07.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.580+0000 7fb305267640 1 -- 172.21.15.46:0/2804311775 >> 172.21.15.46:0/2804311775 conn(0x7fb3000a6340 msgr2=0x7fb30013b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:07.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.580+0000 7fb305267640 1 -- 172.21.15.46:0/2804311775 shutdown_connections 2024-09-16T04:39:07.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:07.580+0000 7fb305267640 1 -- 172.21.15.46:0/2804311775 wait complete. 2024-09-16T04:39:10.594 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:39:10.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.706+0000 7fdd22ec9640 10 monclient: get_monmap_and_config 2024-09-16T04:39:10.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.706+0000 7fdd22ec9640 10 monclient: build_initial_monmap 2024-09-16T04:39:10.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.706+0000 7fdd22ec9640 10 monclient: monmap: 2024-09-16T04:39:10.708 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:10.708 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:10.708 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:10.706953+0000 2024-09-16T04:39:10.708 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:10.706953+0000 2024-09-16T04:39:10.708 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:10.708 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:10.708 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:10.708 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:10.708 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:10.708 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:10.708 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:10.708 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:10.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.707+0000 7fdd22ec9640 1 Processor -- start 2024-09-16T04:39:10.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.707+0000 7fdd22ec9640 1 -- start start 2024-09-16T04:39:10.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.707+0000 7fdd22ec9640 10 monclient: init 2024-09-16T04:39:10.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.707+0000 7fdd22ec9640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:10.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.707+0000 7fdd22ec9640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-16T04:39:10.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.707+0000 7fdd22ec9640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdd1c136e50 0x7fdd1c137230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:10.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.707+0000 7fdd22ec9640 10 monclient(hunting): picked mon.noname-c con 0x7fdd1c136e50 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:10.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.707+0000 7fdd22ec9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdd1c0a3500 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:10.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.707+0000 7fdd22ec9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdd1c0a4350 0x7fdd1c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:10.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.707+0000 7fdd22ec9640 10 monclient(hunting): picked mon.noname-a con 0x7fdd1c0a4350 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:10.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.707+0000 7fdd22ec9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd1c0a4e90 con 0x7fdd1c0a3500 2024-09-16T04:39:10.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.707+0000 7fdd22ec9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:10.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.707+0000 7fdd22ec9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fdd1c0a5410 con 0x7fdd1c0a4350 2024-09-16T04:39:10.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.707+0000 7fdd22ec9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:10.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.707+0000 7fdd22ec9640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fdd1c0a4cb0 con 0x7fdd1c136e50 2024-09-16T04:39:10.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.707+0000 7fdd22ec9640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:10.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.707+0000 7fdd22ec9640 10 monclient(hunting): authenticate will time out at 749.122898s 2024-09-16T04:39:10.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd21466640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdd1c0a4350 0x7fdd1c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:10.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd1bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdd1c0a3500 0x7fdd1c0a3c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35414/0 (socket says 172.21.15.46:35414) 2024-09-16T04:39:10.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd1bfff640 1 -- 172.21.15.46:0/1788165297 learned_addr learned my addr 172.21.15.46:0/1788165297 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:10.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd21466640 10 monclient(hunting): get_auth_request con 0x7fdd1c0a4350 auth_method 0 2024-09-16T04:39:10.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd21466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:10.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd21466640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:10.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd21466640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:10.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd20c65640 1 --2- 172.21.15.46:0/1788165297 >> v2:172.21.15.114:3300/0 conn(0x7fdd1c136e50 0x7fdd1c137230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:10.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd20c65640 10 monclient(hunting): get_auth_request con 0x7fdd1c136e50 auth_method 0 2024-09-16T04:39:10.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd20c65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:10.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd20c65640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:10.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd20c65640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:10.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd21466640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:10.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd21466640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:10.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd21466640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:10.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd20c65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:10.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd20c65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:10.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd20c65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:10.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd21466640 10 monclient(hunting): handle_auth_done global_id 4470 payload 274 2024-09-16T04:39:10.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd21466640 1 -- 172.21.15.46:0/1788165297 >> v2:172.21.15.114:3300/0 conn(0x7fdd1c136e50 msgr2=0x7fdd1c137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:10.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd21466640 1 --2- 172.21.15.46:0/1788165297 >> v2:172.21.15.114:3300/0 conn(0x7fdd1c136e50 0x7fdd1c137230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:10.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd21466640 1 -- 172.21.15.46:0/1788165297 >> v1:172.21.15.46:6789/0 conn(0x7fdd1c0a3500 legacy=0x7fdd1c0a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:10.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd21466640 10 monclient: _finish_hunting 0 2024-09-16T04:39:10.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd21466640 1 monclient: found mon.noname-a 2024-09-16T04:39:10.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd21466640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:10.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd21466640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:10.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.708+0000 7fdd21466640 1 -- 172.21.15.46:0/1788165297 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd1c0a0dc0 con 0x7fdd1c0a4350 2024-09-16T04:39:10.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd21466640 1 --2- 172.21.15.46:0/1788165297 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a4350 0x7fdd1c0a4770 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c0686f0 tx=0x7fdd1402a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23be47d133c15d60 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:10.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd1b7fe640 1 -- 172.21.15.46:0/1788165297 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd14037070 con 0x7fdd1c0a4350 2024-09-16T04:39:10.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:10.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd1b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:10.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd1b7fe640 10 monclient: dump: 2024-09-16T04:39:10.719 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:10.719 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:10.719 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:10.719 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:10.719 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:10.720 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:10.720 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:10.720 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:10.720 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:10.720 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:10.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd1b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:39:10.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd1b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:39:10.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:10.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd22ec9640 5 monclient: authenticate success, global_id 4470 2024-09-16T04:39:10.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd22ec9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:10.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd1b7fe640 1 -- 172.21.15.46:0/1788165297 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd1402add0 con 0x7fdd1c0a4350 2024-09-16T04:39:10.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd1b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:10.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd22ec9640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:10.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd22ec9640 4 set_mon_vals no callback set 2024-09-16T04:39:10.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd1b7fe640 1 -- 172.21.15.46:0/1788165297 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd14032660 con 0x7fdd1c0a4350 2024-09-16T04:39:10.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:10.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd1b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:10.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd1b7fe640 10 monclient: dump: 2024-09-16T04:39:10.721 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:10.721 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:10.721 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:10.721 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:10.721 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:10.721 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:10.721 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:10.721 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:10.721 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:10.721 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:10.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd22ec9640 10 monclient: shutdown 2024-09-16T04:39:10.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd22ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:10.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd22ec9640 1 -- 172.21.15.46:0/1788165297 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a4350 msgr2=0x7fdd1c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:10.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd22ec9640 1 --2- 172.21.15.46:0/1788165297 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a4350 0x7fdd1c0a4770 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c0686f0 tx=0x7fdd1402a080 comp rx=0 tx=0).stop 2024-09-16T04:39:10.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd22ec9640 1 -- 172.21.15.46:0/1788165297 shutdown_connections 2024-09-16T04:39:10.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd22ec9640 1 --2- 172.21.15.46:0/1788165297 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a4350 0x7fdd1c0a4770 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:10.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd22ec9640 1 --2- 172.21.15.46:0/1788165297 >> v2:172.21.15.114:3300/0 conn(0x7fdd1c136e50 0x7fdd1c137230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:10.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd22ec9640 1 -- 172.21.15.46:0/1788165297 >> 172.21.15.46:0/1788165297 conn(0x7fdd1c0a5f20 msgr2=0x7fdd1c0a6330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:10.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.709+0000 7fdd22ec9640 1 -- 172.21.15.46:0/1788165297 shutdown_connections 2024-09-16T04:39:10.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.710+0000 7fdd22ec9640 1 -- 172.21.15.46:0/1788165297 wait complete. 2024-09-16T04:39:10.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.710+0000 7fdd22ec9640 10 monclient: build_initial_monmap 2024-09-16T04:39:10.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.710+0000 7fdd22ec9640 10 monclient: monmap: 2024-09-16T04:39:10.722 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:10.722 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:10.723 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.710+0000 7fdd22ec9640 1 Processor -- start 2024-09-16T04:39:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.710+0000 7fdd22ec9640 1 -- start start 2024-09-16T04:39:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd22ec9640 10 monclient: init 2024-09-16T04:39:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd22ec9640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd22ec9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:39:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd22ec9640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdd1c0a3500 0x7fdd1c1cbc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:10.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd22ec9640 10 monclient(hunting): picked mon.noname-c con 0x7fdd1c0a3500 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:10.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd22ec9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a4350 0x7fdd1c135d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:10.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd22ec9640 10 monclient(hunting): picked mon.noname-a con 0x7fdd1c0a4350 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:10.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd22ec9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd1c136e50 0x7fdd1c1362d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:10.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd22ec9640 10 monclient(hunting): picked mon.noname-b con 0x7fdd1c136e50 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:10.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd22ec9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:10.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd22ec9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1c0a0b30 con 0x7fdd1c0a4350 2024-09-16T04:39:10.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd20c65640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdd1c0a3500 msgr2=0x7fdd1c1cbc80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:39:10.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd22ec9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:10.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd22ec9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1c0a09b0 con 0x7fdd1c136e50 2024-09-16T04:39:10.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd22ec9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:10.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd22ec9640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fdd1c0a1b60 con 0x7fdd1c0a3500 2024-09-16T04:39:10.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd20c65640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdd1c0a3500 0x7fdd1c1cbc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:10.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd22ec9640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd22ec9640 10 monclient(hunting): authenticate will time out at 749.126710s 2024-09-16T04:39:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd1bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a4350 0x7fdd1c135d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd1bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a4350 0x7fdd1c135d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35050/0 (socket says 172.21.15.46:35050) 2024-09-16T04:39:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd1bfff640 1 -- 172.21.15.46:0/1266221695 learned_addr learned my addr 172.21.15.46:0/1266221695 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd1bfff640 10 monclient(hunting): get_auth_request con 0x7fdd1c0a4350 auth_method 0 2024-09-16T04:39:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.711+0000 7fdd1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd1bfff640 10 monclient(hunting): handle_auth_done global_id 4476 payload 824 2024-09-16T04:39:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd1bfff640 1 -- 172.21.15.46:0/1266221695 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdd1c0a3500 msgr2=0x7fdd1c1cbc80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:39:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd1bfff640 1 --2- 172.21.15.46:0/1266221695 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdd1c0a3500 0x7fdd1c1cbc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd1bfff640 1 -- 172.21.15.46:0/1266221695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd1c136e50 msgr2=0x7fdd1c1362d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd1bfff640 1 --2- 172.21.15.46:0/1266221695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd1c136e50 0x7fdd1c1362d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd1bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:39:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd1bfff640 1 monclient: found mon.noname-a 2024-09-16T04:39:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd1bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd1bfff640 1 -- 172.21.15.46:0/1266221695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd1c123470 con 0x7fdd1c0a4350 2024-09-16T04:39:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd1bfff640 1 --2- 172.21.15.46:0/1266221695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a4350 0x7fdd1c135d90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00cb10 tx=0x7fdd0c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd197fa640 1 -- 172.21.15.46:0/1266221695 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c017070 con 0x7fdd1c0a4350 2024-09-16T04:39:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd197fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd197fa640 10 monclient: dump: 2024-09-16T04:39:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:10.727 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd197fa640 10 monclient: _finish_auth 0 2024-09-16T04:39:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd197fa640 10 monclient: _check_auth_tickets 2024-09-16T04:39:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd197fa640 1 -- 172.21.15.46:0/1266221695 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd0c00ac70 con 0x7fdd1c0a4350 2024-09-16T04:39:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd22ec9640 5 monclient: authenticate success, global_id 4476 2024-09-16T04:39:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd197fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:10.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd197fa640 1 -- 172.21.15.46:0/1266221695 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c012620 con 0x7fdd1c0a4350 2024-09-16T04:39:10.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd22ec9640 10 monclient: _renew_subs 2024-09-16T04:39:10.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd22ec9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:10.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd22ec9640 1 -- 172.21.15.46:0/1266221695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd1c1cc1c0 con 0x7fdd1c0a4350 2024-09-16T04:39:10.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:10.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:10.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd197fa640 10 monclient: dump: 2024-09-16T04:39:10.728 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:10.728 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:10.728 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:10.728 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:10.728 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:10.728 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:10.728 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:10.729 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd1affd640 4 set_mon_vals no callback set 2024-09-16T04:39:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd22ec9640 10 monclient: _renew_subs 2024-09-16T04:39:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd22ec9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.712+0000 7fdd22ec9640 1 -- 172.21.15.46:0/1266221695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd1c09f690 con 0x7fdd1c0a4350 2024-09-16T04:39:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.713+0000 7fdd197fa640 1 -- 172.21.15.46:0/1266221695 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fdd0c01a070 con 0x7fdd1c0a4350 2024-09-16T04:39:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.713+0000 7fdd22ec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.713+0000 7fdd22ec9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.713+0000 7fdd22ec9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.713+0000 7fdd22ec9640 1 -- 172.21.15.46:0/1266221695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd1c09f470 con 0x7fdd1c0a4350 2024-09-16T04:39:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.713+0000 7fdd197fa640 1 --2- 172.21.15.46:0/1266221695 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdcfc038360 0x7fdcfc03a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.713+0000 7fdd197fa640 1 -- 172.21.15.46:0/1266221695 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4738+0+0 (secure 0 0 0) 0x7fdd0c04b510 con 0x7fdd1c0a4350 2024-09-16T04:39:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.714+0000 7fdd20c65640 1 --2- 172.21.15.46:0/1266221695 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdcfc038360 0x7fdcfc03a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.714+0000 7fdd20c65640 10 monclient: get_auth_request con 0x7fdcfc038360 auth_method 0 2024-09-16T04:39:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.714+0000 7fdd20c65640 1 --2- 172.21.15.46:0/1266221695 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdcfc038360 0x7fdcfc03a820 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdd040094f0 tx=0x7fdd04009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.718+0000 7fdd197fa640 1 -- 172.21.15.46:0/1266221695 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fdd0c01a520 con 0x7fdd1c0a4350 2024-09-16T04:39:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.718+0000 7fdd197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.718+0000 7fdd197fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:10.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.964+0000 7fdd22ec9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:39:10.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.964+0000 7fdd22ec9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:39:10.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.964+0000 7fdd22ec9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:10.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.964+0000 7fdd22ec9640 1 -- 172.21.15.46:0/1266221695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdd1c0a4770 con 0x7fdd1c0a4350 2024-09-16T04:39:10.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.964+0000 7fdd197fa640 1 -- 172.21.15.46:0/1266221695 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fdd0c04d6e0 con 0x7fdd1c0a4350 2024-09-16T04:39:10.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.964+0000 7fdd197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:39:10.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.964+0000 7fdd197fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:10.966 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":31,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:39:10.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.967+0000 7fdd22ec9640 1 -- 172.21.15.46:0/1266221695 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdcfc038360 msgr2=0x7fdcfc03a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:10.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.967+0000 7fdd22ec9640 1 --2- 172.21.15.46:0/1266221695 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdcfc038360 0x7fdcfc03a820 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdd040094f0 tx=0x7fdd04009190 comp rx=0 tx=0).stop 2024-09-16T04:39:10.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.967+0000 7fdd22ec9640 10 monclient: shutdown 2024-09-16T04:39:10.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.967+0000 7fdd22ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:10.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.967+0000 7fdd22ec9640 1 -- 172.21.15.46:0/1266221695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a4350 msgr2=0x7fdd1c135d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:10.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.967+0000 7fdd22ec9640 1 --2- 172.21.15.46:0/1266221695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a4350 0x7fdd1c135d90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00cb10 tx=0x7fdd0c00a060 comp rx=0 tx=0).stop 2024-09-16T04:39:10.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.967+0000 7fdd22ec9640 1 -- 172.21.15.46:0/1266221695 shutdown_connections 2024-09-16T04:39:10.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.967+0000 7fdd22ec9640 1 --2- 172.21.15.46:0/1266221695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd1c136e50 0x7fdd1c1362d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:10.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.967+0000 7fdd22ec9640 1 --2- 172.21.15.46:0/1266221695 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdcfc038360 0x7fdcfc03a820 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:10.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.967+0000 7fdd22ec9640 1 --2- 172.21.15.46:0/1266221695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a4350 0x7fdd1c135d90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:10.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.967+0000 7fdd22ec9640 1 --2- 172.21.15.46:0/1266221695 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdd1c0a3500 0x7fdd1c1cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:10.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.967+0000 7fdd22ec9640 1 -- 172.21.15.46:0/1266221695 >> 172.21.15.46:0/1266221695 conn(0x7fdd1c0a5f20 msgr2=0x7fdd1c1be6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:10.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.967+0000 7fdd22ec9640 1 -- 172.21.15.46:0/1266221695 shutdown_connections 2024-09-16T04:39:10.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:10.967+0000 7fdd22ec9640 1 -- 172.21.15.46:0/1266221695 wait complete. 2024-09-16T04:39:13.980 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:39:14.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.091+0000 7fdaaffa6640 10 monclient: get_monmap_and_config 2024-09-16T04:39:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.091+0000 7fdaaffa6640 10 monclient: build_initial_monmap 2024-09-16T04:39:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.091+0000 7fdaaffa6640 10 monclient: monmap: 2024-09-16T04:39:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:14.093344+0000 2024-09-16T04:39:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:14.093344+0000 2024-09-16T04:39:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:14.096 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:14.096 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:14.096 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:14.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.092+0000 7fdaaffa6640 1 Processor -- start 2024-09-16T04:39:14.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.092+0000 7fdaaffa6640 1 -- start start 2024-09-16T04:39:14.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.092+0000 7fdaaffa6640 10 monclient: init 2024-09-16T04:39:14.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaaffa6640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:14.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaaffa6640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-16T04:39:14.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaaffa6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdaa80a3c10 0x7fdaa809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:14.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaaffa6640 10 monclient(hunting): picked mon.noname-b con 0x7fdaa80a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:14.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaaffa6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdaa80a3ff0 0x7fdaa80a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:14.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaaffa6640 10 monclient(hunting): picked mon.noname-a con 0x7fdaa80a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:14.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaaffa6640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdaa81369d0 0x7fdaa8136e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:14.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaaffa6640 10 monclient(hunting): picked mon.noname-c con 0x7fdaa81369d0 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:14.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaaffa6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:14.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaaffa6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fdaa809b940 con 0x7fdaa80a3ff0 2024-09-16T04:39:14.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaaffa6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:14.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaaffa6640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fdaa809ab00 con 0x7fdaa81369d0 2024-09-16T04:39:14.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaaffa6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:14.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaaffa6640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fdaa809b030 con 0x7fdaa80a3c10 2024-09-16T04:39:14.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaaffa6640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:14.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaaffa6640 10 monclient(hunting): authenticate will time out at 752.509300s 2024-09-16T04:39:14.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaadd42640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdaa80a3c10 msgr2=0x7fdaa809bcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:39:14.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaadd42640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdaa80a3c10 0x7fdaa809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:14.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaad541640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdaa80a3ff0 0x7fdaa80a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:14.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaad541640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdaa80a3ff0 0x7fdaa80a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35062/0 (socket says 172.21.15.46:35062) 2024-09-16T04:39:14.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaad541640 1 -- 172.21.15.46:0/1857665890 learned_addr learned my addr 172.21.15.46:0/1857665890 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:14.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaad541640 10 monclient(hunting): get_auth_request con 0x7fdaa80a3ff0 auth_method 0 2024-09-16T04:39:14.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaad541640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:14.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaad541640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:14.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaad541640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:14.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaad541640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:14.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaad541640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:14.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaad541640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:14.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaae543640 1 --2- 172.21.15.46:0/1857665890 >> v2:172.21.15.114:3300/0 conn(0x7fdaa81369d0 0x7fdaa8136e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:14.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaae543640 10 monclient(hunting): get_auth_request con 0x7fdaa81369d0 auth_method 0 2024-09-16T04:39:14.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaae543640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaae543640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaae543640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaad541640 10 monclient(hunting): handle_auth_done global_id 4479 payload 274 2024-09-16T04:39:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaad541640 1 -- 172.21.15.46:0/1857665890 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdaa80a3c10 msgr2=0x7fdaa809bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:39:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaad541640 1 --2- 172.21.15.46:0/1857665890 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdaa80a3c10 0x7fdaa809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaad541640 1 -- 172.21.15.46:0/1857665890 >> v2:172.21.15.114:3300/0 conn(0x7fdaa81369d0 msgr2=0x7fdaa8136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaad541640 1 --2- 172.21.15.46:0/1857665890 >> v2:172.21.15.114:3300/0 conn(0x7fdaa81369d0 0x7fdaa8136e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaad541640 10 monclient: _finish_hunting 0 2024-09-16T04:39:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaad541640 1 monclient: found mon.noname-a 2024-09-16T04:39:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaad541640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaad541640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaad541640 1 -- 172.21.15.46:0/1857665890 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa809b310 con 0x7fdaa80a3ff0 2024-09-16T04:39:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.093+0000 7fdaae543640 1 --2- 172.21.15.46:0/1857665890 >> v2:172.21.15.114:3300/0 conn(0x7fdaa81369d0 0x7fdaa8136e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:39:14.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.094+0000 7fdaad541640 1 --2- 172.21.15.46:0/1857665890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaa80a3ff0 0x7fdaa80a43d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8068160 tx=0x7fda9802a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4757647257e4200 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:14.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.094+0000 7fdaacd40640 1 -- 172.21.15.46:0/1857665890 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda98037070 con 0x7fdaa80a3ff0 2024-09-16T04:39:14.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.094+0000 7fdaacd40640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:14.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.094+0000 7fdaacd40640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:14.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.094+0000 7fdaacd40640 10 monclient: dump: 2024-09-16T04:39:14.110 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:14.110 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:14.110 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:14.110 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:14.110 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:14.110 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:14.111 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:14.111 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:14.111 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:14.111 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:14.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.094+0000 7fdaacd40640 10 monclient: _finish_auth 0 2024-09-16T04:39:14.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.094+0000 7fdaacd40640 10 monclient: _check_auth_tickets 2024-09-16T04:39:14.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.094+0000 7fdaacd40640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:14.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.094+0000 7fdaaffa6640 5 monclient: authenticate success, global_id 4479 2024-09-16T04:39:14.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.094+0000 7fdaacd40640 1 -- 172.21.15.46:0/1857665890 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda9802ae20 con 0x7fdaa80a3ff0 2024-09-16T04:39:14.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.094+0000 7fdaaffa6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:14.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.094+0000 7fdaacd40640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:14.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.094+0000 7fdaacd40640 1 -- 172.21.15.46:0/1857665890 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda98032660 con 0x7fdaa80a3ff0 2024-09-16T04:39:14.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.094+0000 7fdaacd40640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:14.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.094+0000 7fdaacd40640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:14.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.094+0000 7fdaacd40640 10 monclient: dump: 2024-09-16T04:39:14.112 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:14.112 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:14.112 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:14.112 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:14.112 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:14.112 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:14.112 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:14.112 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:14.112 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:14.112 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.094+0000 7fdaaffa6640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.094+0000 7fdaaffa6640 4 set_mon_vals no callback set 2024-09-16T04:39:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.095+0000 7fdaaffa6640 10 monclient: shutdown 2024-09-16T04:39:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.095+0000 7fdaaffa6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.095+0000 7fdaaffa6640 1 -- 172.21.15.46:0/1857665890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaa80a3ff0 msgr2=0x7fdaa80a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.095+0000 7fdaaffa6640 1 --2- 172.21.15.46:0/1857665890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaa80a3ff0 0x7fdaa80a43d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8068160 tx=0x7fda9802a0d0 comp rx=0 tx=0).stop 2024-09-16T04:39:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.095+0000 7fdaaffa6640 1 -- 172.21.15.46:0/1857665890 shutdown_connections 2024-09-16T04:39:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.095+0000 7fdaaffa6640 1 --2- 172.21.15.46:0/1857665890 >> v2:172.21.15.114:3300/0 conn(0x7fdaa81369d0 0x7fdaa8136e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.095+0000 7fdaaffa6640 1 --2- 172.21.15.46:0/1857665890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaa80a3ff0 0x7fdaa80a43d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.095+0000 7fdaaffa6640 1 --2- 172.21.15.46:0/1857665890 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdaa80a3c10 0x7fdaa809bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.095+0000 7fdaaffa6640 1 -- 172.21.15.46:0/1857665890 >> 172.21.15.46:0/1857665890 conn(0x7fdaa80a6340 msgr2=0x7fdaa806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.095+0000 7fdaaffa6640 1 -- 172.21.15.46:0/1857665890 shutdown_connections 2024-09-16T04:39:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.095+0000 7fdaaffa6640 1 -- 172.21.15.46:0/1857665890 wait complete. 2024-09-16T04:39:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.096+0000 7fdaaffa6640 10 monclient: build_initial_monmap 2024-09-16T04:39:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.096+0000 7fdaaffa6640 10 monclient: monmap: 2024-09-16T04:39:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:14.115 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.096+0000 7fdaaffa6640 1 Processor -- start 2024-09-16T04:39:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.096+0000 7fdaaffa6640 1 -- start start 2024-09-16T04:39:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.096+0000 7fdaaffa6640 10 monclient: init 2024-09-16T04:39:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.096+0000 7fdaaffa6640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.096+0000 7fdaaffa6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:39:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.096+0000 7fdaaffa6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaa80a3c10 0x7fdaa81cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.096+0000 7fdaaffa6640 10 monclient(hunting): picked mon.noname-a con 0x7fdaa80a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaaffa6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdaa80a3ff0 0x7fdaa81cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaaffa6640 10 monclient(hunting): picked mon.noname-c con 0x7fdaa80a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaaffa6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdaa81369d0 0x7fdaa81232b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaaffa6640 10 monclient(hunting): picked mon.noname-b con 0x7fdaa81369d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaaffa6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaaffa6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa80a4910 con 0x7fdaa80a3c10 2024-09-16T04:39:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaaffa6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaaffa6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa80a69c0 con 0x7fdaa81369d0 2024-09-16T04:39:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaaffa6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaaffa6640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fdaa81265d0 con 0x7fdaa80a3ff0 2024-09-16T04:39:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaaffa6640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaaffa6640 10 monclient(hunting): authenticate will time out at 752.513055s 2024-09-16T04:39:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaad541640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdaa80a3ff0 msgr2=0x7fdaa81cc420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:39:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaad541640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdaa80a3ff0 0x7fdaa81cc420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaadd42640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaa80a3c10 0x7fdaa81cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaadd42640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaa80a3c10 0x7fdaa81cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35076/0 (socket says 172.21.15.46:35076) 2024-09-16T04:39:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaadd42640 1 -- 172.21.15.46:0/3315124753 learned_addr learned my addr 172.21.15.46:0/3315124753 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaadd42640 10 monclient(hunting): get_auth_request con 0x7fdaa80a3c10 auth_method 0 2024-09-16T04:39:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaadd42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaadd42640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaadd42640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaadd42640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaadd42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaadd42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaadd42640 10 monclient(hunting): handle_auth_done global_id 4482 payload 824 2024-09-16T04:39:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaae543640 1 --2- 172.21.15.46:0/3315124753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdaa81369d0 0x7fdaa81232b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaadd42640 1 -- 172.21.15.46:0/3315124753 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdaa80a3ff0 msgr2=0x7fdaa81cc420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:39:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaadd42640 1 --2- 172.21.15.46:0/3315124753 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdaa80a3ff0 0x7fdaa81cc420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaadd42640 1 -- 172.21.15.46:0/3315124753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdaa81369d0 msgr2=0x7fdaa81232b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaadd42640 1 --2- 172.21.15.46:0/3315124753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdaa81369d0 0x7fdaa81232b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaadd42640 10 monclient: _finish_hunting 0 2024-09-16T04:39:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaadd42640 1 monclient: found mon.noname-a 2024-09-16T04:39:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaadd42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaadd42640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaadd42640 1 -- 172.21.15.46:0/3315124753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa8136120 con 0x7fdaa80a3c10 2024-09-16T04:39:14.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaadd42640 1 --2- 172.21.15.46:0/3315124753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaa80a3c10 0x7fdaa81cbee0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdaa400b580 tx=0x7fdaa400ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:14.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fda9e7fc640 1 -- 172.21.15.46:0/3315124753 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4003dc0 con 0x7fdaa80a3c10 2024-09-16T04:39:14.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fda9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:14.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fda9e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:14.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fda9e7fc640 10 monclient: dump: 2024-09-16T04:39:14.119 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:14.119 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:14.119 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:14.119 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:14.119 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:14.119 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:14.119 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:14.120 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:14.120 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:14.120 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:14.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fda9e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:39:14.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fda9e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:39:14.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fda9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:14.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fda9e7fc640 1 -- 172.21.15.46:0/3315124753 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaa4015070 con 0x7fdaa80a3c10 2024-09-16T04:39:14.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fda9e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:14.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fda9e7fc640 1 -- 172.21.15.46:0/3315124753 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4005330 con 0x7fdaa80a3c10 2024-09-16T04:39:14.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fda9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:14.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fda9e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:14.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fda9e7fc640 10 monclient: dump: 2024-09-16T04:39:14.121 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:14.121 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:14.121 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:14.121 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:14.121 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:14.122 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:14.122 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:14.122 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:14.122 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:14.123 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:14.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaaffa6640 5 monclient: authenticate success, global_id 4482 2024-09-16T04:39:14.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaaffa6640 10 monclient: _renew_subs 2024-09-16T04:39:14.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaaffa6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:14.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaaffa6640 1 -- 172.21.15.46:0/3315124753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaa81259f0 con 0x7fdaa80a3c10 2024-09-16T04:39:14.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fda9ffff640 4 set_mon_vals no callback set 2024-09-16T04:39:14.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaaffa6640 10 monclient: _renew_subs 2024-09-16T04:39:14.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaaffa6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:14.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.097+0000 7fdaaffa6640 1 -- 172.21.15.46:0/3315124753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaa80a0400 con 0x7fdaa80a3c10 2024-09-16T04:39:14.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.098+0000 7fda9e7fc640 1 -- 172.21.15.46:0/3315124753 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fdaa40054d0 con 0x7fdaa80a3c10 2024-09-16T04:39:14.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.098+0000 7fdaaffa6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:14.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.098+0000 7fdaaffa6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:14.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.098+0000 7fdaaffa6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:14.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.098+0000 7fdaaffa6640 1 -- 172.21.15.46:0/3315124753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaa809bcf0 con 0x7fdaa80a3c10 2024-09-16T04:39:14.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.099+0000 7fda9e7fc640 1 --2- 172.21.15.46:0/3315124753 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda8c038360 0x7fda8c03a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:14.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.099+0000 7fda9e7fc640 1 -- 172.21.15.46:0/3315124753 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4738+0+0 (secure 0 0 0) 0x7fdaa404b5c0 con 0x7fdaa80a3c10 2024-09-16T04:39:14.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.099+0000 7fdaad541640 1 --2- 172.21.15.46:0/3315124753 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda8c038360 0x7fda8c03a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:14.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.099+0000 7fdaad541640 10 monclient: get_auth_request con 0x7fda8c038360 auth_method 0 2024-09-16T04:39:14.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.099+0000 7fdaad541640 1 --2- 172.21.15.46:0/3315124753 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda8c038360 0x7fda8c03a820 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fda98034000 tx=0x7fda9802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:14.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.103+0000 7fda9e7fc640 1 -- 172.21.15.46:0/3315124753 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fdaa40193c0 con 0x7fdaa80a3c10 2024-09-16T04:39:14.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.103+0000 7fda9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:14.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.103+0000 7fda9e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:14.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.346+0000 7fdaaffa6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:39:14.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.346+0000 7fdaaffa6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:39:14.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.346+0000 7fdaaffa6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:14.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.346+0000 7fdaaffa6640 1 -- 172.21.15.46:0/3315124753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdaa8067e20 con 0x7fdaa80a3c10 2024-09-16T04:39:14.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.346+0000 7fda9e7fc640 1 -- 172.21.15.46:0/3315124753 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fdaa400de10 con 0x7fdaa80a3c10 2024-09-16T04:39:14.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.346+0000 7fda9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:39:14.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.346+0000 7fda9e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:14.350 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":35,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:39:14.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.348+0000 7fdaaffa6640 1 -- 172.21.15.46:0/3315124753 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda8c038360 msgr2=0x7fda8c03a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:14.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.348+0000 7fdaaffa6640 1 --2- 172.21.15.46:0/3315124753 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda8c038360 0x7fda8c03a820 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fda98034000 tx=0x7fda9802e000 comp rx=0 tx=0).stop 2024-09-16T04:39:14.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.348+0000 7fdaaffa6640 10 monclient: shutdown 2024-09-16T04:39:14.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.348+0000 7fdaaffa6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:14.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.348+0000 7fdaaffa6640 1 -- 172.21.15.46:0/3315124753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaa80a3c10 msgr2=0x7fdaa81cbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:14.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.348+0000 7fdaaffa6640 1 --2- 172.21.15.46:0/3315124753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaa80a3c10 0x7fdaa81cbee0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdaa400b580 tx=0x7fdaa400ba40 comp rx=0 tx=0).stop 2024-09-16T04:39:14.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.348+0000 7fdaaffa6640 1 -- 172.21.15.46:0/3315124753 shutdown_connections 2024-09-16T04:39:14.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.348+0000 7fdaaffa6640 1 --2- 172.21.15.46:0/3315124753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdaa81369d0 0x7fdaa81232b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:14.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.348+0000 7fdaaffa6640 1 --2- 172.21.15.46:0/3315124753 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdaa80a3ff0 0x7fdaa81cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:14.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.348+0000 7fdaaffa6640 1 --2- 172.21.15.46:0/3315124753 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda8c038360 0x7fda8c03a820 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:14.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.348+0000 7fdaaffa6640 1 --2- 172.21.15.46:0/3315124753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaa80a3c10 0x7fdaa81cbee0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:14.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.348+0000 7fdaaffa6640 1 -- 172.21.15.46:0/3315124753 >> 172.21.15.46:0/3315124753 conn(0x7fdaa80a6340 msgr2=0x7fdaa80a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:14.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.349+0000 7fdaaffa6640 1 -- 172.21.15.46:0/3315124753 shutdown_connections 2024-09-16T04:39:14.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:14.349+0000 7fdaaffa6640 1 -- 172.21.15.46:0/3315124753 wait complete. 2024-09-16T04:39:17.362 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:39:17.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.473+0000 7f735c99f640 10 monclient: get_monmap_and_config 2024-09-16T04:39:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.473+0000 7f735c99f640 10 monclient: build_initial_monmap 2024-09-16T04:39:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.473+0000 7f735c99f640 10 monclient: monmap: 2024-09-16T04:39:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:17.475164+0000 2024-09-16T04:39:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:17.475164+0000 2024-09-16T04:39:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:17.477 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.474+0000 7f735c99f640 1 Processor -- start 2024-09-16T04:39:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.474+0000 7f735c99f640 1 -- start start 2024-09-16T04:39:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.474+0000 7f735c99f640 10 monclient: init 2024-09-16T04:39:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.474+0000 7f735c99f640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.474+0000 7f735c99f640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-16T04:39:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f735c99f640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 0x7f735809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f735c99f640 10 monclient(hunting): picked mon.noname-c con 0x7f73580a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f735c99f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 0x7f73580a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f735c99f640 10 monclient(hunting): picked mon.noname-a con 0x7f73580a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f735c99f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f73581369d0 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f735c99f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f735809b940 con 0x7f73581369d0 2024-09-16T04:39:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f735c99f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f735c99f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f735809ab00 con 0x7f73580a3ff0 2024-09-16T04:39:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f735c99f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f735c99f640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f735809b030 con 0x7f73580a3c10 2024-09-16T04:39:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f735c99f640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f735c99f640 10 monclient(hunting): authenticate will time out at 755.891082s 2024-09-16T04:39:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f7355d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 0x7f73580a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f7356d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f73581369d0 0x7f7358136e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42180/0 (socket says 172.21.15.46:42180) 2024-09-16T04:39:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f7356d9d640 1 -- 172.21.15.46:0/127582049 learned_addr learned my addr 172.21.15.46:0/127582049 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f7355d9b640 10 monclient(hunting): get_auth_request con 0x7f73580a3ff0 auth_method 0 2024-09-16T04:39:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f7355d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f7355d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f7355d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f7355d9b640 1 -- 172.21.15.46:0/127582049 >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 msgr2=0x7f73580a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:39:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f7355d9b640 1 -- 172.21.15.46:0/127582049 >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 msgr2=0x7f73580a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:39:17.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f7355d9b640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 0x7f73580a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:17.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f7355d9b640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 0x7f73580a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:17.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f735659c640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 0x7f735809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:17.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f735659c640 10 monclient(hunting): get_auth_request con 0x7f73580a3c10 auth_method 0 2024-09-16T04:39:17.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f735659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:17.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f735659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:17.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f735659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:17.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f735659c640 1 -- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 msgr2=0x7f735809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:39:17.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f735659c640 1 -- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 msgr2=0x7f735809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:39:17.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f735659c640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 0x7f735809bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:17.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.475+0000 7f735659c640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 0x7f735809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.676+0000 7f7355d9b640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 0x7f73580a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:17.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.676+0000 7f7355d9b640 10 monclient(hunting): get_auth_request con 0x7f73580a3ff0 auth_method 0 2024-09-16T04:39:17.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.676+0000 7f7355d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:17.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.676+0000 7f7355d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:17.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.676+0000 7f7355d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:17.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.676+0000 7f7355d9b640 1 -- 172.21.15.46:0/127582049 >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 msgr2=0x7f73580a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:39:17.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.676+0000 7f7355d9b640 1 -- 172.21.15.46:0/127582049 >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 msgr2=0x7f73580a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:39:17.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.676+0000 7f7355d9b640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 0x7f73580a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:17.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.676+0000 7f7355d9b640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 0x7f73580a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:39:17.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.676+0000 7f735659c640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 0x7f735809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:17.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.676+0000 7f735659c640 10 monclient(hunting): get_auth_request con 0x7f73580a3c10 auth_method 0 2024-09-16T04:39:17.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.676+0000 7f735659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:17.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.676+0000 7f735659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:17.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.676+0000 7f735659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:17.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.676+0000 7f735659c640 1 -- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 msgr2=0x7f735809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:39:17.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.676+0000 7f735659c640 1 -- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 msgr2=0x7f735809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:39:17.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.676+0000 7f735659c640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 0x7f735809bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:17.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:17.676+0000 7f735659c640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 0x7f735809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:39:18.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.077+0000 7f7355d9b640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 0x7f73580a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:18.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.077+0000 7f7355d9b640 10 monclient(hunting): get_auth_request con 0x7f73580a3ff0 auth_method 0 2024-09-16T04:39:18.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.077+0000 7f7355d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:18.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.077+0000 7f7355d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:18.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.077+0000 7f7355d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:18.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.077+0000 7f7355d9b640 1 -- 172.21.15.46:0/127582049 >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 msgr2=0x7f73580a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:39:18.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.077+0000 7f7355d9b640 1 -- 172.21.15.46:0/127582049 >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 msgr2=0x7f73580a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:39:18.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.077+0000 7f7355d9b640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 0x7f73580a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:18.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.077+0000 7f7355d9b640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 0x7f73580a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:39:18.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.077+0000 7f735659c640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 0x7f735809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:18.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.077+0000 7f735659c640 10 monclient(hunting): get_auth_request con 0x7f73580a3c10 auth_method 0 2024-09-16T04:39:18.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.077+0000 7f735659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:18.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.077+0000 7f735659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:18.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.077+0000 7f735659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:18.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.077+0000 7f735659c640 1 -- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 msgr2=0x7f735809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:39:18.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.077+0000 7f735659c640 1 -- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 msgr2=0x7f735809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:39:18.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.077+0000 7f735659c640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 0x7f735809bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:18.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.077+0000 7f735659c640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 0x7f735809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:39:18.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.878+0000 7f7355d9b640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 0x7f73580a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:18.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.878+0000 7f7355d9b640 10 monclient(hunting): get_auth_request con 0x7f73580a3ff0 auth_method 0 2024-09-16T04:39:18.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.878+0000 7f7355d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:18.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.878+0000 7f7355d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:18.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.878+0000 7f7355d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:18.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.878+0000 7f7355d9b640 1 -- 172.21.15.46:0/127582049 >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 msgr2=0x7f73580a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:39:18.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.878+0000 7f7355d9b640 1 -- 172.21.15.46:0/127582049 >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 msgr2=0x7f73580a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:39:18.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.878+0000 7f7355d9b640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 0x7f73580a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:18.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.878+0000 7f7355d9b640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 0x7f73580a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:39:18.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.878+0000 7f735659c640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 0x7f735809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:18.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.878+0000 7f735659c640 10 monclient(hunting): get_auth_request con 0x7f73580a3c10 auth_method 0 2024-09-16T04:39:18.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.878+0000 7f735659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:18.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.878+0000 7f735659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:18.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.878+0000 7f735659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:18.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.878+0000 7f735659c640 1 -- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 msgr2=0x7f735809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:39:18.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.878+0000 7f735659c640 1 -- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 msgr2=0x7f735809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:39:18.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.878+0000 7f735659c640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 0x7f735809bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:18.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:18.878+0000 7f735659c640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 0x7f735809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:39:20.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:20.475+0000 7f7347fff640 10 monclient(hunting): tick 2024-09-16T04:39:20.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:20.475+0000 7f7347fff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:39:20.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:20.475+0000 7f7347fff640 1 monclient(hunting): continuing hunt 2024-09-16T04:39:20.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:20.475+0000 7f7347fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:39:20.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:20.475+0000 7f7347fff640 1 -- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 msgr2=0x7f735809bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:39:20.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:20.475+0000 7f7347fff640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.114:3300/0 conn(0x7f73580a3c10 0x7f735809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:20.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:20.475+0000 7f7347fff640 1 -- 172.21.15.46:0/127582049 >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 msgr2=0x7f73580a43d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:39:20.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:20.475+0000 7f7347fff640 1 --2- 172.21.15.46:0/127582049 >> v2:172.21.15.46:3300/0 conn(0x7f73580a3ff0 0x7f73580a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:20.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:20.475+0000 7f7347fff640 1 -- 172.21.15.46:0/127582049 >> v1:172.21.15.46:6789/0 conn(0x7f73581369d0 legacy=0x7f7358136e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:20.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:20.475+0000 7f7347fff640 10 monclient: _add_conns ranks=[1,2] 2024-09-16T04:39:20.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:20.475+0000 7f7347fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f73380036b0 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:20.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:20.475+0000 7f7347fff640 1 --2- 172.21.15.46:0/127582049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7338007360 0x7f7338009820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:20.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:20.475+0000 7f7347fff640 10 monclient(hunting): picked mon.noname-b con 0x7f7338007360 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:20.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:20.475+0000 7f7347fff640 1 -- 172.21.15.46:0/127582049 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f735809b310 con 0x7f73380036b0 2024-09-16T04:39:20.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:20.475+0000 7f7347fff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:20.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:20.475+0000 7f7347fff640 1 -- 172.21.15.46:0/127582049 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f735809ab00 con 0x7f7338007360 2024-09-16T04:39:20.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:20.475+0000 7f7347fff640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:20.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:20.475+0000 7f7355d9b640 1 -- 172.21.15.46:0/127582049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7338007360 msgr2=0x7f7338009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:39:20.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:20.475+0000 7f7355d9b640 1 --2- 172.21.15.46:0/127582049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7338007360 0x7f7338009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:20.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:20.676+0000 7f7355d9b640 1 -- 172.21.15.46:0/127582049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7338007360 msgr2=0x7f7338009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:39:20.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:20.676+0000 7f7355d9b640 1 --2- 172.21.15.46:0/127582049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7338007360 0x7f7338009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:39:21.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.077+0000 7f7355d9b640 1 -- 172.21.15.46:0/127582049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7338007360 msgr2=0x7f7338009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:39:21.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.077+0000 7f7355d9b640 1 --2- 172.21.15.46:0/127582049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7338007360 0x7f7338009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:39:21.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 1 -- 172.21.15.46:0/127582049 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3264416002 0 0) 0x7f735809b310 con 0x7f73380036b0 2024-09-16T04:39:21.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 10 monclient(hunting): _negotiate 2024-09-16T04:39:21.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:21.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:21.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 10 monclient(hunting): my global_id is 4456 2024-09-16T04:39:21.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 1 -- 172.21.15.46:0/127582049 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7340003520 con 0x7f73380036b0 2024-09-16T04:39:21.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 1 -- 172.21.15.46:0/127582049 <== mon.1 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 442764030 0 0) 0x7f7340003520 con 0x7f73380036b0 2024-09-16T04:39:21.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 1 -- 172.21.15.46:0/127582049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7338007360 msgr2=0x7f7338009820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:39:21.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 1 --2- 172.21.15.46:0/127582049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7338007360 0x7f7338009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:21.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 10 monclient: _finish_hunting 0 2024-09-16T04:39:21.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 1 monclient: found mon.noname-c-legacy 2024-09-16T04:39:21.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:21.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.114:6789/0 2024-09-16T04:39:21.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 1 -- 172.21.15.46:0/127582049 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f733800aff0 con 0x7f73380036b0 2024-09-16T04:39:21.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 10 monclient: _finish_auth 0 2024-09-16T04:39:21.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 10 monclient: _check_auth_tickets 2024-09-16T04:39:21.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:21.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 1 -- 172.21.15.46:0/127582049 <== mon.1 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f734800d580 con 0x7f73380036b0 2024-09-16T04:39:21.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:21.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T04:39:21.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 10 monclient: dump: 2024-09-16T04:39:21.564 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:21.564 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:21.564 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.564 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.565 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:21.565 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:21.565 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:21.565 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:21.565 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:21.565 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:21.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T04:39:21.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:21.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 1 -- 172.21.15.46:0/127582049 >> v1:172.21.15.114:6789/0 conn(0x7f73380036b0 legacy=0x7f7338005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:21.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:39:21.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f7356d9d640 1 -- 172.21.15.46:0/127582049 reap_dead start 2024-09-16T04:39:21.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 1 --2- 172.21.15.46:0/127582049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7338007360 0x7f73400039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:21.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 10 monclient(hunting): picked mon.c con 0x7f7338007360 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:21.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 1 --2- 172.21.15.46:0/127582049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73400040c0 0x7f7340004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:21.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 10 monclient(hunting): picked mon.a con 0x7f73400040c0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:21.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735659c640 1 -- 172.21.15.46:0/127582049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7338007360 msgr2=0x7f73400039b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:39:21.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 1 --2- 172.21.15.46:0/127582049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7340004d90 0x7f7340005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:21.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 10 monclient(hunting): picked mon.b con 0x7f7340004d90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:21.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:21.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735659c640 1 --2- 172.21.15.46:0/127582049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7338007360 0x7f73400039b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:21.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 1 -- 172.21.15.46:0/127582049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f735809ab00 con 0x7f73400040c0 2024-09-16T04:39:21.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:21.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 1 -- 172.21.15.46:0/127582049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f734000bca0 con 0x7f7340004d90 2024-09-16T04:39:21.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:21.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 1 -- 172.21.15.46:0/127582049 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f734000bea0 con 0x7f7338007360 2024-09-16T04:39:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.558+0000 7f735559a640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:39:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7355d9b640 1 --2- 172.21.15.46:0/127582049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73400040c0 0x7f7340004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7355d9b640 10 monclient(hunting): get_auth_request con 0x7f73400040c0 auth_method 0 2024-09-16T04:39:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7356d9d640 1 --2- 172.21.15.46:0/127582049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7340004d90 0x7f7340005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7355d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7355d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7355d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7356d9d640 10 monclient(hunting): get_auth_request con 0x7f7340004d90 auth_method 0 2024-09-16T04:39:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7356d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7356d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7356d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7355d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7355d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7355d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7356d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7356d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7356d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7355d9b640 10 monclient(hunting): handle_auth_done global_id 4456 payload 293 2024-09-16T04:39:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7355d9b640 1 -- 172.21.15.46:0/127582049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7338007360 msgr2=0x7f73400039b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:39:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7355d9b640 1 --2- 172.21.15.46:0/127582049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7338007360 0x7f73400039b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7355d9b640 1 -- 172.21.15.46:0/127582049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7340004d90 msgr2=0x7f7340005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7355d9b640 1 --2- 172.21.15.46:0/127582049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7340004d90 0x7f7340005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7355d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:39:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7355d9b640 1 monclient: found mon.a 2024-09-16T04:39:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7355d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7355d9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7355d9b640 1 -- 172.21.15.46:0/127582049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7340003520 con 0x7f73400040c0 2024-09-16T04:39:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7355d9b640 10 monclient: _finish_auth 0 2024-09-16T04:39:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7355d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:39:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7355d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f735c99f640 5 monclient: authenticate success, global_id 4456 2024-09-16T04:39:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f735c99f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.559+0000 7f7356d9d640 1 --2- 172.21.15.46:0/127582049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7340004d90 0x7f7340005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.560+0000 7f7355d9b640 1 --2- 172.21.15.46:0/127582049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73400040c0 0x7f7340004800 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f734c003a10 tx=0x7f7340005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea317ea7064a99c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:21.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.560+0000 7f735559a640 1 -- 172.21.15.46:0/127582049 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734c031070 con 0x7f73400040c0 2024-09-16T04:39:21.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.560+0000 7f735559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:21.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.560+0000 7f735559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:21.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.560+0000 7f735559a640 10 monclient: dump: 2024-09-16T04:39:21.570 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:21.570 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:21.570 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.570 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.570 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:21.570 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:21.570 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:21.570 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:21.571 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:21.571 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:21.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.560+0000 7f735c99f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:21.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.561+0000 7f735559a640 1 -- 172.21.15.46:0/127582049 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f734c02b990 con 0x7f73400040c0 2024-09-16T04:39:21.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.561+0000 7f735559a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:21.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.561+0000 7f735559a640 1 -- 172.21.15.46:0/127582049 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734c003610 con 0x7f73400040c0 2024-09-16T04:39:21.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.561+0000 7f735559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:21.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.561+0000 7f735559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:21.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.561+0000 7f735559a640 10 monclient: dump: 2024-09-16T04:39:21.577 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:21.577 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:21.577 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.577 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.578 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:21.578 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:21.578 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:21.578 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:21.579 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:21.579 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:21.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.561+0000 7f735c99f640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:21.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.561+0000 7f735c99f640 4 set_mon_vals no callback set 2024-09-16T04:39:21.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.562+0000 7f735c99f640 10 monclient: shutdown 2024-09-16T04:39:21.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.562+0000 7f735c99f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:21.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.562+0000 7f735c99f640 1 -- 172.21.15.46:0/127582049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73400040c0 msgr2=0x7f7340004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:21.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.562+0000 7f735c99f640 1 --2- 172.21.15.46:0/127582049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73400040c0 0x7f7340004800 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f734c003a10 tx=0x7f7340005830 comp rx=0 tx=0).stop 2024-09-16T04:39:21.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.562+0000 7f735c99f640 1 -- 172.21.15.46:0/127582049 shutdown_connections 2024-09-16T04:39:21.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.562+0000 7f735c99f640 1 --2- 172.21.15.46:0/127582049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7340004d90 0x7f7340005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:21.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.562+0000 7f735c99f640 1 --2- 172.21.15.46:0/127582049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73400040c0 0x7f7340004800 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:21.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.562+0000 7f735c99f640 1 --2- 172.21.15.46:0/127582049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7338007360 0x7f73400039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:21.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.562+0000 7f735c99f640 1 -- 172.21.15.46:0/127582049 >> 172.21.15.46:0/127582049 conn(0x7f73580a6340 msgr2=0x7f735806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:21.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.562+0000 7f735c99f640 1 -- 172.21.15.46:0/127582049 shutdown_connections 2024-09-16T04:39:21.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.562+0000 7f735c99f640 1 -- 172.21.15.46:0/127582049 wait complete. 2024-09-16T04:39:21.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.563+0000 7f735c99f640 10 monclient: build_initial_monmap 2024-09-16T04:39:21.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.563+0000 7f735c99f640 10 monclient: monmap: 2024-09-16T04:39:21.580 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:21.580 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:21.580 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:21.580 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:21.580 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:21.580 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:21.580 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:21.580 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:21.580 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:21.580 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.564+0000 7f735c99f640 1 Processor -- start 2024-09-16T04:39:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.564+0000 7f735c99f640 1 -- start start 2024-09-16T04:39:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.564+0000 7f735c99f640 10 monclient: init 2024-09-16T04:39:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.564+0000 7f735c99f640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.564+0000 7f735c99f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:39:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.564+0000 7f735c99f640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7338007360 0x7f73581c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.564+0000 7f735c99f640 10 monclient(hunting): picked mon.noname-c con 0x7f7338007360 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.564+0000 7f735c99f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73400040c0 0x7f7358136040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.564+0000 7f735c99f640 10 monclient(hunting): picked mon.noname-a con 0x7f73400040c0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.564+0000 7f735c99f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7340004d90 0x7f73581cb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.564+0000 7f735c99f640 10 monclient(hunting): picked mon.noname-b con 0x7f7340004d90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.564+0000 7f735c99f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.564+0000 7f735c99f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f735809be70 con 0x7f73400040c0 2024-09-16T04:39:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.564+0000 7f735c99f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:21.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.564+0000 7f735c99f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f735809bcf0 con 0x7f7340004d90 2024-09-16T04:39:21.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.564+0000 7f735c99f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:21.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.564+0000 7f735c99f640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f735809bff0 con 0x7f7338007360 2024-09-16T04:39:21.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.564+0000 7f735c99f640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:21.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.564+0000 7f735c99f640 10 monclient(hunting): authenticate will time out at 759.980995s 2024-09-16T04:39:21.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f735659c640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7338007360 msgr2=0x7f73581c7e10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:39:21.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f735659c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7338007360 0x7f73581c7e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:21.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7355d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73400040c0 0x7f7358136040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:21.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7355d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73400040c0 0x7f7358136040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38304/0 (socket says 172.21.15.46:38304) 2024-09-16T04:39:21.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7355d9b640 1 -- 172.21.15.46:0/3570133609 learned_addr learned my addr 172.21.15.46:0/3570133609 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:21.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7355d9b640 10 monclient(hunting): get_auth_request con 0x7f73400040c0 auth_method 0 2024-09-16T04:39:21.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7355d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:21.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7355d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:21.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7355d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:21.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7355d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:21.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7355d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:21.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7355d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:21.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7356d9d640 1 --2- 172.21.15.46:0/3570133609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7340004d90 0x7f73581cb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:21.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7356d9d640 10 monclient(hunting): get_auth_request con 0x7f7340004d90 auth_method 0 2024-09-16T04:39:21.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7356d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:21.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7356d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:21.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7356d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:21.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7356d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:21.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7356d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:21.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7356d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:21.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7355d9b640 10 monclient(hunting): handle_auth_done global_id 4485 payload 824 2024-09-16T04:39:21.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7355d9b640 1 -- 172.21.15.46:0/3570133609 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7338007360 msgr2=0x7f73581c7e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:39:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7355d9b640 1 --2- 172.21.15.46:0/3570133609 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7338007360 0x7f73581c7e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7355d9b640 1 -- 172.21.15.46:0/3570133609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7340004d90 msgr2=0x7f73581cb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7355d9b640 1 --2- 172.21.15.46:0/3570133609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7340004d90 0x7f73581cb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7355d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:39:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7355d9b640 1 monclient: found mon.noname-a 2024-09-16T04:39:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7355d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7355d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7355d9b640 1 -- 172.21.15.46:0/3570133609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7358121f80 con 0x7f73400040c0 2024-09-16T04:39:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.565+0000 7f7355d9b640 1 --2- 172.21.15.46:0/3570133609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73400040c0 0x7f7358136040 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f734c003b30 tx=0x7f734c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.566+0000 7f7346ffd640 1 -- 172.21.15.46:0/3570133609 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734c044020 con 0x7f73400040c0 2024-09-16T04:39:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.566+0000 7f7346ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.566+0000 7f7346ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.566+0000 7f7346ffd640 10 monclient: dump: 2024-09-16T04:39:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:21.585 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.566+0000 7f7346ffd640 10 monclient: _finish_auth 0 2024-09-16T04:39:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.566+0000 7f7346ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:39:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.566+0000 7f7346ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.566+0000 7f7346ffd640 1 -- 172.21.15.46:0/3570133609 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f734c02b790 con 0x7f73400040c0 2024-09-16T04:39:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.566+0000 7f735c99f640 5 monclient: authenticate success, global_id 4485 2024-09-16T04:39:21.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.566+0000 7f735c99f640 10 monclient: _renew_subs 2024-09-16T04:39:21.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.566+0000 7f735c99f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:21.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.566+0000 7f735c99f640 1 -- 172.21.15.46:0/3570133609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7358124a10 con 0x7f73400040c0 2024-09-16T04:39:21.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.566+0000 7f7346ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:21.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.566+0000 7f7346ffd640 1 -- 172.21.15.46:0/3570133609 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734c0408c0 con 0x7f73400040c0 2024-09-16T04:39:21.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.566+0000 7f7346ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:21.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.566+0000 7f7346ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:21.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.566+0000 7f7346ffd640 10 monclient: dump: 2024-09-16T04:39:21.586 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:21.586 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:21.586 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.586 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.586 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:21.586 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:21.587 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:21.587 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:21.587 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:21.587 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:21.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.566+0000 7f7354d99640 4 set_mon_vals no callback set 2024-09-16T04:39:21.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.566+0000 7f735c99f640 10 monclient: _renew_subs 2024-09-16T04:39:21.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.566+0000 7f735c99f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:21.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.566+0000 7f735c99f640 1 -- 172.21.15.46:0/3570133609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7358122720 con 0x7f73400040c0 2024-09-16T04:39:21.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.567+0000 7f7346ffd640 1 -- 172.21.15.46:0/3570133609 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f734c047020 con 0x7f73400040c0 2024-09-16T04:39:21.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.567+0000 7f735c99f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:21.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.567+0000 7f735c99f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:21.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.567+0000 7f735c99f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:21.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.567+0000 7f735c99f640 1 -- 172.21.15.46:0/3570133609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73580a43d0 con 0x7f73400040c0 2024-09-16T04:39:21.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.567+0000 7f7346ffd640 1 --2- 172.21.15.46:0/3570133609 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7338038c60 0x7f733803b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:21.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.567+0000 7f7346ffd640 1 -- 172.21.15.46:0/3570133609 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4738+0+0 (secure 0 0 0) 0x7f734c06c720 con 0x7f73400040c0 2024-09-16T04:39:21.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.567+0000 7f735659c640 1 --2- 172.21.15.46:0/3570133609 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7338038c60 0x7f733803b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:21.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.567+0000 7f735659c640 10 monclient: get_auth_request con 0x7f7338038c60 auth_method 0 2024-09-16T04:39:21.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.568+0000 7f735659c640 1 --2- 172.21.15.46:0/3570133609 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7338038c60 0x7f733803b120 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f73581cc3d0 tx=0x7f7348008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:21.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.572+0000 7f7346ffd640 1 -- 172.21.15.46:0/3570133609 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f734c04e030 con 0x7f73400040c0 2024-09-16T04:39:21.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.572+0000 7f7346ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:21.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.572+0000 7f7346ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:21.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.819+0000 7f735c99f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:39:21.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.819+0000 7f735c99f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:39:21.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.819+0000 7f735c99f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:21.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.819+0000 7f735c99f640 1 -- 172.21.15.46:0/3570133609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f73581234c0 con 0x7f73400040c0 2024-09-16T04:39:21.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.820+0000 7f7346ffd640 1 -- 172.21.15.46:0/3570133609 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f734c03fa10 con 0x7f73400040c0 2024-09-16T04:39:21.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.820+0000 7f7346ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:39:21.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.820+0000 7f7346ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:21.822 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":12,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:39:21.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.822+0000 7f735c99f640 1 -- 172.21.15.46:0/3570133609 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7338038c60 msgr2=0x7f733803b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:21.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.822+0000 7f735c99f640 1 --2- 172.21.15.46:0/3570133609 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7338038c60 0x7f733803b120 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f73581cc3d0 tx=0x7f7348008040 comp rx=0 tx=0).stop 2024-09-16T04:39:21.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.822+0000 7f735c99f640 10 monclient: shutdown 2024-09-16T04:39:21.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.822+0000 7f735c99f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:21.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.822+0000 7f735c99f640 1 -- 172.21.15.46:0/3570133609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73400040c0 msgr2=0x7f7358136040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:21.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.822+0000 7f735c99f640 1 --2- 172.21.15.46:0/3570133609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73400040c0 0x7f7358136040 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f734c003b30 tx=0x7f734c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:39:21.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.822+0000 7f735c99f640 1 -- 172.21.15.46:0/3570133609 shutdown_connections 2024-09-16T04:39:21.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.822+0000 7f735c99f640 1 --2- 172.21.15.46:0/3570133609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7340004d90 0x7f73581cb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:21.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.822+0000 7f735c99f640 1 --2- 172.21.15.46:0/3570133609 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7338038c60 0x7f733803b120 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:21.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.822+0000 7f735c99f640 1 --2- 172.21.15.46:0/3570133609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73400040c0 0x7f7358136040 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:21.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.822+0000 7f735c99f640 1 --2- 172.21.15.46:0/3570133609 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7338007360 0x7f73581c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:21.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.822+0000 7f735c99f640 1 -- 172.21.15.46:0/3570133609 >> 172.21.15.46:0/3570133609 conn(0x7f73580a6340 msgr2=0x7f73580a2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:21.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.822+0000 7f735c99f640 1 -- 172.21.15.46:0/3570133609 shutdown_connections 2024-09-16T04:39:21.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.822+0000 7f735c99f640 1 -- 172.21.15.46:0/3570133609 wait complete. 2024-09-16T04:39:21.835 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:39:21.835 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:39:21.836 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:39:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.935+0000 7fb4b9245640 10 monclient: get_monmap_and_config 2024-09-16T04:39:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.935+0000 7fb4b9245640 10 monclient: build_initial_monmap 2024-09-16T04:39:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.936+0000 7fb4b9245640 10 monclient: monmap: 2024-09-16T04:39:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:21.937469+0000 2024-09-16T04:39:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:21.937469+0000 2024-09-16T04:39:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:21.939 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:21.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.936+0000 7fb4b9245640 1 Processor -- start 2024-09-16T04:39:21.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.936+0000 7fb4b9245640 1 -- start start 2024-09-16T04:39:21.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.936+0000 7fb4b9245640 10 monclient: init 2024-09-16T04:39:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4b9245640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4b9245640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-16T04:39:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4b9245640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb4b40a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4b9245640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb4b40a3ff0 0x7fb4b40a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4b9245640 10 monclient(hunting): picked mon.noname-c con 0x7fb4b40a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4b9245640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb4b413be90 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4b9245640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4b409b940 con 0x7fb4b40a3c10 2024-09-16T04:39:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4b9245640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4b409ab00 con 0x7fb4b413be90 2024-09-16T04:39:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4b9245640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4b9245640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fb4b409b030 con 0x7fb4b40a3ff0 2024-09-16T04:39:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4b9245640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4b9245640 10 monclient(hunting): authenticate will time out at 760.353151s 2024-09-16T04:39:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4b2d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb4b40a3c10 0x7fb4b409bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42186/0 (socket says 172.21.15.46:42186) 2024-09-16T04:39:21.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4b2d9d640 1 -- 172.21.15.46:0/4002065616 learned_addr learned my addr 172.21.15.46:0/4002065616 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:21.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4aa59c640 1 --2- 172.21.15.46:0/4002065616 >> v2:172.21.15.114:3300/0 conn(0x7fb4b40a3ff0 0x7fb4b40a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:21.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4aa59c640 10 monclient(hunting): get_auth_request con 0x7fb4b40a3ff0 auth_method 0 2024-09-16T04:39:21.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:21.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:21.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:21.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4b259c640 1 -- 172.21.15.46:0/4002065616 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3465357337 0 0) 0x7fb4b409b940 con 0x7fb4b40a3c10 2024-09-16T04:39:21.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:21.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:21.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:21.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4b259c640 10 monclient(hunting): _negotiate 2024-09-16T04:39:21.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4b259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4b259c640 10 monclient(hunting): my global_id is 4488 2024-09-16T04:39:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4b259c640 1 -- 172.21.15.46:0/4002065616 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4a0003520 con 0x7fb4b40a3c10 2024-09-16T04:39:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4aa59c640 10 monclient(hunting): handle_auth_done global_id 4462 payload 274 2024-09-16T04:39:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4aa59c640 1 -- 172.21.15.46:0/4002065616 >> v1:172.21.15.114:6789/0 conn(0x7fb4b413be90 legacy=0x7fb4b413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4aa59c640 1 -- 172.21.15.46:0/4002065616 >> v1:172.21.15.46:6789/0 conn(0x7fb4b40a3c10 legacy=0x7fb4b409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4aa59c640 10 monclient: _finish_hunting 0 2024-09-16T04:39:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4aa59c640 1 monclient: found mon.noname-c 2024-09-16T04:39:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4aa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4aa59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:39:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.937+0000 7fb4aa59c640 1 -- 172.21.15.46:0/4002065616 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b409b310 con 0x7fb4b40a3ff0 2024-09-16T04:39:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.938+0000 7fb4aa59c640 1 --2- 172.21.15.46:0/4002065616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b40a3ff0 0x7fb4b40a43d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb49c0097a0 tx=0x7fb49c02a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=931f2515f6fd5cd0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.938+0000 7fb4b259c640 1 -- 172.21.15.46:0/4002065616 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49c037070 con 0x7fb4b40a3ff0 2024-09-16T04:39:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.938+0000 7fb4b259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.938+0000 7fb4b259c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:39:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.938+0000 7fb4b259c640 10 monclient: dump: 2024-09-16T04:39:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:21.957 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.938+0000 7fb4b259c640 10 monclient: _finish_auth 0 2024-09-16T04:39:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.938+0000 7fb4b259c640 10 monclient: _check_auth_tickets 2024-09-16T04:39:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.938+0000 7fb4b259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.938+0000 7fb4b259c640 1 -- 172.21.15.46:0/4002065616 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb49c032440 con 0x7fb4b40a3ff0 2024-09-16T04:39:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.938+0000 7fb4b259c640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.938+0000 7fb4b9245640 5 monclient: authenticate success, global_id 4462 2024-09-16T04:39:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.938+0000 7fb4b9245640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.938+0000 7fb4b9245640 4 set_mon_vals no callback set 2024-09-16T04:39:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.938+0000 7fb4b259c640 1 -- 172.21.15.46:0/4002065616 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49c0327f0 con 0x7fb4b40a3ff0 2024-09-16T04:39:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.938+0000 7fb4b259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.938+0000 7fb4b259c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.938+0000 7fb4b259c640 10 monclient: dump: 2024-09-16T04:39:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:21.959 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.939+0000 7fb4b9245640 10 monclient: shutdown 2024-09-16T04:39:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.939+0000 7fb4b9245640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.939+0000 7fb4b9245640 1 -- 172.21.15.46:0/4002065616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b40a3ff0 msgr2=0x7fb4b40a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.939+0000 7fb4b9245640 1 --2- 172.21.15.46:0/4002065616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b40a3ff0 0x7fb4b40a43d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb49c0097a0 tx=0x7fb49c02a1a0 comp rx=0 tx=0).stop 2024-09-16T04:39:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.939+0000 7fb4b9245640 1 -- 172.21.15.46:0/4002065616 shutdown_connections 2024-09-16T04:39:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.939+0000 7fb4b9245640 1 --2- 172.21.15.46:0/4002065616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b40a3ff0 0x7fb4b40a43d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.939+0000 7fb4b9245640 1 -- 172.21.15.46:0/4002065616 >> 172.21.15.46:0/4002065616 conn(0x7fb4b40a6340 msgr2=0x7fb4b406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.939+0000 7fb4b9245640 1 -- 172.21.15.46:0/4002065616 shutdown_connections 2024-09-16T04:39:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.939+0000 7fb4b9245640 1 -- 172.21.15.46:0/4002065616 wait complete. 2024-09-16T04:39:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.939+0000 7fb4b9245640 10 monclient: build_initial_monmap 2024-09-16T04:39:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.939+0000 7fb4b9245640 10 monclient: monmap: 2024-09-16T04:39:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:21.961 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:21.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.940+0000 7fb4b9245640 1 Processor -- start 2024-09-16T04:39:21.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.940+0000 7fb4b9245640 1 -- start start 2024-09-16T04:39:21.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.940+0000 7fb4b9245640 10 monclient: init 2024-09-16T04:39:21.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.940+0000 7fb4b9245640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:21.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.940+0000 7fb4b9245640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:39:21.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.940+0000 7fb4b9245640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b40a3c10 0x7fb4b41cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:21.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.940+0000 7fb4b9245640 10 monclient(hunting): picked mon.noname-b con 0x7fb4b40a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:21.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.940+0000 7fb4b9245640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb4b40a3ff0 0x7fb4b4136dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:21.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.940+0000 7fb4b9245640 10 monclient(hunting): picked mon.noname-c con 0x7fb4b40a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:21.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.940+0000 7fb4b9245640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b413be90 0x7fb4b4138ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:21.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.940+0000 7fb4b9245640 10 monclient(hunting): picked mon.noname-a con 0x7fb4b413be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:21.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.940+0000 7fb4b9245640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:21.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.940+0000 7fb4b9245640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b40a6b40 con 0x7fb4b413be90 2024-09-16T04:39:21.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.940+0000 7fb4b9245640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:21.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.940+0000 7fb4b9245640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b40a69c0 con 0x7fb4b40a3c10 2024-09-16T04:39:21.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.940+0000 7fb4b9245640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:21.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.940+0000 7fb4b9245640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fb4b413f9a0 con 0x7fb4b40a3ff0 2024-09-16T04:39:21.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.940+0000 7fb4b9245640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:21.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.940+0000 7fb4b9245640 10 monclient(hunting): authenticate will time out at 760.356968s 2024-09-16T04:39:21.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.940+0000 7fb4aa59c640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb4b40a3ff0 msgr2=0x7fb4b4136dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:39:21.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4aa59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb4b40a3ff0 0x7fb4b4136dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:21.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b359e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b413be90 0x7fb4b4138ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:21.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b359e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b413be90 0x7fb4b4138ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38306/0 (socket says 172.21.15.46:38306) 2024-09-16T04:39:21.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b359e640 1 -- 172.21.15.46:0/2482663455 learned_addr learned my addr 172.21.15.46:0/2482663455 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:21.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b359e640 10 monclient(hunting): get_auth_request con 0x7fb4b413be90 auth_method 0 2024-09-16T04:39:21.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:21.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:21.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:21.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:21.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:21.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:21.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b2d9d640 1 --2- 172.21.15.46:0/2482663455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b40a3c10 0x7fb4b41cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:21.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b2d9d640 10 monclient(hunting): get_auth_request con 0x7fb4b40a3c10 auth_method 0 2024-09-16T04:39:21.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:21.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:21.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:21.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b359e640 10 monclient(hunting): handle_auth_done global_id 4491 payload 824 2024-09-16T04:39:21.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b359e640 1 -- 172.21.15.46:0/2482663455 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb4b40a3ff0 msgr2=0x7fb4b4136dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:39:21.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b359e640 1 --2- 172.21.15.46:0/2482663455 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb4b40a3ff0 0x7fb4b4136dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:21.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b359e640 1 -- 172.21.15.46:0/2482663455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b40a3c10 msgr2=0x7fb4b41cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:21.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b359e640 1 --2- 172.21.15.46:0/2482663455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b40a3c10 0x7fb4b41cbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:21.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b359e640 10 monclient: _finish_hunting 0 2024-09-16T04:39:21.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b359e640 1 monclient: found mon.noname-a 2024-09-16T04:39:21.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:21.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b359e640 1 -- 172.21.15.46:0/2482663455 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b41cc9c0 con 0x7fb4b413be90 2024-09-16T04:39:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b359e640 1 --2- 172.21.15.46:0/2482663455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b413be90 0x7fb4b4138ef0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac00c730 tx=0x7fb4ac00cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4abfff640 1 -- 172.21.15.46:0/2482663455 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac003b70 con 0x7fb4b413be90 2024-09-16T04:39:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4abfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4abfff640 10 monclient: dump: 2024-09-16T04:39:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:21.966 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:21.966 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:21.966 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:21.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4abfff640 10 monclient: _finish_auth 0 2024-09-16T04:39:21.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4abfff640 10 monclient: _check_auth_tickets 2024-09-16T04:39:21.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:21.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4abfff640 1 -- 172.21.15.46:0/2482663455 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4ac003d10 con 0x7fb4b413be90 2024-09-16T04:39:21.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b9245640 5 monclient: authenticate success, global_id 4491 2024-09-16T04:39:21.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4abfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:21.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b9245640 10 monclient: _renew_subs 2024-09-16T04:39:21.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4abfff640 1 -- 172.21.15.46:0/2482663455 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac005310 con 0x7fb4b413be90 2024-09-16T04:39:21.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b9245640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:21.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b9245640 1 -- 172.21.15.46:0/2482663455 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4b40a0780 con 0x7fb4b413be90 2024-09-16T04:39:21.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:21.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:21.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4abfff640 10 monclient: dump: 2024-09-16T04:39:21.967 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:21.967 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:21.967 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.967 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.967 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:21.967 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:21.967 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:21.967 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:21.967 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:21.967 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b1d9b640 4 set_mon_vals no callback set 2024-09-16T04:39:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b9245640 10 monclient: _renew_subs 2024-09-16T04:39:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b9245640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.941+0000 7fb4b9245640 1 -- 172.21.15.46:0/2482663455 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4b413f4c0 con 0x7fb4b413be90 2024-09-16T04:39:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.942+0000 7fb4b9245640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.942+0000 7fb4b9245640 1 --2- 172.21.15.46:0/2482663455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b41200e0 0x7fb4b4139430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.942+0000 7fb4b9245640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.942+0000 7fb4b9245640 1 -- 172.21.15.46:0/2482663455 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b413b5b0 con 0x7fb4b41200e0 2024-09-16T04:39:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.942+0000 7fb4abfff640 1 -- 172.21.15.46:0/2482663455 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fb4ac0054b0 con 0x7fb4b413be90 2024-09-16T04:39:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.942+0000 7fb4b2d9d640 1 --2- 172.21.15.46:0/2482663455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b41200e0 0x7fb4b4139430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.942+0000 7fb4b2d9d640 10 monclient: get_auth_request con 0x7fb4b41200e0 auth_method 0 2024-09-16T04:39:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.942+0000 7fb4b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.942+0000 7fb4b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.942+0000 7fb4b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.942+0000 7fb4b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.942+0000 7fb4b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.942+0000 7fb4b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.943+0000 7fb4abfff640 1 --2- 172.21.15.46:0/2482663455 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb494038360 0x7fb49403a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.943+0000 7fb4abfff640 1 -- 172.21.15.46:0/2482663455 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4738+0+0 (secure 0 0 0) 0x7fb4ac04b660 con 0x7fb4b413be90 2024-09-16T04:39:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.943+0000 7fb4b2d9d640 10 monclient(hunting): handle_auth_done global_id 4494 payload 824 2024-09-16T04:39:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.943+0000 7fb4b2d9d640 1 -- 172.21.15.46:0/2482663455 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb4b40a87e0 con 0x7fb4b41200e0 2024-09-16T04:39:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.943+0000 7fb4b2d9d640 1 --2- 172.21.15.46:0/2482663455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b41200e0 0x7fb4b4139430 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb4a40011e0 tx=0x7fb4a400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.943+0000 7fb4aa59c640 1 --2- 172.21.15.46:0/2482663455 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb494038360 0x7fb49403a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.943+0000 7fb4aa59c640 10 monclient: get_auth_request con 0x7fb494038360 auth_method 0 2024-09-16T04:39:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.943+0000 7fb4abfff640 1 -- 172.21.15.46:0/2482663455 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a4005890 con 0x7fb4b41200e0 2024-09-16T04:39:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.943+0000 7fb4abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:21.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.943+0000 7fb4abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:21.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.943+0000 7fb4abfff640 10 monclient: dump: 2024-09-16T04:39:21.970 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:21.970 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:21.970 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.970 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.970 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:21.970 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:21.970 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:21.970 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:21.970 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:21.970 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:21.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.943+0000 7fb4abfff640 1 -- 172.21.15.46:0/2482663455 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb4a4018070 con 0x7fb4b41200e0 2024-09-16T04:39:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.943+0000 7fb4abfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.943+0000 7fb4abfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.943+0000 7fb4abfff640 1 -- 172.21.15.46:0/2482663455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b41200e0 msgr2=0x7fb4b4139430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.943+0000 7fb4abfff640 1 --2- 172.21.15.46:0/2482663455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b41200e0 0x7fb4b4139430 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb4a40011e0 tx=0x7fb4a400eb90 comp rx=0 tx=0).stop 2024-09-16T04:39:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.943+0000 7fb4abfff640 1 -- 172.21.15.46:0/2482663455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b41200e0 msgr2=0x7fb4b4139430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.943+0000 7fb4abfff640 1 --2- 172.21.15.46:0/2482663455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b41200e0 0x7fb4b4139430 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.943+0000 7fb4aa59c640 1 --2- 172.21.15.46:0/2482663455 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb494038360 0x7fb49403a820 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb49c034000 tx=0x7fb49c02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.949+0000 7fb4b9245640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:39:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.949+0000 7fb4b9245640 1 --2- 172.21.15.46:0/2482663455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b4139970 0x7fb4b413c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.949+0000 7fb4b9245640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.949+0000 7fb4b9245640 1 -- 172.21.15.46:0/2482663455 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b4139d80 con 0x7fb4b4139970 2024-09-16T04:39:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.949+0000 7fb4b2d9d640 1 --2- 172.21.15.46:0/2482663455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b4139970 0x7fb4b413c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.950+0000 7fb4b2d9d640 10 monclient: get_auth_request con 0x7fb4b4139970 auth_method 0 2024-09-16T04:39:21.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.950+0000 7fb4b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:21.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.950+0000 7fb4b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:21.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.950+0000 7fb4b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:21.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.950+0000 7fb4b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:21.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.950+0000 7fb4b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:21.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.950+0000 7fb4b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:21.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.950+0000 7fb4b2d9d640 10 monclient(hunting): handle_auth_done global_id 4497 payload 824 2024-09-16T04:39:21.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.950+0000 7fb4b2d9d640 1 -- 172.21.15.46:0/2482663455 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb4b40a0400 con 0x7fb4b4139970 2024-09-16T04:39:21.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.950+0000 7fb4b2d9d640 1 --2- 172.21.15.46:0/2482663455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b4139970 0x7fb4b413c270 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4017000 tx=0x7fb4a4001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:21.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.950+0000 7fb4abfff640 1 -- 172.21.15.46:0/2482663455 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a401d7c0 con 0x7fb4b4139970 2024-09-16T04:39:21.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.950+0000 7fb4abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:21.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.950+0000 7fb4abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:21.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.950+0000 7fb4abfff640 10 monclient: dump: 2024-09-16T04:39:21.973 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:21.973 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:21.973 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.973 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:21.973 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:21.973 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:21.973 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:21.973 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:21.973 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:21.973 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:21.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.950+0000 7fb4abfff640 1 -- 172.21.15.46:0/2482663455 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4702 (secure 0 0 0) 0x7fb4a401d7c0 con 0x7fb4b4139970 2024-09-16T04:39:21.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.950+0000 7fb4abfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:39:21.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.950+0000 7fb4abfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:21.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.950+0000 7fb4abfff640 1 -- 172.21.15.46:0/2482663455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b4139970 msgr2=0x7fb4b413c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:21.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.950+0000 7fb4abfff640 1 --2- 172.21.15.46:0/2482663455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b4139970 0x7fb4b413c270 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4017000 tx=0x7fb4a4001150 comp rx=0 tx=0).stop 2024-09-16T04:39:21.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.950+0000 7fb4abfff640 1 -- 172.21.15.46:0/2482663455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b4139970 msgr2=0x7fb4b413c270 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:21.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.950+0000 7fb4abfff640 1 --2- 172.21.15.46:0/2482663455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b4139970 0x7fb4b413c270 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4017000 tx=0x7fb4a4001150 comp rx=0 tx=0).stop 2024-09-16T04:39:21.974 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:39:21.974 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:39:21.974 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:39:21.974 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:39:21.974 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 12, 2024-09-16T04:39:21.975 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 47889, 2024-09-16T04:39:21.975 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:39:21.975 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:39:21.975 INFO:teuthology.orchestra.run.smithi046.stdout: 1 2024-09-16T04:39:21.975 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:21.975 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:39:21.975 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:21.975 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:39:21.975 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:39:21.975 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:21.975 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:21.975 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:21.975 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:21.975 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:21.976 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:21.976 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:21.976 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:21.976 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:21.976 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:21.976 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:21.976 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:21.976 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:39:21.976 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:39:21.976 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:21.976 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:21.976 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:21.976 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:21.977 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:21.977 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:21.977 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:21.977 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:21.977 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:21.977 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:21.977 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:21.977 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:21.977 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:21.977 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:39:21.977 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:39:21.977 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:39:21.977 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:39:21.977 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:39:21.978 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:39:21.978 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:21.978 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:21.978 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:39:21.978 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:39:21.978 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:39:21.978 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:39:21.978 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:39:21.978 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:39:21.978 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:39:21.978 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:21.978 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:39:21.978 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:21.979 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:21.979 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:21.979 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:21.979 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:21.979 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:21.979 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:21.979 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:21.979 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:21.979 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:21.979 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:21.979 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:21.979 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:39:21.979 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:21.980 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:39:21.980 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:21.980 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:39:21.980 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:39:21.980 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:21.980 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:21.980 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:21.980 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:21.980 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:39:21.980 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:21.981 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:21.981 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:21.981 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:21.981 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:39:21.981 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:21.981 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:21.981 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:21.981 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:21.981 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:39:21.981 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:39:21.981 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:21.981 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:21.981 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:21.982 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:21.982 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:21.982 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:39:21.982 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:39:21.982 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:21.982 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:21.982 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:21.982 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:21.982 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:39:21.982 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:21.982 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:21.982 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:21.982 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:21.982 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:39:21.983 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:21.983 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:21.983 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:21.983 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:21.983 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:39:21.983 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:39:21.983 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:21.983 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:21.983 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:21.983 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:21.983 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:21.983 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:39:21.983 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:39:21.983 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:21.984 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:21.984 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:21.984 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:21.984 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:39:21.984 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:21.984 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:21.984 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:21.984 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:21.984 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:39:21.984 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:21.984 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:21.984 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:21.984 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:21.985 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:39:21.985 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:39:21.985 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:21.985 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:21.985 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:21.985 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:21.985 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:21.985 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:21.985 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:39:21.985 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:39:21.985 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:21.985 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:21.985 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:21.985 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:39:21.986 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:21.986 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:21.986 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:39:21.986 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:21.986 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:21.986 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:21.986 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:39:21.986 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:21.986 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:21.986 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:39:21.986 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:21.986 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:21.986 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:21.986 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:39:21.987 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:21.987 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:21.987 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:21.987 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:39:21.987 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:39:21.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.951+0000 7fb4b9245640 1 -- 172.21.15.46:0/2482663455 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb494038360 msgr2=0x7fb49403a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:21.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.951+0000 7fb4b9245640 1 --2- 172.21.15.46:0/2482663455 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb494038360 0x7fb49403a820 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb49c034000 tx=0x7fb49c02e000 comp rx=0 tx=0).stop 2024-09-16T04:39:21.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.951+0000 7fb4b9245640 10 monclient: shutdown 2024-09-16T04:39:21.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.951+0000 7fb4b9245640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:21.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.951+0000 7fb4b9245640 1 -- 172.21.15.46:0/2482663455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b413be90 msgr2=0x7fb4b4138ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:21.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.951+0000 7fb4b9245640 1 --2- 172.21.15.46:0/2482663455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b413be90 0x7fb4b4138ef0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac00c730 tx=0x7fb4ac00cb30 comp rx=0 tx=0).stop 2024-09-16T04:39:21.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.951+0000 7fb4b359e640 1 -- 172.21.15.46:0/2482663455 reap_dead start 2024-09-16T04:39:21.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.951+0000 7fb4b359e640 1 -- 172.21.15.46:0/2482663455 reap_dead start 2024-09-16T04:39:21.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.951+0000 7fb4b9245640 1 -- 172.21.15.46:0/2482663455 shutdown_connections 2024-09-16T04:39:21.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.951+0000 7fb4b9245640 1 -- 172.21.15.46:0/2482663455 >> 172.21.15.46:0/2482663455 conn(0x7fb4b40a6340 msgr2=0x7fb4b40a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:21.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.951+0000 7fb4b9245640 1 -- 172.21.15.46:0/2482663455 shutdown_connections 2024-09-16T04:39:21.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:21.951+0000 7fb4b9245640 1 -- 172.21.15.46:0/2482663455 wait complete. 2024-09-16T04:39:21.989 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:39:21.990 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:39:21.990 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:39:22.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.089+0000 7f730acba640 10 monclient: get_monmap_and_config 2024-09-16T04:39:22.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.089+0000 7f730acba640 10 monclient: build_initial_monmap 2024-09-16T04:39:22.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.089+0000 7f730acba640 10 monclient: monmap: 2024-09-16T04:39:22.092 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:22.092 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:22.093 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:22.091330+0000 2024-09-16T04:39:22.093 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:22.091330+0000 2024-09-16T04:39:22.093 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:22.093 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:22.093 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:22.093 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:22.093 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:22.094 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:22.094 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:22.094 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.090+0000 7f730acba640 1 Processor -- start 2024-09-16T04:39:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.090+0000 7f730acba640 1 -- start start 2024-09-16T04:39:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.090+0000 7f730acba640 10 monclient: init 2024-09-16T04:39:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f730acba640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f730acba640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-16T04:39:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f730acba640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73040a3c10 0x7f730409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f730acba640 10 monclient(hunting): picked mon.noname-a con 0x7f73040a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f730acba640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f73040a3ff0 0x7f73040a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f730acba640 10 monclient(hunting): picked mon.noname-b con 0x7f73040a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f730acba640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f73041369d0 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f730acba640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f730409b940 con 0x7f73041369d0 2024-09-16T04:39:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f730acba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f730acba640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f730409ab00 con 0x7f73040a3c10 2024-09-16T04:39:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f730acba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f730acba640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f730409b030 con 0x7f73040a3ff0 2024-09-16T04:39:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f730acba640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f730acba640 10 monclient(hunting): authenticate will time out at 760.507184s 2024-09-16T04:39:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f7303fff640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f73040a3ff0 msgr2=0x7f73040a43d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:39:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f7303fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f73040a3ff0 0x7f73040a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f7308a56640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73040a3c10 0x7f730409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f7309257640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f73041369d0 0x7f7304136e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42194/0 (socket says 172.21.15.46:42194) 2024-09-16T04:39:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f7309257640 1 -- 172.21.15.46:0/3305743167 learned_addr learned my addr 172.21.15.46:0/3305743167 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f7308a56640 10 monclient(hunting): get_auth_request con 0x7f73040a3c10 auth_method 0 2024-09-16T04:39:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f7308a56640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f7308a56640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f7308a56640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f7308a56640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f7308a56640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f7308a56640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f7308a56640 10 monclient(hunting): handle_auth_done global_id 4500 payload 274 2024-09-16T04:39:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f7308a56640 1 -- 172.21.15.46:0/3305743167 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f73040a3ff0 msgr2=0x7f73040a43d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:39:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f7308a56640 1 --2- 172.21.15.46:0/3305743167 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f73040a3ff0 0x7f73040a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f7308a56640 1 -- 172.21.15.46:0/3305743167 >> v1:172.21.15.46:6789/0 conn(0x7f73041369d0 legacy=0x7f7304136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f7308a56640 10 monclient: _finish_hunting 0 2024-09-16T04:39:22.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f7308a56640 1 monclient: found mon.noname-a 2024-09-16T04:39:22.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f7308a56640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:22.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f7308a56640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:22.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f7308a56640 1 -- 172.21.15.46:0/3305743167 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f730409b310 con 0x7f73040a3c10 2024-09-16T04:39:22.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f73037fe640 1 -- 172.21.15.46:0/3305743167 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2211331964 0 0) 0x7f730409b940 con 0x7f73041369d0 2024-09-16T04:39:22.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.091+0000 7f73037fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:39:22.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.092+0000 7f7308a56640 1 --2- 172.21.15.46:0/3305743167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73040a3c10 0x7f730409bcf0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f72f40097a0 tx=0x7f72f402a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c610d10edbda3558 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:22.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.092+0000 7f73037fe640 1 -- 172.21.15.46:0/3305743167 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f4037070 con 0x7f73040a3c10 2024-09-16T04:39:22.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.092+0000 7f73037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:22.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.092+0000 7f73037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:22.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.092+0000 7f73037fe640 10 monclient: dump: 2024-09-16T04:39:22.106 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:22.106 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:22.107 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:22.107 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:22.107 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:22.107 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:22.107 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:22.107 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:22.107 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:22.107 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:22.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.092+0000 7f73037fe640 10 monclient: _finish_auth 0 2024-09-16T04:39:22.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.092+0000 7f73037fe640 10 monclient: _check_auth_tickets 2024-09-16T04:39:22.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.092+0000 7f73037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:22.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.092+0000 7f73037fe640 1 -- 172.21.15.46:0/3305743167 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72f402add0 con 0x7f73040a3c10 2024-09-16T04:39:22.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.092+0000 7f730acba640 5 monclient: authenticate success, global_id 4500 2024-09-16T04:39:22.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.092+0000 7f730acba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:22.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.092+0000 7f73037fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.092+0000 7f730acba640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.092+0000 7f73037fe640 1 -- 172.21.15.46:0/3305743167 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f4032660 con 0x7f73040a3c10 2024-09-16T04:39:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.092+0000 7f730acba640 4 set_mon_vals no callback set 2024-09-16T04:39:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.092+0000 7f73037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.092+0000 7f73037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.092+0000 7f73037fe640 10 monclient: dump: 2024-09-16T04:39:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:22.109 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.093+0000 7f730acba640 10 monclient: shutdown 2024-09-16T04:39:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.093+0000 7f730acba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.093+0000 7f730acba640 1 -- 172.21.15.46:0/3305743167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73040a3c10 msgr2=0x7f730409bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.093+0000 7f730acba640 1 --2- 172.21.15.46:0/3305743167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73040a3c10 0x7f730409bcf0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f72f40097a0 tx=0x7f72f402a080 comp rx=0 tx=0).stop 2024-09-16T04:39:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.093+0000 7f730acba640 1 -- 172.21.15.46:0/3305743167 shutdown_connections 2024-09-16T04:39:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.093+0000 7f730acba640 1 --2- 172.21.15.46:0/3305743167 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f73040a3ff0 0x7f73040a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:22.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.093+0000 7f730acba640 1 --2- 172.21.15.46:0/3305743167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73040a3c10 0x7f730409bcf0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:22.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.093+0000 7f730acba640 1 -- 172.21.15.46:0/3305743167 >> 172.21.15.46:0/3305743167 conn(0x7f73040a6340 msgr2=0x7f730406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:22.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.093+0000 7f730acba640 1 -- 172.21.15.46:0/3305743167 shutdown_connections 2024-09-16T04:39:22.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.093+0000 7f730acba640 1 -- 172.21.15.46:0/3305743167 wait complete. 2024-09-16T04:39:22.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.093+0000 7f730acba640 10 monclient: build_initial_monmap 2024-09-16T04:39:22.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.093+0000 7f730acba640 10 monclient: monmap: 2024-09-16T04:39:22.111 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:22.111 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:22.111 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:22.111 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:22.111 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:22.111 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:22.111 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:22.111 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:22.111 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:22.111 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:22.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f730acba640 1 Processor -- start 2024-09-16T04:39:22.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f730acba640 1 -- start start 2024-09-16T04:39:22.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f730acba640 10 monclient: init 2024-09-16T04:39:22.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f730acba640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:22.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f730acba640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:39:22.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f730acba640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f73040a3c10 0x7f73041cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:22.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f730acba640 10 monclient(hunting): picked mon.noname-c con 0x7f73040a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:22.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f730acba640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73040a3ff0 0x7f73041cc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:22.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f730acba640 10 monclient(hunting): picked mon.noname-b con 0x7f73040a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:22.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f730acba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73041369d0 0x7f7304123180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:22.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f730acba640 10 monclient(hunting): picked mon.noname-a con 0x7f73041369d0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:22.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f730acba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:22.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f730acba640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f73040a4910 con 0x7f73041369d0 2024-09-16T04:39:22.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f730acba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:22.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f7308a56640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f73040a3c10 msgr2=0x7f73041cbf50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:39:22.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f730acba640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f73040a69c0 con 0x7f73040a3ff0 2024-09-16T04:39:22.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f730acba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:22.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f730acba640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f7304126df0 con 0x7f73040a3c10 2024-09-16T04:39:22.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f7308a56640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f73040a3c10 0x7f73041cbf50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:22.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f730acba640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:22.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f730acba640 10 monclient(hunting): authenticate will time out at 760.510784s 2024-09-16T04:39:22.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f7309257640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73041369d0 0x7f7304123180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:22.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f7309257640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73041369d0 0x7f7304123180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38346/0 (socket says 172.21.15.46:38346) 2024-09-16T04:39:22.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f7309257640 1 -- 172.21.15.46:0/465123543 learned_addr learned my addr 172.21.15.46:0/465123543 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:22.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f7309257640 10 monclient(hunting): get_auth_request con 0x7f73041369d0 auth_method 0 2024-09-16T04:39:22.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f7309257640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:22.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f7309257640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:22.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f7309257640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:22.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f7309257640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:22.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f7309257640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:22.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.094+0000 7f7309257640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:22.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f7303fff640 1 --2- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73040a3ff0 0x7f73041cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:22.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f7303fff640 10 monclient(hunting): get_auth_request con 0x7f73040a3ff0 auth_method 0 2024-09-16T04:39:22.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f7303fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:22.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f7303fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:22.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f7303fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:22.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f7309257640 10 monclient(hunting): handle_auth_done global_id 4506 payload 824 2024-09-16T04:39:22.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f7309257640 1 -- 172.21.15.46:0/465123543 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f73040a3c10 msgr2=0x7f73041cbf50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:39:22.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f7309257640 1 --2- 172.21.15.46:0/465123543 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f73040a3c10 0x7f73041cbf50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:22.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f7309257640 1 -- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73040a3ff0 msgr2=0x7f73041cc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:22.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f7309257640 1 --2- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73040a3ff0 0x7f73041cc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:22.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f7309257640 10 monclient: _finish_hunting 0 2024-09-16T04:39:22.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f7309257640 1 monclient: found mon.noname-a 2024-09-16T04:39:22.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f7309257640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:22.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f7309257640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:22.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f7309257640 1 -- 172.21.15.46:0/465123543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73040a2ad0 con 0x7f73041369d0 2024-09-16T04:39:22.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f7309257640 1 --2- 172.21.15.46:0/465123543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73041369d0 0x7f7304123180 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f72fc00c830 tx=0x7f72fc00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:22.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f73017fa640 1 -- 172.21.15.46:0/465123543 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72fc018070 con 0x7f73041369d0 2024-09-16T04:39:22.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f73017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:22.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f73017fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:22.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f73017fa640 10 monclient: dump: 2024-09-16T04:39:22.116 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:22.116 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:22.116 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:22.116 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:22.116 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:22.116 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:22.116 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:22.116 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:22.117 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:22.117 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:22.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f73017fa640 10 monclient: _finish_auth 0 2024-09-16T04:39:22.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f73017fa640 10 monclient: _check_auth_tickets 2024-09-16T04:39:22.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f73017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:22.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f73017fa640 1 -- 172.21.15.46:0/465123543 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72fc00ce20 con 0x7f73041369d0 2024-09-16T04:39:22.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f730acba640 5 monclient: authenticate success, global_id 4506 2024-09-16T04:39:22.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f730acba640 10 monclient: _renew_subs 2024-09-16T04:39:22.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f730acba640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:22.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f730acba640 1 -- 172.21.15.46:0/465123543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7304126b10 con 0x7f73041369d0 2024-09-16T04:39:22.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f73017fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:22.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f73017fa640 1 -- 172.21.15.46:0/465123543 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72fc00a880 con 0x7f73041369d0 2024-09-16T04:39:22.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f73017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:22.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f73017fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:22.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f73017fa640 10 monclient: dump: 2024-09-16T04:39:22.118 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:22.118 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:22.118 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:22.118 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:22.118 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:22.118 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:22.118 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:22.118 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:22.118 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:22.118 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:22.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f730acba640 10 monclient: _renew_subs 2024-09-16T04:39:22.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f7302ffd640 4 set_mon_vals no callback set 2024-09-16T04:39:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f730acba640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.095+0000 7f730acba640 1 -- 172.21.15.46:0/465123543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7304125220 con 0x7f73041369d0 2024-09-16T04:39:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.096+0000 7f730acba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.096+0000 7f730acba640 1 --2- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73041236c0 0x7f7304123aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.096+0000 7f730acba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.096+0000 7f730acba640 1 -- 172.21.15.46:0/465123543 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f73040a4a90 con 0x7f73041236c0 2024-09-16T04:39:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.096+0000 7f73017fa640 1 -- 172.21.15.46:0/465123543 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f72fc01b070 con 0x7f73041369d0 2024-09-16T04:39:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.096+0000 7f7308a56640 1 --2- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73041236c0 0x7f7304123aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.096+0000 7f7308a56640 10 monclient: get_auth_request con 0x7f73041236c0 auth_method 0 2024-09-16T04:39:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.096+0000 7f7308a56640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.096+0000 7f7308a56640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.096+0000 7f7308a56640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.096+0000 7f73017fa640 1 --2- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f72f0038360 0x7f72f003a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.096+0000 7f7308a56640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.096+0000 7f7308a56640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.096+0000 7f7308a56640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.096+0000 7f73017fa640 1 -- 172.21.15.46:0/465123543 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4738+0+0 (secure 0 0 0) 0x7f72fc04acc0 con 0x7f73041369d0 2024-09-16T04:39:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.096+0000 7f7308a56640 10 monclient(hunting): handle_auth_done global_id 4474 payload 824 2024-09-16T04:39:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.096+0000 7f7303fff640 1 --2- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f72f0038360 0x7f72f003a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.096+0000 7f7308a56640 1 -- 172.21.15.46:0/465123543 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f730409bd70 con 0x7f73041236c0 2024-09-16T04:39:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.096+0000 7f7303fff640 10 monclient: get_auth_request con 0x7f72f0038360 auth_method 0 2024-09-16T04:39:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.097+0000 7f7308a56640 1 --2- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73041236c0 0x7f7304123aa0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7304068160 tx=0x7f72f40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.097+0000 7f73017fa640 1 -- 172.21.15.46:0/465123543 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f4042070 con 0x7f73041236c0 2024-09-16T04:39:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.097+0000 7f73017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.097+0000 7f73017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.097+0000 7f73017fa640 10 monclient: dump: 2024-09-16T04:39:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:22.121 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.097+0000 7f7303fff640 1 --2- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f72f0038360 0x7f72f003a820 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f72f8006fd0 tx=0x7f72f8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.097+0000 7f73017fa640 1 -- 172.21.15.46:0/465123543 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f72f4032d90 con 0x7f73041236c0 2024-09-16T04:39:22.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.097+0000 7f73017fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:22.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.097+0000 7f73017fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:22.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.097+0000 7f73017fa640 1 -- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73041236c0 msgr2=0x7f7304123aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:22.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.097+0000 7f73017fa640 1 --2- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73041236c0 0x7f7304123aa0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7304068160 tx=0x7f72f40046c0 comp rx=0 tx=0).stop 2024-09-16T04:39:22.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.097+0000 7f73017fa640 1 -- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73041236c0 msgr2=0x7f7304123aa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:22.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.097+0000 7f73017fa640 1 --2- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73041236c0 0x7f7304123aa0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:22.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.103+0000 7f730acba640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:39:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.103+0000 7f730acba640 1 --2- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7304123fe0 0x7f730411f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.103+0000 7f730acba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.103+0000 7f730acba640 1 -- 172.21.15.46:0/465123543 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7304126890 con 0x7f7304123fe0 2024-09-16T04:39:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.104+0000 7f7308a56640 1 --2- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7304123fe0 0x7f730411f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.104+0000 7f7308a56640 10 monclient: get_auth_request con 0x7f7304123fe0 auth_method 0 2024-09-16T04:39:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.104+0000 7f7308a56640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.104+0000 7f7308a56640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.104+0000 7f7308a56640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.104+0000 7f7308a56640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.104+0000 7f7308a56640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.104+0000 7f7308a56640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.104+0000 7f7308a56640 10 monclient(hunting): handle_auth_done global_id 4477 payload 824 2024-09-16T04:39:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.104+0000 7f7308a56640 1 -- 172.21.15.46:0/465123543 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7304067e20 con 0x7f7304123fe0 2024-09-16T04:39:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.104+0000 7f7308a56640 1 --2- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7304123fe0 0x7f730411f860 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f72f4000c00 tx=0x7f72f4008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.104+0000 7f73017fa640 1 -- 172.21.15.46:0/465123543 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f4037040 con 0x7f7304123fe0 2024-09-16T04:39:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.104+0000 7f73017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.104+0000 7f73017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.104+0000 7f73017fa640 10 monclient: dump: 2024-09-16T04:39:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:22.127 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.105+0000 7f73017fa640 1 -- 172.21.15.46:0/465123543 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4867 (secure 0 0 0) 0x7f72f403c3e0 con 0x7f7304123fe0 2024-09-16T04:39:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.105+0000 7f73017fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:39:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.105+0000 7f73017fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.105+0000 7f73017fa640 1 -- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7304123fe0 msgr2=0x7f730411f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.105+0000 7f73017fa640 1 --2- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7304123fe0 0x7f730411f860 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f72f4000c00 tx=0x7f72f4008030 comp rx=0 tx=0).stop 2024-09-16T04:39:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.105+0000 7f73017fa640 1 -- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7304123fe0 msgr2=0x7f730411f860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.105+0000 7f73017fa640 1 --2- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7304123fe0 0x7f730411f860 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:22.127 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:39:22.128 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:39:22.128 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:39:22.128 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:39:22.128 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 12, 2024-09-16T04:39:22.128 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 48006, 2024-09-16T04:39:22.128 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:39:22.128 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:39:22.128 INFO:teuthology.orchestra.run.smithi046.stdout: 1 2024-09-16T04:39:22.128 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:22.128 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:39:22.128 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:22.128 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:39:22.129 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:39:22.129 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:22.129 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:22.129 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:22.129 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:22.129 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:22.129 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:22.129 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:22.129 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:22.129 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:22.129 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:22.129 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:22.130 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:22.130 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:39:22.130 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:39:22.130 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:22.130 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:22.130 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:22.130 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:22.130 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:22.130 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:22.130 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:22.130 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:22.130 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:22.130 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:22.130 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:39:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:39:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:39:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:39:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:39:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:39:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:39:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:39:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:39:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:39:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:39:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:39:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:39:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:39:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:39:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:39:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:39:22.134 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:39:22.134 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:22.134 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:22.134 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:22.134 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:22.134 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:39:22.134 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:22.134 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:22.134 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:22.134 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:22.134 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:39:22.134 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:22.134 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:22.134 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:22.135 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:22.135 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:39:22.135 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:39:22.135 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:22.135 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:22.135 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:22.135 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:22.135 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:22.135 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:39:22.135 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:39:22.135 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:22.135 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:22.135 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:22.135 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:22.136 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:39:22.136 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:22.136 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:22.136 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:22.136 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:22.136 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:39:22.136 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:22.136 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:22.136 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:22.137 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:22.137 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:39:22.137 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:39:22.137 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:22.137 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:22.137 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:22.137 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:22.137 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:22.137 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:39:22.137 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:39:22.137 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:22.137 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:22.137 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:22.137 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:22.138 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:39:22.138 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:22.138 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:22.138 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:22.138 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:22.138 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:39:22.138 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:22.139 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:22.139 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:22.139 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:22.139 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:39:22.139 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:39:22.139 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:22.139 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:22.139 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:22.139 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:22.139 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:22.139 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:22.139 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:39:22.140 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:39:22.140 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:22.140 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:22.140 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:22.140 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:39:22.140 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:22.140 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:22.140 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:39:22.140 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:22.140 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:22.140 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:22.140 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:39:22.140 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:22.140 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:22.141 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:39:22.141 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:22.141 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:22.141 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:22.141 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6 2024-09-16T04:39:22.141 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:22.141 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:22.141 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:39:22.141 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:22.141 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:22.141 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:22.141 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:39:22.142 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:22.142 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:22.142 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:22.142 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:39:22.142 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:39:22.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.105+0000 7f730acba640 1 -- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f72f0038360 msgr2=0x7f72f003a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:22.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.105+0000 7f730acba640 1 --2- 172.21.15.46:0/465123543 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f72f0038360 0x7f72f003a820 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f72f8006fd0 tx=0x7f72f8008040 comp rx=0 tx=0).stop 2024-09-16T04:39:22.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.105+0000 7f730acba640 10 monclient: shutdown 2024-09-16T04:39:22.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.105+0000 7f730acba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:22.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.105+0000 7f730acba640 1 -- 172.21.15.46:0/465123543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73041369d0 msgr2=0x7f7304123180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:22.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.105+0000 7f730acba640 1 --2- 172.21.15.46:0/465123543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73041369d0 0x7f7304123180 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f72fc00c830 tx=0x7f72fc00ccf0 comp rx=0 tx=0).stop 2024-09-16T04:39:22.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.105+0000 7f7309257640 1 -- 172.21.15.46:0/465123543 reap_dead start 2024-09-16T04:39:22.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.105+0000 7f730acba640 1 -- 172.21.15.46:0/465123543 shutdown_connections 2024-09-16T04:39:22.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.105+0000 7f730acba640 1 --2- 172.21.15.46:0/465123543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73041369d0 0x7f7304123180 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:22.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.105+0000 7f730acba640 1 -- 172.21.15.46:0/465123543 >> 172.21.15.46:0/465123543 conn(0x7f73040a6340 msgr2=0x7f73040a2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:22.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.105+0000 7f730acba640 1 -- 172.21.15.46:0/465123543 shutdown_connections 2024-09-16T04:39:22.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:22.105+0000 7f730acba640 1 -- 172.21.15.46:0/465123543 wait complete. 2024-09-16T04:39:22.145 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:39:22.145 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:39:22.656 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35319]: starting. 2024-09-16T04:39:22.656 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35319]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-35318 2024-09-16T04:39:22.656 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35319]: created object 0... 2024-09-16T04:39:22.656 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35319]: created object 25... 2024-09-16T04:39:22.656 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35319]: created object 49... 2024-09-16T04:39:22.656 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35319]: finishing. 2024-09-16T04:39:22.656 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35319]: shutting down. 2024-09-16T04:39:23.571 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[35320]: starting. 2024-09-16T04:39:23.571 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[35320]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-35318 2024-09-16T04:39:23.571 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[35320]: shutting down. 2024-09-16T04:39:25.663 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:25.663 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35365]: starting. 2024-09-16T04:39:25.663 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35365]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-35318 2024-09-16T04:39:25.663 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35365]: created object 0... 2024-09-16T04:39:25.663 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35365]: created object 25... 2024-09-16T04:39:25.663 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35365]: created object 49... 2024-09-16T04:39:25.663 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35365]: finishing. 2024-09-16T04:39:25.663 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35365]: shutting down. 2024-09-16T04:39:25.676 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:25.676 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35367]: starting. 2024-09-16T04:39:25.676 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35367]: listing objects. 2024-09-16T04:39:25.676 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35367]: listed object 0... 2024-09-16T04:39:25.676 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35367]: listed object 25... 2024-09-16T04:39:25.676 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35367]: saw 50 objects 2024-09-16T04:39:25.676 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35367]: shutting down. 2024-09-16T04:39:26.580 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:26.580 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[35366]: starting. 2024-09-16T04:39:26.580 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[35366]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-35318 2024-09-16T04:39:26.580 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[35366]: shutting down. 2024-09-16T04:39:26.582 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:26.582 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:26.582 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:39:26.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:39:28.672 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35435]: starting. 2024-09-16T04:39:28.672 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35435]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-35434 2024-09-16T04:39:28.672 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35435]: created object 0... 2024-09-16T04:39:28.673 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35435]: created object 25... 2024-09-16T04:39:28.673 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35435]: created object 49... 2024-09-16T04:39:28.673 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35435]: finishing. 2024-09-16T04:39:28.673 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35435]: shutting down. 2024-09-16T04:39:29.593 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[35436]: starting. 2024-09-16T04:39:29.593 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[35436]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-35434 2024-09-16T04:39:29.593 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[35436]: shutting down. 2024-09-16T04:39:31.895 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:31.895 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35469]: starting. 2024-09-16T04:39:31.895 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35469]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-35434 2024-09-16T04:39:31.896 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35469]: created object 0... 2024-09-16T04:39:31.896 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35469]: created object 25... 2024-09-16T04:39:31.896 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35469]: created object 49... 2024-09-16T04:39:31.896 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35469]: finishing. 2024-09-16T04:39:31.896 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35469]: shutting down. 2024-09-16T04:39:31.909 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:31.909 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35471]: starting. 2024-09-16T04:39:31.909 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35471]: listing objects. 2024-09-16T04:39:31.909 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35471]: listed object 0... 2024-09-16T04:39:31.909 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35471]: listed object 25... 2024-09-16T04:39:31.909 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35471]: saw 50 objects 2024-09-16T04:39:31.909 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35471]: shutting down. 2024-09-16T04:39:32.606 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:32.606 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[35470]: starting. 2024-09-16T04:39:32.607 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[35470]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-35434 2024-09-16T04:39:32.607 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[35470]: shutting down. 2024-09-16T04:39:32.607 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:32.608 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:32.608 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:39:32.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:39:34.707 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35538]: starting. 2024-09-16T04:39:34.707 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35538]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-35537 2024-09-16T04:39:34.707 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35538]: created object 0... 2024-09-16T04:39:34.707 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35538]: created object 25... 2024-09-16T04:39:34.707 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35538]: created object 49... 2024-09-16T04:39:34.707 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35538]: finishing. 2024-09-16T04:39:34.707 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35538]: shutting down. 2024-09-16T04:39:35.623 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[35539]: starting. 2024-09-16T04:39:35.623 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[35539]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-35537 2024-09-16T04:39:35.623 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[35539]: shutting down. 2024-09-16T04:39:37.726 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:37.727 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35572]: starting. 2024-09-16T04:39:37.727 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35572]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-35537 2024-09-16T04:39:37.727 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35572]: created object 0... 2024-09-16T04:39:37.727 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35572]: created object 25... 2024-09-16T04:39:37.727 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35572]: created object 49... 2024-09-16T04:39:37.727 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35572]: finishing. 2024-09-16T04:39:37.727 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35572]: shutting down. 2024-09-16T04:39:37.738 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:37.738 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35574]: starting. 2024-09-16T04:39:37.738 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35574]: listing objects. 2024-09-16T04:39:37.738 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35574]: listed object 0... 2024-09-16T04:39:37.738 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35574]: listed object 25... 2024-09-16T04:39:37.739 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35574]: saw 50 objects 2024-09-16T04:39:37.739 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35574]: shutting down. 2024-09-16T04:39:38.634 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:38.634 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[35573]: starting. 2024-09-16T04:39:38.634 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[35573]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-35537 2024-09-16T04:39:38.634 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[35573]: shutting down. 2024-09-16T04:39:38.635 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:38.635 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:38.635 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:39:38.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:39:40.731 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35641]: starting. 2024-09-16T04:39:40.731 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35641]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-35640 2024-09-16T04:39:40.732 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35641]: created object 0... 2024-09-16T04:39:40.732 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35641]: created object 25... 2024-09-16T04:39:40.732 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35641]: created object 49... 2024-09-16T04:39:40.732 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35641]: finishing. 2024-09-16T04:39:40.732 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35641]: shutting down. 2024-09-16T04:39:41.657 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[35642]: starting. 2024-09-16T04:39:41.658 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[35642]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-35640 2024-09-16T04:39:41.658 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[35642]: shutting down. 2024-09-16T04:39:42.146 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:39:42.146 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T04:39:42.146 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:39:42.147 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:39:42.149 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:39:42.149 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:39:42.150 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:39:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.262+0000 7f82da328640 10 monclient: get_monmap_and_config 2024-09-16T04:39:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.262+0000 7f82da328640 10 monclient: build_initial_monmap 2024-09-16T04:39:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.262+0000 7f82da328640 10 monclient: monmap: 2024-09-16T04:39:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:42.263803+0000 2024-09-16T04:39:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:42.263803+0000 2024-09-16T04:39:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:42.265 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:42.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82da328640 1 Processor -- start 2024-09-16T04:39:42.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82da328640 1 -- start start 2024-09-16T04:39:42.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82da328640 10 monclient: init 2024-09-16T04:39:42.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82da328640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:42.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82da328640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-16T04:39:42.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82da328640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d40a3c10 0x7f82d409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82da328640 10 monclient(hunting): picked mon.noname-b con 0x7f82d40a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:42.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82da328640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f82d40a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:42.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82da328640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f82d413be90 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:42.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82da328640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82d409ab70 con 0x7f82d413be90 2024-09-16T04:39:42.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82da328640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82d40a4d60 con 0x7f82d40a3ff0 2024-09-16T04:39:42.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82da328640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:42.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82da328640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f82d409b030 con 0x7f82d40a3c10 2024-09-16T04:39:42.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82da328640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82da328640 10 monclient(hunting): authenticate will time out at 780.679762s 2024-09-16T04:39:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82d88c5640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f82d413be90 0x7f82d413e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33870/0 (socket says 172.21.15.46:33870) 2024-09-16T04:39:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82d3fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d40a3c10 0x7f82d409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82d88c5640 1 -- 172.21.15.46:0/1127334427 learned_addr learned my addr 172.21.15.46:0/1127334427 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82d3fff640 10 monclient(hunting): get_auth_request con 0x7f82d40a3c10 auth_method 0 2024-09-16T04:39:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.263+0000 7f82d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d3fff640 1 -- 172.21.15.46:0/1127334427 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d40a3c10 msgr2=0x7f82d409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:39:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d3fff640 1 -- 172.21.15.46:0/1127334427 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d40a3c10 msgr2=0x7f82d409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:39:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d3fff640 1 --2- 172.21.15.46:0/1127334427 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d40a3c10 0x7f82d409bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d3fff640 1 --2- 172.21.15.46:0/1127334427 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d40a3c10 0x7f82d409bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:42.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 1 -- 172.21.15.46:0/1127334427 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1700029266 0 0) 0x7f82d409ab70 con 0x7f82d413be90 2024-09-16T04:39:42.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:39:42.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:42.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient(hunting): my global_id is 4626 2024-09-16T04:39:42.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 1 -- 172.21.15.46:0/1127334427 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82bc003520 con 0x7f82d413be90 2024-09-16T04:39:42.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 1 -- 172.21.15.46:0/1127334427 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 555826646 0 0) 0x7f82d40a4d60 con 0x7f82d40a3ff0 2024-09-16T04:39:42.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:39:42.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:42.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient(hunting): my global_id is 4603 2024-09-16T04:39:42.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 1 -- 172.21.15.46:0/1127334427 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82d409ab70 con 0x7f82d40a3ff0 2024-09-16T04:39:42.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 1 -- 172.21.15.46:0/1127334427 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2084272821 0 0) 0x7f82bc003520 con 0x7f82d413be90 2024-09-16T04:39:42.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 1 -- 172.21.15.46:0/1127334427 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d40a3c10 msgr2=0x7f82d409bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:39:42.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 1 --2- 172.21.15.46:0/1127334427 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d40a3c10 0x7f82d409bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:42.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 1 -- 172.21.15.46:0/1127334427 >> v1:172.21.15.114:6789/0 conn(0x7f82d40a3ff0 legacy=0x7f82d40a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:42.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:39:42.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:39:42.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:42.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:39:42.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 1 -- 172.21.15.46:0/1127334427 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d409b310 con 0x7f82d413be90 2024-09-16T04:39:42.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:39:42.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:39:42.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:42.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 1 -- 172.21.15.46:0/1127334427 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f82cc0019f0 con 0x7f82d413be90 2024-09-16T04:39:42.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:42.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:39:42.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient: dump: 2024-09-16T04:39:42.283 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:42.283 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:42.283 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.283 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.283 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:42.283 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:42.283 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:42.283 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:42.283 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:42.284 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:42.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:39:42.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:42.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 1 -- 172.21.15.46:0/1127334427 >> v1:172.21.15.46:6789/0 conn(0x7f82d413be90 legacy=0x7f82d413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:42.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:39:42.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 1 --2- 172.21.15.46:0/1127334427 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d40a3c10 0x7f82bc003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient(hunting): picked mon.c con 0x7f82d40a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:42.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 1 --2- 172.21.15.46:0/1127334427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82bc0040d0 0x7f82bc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient(hunting): picked mon.a con 0x7f82bc0040d0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:42.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 1 --2- 172.21.15.46:0/1127334427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82bc004d90 0x7f82bc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient(hunting): picked mon.b con 0x7f82bc004d90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:42.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:42.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 1 -- 172.21.15.46:0/1127334427 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f82d409b030 con 0x7f82bc0040d0 2024-09-16T04:39:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 1 -- 172.21.15.46:0/1127334427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f82bc00bc70 con 0x7f82bc004d90 2024-09-16T04:39:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 1 -- 172.21.15.46:0/1127334427 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f82bc00be70 con 0x7f82d40a3c10 2024-09-16T04:39:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:39:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d37fe640 1 --2- 172.21.15.46:0/1127334427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82bc0040d0 0x7f82bc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d3fff640 1 --2- 172.21.15.46:0/1127334427 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d40a3c10 0x7f82bc003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d37fe640 10 monclient(hunting): get_auth_request con 0x7f82bc0040d0 auth_method 0 2024-09-16T04:39:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d3fff640 10 monclient(hunting): get_auth_request con 0x7f82d40a3c10 auth_method 0 2024-09-16T04:39:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d88c5640 1 --2- 172.21.15.46:0/1127334427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82bc004d90 0x7f82bc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d88c5640 10 monclient(hunting): get_auth_request con 0x7f82bc004d90 auth_method 0 2024-09-16T04:39:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d88c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d88c5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.264+0000 7f82d88c5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.265+0000 7f82d88c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.265+0000 7f82d88c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.265+0000 7f82d88c5640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.265+0000 7f82d37fe640 10 monclient(hunting): handle_auth_done global_id 4626 payload 293 2024-09-16T04:39:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.265+0000 7f82d37fe640 1 -- 172.21.15.46:0/1127334427 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d40a3c10 msgr2=0x7f82bc003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.265+0000 7f82d37fe640 1 --2- 172.21.15.46:0/1127334427 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d40a3c10 0x7f82bc003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.265+0000 7f82d37fe640 1 -- 172.21.15.46:0/1127334427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82bc004d90 msgr2=0x7f82bc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.265+0000 7f82d37fe640 1 --2- 172.21.15.46:0/1127334427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82bc004d90 0x7f82bc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.265+0000 7f82d37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:39:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.265+0000 7f82d37fe640 1 monclient: found mon.a 2024-09-16T04:39:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.265+0000 7f82d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.265+0000 7f82d37fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.265+0000 7f82d37fe640 1 -- 172.21.15.46:0/1127334427 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82bc003520 con 0x7f82bc0040d0 2024-09-16T04:39:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.265+0000 7f82d37fe640 10 monclient: _finish_auth 0 2024-09-16T04:39:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.265+0000 7f82d37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:39:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.265+0000 7f82d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.265+0000 7f82d3fff640 1 --2- 172.21.15.46:0/1127334427 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d40a3c10 0x7f82bc003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.265+0000 7f82da328640 5 monclient: authenticate success, global_id 4626 2024-09-16T04:39:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.265+0000 7f82da328640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.265+0000 7f82d37fe640 1 --2- 172.21.15.46:0/1127334427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82bc0040d0 0x7f82bc004800 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f82c80018d0 tx=0x7f82bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a6aafe53ad9a597 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82d2ffd640 1 -- 172.21.15.46:0/1127334427 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c80014f0 con 0x7f82bc0040d0 2024-09-16T04:39:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82d2ffd640 10 monclient: dump: 2024-09-16T04:39:42.288 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:42.289 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82d2ffd640 1 -- 172.21.15.46:0/1127334427 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82c80052d0 con 0x7f82bc0040d0 2024-09-16T04:39:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82da328640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82d2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82d2ffd640 1 -- 172.21.15.46:0/1127334427 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c802aca0 con 0x7f82bc0040d0 2024-09-16T04:39:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82da328640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82da328640 4 set_mon_vals no callback set 2024-09-16T04:39:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82d2ffd640 10 monclient: dump: 2024-09-16T04:39:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:42.291 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82da328640 10 monclient: shutdown 2024-09-16T04:39:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82da328640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82da328640 1 -- 172.21.15.46:0/1127334427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82bc0040d0 msgr2=0x7f82bc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82da328640 1 --2- 172.21.15.46:0/1127334427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82bc0040d0 0x7f82bc004800 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f82c80018d0 tx=0x7f82bc005830 comp rx=0 tx=0).stop 2024-09-16T04:39:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82d88c5640 1 -- 172.21.15.46:0/1127334427 reap_dead start 2024-09-16T04:39:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82da328640 1 -- 172.21.15.46:0/1127334427 shutdown_connections 2024-09-16T04:39:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82da328640 1 -- 172.21.15.46:0/1127334427 >> 172.21.15.46:0/1127334427 conn(0x7f82d40a5d10 msgr2=0x7f82d40a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82da328640 1 -- 172.21.15.46:0/1127334427 shutdown_connections 2024-09-16T04:39:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.266+0000 7f82da328640 1 -- 172.21.15.46:0/1127334427 wait complete. 2024-09-16T04:39:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.267+0000 7f82da328640 10 monclient: build_initial_monmap 2024-09-16T04:39:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.267+0000 7f82da328640 10 monclient: monmap: 2024-09-16T04:39:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:42.292 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.267+0000 7f82da328640 1 Processor -- start 2024-09-16T04:39:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82da328640 1 -- start start 2024-09-16T04:39:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82da328640 10 monclient: init 2024-09-16T04:39:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82da328640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82da328640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:39:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82da328640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82bc0040d0 0x7f82d41cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82da328640 10 monclient(hunting): picked mon.noname-b con 0x7f82bc0040d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82da328640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d41cc7e0 0x7f82d411fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82da328640 10 monclient(hunting): picked mon.noname-c con 0x7f82d41cc7e0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82da328640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d41ccc60 0x7f82d4120400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82da328640 10 monclient(hunting): picked mon.noname-a con 0x7f82d41ccc60 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82da328640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82da328640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f82d409be70 con 0x7f82d41ccc60 2024-09-16T04:39:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82da328640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82da328640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f82d409bcf0 con 0x7f82bc0040d0 2024-09-16T04:39:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82da328640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82da328640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f82d409bff0 con 0x7f82d41cc7e0 2024-09-16T04:39:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82da328640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82da328640 10 monclient(hunting): authenticate will time out at 780.684602s 2024-09-16T04:39:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d37fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d41cc7e0 0x7f82d411fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d88c5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d41ccc60 0x7f82d4120400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d37fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d41cc7e0 0x7f82d411fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39370/0 (socket says 172.21.15.46:39370) 2024-09-16T04:39:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d88c5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d41ccc60 0x7f82d4120400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33672/0 (socket says 172.21.15.46:33672) 2024-09-16T04:39:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d37fe640 1 -- 172.21.15.46:0/2836637341 learned_addr learned my addr 172.21.15.46:0/2836637341 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d37fe640 10 monclient(hunting): get_auth_request con 0x7f82d41cc7e0 auth_method 0 2024-09-16T04:39:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d88c5640 10 monclient(hunting): get_auth_request con 0x7f82d41ccc60 auth_method 0 2024-09-16T04:39:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d88c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d88c5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d88c5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d3fff640 1 --2- 172.21.15.46:0/2836637341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82bc0040d0 0x7f82d41cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d3fff640 10 monclient(hunting): get_auth_request con 0x7f82bc0040d0 auth_method 0 2024-09-16T04:39:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d88c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d88c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d88c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d37fe640 1 -- 172.21.15.46:0/2836637341 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d41cc7e0 msgr2=0x7f82d411fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:39:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d37fe640 1 -- 172.21.15.46:0/2836637341 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d41cc7e0 msgr2=0x7f82d411fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:39:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d37fe640 1 --2- 172.21.15.46:0/2836637341 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d41cc7e0 0x7f82d411fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.268+0000 7f82d37fe640 1 --2- 172.21.15.46:0/2836637341 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d41cc7e0 0x7f82d411fec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d3fff640 10 monclient(hunting): handle_auth_done global_id 4606 payload 824 2024-09-16T04:39:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d3fff640 1 -- 172.21.15.46:0/2836637341 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d41cc7e0 msgr2=0x7f82d411fec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:39:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d3fff640 1 --2- 172.21.15.46:0/2836637341 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d41cc7e0 0x7f82d411fec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d3fff640 1 -- 172.21.15.46:0/2836637341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d41ccc60 msgr2=0x7f82d4120400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d3fff640 1 --2- 172.21.15.46:0/2836637341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d41ccc60 0x7f82d4120400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d3fff640 10 monclient: _finish_hunting 0 2024-09-16T04:39:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d3fff640 1 monclient: found mon.noname-b 2024-09-16T04:39:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3300/0 2024-09-16T04:39:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d3fff640 1 -- 172.21.15.46:0/2836637341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d4123e50 con 0x7f82bc0040d0 2024-09-16T04:39:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d88c5640 1 --2- 172.21.15.46:0/2836637341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d41ccc60 0x7f82d4120400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d3fff640 1 --2- 172.21.15.46:0/2836637341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82bc0040d0 0x7f82d41cc2a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f82c4010a00 tx=0x7f82c4010ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d0ff9640 1 -- 172.21.15.46:0/2836637341 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c401a020 con 0x7f82bc0040d0 2024-09-16T04:39:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d0ff9640 10 monclient: dump: 2024-09-16T04:39:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:42.299 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:42.300 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:42.300 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:42.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d0ff9640 10 monclient: _finish_auth 0 2024-09-16T04:39:42.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d0ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:39:42.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:42.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d0ff9640 1 -- 172.21.15.46:0/2836637341 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82c400eb40 con 0x7f82bc0040d0 2024-09-16T04:39:42.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:42.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d0ff9640 1 -- 172.21.15.46:0/2836637341 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c4016480 con 0x7f82bc0040d0 2024-09-16T04:39:42.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82da328640 5 monclient: authenticate success, global_id 4606 2024-09-16T04:39:42.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:42.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:42.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d0ff9640 10 monclient: dump: 2024-09-16T04:39:42.301 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:42.301 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:42.301 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.301 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.301 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:42.301 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:42.301 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:42.301 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:42.301 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:42.301 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:42.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82da328640 10 monclient: _renew_subs 2024-09-16T04:39:42.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82d27fc640 4 set_mon_vals no callback set 2024-09-16T04:39:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82da328640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82da328640 1 -- 172.21.15.46:0/2836637341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82d40a0400 con 0x7f82bc0040d0 2024-09-16T04:39:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82da328640 10 monclient: _renew_subs 2024-09-16T04:39:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82da328640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.269+0000 7f82da328640 1 -- 172.21.15.46:0/2836637341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82d40a0d90 con 0x7f82bc0040d0 2024-09-16T04:39:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.270+0000 7f82d0ff9640 1 -- 172.21.15.46:0/2836637341 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f82c401d020 con 0x7f82bc0040d0 2024-09-16T04:39:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.270+0000 7f82da328640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.270+0000 7f82da328640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.270+0000 7f82da328640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.270+0000 7f82da328640 1 -- 172.21.15.46:0/2836637341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82d40a43d0 con 0x7f82bc0040d0 2024-09-16T04:39:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.271+0000 7f82d0ff9640 1 --2- 172.21.15.46:0/2836637341 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f82c00383b0 0x7f82c003a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.271+0000 7f82d0ff9640 1 -- 172.21.15.46:0/2836637341 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4346+0+0 (secure 0 0 0) 0x7f82c404c080 con 0x7f82bc0040d0 2024-09-16T04:39:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.271+0000 7f82d37fe640 1 --2- 172.21.15.46:0/2836637341 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f82c00383b0 0x7f82c003a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.271+0000 7f82d37fe640 10 monclient: get_auth_request con 0x7f82c00383b0 auth_method 0 2024-09-16T04:39:42.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.271+0000 7f82d37fe640 1 --2- 172.21.15.46:0/2836637341 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f82c00383b0 0x7f82c003a870 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f82c80010e0 tx=0x7f82c8009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:42.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.276+0000 7f82d0ff9640 1 -- 172.21.15.46:0/2836637341 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f82c4024020 con 0x7f82bc0040d0 2024-09-16T04:39:42.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.276+0000 7f82d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:42.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.276+0000 7f82d0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:42.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.523+0000 7f82da328640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:39:42.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.523+0000 7f82da328640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:39:42.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.523+0000 7f82da328640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:42.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.523+0000 7f82da328640 1 -- 172.21.15.46:0/2836637341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f82d4069150 con 0x7f82bc0040d0 2024-09-16T04:39:42.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.523+0000 7f82d0ff9640 1 -- 172.21.15.46:0/2836637341 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f82c4013590 con 0x7f82bc0040d0 2024-09-16T04:39:42.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.523+0000 7f82d0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:39:42.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.523+0000 7f82d0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:42.527 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:39:42.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.525+0000 7f82da328640 1 -- 172.21.15.46:0/2836637341 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f82c00383b0 msgr2=0x7f82c003a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:42.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.525+0000 7f82da328640 1 --2- 172.21.15.46:0/2836637341 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f82c00383b0 0x7f82c003a870 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f82c80010e0 tx=0x7f82c8009030 comp rx=0 tx=0).stop 2024-09-16T04:39:42.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.525+0000 7f82da328640 10 monclient: shutdown 2024-09-16T04:39:42.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.525+0000 7f82da328640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:42.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.525+0000 7f82da328640 1 -- 172.21.15.46:0/2836637341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82bc0040d0 msgr2=0x7f82d41cc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:42.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.525+0000 7f82da328640 1 --2- 172.21.15.46:0/2836637341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82bc0040d0 0x7f82d41cc2a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f82c4010a00 tx=0x7f82c4010ec0 comp rx=0 tx=0).stop 2024-09-16T04:39:42.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.525+0000 7f82da328640 1 -- 172.21.15.46:0/2836637341 shutdown_connections 2024-09-16T04:39:42.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.525+0000 7f82da328640 1 --2- 172.21.15.46:0/2836637341 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f82c00383b0 0x7f82c003a870 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:42.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.525+0000 7f82da328640 1 --2- 172.21.15.46:0/2836637341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d41ccc60 0x7f82d4120400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:42.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.525+0000 7f82da328640 1 --2- 172.21.15.46:0/2836637341 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82d41cc7e0 0x7f82d411fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:42.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.525+0000 7f82da328640 1 --2- 172.21.15.46:0/2836637341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82bc0040d0 0x7f82d41cc2a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:42.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.525+0000 7f82da328640 1 -- 172.21.15.46:0/2836637341 >> 172.21.15.46:0/2836637341 conn(0x7f82d40a5d10 msgr2=0x7f82d409f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:42.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.525+0000 7f82da328640 1 -- 172.21.15.46:0/2836637341 shutdown_connections 2024-09-16T04:39:42.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.526+0000 7f82da328640 1 -- 172.21.15.46:0/2836637341 wait complete. 2024-09-16T04:39:42.541 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:39:42.541 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:39:42.541 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:39:42.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.641+0000 7f69f68ba640 10 monclient: get_monmap_and_config 2024-09-16T04:39:42.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.641+0000 7f69f68ba640 10 monclient: build_initial_monmap 2024-09-16T04:39:42.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.641+0000 7f69f68ba640 10 monclient: monmap: 2024-09-16T04:39:42.644 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:42.644 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:42.644 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:42.643093+0000 2024-09-16T04:39:42.644 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:42.643093+0000 2024-09-16T04:39:42.644 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:42.644 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:42.644 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:42.644 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:42.644 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:42.644 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:42.644 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:42.645 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:42.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.642+0000 7f69f68ba640 1 Processor -- start 2024-09-16T04:39:42.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.642+0000 7f69f68ba640 1 -- start start 2024-09-16T04:39:42.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.642+0000 7f69f68ba640 10 monclient: init 2024-09-16T04:39:42.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.642+0000 7f69f68ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:42.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.642+0000 7f69f68ba640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-16T04:39:42.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.642+0000 7f69f68ba640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69f00a3c10 0x7f69f009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.642+0000 7f69f68ba640 10 monclient(hunting): picked mon.noname-b con 0x7f69f00a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:42.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.642+0000 7f69f68ba640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f69f00a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:42.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.642+0000 7f69f68ba640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f69f013be90 0x7f69f013e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.642+0000 7f69f68ba640 10 monclient(hunting): picked mon.noname-c con 0x7f69f013be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:42.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.642+0000 7f69f68ba640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69f009b940 con 0x7f69f00a3ff0 2024-09-16T04:39:42.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.642+0000 7f69f68ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:42.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.642+0000 7f69f68ba640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f69f009ab00 con 0x7f69f013be90 2024-09-16T04:39:42.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.642+0000 7f69f68ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:42.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.642+0000 7f69f68ba640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f69f009b030 con 0x7f69f00a3c10 2024-09-16T04:39:42.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.642+0000 7f69f68ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:42.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.642+0000 7f69f68ba640 10 monclient(hunting): authenticate will time out at 781.058919s 2024-09-16T04:39:42.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69effff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69f00a3c10 0x7f69f009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69effff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69f00a3c10 0x7f69f009bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39374/0 (socket says 172.21.15.46:39374) 2024-09-16T04:39:42.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69effff640 1 -- 172.21.15.46:0/2879739358 learned_addr learned my addr 172.21.15.46:0/2879739358 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:42.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69effff640 10 monclient(hunting): get_auth_request con 0x7f69f00a3c10 auth_method 0 2024-09-16T04:39:42.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:42.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69effff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:42.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69effff640 1 -- 172.21.15.46:0/2879739358 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69f00a3c10 msgr2=0x7f69f009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:39:42.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69effff640 1 -- 172.21.15.46:0/2879739358 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69f00a3c10 msgr2=0x7f69f009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:39:42.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69effff640 1 --2- 172.21.15.46:0/2879739358 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69f00a3c10 0x7f69f009bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:42.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69effff640 1 --2- 172.21.15.46:0/2879739358 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69f00a3c10 0x7f69f009bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69f4e57640 1 --2- 172.21.15.46:0/2879739358 >> v2:172.21.15.114:3300/0 conn(0x7f69f013be90 0x7f69f013e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69f4e57640 10 monclient(hunting): get_auth_request con 0x7f69f013be90 auth_method 0 2024-09-16T04:39:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69f4e57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69f4e57640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69f4e57640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69f4e57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69f4e57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69f4e57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69f4e57640 10 monclient(hunting): handle_auth_done global_id 4609 payload 274 2024-09-16T04:39:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69f4e57640 1 -- 172.21.15.46:0/2879739358 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69f00a3c10 msgr2=0x7f69f009bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:39:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69f4e57640 1 --2- 172.21.15.46:0/2879739358 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69f00a3c10 0x7f69f009bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69f4e57640 1 -- 172.21.15.46:0/2879739358 >> v1:172.21.15.114:6789/0 conn(0x7f69f00a3ff0 legacy=0x7f69f00a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:42.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69f4e57640 10 monclient: _finish_hunting 0 2024-09-16T04:39:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69f4e57640 1 monclient: found mon.noname-c 2024-09-16T04:39:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69f4e57640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69f4e57640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:39:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69f4e57640 1 -- 172.21.15.46:0/2879739358 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69f009b310 con 0x7f69f013be90 2024-09-16T04:39:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69eeffd640 1 -- 172.21.15.46:0/2879739358 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4256915080 0 0) 0x7f69f009b940 con 0x7f69f00a3ff0 2024-09-16T04:39:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.643+0000 7f69eeffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:39:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.644+0000 7f69f4e57640 1 --2- 172.21.15.46:0/2879739358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f013be90 0x7f69f013e280 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f69e8009d90 tx=0x7f69e802a530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43aac158f04d7916 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.644+0000 7f69eeffd640 1 -- 172.21.15.46:0/2879739358 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e8007740 con 0x7f69f013be90 2024-09-16T04:39:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.644+0000 7f69eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.644+0000 7f69eeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:39:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.644+0000 7f69eeffd640 10 monclient: dump: 2024-09-16T04:39:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:42.658 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.644+0000 7f69eeffd640 10 monclient: _finish_auth 0 2024-09-16T04:39:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.644+0000 7f69eeffd640 10 monclient: _check_auth_tickets 2024-09-16T04:39:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.644+0000 7f69eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.644+0000 7f69eeffd640 1 -- 172.21.15.46:0/2879739358 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69e80078e0 con 0x7f69f013be90 2024-09-16T04:39:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.644+0000 7f69eeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.644+0000 7f69eeffd640 1 -- 172.21.15.46:0/2879739358 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e8007cb0 con 0x7f69f013be90 2024-09-16T04:39:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.644+0000 7f69f68ba640 5 monclient: authenticate success, global_id 4609 2024-09-16T04:39:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.644+0000 7f69f68ba640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.644+0000 7f69eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.644+0000 7f69f68ba640 4 set_mon_vals no callback set 2024-09-16T04:39:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.644+0000 7f69eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.644+0000 7f69eeffd640 10 monclient: dump: 2024-09-16T04:39:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:42.659 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:42.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.645+0000 7f69f68ba640 10 monclient: shutdown 2024-09-16T04:39:42.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.645+0000 7f69f68ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:42.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.645+0000 7f69f68ba640 1 -- 172.21.15.46:0/2879739358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f013be90 msgr2=0x7f69f013e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:42.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.645+0000 7f69f68ba640 1 --2- 172.21.15.46:0/2879739358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f013be90 0x7f69f013e280 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f69e8009d90 tx=0x7f69e802a530 comp rx=0 tx=0).stop 2024-09-16T04:39:42.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.645+0000 7f69f68ba640 1 -- 172.21.15.46:0/2879739358 shutdown_connections 2024-09-16T04:39:42.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.645+0000 7f69f68ba640 1 --2- 172.21.15.46:0/2879739358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f013be90 0x7f69f013e280 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:42.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.645+0000 7f69f68ba640 1 --2- 172.21.15.46:0/2879739358 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69f00a3c10 0x7f69f009bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:42.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.645+0000 7f69f68ba640 1 -- 172.21.15.46:0/2879739358 >> 172.21.15.46:0/2879739358 conn(0x7f69f00a6340 msgr2=0x7f69f006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:42.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.645+0000 7f69f68ba640 1 -- 172.21.15.46:0/2879739358 shutdown_connections 2024-09-16T04:39:42.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.645+0000 7f69f68ba640 1 -- 172.21.15.46:0/2879739358 wait complete. 2024-09-16T04:39:42.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.645+0000 7f69f68ba640 10 monclient: build_initial_monmap 2024-09-16T04:39:42.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.645+0000 7f69f68ba640 10 monclient: monmap: 2024-09-16T04:39:42.660 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:42.660 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:42.661 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:42.661 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:42.661 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:42.661 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:42.661 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:42.661 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:42.661 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:42.661 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:42.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.646+0000 7f69f68ba640 1 Processor -- start 2024-09-16T04:39:42.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.646+0000 7f69f68ba640 1 -- start start 2024-09-16T04:39:42.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.646+0000 7f69f68ba640 10 monclient: init 2024-09-16T04:39:42.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.646+0000 7f69f68ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:42.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.646+0000 7f69f68ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:39:42.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.646+0000 7f69f68ba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f00a3c10 0x7f69f01cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.646+0000 7f69f68ba640 10 monclient(hunting): picked mon.noname-a con 0x7f69f00a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:42.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.646+0000 7f69f68ba640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f00a3ff0 0x7f69f01cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.646+0000 7f69f68ba640 10 monclient(hunting): picked mon.noname-b con 0x7f69f00a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:42.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.646+0000 7f69f68ba640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69f013be90 0x7f69f009e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.646+0000 7f69f68ba640 10 monclient(hunting): picked mon.noname-c con 0x7f69f013be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:42.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.646+0000 7f69f68ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:42.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.646+0000 7f69f68ba640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f69f00a6b40 con 0x7f69f00a3c10 2024-09-16T04:39:42.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.646+0000 7f69f68ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:42.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.646+0000 7f69f68ba640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f69f00a69c0 con 0x7f69f00a3ff0 2024-09-16T04:39:42.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.646+0000 7f69f68ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:42.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.646+0000 7f69f68ba640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f69f013e7c0 con 0x7f69f013be90 2024-09-16T04:39:42.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.646+0000 7f69f68ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:42.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.646+0000 7f69f68ba640 10 monclient(hunting): authenticate will time out at 781.062941s 2024-09-16T04:39:42.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69f4e57640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69f013be90 0x7f69f009e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69effff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f00a3c10 0x7f69f01cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69effff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f00a3c10 0x7f69f01cb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33688/0 (socket says 172.21.15.46:33688) 2024-09-16T04:39:42.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69effff640 1 -- 172.21.15.46:0/4044259140 learned_addr learned my addr 172.21.15.46:0/4044259140 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:42.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69f4e57640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69f013be90 0x7f69f009e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39386/0 (socket says 172.21.15.46:39386) 2024-09-16T04:39:42.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69effff640 10 monclient(hunting): get_auth_request con 0x7f69f00a3c10 auth_method 0 2024-09-16T04:39:42.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:42.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69effff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:42.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69f4e57640 10 monclient(hunting): get_auth_request con 0x7f69f013be90 auth_method 0 2024-09-16T04:39:42.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69f4e57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:42.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69f4e57640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69f4e57640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:42.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69ef7fe640 1 --2- 172.21.15.46:0/4044259140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f00a3ff0 0x7f69f01cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69ef7fe640 10 monclient(hunting): get_auth_request con 0x7f69f00a3ff0 auth_method 0 2024-09-16T04:39:42.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69f4e57640 1 -- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69f013be90 msgr2=0x7f69f009e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:39:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69f4e57640 1 -- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69f013be90 msgr2=0x7f69f009e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:39:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69f4e57640 1 --2- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69f013be90 0x7f69f009e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69f4e57640 1 --2- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69f013be90 0x7f69f009e040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69effff640 10 monclient(hunting): handle_auth_done global_id 4632 payload 824 2024-09-16T04:39:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69effff640 1 -- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69f013be90 msgr2=0x7f69f009e040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:39:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69effff640 1 --2- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69f013be90 0x7f69f009e040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69effff640 1 -- 172.21.15.46:0/4044259140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f00a3ff0 msgr2=0x7f69f01cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69effff640 1 --2- 172.21.15.46:0/4044259140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f00a3ff0 0x7f69f01cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69effff640 10 monclient: _finish_hunting 0 2024-09-16T04:39:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69effff640 1 monclient: found mon.noname-a 2024-09-16T04:39:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69effff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69effff640 1 -- 172.21.15.46:0/4044259140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69f0121410 con 0x7f69f00a3c10 2024-09-16T04:39:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.647+0000 7f69effff640 1 --2- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f00a3c10 0x7f69f01cb7b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f69e000e840 tx=0x7f69e000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.648+0000 7f69ecff9640 1 -- 172.21.15.46:0/4044259140 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e0017070 con 0x7f69f00a3c10 2024-09-16T04:39:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.648+0000 7f69ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.648+0000 7f69ecff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.648+0000 7f69ecff9640 10 monclient: dump: 2024-09-16T04:39:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:42.666 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.648+0000 7f69ecff9640 10 monclient: _finish_auth 0 2024-09-16T04:39:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.648+0000 7f69ecff9640 10 monclient: _check_auth_tickets 2024-09-16T04:39:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.648+0000 7f69ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.648+0000 7f69ecff9640 1 -- 172.21.15.46:0/4044259140 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69e0004020 con 0x7f69f00a3c10 2024-09-16T04:39:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.648+0000 7f69f68ba640 5 monclient: authenticate success, global_id 4632 2024-09-16T04:39:42.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.648+0000 7f69f68ba640 10 monclient: _renew_subs 2024-09-16T04:39:42.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.648+0000 7f69f68ba640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:42.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.648+0000 7f69f68ba640 1 -- 172.21.15.46:0/4044259140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69f009fc60 con 0x7f69f00a3c10 2024-09-16T04:39:42.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.648+0000 7f69ecff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:42.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.648+0000 7f69ecff9640 1 -- 172.21.15.46:0/4044259140 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e000f040 con 0x7f69f00a3c10 2024-09-16T04:39:42.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.648+0000 7f69ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:42.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.648+0000 7f69ecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:42.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.648+0000 7f69ecff9640 10 monclient: dump: 2024-09-16T04:39:42.667 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:42.667 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:42.667 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.667 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.667 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:42.667 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:42.667 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:42.668 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:42.668 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:42.668 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:42.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.648+0000 7f69f68ba640 10 monclient: _renew_subs 2024-09-16T04:39:42.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.648+0000 7f69f68ba640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:42.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.648+0000 7f69f68ba640 1 -- 172.21.15.46:0/4044259140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69f011ff60 con 0x7f69f00a3c10 2024-09-16T04:39:42.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.648+0000 7f69ee7fc640 4 set_mon_vals no callback set 2024-09-16T04:39:42.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.649+0000 7f69f68ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:42.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.649+0000 7f69f68ba640 1 --2- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f01cc420 0x7f69f009e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.649+0000 7f69f68ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:42.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.649+0000 7f69f68ba640 1 -- 172.21.15.46:0/4044259140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f69f009bd70 con 0x7f69f01cc420 2024-09-16T04:39:42.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.649+0000 7f69ef7fe640 1 --2- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f01cc420 0x7f69f009e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.649+0000 7f69ecff9640 1 -- 172.21.15.46:0/4044259140 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f69e000c6d0 con 0x7f69f00a3c10 2024-09-16T04:39:42.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.649+0000 7f69ef7fe640 10 monclient: get_auth_request con 0x7f69f01cc420 auth_method 0 2024-09-16T04:39:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.649+0000 7f69ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.649+0000 7f69ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.649+0000 7f69ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.649+0000 7f69ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.649+0000 7f69ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.649+0000 7f69ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.650+0000 7f69ecff9640 1 --2- 172.21.15.46:0/4044259140 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f69dc0383b0 0x7f69dc03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.650+0000 7f69ef7fe640 10 monclient(hunting): handle_auth_done global_id 4635 payload 824 2024-09-16T04:39:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.650+0000 7f69ef7fe640 1 -- 172.21.15.46:0/4044259140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f69f00a0400 con 0x7f69f01cc420 2024-09-16T04:39:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.650+0000 7f69ecff9640 1 -- 172.21.15.46:0/4044259140 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4346+0+0 (secure 0 0 0) 0x7f69e004b1b0 con 0x7f69f00a3c10 2024-09-16T04:39:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.650+0000 7f69ef7fe640 1 --2- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f01cc420 0x7f69f009e580 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f69e40011e0 tx=0x7f69e400fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.650+0000 7f69f4e57640 1 --2- 172.21.15.46:0/4044259140 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f69dc0383b0 0x7f69dc03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.650+0000 7f69f4e57640 10 monclient: get_auth_request con 0x7f69dc0383b0 auth_method 0 2024-09-16T04:39:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.650+0000 7f69ecff9640 1 -- 172.21.15.46:0/4044259140 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e40050d0 con 0x7f69f01cc420 2024-09-16T04:39:42.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.650+0000 7f69ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:42.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.650+0000 7f69ecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:42.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.650+0000 7f69ecff9640 10 monclient: dump: 2024-09-16T04:39:42.670 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:42.670 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:42.670 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.670 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.670 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:42.670 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:42.670 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:42.670 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:42.670 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:42.670 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:42.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.650+0000 7f69ecff9640 1 -- 172.21.15.46:0/4044259140 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f69e4018070 con 0x7f69f01cc420 2024-09-16T04:39:42.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.650+0000 7f69ecff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:42.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.650+0000 7f69ecff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:42.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.650+0000 7f69ecff9640 1 -- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f01cc420 msgr2=0x7f69f009e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:42.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.650+0000 7f69ecff9640 1 --2- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f01cc420 0x7f69f009e580 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f69e40011e0 tx=0x7f69e400fc00 comp rx=0 tx=0).stop 2024-09-16T04:39:42.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.650+0000 7f69ecff9640 1 -- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f01cc420 msgr2=0x7f69f009e580 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:42.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.650+0000 7f69ecff9640 1 --2- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f01cc420 0x7f69f009e580 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f69e40011e0 tx=0x7f69e400fc00 comp rx=0 tx=0).stop 2024-09-16T04:39:42.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.650+0000 7f69f4e57640 1 --2- 172.21.15.46:0/4044259140 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f69dc0383b0 0x7f69dc03a870 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f69e8034000 tx=0x7f69e802f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:42.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.660+0000 7f69f68ba640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:39:42.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.660+0000 7f69f68ba640 1 --2- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f009eac0 0x7f69f0137e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.660+0000 7f69f68ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:42.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.660+0000 7f69f68ba640 1 -- 172.21.15.46:0/4044259140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f69f013c4e0 con 0x7f69f009eac0 2024-09-16T04:39:42.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.661+0000 7f69ef7fe640 1 --2- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f009eac0 0x7f69f0137e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.661+0000 7f69ef7fe640 10 monclient: get_auth_request con 0x7f69f009eac0 auth_method 0 2024-09-16T04:39:42.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.661+0000 7f69ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:42.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.661+0000 7f69ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.661+0000 7f69ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.661+0000 7f69ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.661+0000 7f69ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.661+0000 7f69ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.661+0000 7f69ef7fe640 10 monclient(hunting): handle_auth_done global_id 4638 payload 824 2024-09-16T04:39:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.661+0000 7f69ef7fe640 1 -- 172.21.15.46:0/4044259140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f69f009eea0 con 0x7f69f009eac0 2024-09-16T04:39:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.661+0000 7f69ef7fe640 1 --2- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f009eac0 0x7f69f0137e70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f69e40011e0 tx=0x7f69e4001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.661+0000 7f69ecff9640 1 -- 172.21.15.46:0/4044259140 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e400d770 con 0x7f69f009eac0 2024-09-16T04:39:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.661+0000 7f69ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.661+0000 7f69ecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.661+0000 7f69ecff9640 10 monclient: dump: 2024-09-16T04:39:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:42.675 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.662+0000 7f69ecff9640 1 -- 172.21.15.46:0/4044259140 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f69e400d770 con 0x7f69f009eac0 2024-09-16T04:39:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.662+0000 7f69ecff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:39:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.662+0000 7f69ecff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.662+0000 7f69ecff9640 1 -- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f009eac0 msgr2=0x7f69f0137e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.662+0000 7f69ecff9640 1 --2- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f009eac0 0x7f69f0137e70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f69e40011e0 tx=0x7f69e4001080 comp rx=0 tx=0).stop 2024-09-16T04:39:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.662+0000 7f69ecff9640 1 -- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f009eac0 msgr2=0x7f69f0137e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.662+0000 7f69ecff9640 1 --2- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f009eac0 0x7f69f0137e70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:42.676 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:39:42.676 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:39:42.676 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:39:42.676 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:39:42.676 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 14, 2024-09-16T04:39:42.676 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 68600, 2024-09-16T04:39:42.676 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:39:42.676 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:39:42.676 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:39:42.676 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:39:42.676 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:39:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:39:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:39:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:42.677 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:42.678 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:42.678 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:42.678 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:39:42.678 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:39:42.678 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:42.678 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:42.678 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:42.678 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:42.678 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:42.678 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:42.678 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:42.678 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:42.678 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:42.678 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:42.679 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:42.679 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:42.679 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:42.679 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:39:42.679 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:39:42.679 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:39:42.679 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:39:42.679 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:39:42.679 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:39:42.679 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:42.679 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:42.679 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:39:42.679 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:39:42.680 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:39:42.680 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:39:42.680 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:39:42.680 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:39:42.680 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:39:42.680 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:42.680 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:39:42.680 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:42.680 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:42.680 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:42.680 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:42.680 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:42.680 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:42.680 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:42.681 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:42.681 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:42.681 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:42.681 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:42.681 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:42.681 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:39:42.681 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:42.681 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:39:42.681 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:42.681 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:39:42.681 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:39:42.681 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:42.681 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:42.682 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:42.682 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:42.682 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:39:42.682 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:42.682 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:42.682 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:42.682 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:42.682 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:39:42.682 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:42.682 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:42.682 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:42.682 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:42.682 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:39:42.682 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:39:42.683 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:42.683 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:42.683 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:42.683 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:42.683 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:42.683 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:39:42.683 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:39:42.683 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:42.683 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:42.683 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:42.683 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:42.683 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:39:42.683 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:42.683 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:42.684 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:42.684 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:42.684 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:39:42.684 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:42.684 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:42.684 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:42.684 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:42.684 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:39:42.684 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:39:42.684 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:42.684 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:42.684 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:42.684 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:42.684 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:42.684 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:39:42.685 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:39:42.685 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:42.685 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:42.685 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:42.685 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:42.685 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:39:42.685 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:42.685 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:42.685 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:42.685 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:42.685 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:39:42.685 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:42.685 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:42.685 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:42.686 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:42.686 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:39:42.686 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:39:42.686 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:42.686 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:42.686 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:42.686 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:42.686 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:42.686 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:42.686 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:39:42.686 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:39:42.686 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:42.686 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:42.686 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:42.686 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:39:42.687 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:42.687 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:42.687 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:39:42.687 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:42.687 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:42.687 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:42.687 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:39:42.687 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:42.687 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:42.687 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:39:42.687 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:42.687 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:42.687 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:42.687 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:39:42.688 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:42.688 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:42.688 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:42.688 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:39:42.688 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:39:42.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.662+0000 7f69f68ba640 1 -- 172.21.15.46:0/4044259140 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f69dc0383b0 msgr2=0x7f69dc03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:42.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.662+0000 7f69f68ba640 1 --2- 172.21.15.46:0/4044259140 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f69dc0383b0 0x7f69dc03a870 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f69e8034000 tx=0x7f69e802f000 comp rx=0 tx=0).stop 2024-09-16T04:39:42.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.662+0000 7f69f68ba640 10 monclient: shutdown 2024-09-16T04:39:42.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.662+0000 7f69f68ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:42.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.662+0000 7f69f68ba640 1 -- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f00a3c10 msgr2=0x7f69f01cb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:42.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.662+0000 7f69f68ba640 1 --2- 172.21.15.46:0/4044259140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f00a3c10 0x7f69f01cb7b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f69e000e840 tx=0x7f69e000ed00 comp rx=0 tx=0).stop 2024-09-16T04:39:42.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.662+0000 7f69f4e57640 1 -- 172.21.15.46:0/4044259140 reap_dead start 2024-09-16T04:39:42.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.662+0000 7f69f4e57640 1 -- 172.21.15.46:0/4044259140 reap_dead start 2024-09-16T04:39:42.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.662+0000 7f69f68ba640 1 -- 172.21.15.46:0/4044259140 shutdown_connections 2024-09-16T04:39:42.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.662+0000 7f69f68ba640 1 -- 172.21.15.46:0/4044259140 >> 172.21.15.46:0/4044259140 conn(0x7f69f00a6340 msgr2=0x7f69f00a8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:42.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.662+0000 7f69f68ba640 1 -- 172.21.15.46:0/4044259140 shutdown_connections 2024-09-16T04:39:42.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.662+0000 7f69f68ba640 1 -- 172.21.15.46:0/4044259140 wait complete. 2024-09-16T04:39:42.690 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:39:42.690 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:39:42.691 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:39:42.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.790+0000 7fa0fc4d7640 10 monclient: get_monmap_and_config 2024-09-16T04:39:42.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.790+0000 7fa0fc4d7640 10 monclient: build_initial_monmap 2024-09-16T04:39:42.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.790+0000 7fa0fc4d7640 10 monclient: monmap: 2024-09-16T04:39:42.792 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:42.792 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:42.792 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:42.791649+0000 2024-09-16T04:39:42.792 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:42.791649+0000 2024-09-16T04:39:42.793 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:42.793 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:42.793 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:42.793 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:42.793 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:42.793 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:42.793 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:42.793 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:42.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.790+0000 7fa0fc4d7640 1 Processor -- start 2024-09-16T04:39:42.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.790+0000 7fa0fc4d7640 1 -- start start 2024-09-16T04:39:42.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.790+0000 7fa0fc4d7640 10 monclient: init 2024-09-16T04:39:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fc4d7640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fc4d7640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-16T04:39:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fc4d7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa0f40a3c10 0x7fa0f409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fc4d7640 10 monclient(hunting): picked mon.noname-a con 0x7fa0f40a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fc4d7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa0f40a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fc4d7640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f413be90 0x7fa0f413e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fc4d7640 10 monclient(hunting): picked mon.noname-b con 0x7fa0f413be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fc4d7640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0f409ab70 con 0x7fa0f40a3ff0 2024-09-16T04:39:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fc4d7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fc4d7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa0f40a4d60 con 0x7fa0f40a3c10 2024-09-16T04:39:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fc4d7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fc4d7640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fa0f409b030 con 0x7fa0f413be90 2024-09-16T04:39:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fc4d7640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:42.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fc4d7640 10 monclient(hunting): authenticate will time out at 781.207593s 2024-09-16T04:39:42.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fa273640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa0f40a3c10 0x7fa0f409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0faa74640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f413be90 0x7fa0f413e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fa273640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa0f40a3c10 0x7fa0f409bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33716/0 (socket says 172.21.15.46:33716) 2024-09-16T04:39:42.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fa273640 1 -- 172.21.15.46:0/2276980930 learned_addr learned my addr 172.21.15.46:0/2276980930 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:42.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fa273640 10 monclient(hunting): get_auth_request con 0x7fa0f40a3c10 auth_method 0 2024-09-16T04:39:42.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fa273640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:42.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fa273640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fa273640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:42.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0faa74640 10 monclient(hunting): get_auth_request con 0x7fa0f413be90 auth_method 0 2024-09-16T04:39:42.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0faa74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:42.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0faa74640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0faa74640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:42.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fa273640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:42.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fa273640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:42.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fa273640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:42.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0faa74640 1 -- 172.21.15.46:0/2276980930 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f413be90 msgr2=0x7fa0f413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:39:42.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0faa74640 1 -- 172.21.15.46:0/2276980930 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f413be90 msgr2=0x7fa0f413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:39:42.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0faa74640 1 --2- 172.21.15.46:0/2276980930 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f413be90 0x7fa0f413e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:42.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0faa74640 1 --2- 172.21.15.46:0/2276980930 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f413be90 0x7fa0f413e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:42.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fa273640 10 monclient(hunting): handle_auth_done global_id 4641 payload 274 2024-09-16T04:39:42.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fa273640 1 -- 172.21.15.46:0/2276980930 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f413be90 msgr2=0x7fa0f413e280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:39:42.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fa273640 1 --2- 172.21.15.46:0/2276980930 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f413be90 0x7fa0f413e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:42.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.791+0000 7fa0fa273640 1 -- 172.21.15.46:0/2276980930 >> v1:172.21.15.114:6789/0 conn(0x7fa0f40a3ff0 legacy=0x7fa0f40a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:42.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0fa273640 10 monclient: _finish_hunting 0 2024-09-16T04:39:42.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0fa273640 1 monclient: found mon.noname-a 2024-09-16T04:39:42.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0fa273640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:42.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0fa273640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:42.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0fa273640 1 -- 172.21.15.46:0/2276980930 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f409b310 con 0x7fa0f40a3c10 2024-09-16T04:39:42.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0fa273640 1 --2- 172.21.15.46:0/2276980930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0f40a3c10 0x7fa0f409bcf0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0009870 tx=0x7fa0f002a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3cc977f0aad8383 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:42.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0f9271640 1 -- 172.21.15.46:0/2276980930 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0037070 con 0x7fa0f40a3c10 2024-09-16T04:39:42.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0f9271640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:42.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0f9271640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:42.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0f9271640 10 monclient: dump: 2024-09-16T04:39:42.806 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:42.806 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:42.806 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.807 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.807 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:42.807 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:42.807 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:42.807 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:42.807 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:42.807 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:42.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0f9271640 10 monclient: _finish_auth 0 2024-09-16T04:39:42.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0f9271640 10 monclient: _check_auth_tickets 2024-09-16T04:39:42.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0f9271640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:42.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0f9271640 1 -- 172.21.15.46:0/2276980930 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0f0032440 con 0x7fa0f40a3c10 2024-09-16T04:39:42.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0fc4d7640 5 monclient: authenticate success, global_id 4641 2024-09-16T04:39:42.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0fc4d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:42.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0f9271640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:42.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0fc4d7640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:42.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0f9271640 1 -- 172.21.15.46:0/2276980930 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0032810 con 0x7fa0f40a3c10 2024-09-16T04:39:42.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0f9271640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:42.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0fc4d7640 4 set_mon_vals no callback set 2024-09-16T04:39:42.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0f9271640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:42.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.792+0000 7fa0f9271640 10 monclient: dump: 2024-09-16T04:39:42.809 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:42.809 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:42.809 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.809 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.809 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:42.809 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:42.809 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:42.809 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:42.809 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:42.809 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:42.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.793+0000 7fa0fc4d7640 10 monclient: shutdown 2024-09-16T04:39:42.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.793+0000 7fa0fc4d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:42.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.793+0000 7fa0fc4d7640 1 -- 172.21.15.46:0/2276980930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0f40a3c10 msgr2=0x7fa0f409bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:42.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.793+0000 7fa0fc4d7640 1 --2- 172.21.15.46:0/2276980930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0f40a3c10 0x7fa0f409bcf0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0009870 tx=0x7fa0f002a130 comp rx=0 tx=0).stop 2024-09-16T04:39:42.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.793+0000 7fa0fc4d7640 1 -- 172.21.15.46:0/2276980930 shutdown_connections 2024-09-16T04:39:42.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.793+0000 7fa0fc4d7640 1 --2- 172.21.15.46:0/2276980930 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f413be90 0x7fa0f413e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:42.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.793+0000 7fa0fc4d7640 1 --2- 172.21.15.46:0/2276980930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0f40a3c10 0x7fa0f409bcf0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:42.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.793+0000 7fa0fc4d7640 1 -- 172.21.15.46:0/2276980930 >> 172.21.15.46:0/2276980930 conn(0x7fa0f40a5d10 msgr2=0x7fa0f40a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:42.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.793+0000 7fa0fc4d7640 1 -- 172.21.15.46:0/2276980930 shutdown_connections 2024-09-16T04:39:42.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.793+0000 7fa0fc4d7640 1 -- 172.21.15.46:0/2276980930 wait complete. 2024-09-16T04:39:42.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.794+0000 7fa0fc4d7640 10 monclient: build_initial_monmap 2024-09-16T04:39:42.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.794+0000 7fa0fc4d7640 10 monclient: monmap: 2024-09-16T04:39:42.810 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:42.810 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:42.810 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:42.810 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:42.810 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:42.811 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:42.811 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:42.811 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:42.811 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:42.811 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:42.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.794+0000 7fa0fc4d7640 1 Processor -- start 2024-09-16T04:39:42.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.794+0000 7fa0fc4d7640 1 -- start start 2024-09-16T04:39:42.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.794+0000 7fa0fc4d7640 10 monclient: init 2024-09-16T04:39:42.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.794+0000 7fa0fc4d7640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:42.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.794+0000 7fa0fc4d7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:39:42.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fc4d7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0f40a3c10 0x7fa0f4133d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fc4d7640 10 monclient(hunting): picked mon.noname-a con 0x7fa0f40a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:42.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fc4d7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0f40a3ff0 0x7fa0f41342a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fc4d7640 10 monclient(hunting): picked mon.noname-b con 0x7fa0f40a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:42.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fc4d7640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f413be90 0x7fa0f41cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fc4d7640 10 monclient(hunting): picked mon.noname-c con 0x7fa0f413be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:42.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fc4d7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:42.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fc4d7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f406e550 con 0x7fa0f40a3c10 2024-09-16T04:39:42.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fc4d7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:42.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fc4d7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f406e3d0 con 0x7fa0f40a3ff0 2024-09-16T04:39:42.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fc4d7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:42.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fc4d7640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fa0f413e7c0 con 0x7fa0f413be90 2024-09-16T04:39:42.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fc4d7640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:42.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fc4d7640 10 monclient(hunting): authenticate will time out at 781.211105s 2024-09-16T04:39:42.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fa273640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0f40a3c10 0x7fa0f4133d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fa273640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0f40a3c10 0x7fa0f4133d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33722/0 (socket says 172.21.15.46:33722) 2024-09-16T04:39:42.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fa273640 1 -- 172.21.15.46:0/1478635216 learned_addr learned my addr 172.21.15.46:0/1478635216 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:42.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fa273640 10 monclient(hunting): get_auth_request con 0x7fa0f40a3c10 auth_method 0 2024-09-16T04:39:42.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fa273640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:42.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fa273640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fa273640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:42.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0faa74640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f413be90 0x7fa0f41cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0faa74640 10 monclient(hunting): get_auth_request con 0x7fa0f413be90 auth_method 0 2024-09-16T04:39:42.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0faa74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:42.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0faa74640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0faa74640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:42.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fa273640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:42.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fa273640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:42.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fa273640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:42.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0faa74640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f413be90 msgr2=0x7fa0f41cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:39:42.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0faa74640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f413be90 msgr2=0x7fa0f41cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:39:42.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0faa74640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f413be90 0x7fa0f41cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:42.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0faa74640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f413be90 0x7fa0f41cb960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:42.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0f40a3ff0 0x7fa0f41342a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0f9a72640 10 monclient(hunting): get_auth_request con 0x7fa0f40a3ff0 auth_method 0 2024-09-16T04:39:42.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0f9a72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:42.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0f9a72640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0f9a72640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:42.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fa273640 10 monclient(hunting): handle_auth_done global_id 4644 payload 824 2024-09-16T04:39:42.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fa273640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f413be90 msgr2=0x7fa0f41cb960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:39:42.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fa273640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f413be90 0x7fa0f41cb960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:42.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fa273640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0f40a3ff0 msgr2=0x7fa0f41342a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:42.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fa273640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0f40a3ff0 0x7fa0f41342a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:42.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fa273640 10 monclient: _finish_hunting 0 2024-09-16T04:39:42.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fa273640 1 monclient: found mon.noname-a 2024-09-16T04:39:42.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fa273640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:42.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fa273640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:42.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fa273640 1 -- 172.21.15.46:0/1478635216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f41397b0 con 0x7fa0f40a3c10 2024-09-16T04:39:42.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fa273640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0f40a3c10 0x7fa0f4133d60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0000c00 tx=0x7fa0f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:42.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0eaffd640 1 -- 172.21.15.46:0/1478635216 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0037040 con 0x7fa0f40a3c10 2024-09-16T04:39:42.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:42.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:42.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0eaffd640 10 monclient: dump: 2024-09-16T04:39:42.815 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:42.815 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:42.815 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.815 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.815 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:42.816 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0eaffd640 10 monclient: _finish_auth 0 2024-09-16T04:39:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0eaffd640 10 monclient: _check_auth_tickets 2024-09-16T04:39:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0eaffd640 1 -- 172.21.15.46:0/1478635216 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0f0042070 con 0x7fa0f40a3c10 2024-09-16T04:39:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0eaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0eaffd640 1 -- 172.21.15.46:0/1478635216 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0030980 con 0x7fa0f40a3c10 2024-09-16T04:39:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fc4d7640 5 monclient: authenticate success, global_id 4644 2024-09-16T04:39:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fc4d7640 10 monclient: _renew_subs 2024-09-16T04:39:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fc4d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fc4d7640 1 -- 172.21.15.46:0/1478635216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0f4139a90 con 0x7fa0f40a3c10 2024-09-16T04:39:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0eaffd640 10 monclient: dump: 2024-09-16T04:39:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:42.817 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0f8a70640 4 set_mon_vals no callback set 2024-09-16T04:39:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fc4d7640 10 monclient: _renew_subs 2024-09-16T04:39:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fc4d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.795+0000 7fa0fc4d7640 1 -- 172.21.15.46:0/1478635216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0f41394d0 con 0x7fa0f40a3c10 2024-09-16T04:39:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.796+0000 7fa0eaffd640 1 -- 172.21.15.46:0/1478635216 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fa0f0030b20 con 0x7fa0f40a3c10 2024-09-16T04:39:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.796+0000 7fa0fc4d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.796+0000 7fa0fc4d7640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 0x7fa0f41cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.796+0000 7fa0fc4d7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.796+0000 7fa0fc4d7640 1 -- 172.21.15.46:0/1478635216 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fa0f413c520 con 0x7fa0f4138100 2024-09-16T04:39:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.796+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 0x7fa0f41cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.796+0000 7fa0f9a72640 10 monclient: get_auth_request con 0x7fa0f4138100 auth_method 0 2024-09-16T04:39:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.796+0000 7fa0f9a72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.796+0000 7fa0f9a72640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.796+0000 7fa0f9a72640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.796+0000 7fa0eaffd640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa0d80383b0 0x7fa0d803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:42.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.796+0000 7fa0f9a72640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 msgr2=0x7fa0f41cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:39:42.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.796+0000 7fa0f9a72640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 msgr2=0x7fa0f41cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:39:42.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.796+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 0x7fa0f41cbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:42.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.797+0000 7fa0eaffd640 1 -- 172.21.15.46:0/1478635216 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4738+0+0 (secure 0 0 0) 0x7fa0f006c180 con 0x7fa0f40a3c10 2024-09-16T04:39:42.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.797+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 0x7fa0f41cbea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:42.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.797+0000 7fa0faa74640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa0d80383b0 0x7fa0d803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:42.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.797+0000 7fa0faa74640 10 monclient: get_auth_request con 0x7fa0d80383b0 auth_method 0 2024-09-16T04:39:42.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.797+0000 7fa0faa74640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa0d80383b0 0x7fa0d803a870 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa0f409fc10 tx=0x7fa0ec00b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:42.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.998+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 0x7fa0f41cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:43.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.998+0000 7fa0f9a72640 10 monclient: get_auth_request con 0x7fa0f4138100 auth_method 0 2024-09-16T04:39:43.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.998+0000 7fa0f9a72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:43.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.998+0000 7fa0f9a72640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:43.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.998+0000 7fa0f9a72640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:43.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.998+0000 7fa0f9a72640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 msgr2=0x7fa0f41cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:39:43.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.998+0000 7fa0f9a72640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 msgr2=0x7fa0f41cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:39:43.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.998+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 0x7fa0f41cbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:43.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:42.998+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 0x7fa0f41cbea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:39:43.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:43.399+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 0x7fa0f41cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:43.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:43.399+0000 7fa0f9a72640 10 monclient: get_auth_request con 0x7fa0f4138100 auth_method 0 2024-09-16T04:39:43.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:43.399+0000 7fa0f9a72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:43.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:43.399+0000 7fa0f9a72640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:43.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:43.399+0000 7fa0f9a72640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:43.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:43.399+0000 7fa0f9a72640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 msgr2=0x7fa0f41cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:39:43.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:43.399+0000 7fa0f9a72640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 msgr2=0x7fa0f41cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:39:43.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:43.399+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 0x7fa0f41cbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:43.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:43.399+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 0x7fa0f41cbea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:39:43.756 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:43.757 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35675]: starting. 2024-09-16T04:39:43.757 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35675]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-35640 2024-09-16T04:39:43.757 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35675]: created object 0... 2024-09-16T04:39:43.757 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35675]: created object 25... 2024-09-16T04:39:43.757 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35675]: created object 49... 2024-09-16T04:39:43.757 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35675]: finishing. 2024-09-16T04:39:43.757 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35675]: shutting down. 2024-09-16T04:39:43.769 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:43.769 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35677]: starting. 2024-09-16T04:39:43.769 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35677]: listing objects. 2024-09-16T04:39:43.769 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35677]: listed object 0... 2024-09-16T04:39:43.769 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35677]: listed object 25... 2024-09-16T04:39:43.769 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35677]: saw 50 objects 2024-09-16T04:39:43.769 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35677]: shutting down. 2024-09-16T04:39:44.081 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:44.081 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[35676]: starting. 2024-09-16T04:39:44.081 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[35676]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-35640 2024-09-16T04:39:44.081 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[35676]: shutting down. 2024-09-16T04:39:44.082 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:44.082 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:44.083 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:39:44.085 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:39:44.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:44.200+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 0x7fa0f41cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:44.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:44.200+0000 7fa0f9a72640 10 monclient: get_auth_request con 0x7fa0f4138100 auth_method 0 2024-09-16T04:39:44.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:44.200+0000 7fa0f9a72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:44.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:44.200+0000 7fa0f9a72640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:44.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:44.200+0000 7fa0f9a72640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:44.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:44.200+0000 7fa0f9a72640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 msgr2=0x7fa0f41cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:39:44.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:44.200+0000 7fa0f9a72640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 msgr2=0x7fa0f41cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:39:44.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:44.200+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 0x7fa0f41cbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:44.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:44.200+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 0x7fa0f41cbea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:39:45.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:45.795+0000 7fa0e9ffb640 10 monclient: tick 2024-09-16T04:39:45.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:45.795+0000 7fa0e9ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:39:45.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:45.795+0000 7fa0e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:45.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:45.795+0000 7fa0e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:45.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:45.803+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 0x7fa0f41cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:45.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:45.803+0000 7fa0f9a72640 10 monclient: get_auth_request con 0x7fa0f4138100 auth_method 0 2024-09-16T04:39:45.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:45.803+0000 7fa0f9a72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:45.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:45.803+0000 7fa0f9a72640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:45.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:45.803+0000 7fa0f9a72640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:45.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:45.803+0000 7fa0f9a72640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 msgr2=0x7fa0f41cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:39:45.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:45.803+0000 7fa0f9a72640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 msgr2=0x7fa0f41cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:39:45.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:45.803+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 0x7fa0f41cbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:45.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:45.803+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 0x7fa0f41cbea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-09-16T04:39:46.185 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35744]: starting. 2024-09-16T04:39:46.185 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35744]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-35743 2024-09-16T04:39:46.185 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35744]: created object 0... 2024-09-16T04:39:46.186 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35744]: created object 25... 2024-09-16T04:39:46.186 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35744]: created object 49... 2024-09-16T04:39:46.186 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35744]: finishing. 2024-09-16T04:39:46.186 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35744]: shutting down. 2024-09-16T04:39:46.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.796+0000 7fa0e9ffb640 10 monclient: tick 2024-09-16T04:39:46.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.796+0000 7fa0e9ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:39:46.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.796+0000 7fa0e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:46.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.796+0000 7fa0e9ffb640 5 monclient: _check_tell_commands timeout tell command 1 2024-09-16T04:39:46.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.796+0000 7fa0e9ffb640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 msgr2=0x7fa0f41cbea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:39:46.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.796+0000 7fa0e9ffb640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 0x7fa0f41cbea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:46.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.796+0000 7fa0e9ffb640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0d0003540 0x7fa0d00059d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:46.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.796+0000 7fa0e9ffb640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 msgr2=0x7fa0f41cbea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:46.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.796+0000 7fa0e9ffb640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f4138100 0x7fa0f41cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:46.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.796+0000 7fa0e9ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:46.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.796+0000 7fa0e9ffb640 1 -- 172.21.15.46:0/1478635216 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fa0d0006080 con 0x7fa0d0003540 2024-09-16T04:39:46.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.796+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0d0003540 0x7fa0d00059d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:46.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.796+0000 7fa0f9a72640 10 monclient: get_auth_request con 0x7fa0d0003540 auth_method 0 2024-09-16T04:39:46.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.796+0000 7fa0f9a72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:46.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.796+0000 7fa0f9a72640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:46.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.796+0000 7fa0f9a72640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:46.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.796+0000 7fa0f9a72640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0d0003540 msgr2=0x7fa0d00059d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:39:46.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.796+0000 7fa0f9a72640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0d0003540 msgr2=0x7fa0d00059d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:39:46.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.796+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0d0003540 0x7fa0d00059d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:46.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.796+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0d0003540 0x7fa0d00059d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:46.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.997+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0d0003540 0x7fa0d00059d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:46.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.997+0000 7fa0f9a72640 10 monclient: get_auth_request con 0x7fa0d0003540 auth_method 0 2024-09-16T04:39:46.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.997+0000 7fa0f9a72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:46.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.997+0000 7fa0f9a72640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:47.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.997+0000 7fa0f9a72640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:47.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.997+0000 7fa0f9a72640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0d0003540 msgr2=0x7fa0d00059d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:39:47.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.997+0000 7fa0f9a72640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0d0003540 msgr2=0x7fa0d00059d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:39:47.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.997+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0d0003540 0x7fa0d00059d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:47.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:46.997+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0d0003540 0x7fa0d00059d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:39:47.096 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[35745]: starting. 2024-09-16T04:39:47.096 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[35745]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-35743 2024-09-16T04:39:47.096 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[35745]: shutting down. 2024-09-16T04:39:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.398+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0d0003540 0x7fa0d00059d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.398+0000 7fa0f9a72640 10 monclient: get_auth_request con 0x7fa0d0003540 auth_method 0 2024-09-16T04:39:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.398+0000 7fa0f9a72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.398+0000 7fa0f9a72640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.398+0000 7fa0f9a72640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.398+0000 7fa0f9a72640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.398+0000 7fa0f9a72640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.398+0000 7fa0f9a72640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.398+0000 7fa0f9a72640 10 monclient(hunting): handle_auth_done global_id 14099 payload 824 2024-09-16T04:39:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.398+0000 7fa0f9a72640 1 -- 172.21.15.46:0/1478635216 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa0d0006230 con 0x7fa0d0003540 2024-09-16T04:39:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.398+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0d0003540 0x7fa0d00059d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa0f413d2e0 tx=0x7fa0e4017af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.398+0000 7fa0eaffd640 1 -- 172.21.15.46:0/1478635216 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4015760 con 0x7fa0d0003540 2024-09-16T04:39:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.398+0000 7fa0eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.398+0000 7fa0eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.398+0000 7fa0eaffd640 10 monclient: dump: 2024-09-16T04:39:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:47.403 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:47.403 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:47.403 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:47.403 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:47.403 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:47.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.399+0000 7fa0eaffd640 1 -- 172.21.15.46:0/1478635216 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa0e4015760 con 0x7fa0d0003540 2024-09-16T04:39:47.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.399+0000 7fa0eaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:47.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.399+0000 7fa0eaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:47.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.399+0000 7fa0eaffd640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0d0003540 msgr2=0x7fa0d00059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:47.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.399+0000 7fa0eaffd640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0d0003540 0x7fa0d00059d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa0f413d2e0 tx=0x7fa0e4017af0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.399+0000 7fa0eaffd640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0d0003540 msgr2=0x7fa0d00059d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:47.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.399+0000 7fa0eaffd640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0d0003540 0x7fa0d00059d0 secure :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa0f413d2e0 tx=0x7fa0e4017af0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.406+0000 7fa0fc4d7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:39:47.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.406+0000 7fa0fc4d7640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f41cc3e0 0x7fa0f40a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:47.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.406+0000 7fa0fc4d7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:47.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.406+0000 7fa0fc4d7640 1 -- 172.21.15.46:0/1478635216 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fa0f41cc7f0 con 0x7fa0f41cc3e0 2024-09-16T04:39:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.406+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f41cc3e0 0x7fa0f40a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.406+0000 7fa0f9a72640 10 monclient: get_auth_request con 0x7fa0f41cc3e0 auth_method 0 2024-09-16T04:39:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.406+0000 7fa0f9a72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.406+0000 7fa0f9a72640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.406+0000 7fa0f9a72640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.406+0000 7fa0f9a72640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.406+0000 7fa0f9a72640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.406+0000 7fa0f9a72640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.406+0000 7fa0f9a72640 10 monclient(hunting): handle_auth_done global_id 14102 payload 824 2024-09-16T04:39:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.406+0000 7fa0f9a72640 1 -- 172.21.15.46:0/1478635216 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa0f4069150 con 0x7fa0f41cc3e0 2024-09-16T04:39:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.406+0000 7fa0f9a72640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f41cc3e0 0x7fa0f40a0500 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa0f413d2e0 tx=0x7fa0e4000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.407+0000 7fa0eaffd640 1 -- 172.21.15.46:0/1478635216 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4021630 con 0x7fa0f41cc3e0 2024-09-16T04:39:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.407+0000 7fa0eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.407+0000 7fa0eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:47.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.407+0000 7fa0eaffd640 10 monclient: dump: 2024-09-16T04:39:47.411 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:47.411 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:47.411 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.411 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.411 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:47.411 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:47.411 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:47.411 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:47.411 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:47.411 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:47.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.407+0000 7fa0eaffd640 1 -- 172.21.15.46:0/1478635216 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7fa0e4021630 con 0x7fa0f41cc3e0 2024-09-16T04:39:47.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.407+0000 7fa0eaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:39:47.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.407+0000 7fa0eaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:47.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.407+0000 7fa0eaffd640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f41cc3e0 msgr2=0x7fa0f40a0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:47.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.407+0000 7fa0eaffd640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f41cc3e0 0x7fa0f40a0500 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa0f413d2e0 tx=0x7fa0e4000bc0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.407+0000 7fa0eaffd640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f41cc3e0 msgr2=0x7fa0f40a0500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:47.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.407+0000 7fa0eaffd640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa0f41cc3e0 0x7fa0f40a0500 secure :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa0f413d2e0 tx=0x7fa0e4000bc0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.407+0000 7fa0faa74640 1 -- 172.21.15.46:0/1478635216 reap_dead start 2024-09-16T04:39:47.412 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:39:47.412 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:39:47.412 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:39:47.412 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:39:47.412 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 14, 2024-09-16T04:39:47.413 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 5162, 2024-09-16T04:39:47.413 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:39:47.413 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:39:47.413 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:39:47.413 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:39:47.413 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:47.413 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 4, 2024-09-16T04:39:47.413 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:47.413 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:39:47.413 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:39:47.413 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:47.413 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:47.413 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:47.413 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:47.414 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:47.414 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:47.414 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:47.414 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:47.414 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:47.414 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:47.414 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:47.414 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:47.414 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:39:47.414 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:39:47.414 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:47.414 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:47.414 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:47.414 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:47.415 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:47.415 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:47.415 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:47.415 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:47.415 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:47.415 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:47.415 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:47.415 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:47.415 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.415 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:39:47.415 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:39:47.415 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:39:47.415 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:39:47.415 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:39:47.416 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:39:47.416 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:47.416 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:47.416 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:39:47.416 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:39:47.416 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:39:47.416 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:39:47.416 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:39:47.416 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:39:47.416 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:39:47.416 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:47.416 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:39:47.416 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:47.417 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:47.417 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:47.417 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:47.417 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:47.417 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:47.417 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:47.417 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:47.417 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:47.417 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:47.417 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:47.418 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:47.418 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:39:47.418 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.418 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:39:47.418 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.418 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:39:47.418 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:39:47.418 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:47.418 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:47.418 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.418 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:47.418 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:39:47.418 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:47.418 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.419 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.419 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:47.419 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:39:47.419 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:47.419 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:47.419 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:47.419 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.419 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:39:47.419 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:39:47.419 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:47.419 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:47.419 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:47.419 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.419 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.420 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:39:47.420 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:39:47.420 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:47.420 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:47.420 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.420 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:47.420 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:39:47.420 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:47.420 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.420 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.420 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:47.420 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:39:47.420 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:47.420 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:47.421 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:47.421 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.421 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:39:47.421 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:39:47.421 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:47.421 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:47.421 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:47.421 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.421 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.421 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:39:47.421 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:39:47.421 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:47.421 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:47.421 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.422 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:47.422 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:39:47.422 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:47.422 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.422 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.422 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:47.422 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:39:47.422 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:47.422 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:47.422 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:47.422 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.422 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:39:47.422 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:39:47.422 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:47.422 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:47.423 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:47.423 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:47.423 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:47.423 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.423 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:39:47.423 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:39:47.423 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.423 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:47.423 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:47.423 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:39:47.423 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:47.423 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:47.423 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:39:47.423 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.424 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:47.424 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:47.424 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:39:47.424 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:47.424 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:47.424 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.424 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:39:47.424 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:39:47.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.407+0000 7fa0fc4d7640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa0d80383b0 msgr2=0x7fa0d803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:47.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.407+0000 7fa0fc4d7640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa0d80383b0 0x7fa0d803a870 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa0f409fc10 tx=0x7fa0ec00b040 comp rx=0 tx=0).stop 2024-09-16T04:39:47.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.407+0000 7fa0fc4d7640 10 monclient: shutdown 2024-09-16T04:39:47.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.407+0000 7fa0fc4d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:47.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.407+0000 7fa0fc4d7640 1 -- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0f40a3c10 msgr2=0x7fa0f4133d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:47.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.407+0000 7fa0fc4d7640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0f40a3c10 0x7fa0f4133d60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0000c00 tx=0x7fa0f00046c0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.408+0000 7fa0fc4d7640 1 -- 172.21.15.46:0/1478635216 shutdown_connections 2024-09-16T04:39:47.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.408+0000 7fa0fc4d7640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa0d80383b0 0x7fa0d803a870 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.408+0000 7fa0fc4d7640 1 --2- 172.21.15.46:0/1478635216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0f40a3c10 0x7fa0f4133d60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.408+0000 7fa0fc4d7640 1 -- 172.21.15.46:0/1478635216 >> 172.21.15.46:0/1478635216 conn(0x7fa0f40a5d10 msgr2=0x7fa0f413b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:47.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.408+0000 7fa0fc4d7640 1 -- 172.21.15.46:0/1478635216 shutdown_connections 2024-09-16T04:39:47.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.408+0000 7fa0fc4d7640 1 -- 172.21.15.46:0/1478635216 wait complete. 2024-09-16T04:39:47.427 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:39:47.427 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:39:47.427 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:39:47.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.535+0000 7fd174331640 10 monclient: get_monmap_and_config 2024-09-16T04:39:47.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.535+0000 7fd174331640 10 monclient: build_initial_monmap 2024-09-16T04:39:47.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.535+0000 7fd174331640 10 monclient: monmap: 2024-09-16T04:39:47.537 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:47.537 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:47.537 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:47.536801+0000 2024-09-16T04:39:47.538 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:47.536801+0000 2024-09-16T04:39:47.538 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:47.538 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:47.538 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:47.538 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:47.538 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:47.538 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:47.538 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:47.538 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.535+0000 7fd174331640 1 Processor -- start 2024-09-16T04:39:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd174331640 1 -- start start 2024-09-16T04:39:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd174331640 10 monclient: init 2024-09-16T04:39:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd174331640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd174331640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-16T04:39:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd174331640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd16c0a3c10 0x7fd16c09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd174331640 10 monclient(hunting): picked mon.noname-b con 0x7fd16c0a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd174331640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd16c0a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd174331640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd16c13be90 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd174331640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd16c09b940 con 0x7fd16c13be90 2024-09-16T04:39:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd174331640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd16c09ab00 con 0x7fd16c0a3ff0 2024-09-16T04:39:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd174331640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd174331640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd16c09b030 con 0x7fd16c0a3c10 2024-09-16T04:39:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd174331640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd174331640 10 monclient(hunting): authenticate will time out at 785.952737s 2024-09-16T04:39:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd1720cd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd16c0a3c10 0x7fd16c09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd1728ce640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd16c13be90 0x7fd16c13e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43350/0 (socket says 172.21.15.46:43350) 2024-09-16T04:39:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd1728ce640 1 -- 172.21.15.46:0/1028939584 learned_addr learned my addr 172.21.15.46:0/1028939584 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd1720cd640 10 monclient(hunting): get_auth_request con 0x7fd16c0a3c10 auth_method 0 2024-09-16T04:39:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd1720cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd1720cd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd1720cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd1720cd640 1 -- 172.21.15.46:0/1028939584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd16c0a3c10 msgr2=0x7fd16c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:39:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd1720cd640 1 -- 172.21.15.46:0/1028939584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd16c0a3c10 msgr2=0x7fd16c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:39:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd1720cd640 1 --2- 172.21.15.46:0/1028939584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd16c0a3c10 0x7fd16c09bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:39:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.536+0000 7fd1720cd640 1 --2- 172.21.15.46:0/1028939584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd16c0a3c10 0x7fd16c09bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 1 -- 172.21.15.46:0/1028939584 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 668321565 0 0) 0x7fd16c09b940 con 0x7fd16c13be90 2024-09-16T04:39:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient(hunting): _negotiate 2024-09-16T04:39:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient(hunting): my global_id is 4680 2024-09-16T04:39:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 1 -- 172.21.15.46:0/1028939584 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd158003520 con 0x7fd16c13be90 2024-09-16T04:39:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 1 -- 172.21.15.46:0/1028939584 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4254675274 0 0) 0x7fd16c09ab00 con 0x7fd16c0a3ff0 2024-09-16T04:39:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient(hunting): _negotiate 2024-09-16T04:39:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient(hunting): my global_id is 4651 2024-09-16T04:39:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 1 -- 172.21.15.46:0/1028939584 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd16c09b940 con 0x7fd16c0a3ff0 2024-09-16T04:39:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 1 -- 172.21.15.46:0/1028939584 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 292849865 0 0) 0x7fd158003520 con 0x7fd16c13be90 2024-09-16T04:39:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 1 -- 172.21.15.46:0/1028939584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd16c0a3c10 msgr2=0x7fd16c09bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:39:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 1 --2- 172.21.15.46:0/1028939584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd16c0a3c10 0x7fd16c09bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 1 -- 172.21.15.46:0/1028939584 >> v1:172.21.15.114:6789/0 conn(0x7fd16c0a3ff0 legacy=0x7fd16c0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient: _finish_hunting 0 2024-09-16T04:39:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:39:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:39:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 1 -- 172.21.15.46:0/1028939584 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16c09b310 con 0x7fd16c13be90 2024-09-16T04:39:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient: _finish_auth 0 2024-09-16T04:39:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient: _check_auth_tickets 2024-09-16T04:39:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 1 -- 172.21.15.46:0/1028939584 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7fd164001670 con 0x7fd16c13be90 2024-09-16T04:39:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:47.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:39:47.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient: dump: 2024-09-16T04:39:47.553 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:47.557 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:39:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 1 -- 172.21.15.46:0/1028939584 >> v1:172.21.15.46:6789/0 conn(0x7fd16c13be90 legacy=0x7fd16c13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:39:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 1 --2- 172.21.15.46:0/1028939584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd158003b70 0x7fd158003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient(hunting): picked mon.b con 0x7fd158003b70 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 1 --2- 172.21.15.46:0/1028939584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd16c0a3c10 0x7fd158004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient(hunting): picked mon.c con 0x7fd16c0a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 1 --2- 172.21.15.46:0/1028939584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd158004d90 0x7fd158005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient(hunting): picked mon.a con 0x7fd158004d90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 1 -- 172.21.15.46:0/1028939584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c09b030 con 0x7fd158004d90 2024-09-16T04:39:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 1 -- 172.21.15.46:0/1028939584 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd15800bc20 con 0x7fd158003b70 2024-09-16T04:39:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 1 -- 172.21.15.46:0/1028939584 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd15800be20 con 0x7fd16c0a3c10 2024-09-16T04:39:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1710cb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:39:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1728ce640 1 --2- 172.21.15.46:0/1028939584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd158004d90 0x7fd158005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1720cd640 1 --2- 172.21.15.46:0/1028939584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd158003b70 0x7fd158003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1718cc640 1 --2- 172.21.15.46:0/1028939584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd16c0a3c10 0x7fd158004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:47.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1720cd640 10 monclient(hunting): get_auth_request con 0x7fd158003b70 auth_method 0 2024-09-16T04:39:47.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1720cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:47.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1720cd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:47.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1720cd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:47.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1728ce640 10 monclient(hunting): get_auth_request con 0x7fd158004d90 auth_method 0 2024-09-16T04:39:47.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1728ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:47.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1728ce640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:47.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1728ce640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:47.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1718cc640 10 monclient(hunting): get_auth_request con 0x7fd16c0a3c10 auth_method 0 2024-09-16T04:39:47.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1718cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:47.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1718cc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:47.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1718cc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:39:47.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1728ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:47.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1728ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:47.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1728ce640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:47.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1718cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:47.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1718cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:47.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1718cc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:47.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1720cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:47.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1720cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:47.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1720cd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:39:47.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1728ce640 10 monclient(hunting): handle_auth_done global_id 4680 payload 293 2024-09-16T04:39:47.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1728ce640 1 -- 172.21.15.46:0/1028939584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd16c0a3c10 msgr2=0x7fd158004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:47.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1728ce640 1 --2- 172.21.15.46:0/1028939584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd16c0a3c10 0x7fd158004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1728ce640 1 -- 172.21.15.46:0/1028939584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd158003b70 msgr2=0x7fd158003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:47.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1728ce640 1 --2- 172.21.15.46:0/1028939584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd158003b70 0x7fd158003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1728ce640 10 monclient: _finish_hunting 0 2024-09-16T04:39:47.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1728ce640 1 monclient: found mon.a 2024-09-16T04:39:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1728ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1728ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.537+0000 7fd1728ce640 1 -- 172.21.15.46:0/1028939584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd158003520 con 0x7fd158004d90 2024-09-16T04:39:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.538+0000 7fd1728ce640 10 monclient: _finish_auth 0 2024-09-16T04:39:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.538+0000 7fd1728ce640 10 monclient: _check_auth_tickets 2024-09-16T04:39:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.538+0000 7fd1728ce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.538+0000 7fd174331640 5 monclient: authenticate success, global_id 4680 2024-09-16T04:39:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.538+0000 7fd174331640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.538+0000 7fd1718cc640 1 --2- 172.21.15.46:0/1028939584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd16c0a3c10 0x7fd158004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.538+0000 7fd1728ce640 1 --2- 172.21.15.46:0/1028939584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd158004d90 0x7fd158005210 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd16c09af60 tx=0x7fd158005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5d685eab68d732d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.538+0000 7fd1710cb640 1 -- 172.21.15.46:0/1028939584 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164007cc0 con 0x7fd158004d90 2024-09-16T04:39:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.538+0000 7fd1710cb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.538+0000 7fd1710cb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.538+0000 7fd1710cb640 10 monclient: dump: 2024-09-16T04:39:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:47.562 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.538+0000 7fd174331640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.538+0000 7fd1710cb640 1 -- 172.21.15.46:0/1028939584 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd164005ad0 con 0x7fd158004d90 2024-09-16T04:39:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.538+0000 7fd1710cb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.538+0000 7fd174331640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.538+0000 7fd174331640 4 set_mon_vals no callback set 2024-09-16T04:39:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.538+0000 7fd1710cb640 1 -- 172.21.15.46:0/1028939584 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164007070 con 0x7fd158004d90 2024-09-16T04:39:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.538+0000 7fd1710cb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.538+0000 7fd1710cb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.538+0000 7fd1710cb640 10 monclient: dump: 2024-09-16T04:39:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:47.563 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.539+0000 7fd174331640 10 monclient: shutdown 2024-09-16T04:39:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.539+0000 7fd174331640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.539+0000 7fd174331640 1 -- 172.21.15.46:0/1028939584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd158004d90 msgr2=0x7fd158005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.539+0000 7fd174331640 1 --2- 172.21.15.46:0/1028939584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd158004d90 0x7fd158005210 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd16c09af60 tx=0x7fd158005830 comp rx=0 tx=0).stop 2024-09-16T04:39:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.539+0000 7fd1728ce640 1 -- 172.21.15.46:0/1028939584 reap_dead start 2024-09-16T04:39:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.539+0000 7fd174331640 1 -- 172.21.15.46:0/1028939584 shutdown_connections 2024-09-16T04:39:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.539+0000 7fd174331640 1 -- 172.21.15.46:0/1028939584 >> 172.21.15.46:0/1028939584 conn(0x7fd16c0a6340 msgr2=0x7fd16c06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.539+0000 7fd174331640 1 -- 172.21.15.46:0/1028939584 shutdown_connections 2024-09-16T04:39:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.539+0000 7fd174331640 1 -- 172.21.15.46:0/1028939584 wait complete. 2024-09-16T04:39:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.540+0000 7fd174331640 10 monclient: build_initial_monmap 2024-09-16T04:39:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.540+0000 7fd174331640 10 monclient: monmap: 2024-09-16T04:39:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:47.565 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.540+0000 7fd174331640 1 Processor -- start 2024-09-16T04:39:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.540+0000 7fd174331640 1 -- start start 2024-09-16T04:39:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.540+0000 7fd174331640 10 monclient: init 2024-09-16T04:39:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.540+0000 7fd174331640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.540+0000 7fd174331640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:39:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.540+0000 7fd174331640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd158004d90 0x7fd16c120240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.540+0000 7fd174331640 10 monclient(hunting): picked mon.noname-a con 0x7fd158004d90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd174331640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16c120780 0x7fd16c120bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd174331640 10 monclient(hunting): picked mon.noname-b con 0x7fd16c120780 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd174331640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd16c121100 0x7fd16c1d2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd174331640 10 monclient(hunting): picked mon.noname-c con 0x7fd16c121100 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd174331640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd174331640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c09be70 con 0x7fd158004d90 2024-09-16T04:39:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd174331640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd174331640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c09bcf0 con 0x7fd16c120780 2024-09-16T04:39:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd174331640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd174331640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd16c09bff0 con 0x7fd16c121100 2024-09-16T04:39:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd174331640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd174331640 10 monclient(hunting): authenticate will time out at 785.957073s 2024-09-16T04:39:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1720cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd158004d90 0x7fd16c120240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1720cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd158004d90 0x7fd16c120240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38208/0 (socket says 172.21.15.46:38208) 2024-09-16T04:39:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1720cd640 1 -- 172.21.15.46:0/2359954390 learned_addr learned my addr 172.21.15.46:0/2359954390 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1720cd640 10 monclient(hunting): get_auth_request con 0x7fd158004d90 auth_method 0 2024-09-16T04:39:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1720cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1720cd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1720cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1728ce640 1 --2- 172.21.15.46:0/2359954390 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd16c121100 0x7fd16c1d2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1728ce640 10 monclient(hunting): get_auth_request con 0x7fd16c121100 auth_method 0 2024-09-16T04:39:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1728ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1728ce640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1728ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1720cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1720cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1720cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1728ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1728ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1728ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1718cc640 1 --2- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16c120780 0x7fd16c120bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1718cc640 10 monclient(hunting): get_auth_request con 0x7fd16c120780 auth_method 0 2024-09-16T04:39:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1718cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1718cc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1718cc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1718cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1718cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1718cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1728ce640 10 monclient(hunting): handle_auth_done global_id 14105 payload 824 2024-09-16T04:39:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1728ce640 1 -- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16c120780 msgr2=0x7fd16c120bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1728ce640 1 --2- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16c120780 0x7fd16c120bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1728ce640 1 -- 172.21.15.46:0/2359954390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd158004d90 msgr2=0x7fd16c120240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1728ce640 1 --2- 172.21.15.46:0/2359954390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd158004d90 0x7fd16c120240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1728ce640 10 monclient: _finish_hunting 0 2024-09-16T04:39:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1728ce640 1 monclient: found mon.noname-c 2024-09-16T04:39:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1728ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1728ce640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:39:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1728ce640 1 -- 172.21.15.46:0/2359954390 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16c123c30 con 0x7fd16c121100 2024-09-16T04:39:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1720cd640 1 --2- 172.21.15.46:0/2359954390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd158004d90 0x7fd16c120240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1728ce640 1 --2- 172.21.15.46:0/2359954390 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd16c121100 0x7fd16c1d2060 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd15800bbf0 tx=0x7fd1640019c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd162ffd640 1 -- 172.21.15.46:0/2359954390 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c0014f0 con 0x7fd16c121100 2024-09-16T04:39:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd162ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd162ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:39:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd162ffd640 10 monclient: dump: 2024-09-16T04:39:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:47.570 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd162ffd640 10 monclient: _finish_auth 0 2024-09-16T04:39:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd162ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:39:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd162ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd162ffd640 1 -- 172.21.15.46:0/2359954390 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd164038040 con 0x7fd16c121100 2024-09-16T04:39:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd174331640 5 monclient: authenticate success, global_id 14105 2024-09-16T04:39:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd174331640 10 monclient: _renew_subs 2024-09-16T04:39:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd174331640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd174331640 1 -- 172.21.15.46:0/2359954390 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd16c0a0cf0 con 0x7fd16c121100 2024-09-16T04:39:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd162ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd162ffd640 1 -- 172.21.15.46:0/2359954390 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164004a20 con 0x7fd16c121100 2024-09-16T04:39:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd162ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd162ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd162ffd640 10 monclient: dump: 2024-09-16T04:39:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:47.572 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:47.572 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:47.572 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:47.572 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:47.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.542+0000 7fd174331640 10 monclient: _renew_subs 2024-09-16T04:39:47.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.542+0000 7fd174331640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:47.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.542+0000 7fd174331640 1 -- 172.21.15.46:0/2359954390 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd16c123950 con 0x7fd16c121100 2024-09-16T04:39:47.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.541+0000 7fd1708ca640 4 set_mon_vals no callback set 2024-09-16T04:39:47.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.542+0000 7fd174331640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:47.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.542+0000 7fd174331640 1 --2- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16c09cbe0 0x7fd16c09cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:47.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.542+0000 7fd174331640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:47.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.542+0000 7fd174331640 1 -- 172.21.15.46:0/2359954390 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c0a1240 con 0x7fd16c09cbe0 2024-09-16T04:39:47.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.542+0000 7fd162ffd640 1 -- 172.21.15.46:0/2359954390 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fd164004bc0 con 0x7fd16c121100 2024-09-16T04:39:47.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.543+0000 7fd1720cd640 1 --2- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16c09cbe0 0x7fd16c09cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.543+0000 7fd1720cd640 10 monclient: get_auth_request con 0x7fd16c09cbe0 auth_method 0 2024-09-16T04:39:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.543+0000 7fd162ffd640 1 --2- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd1500383b0 0x7fd15003a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.543+0000 7fd1720cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.543+0000 7fd1720cd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.543+0000 7fd1720cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.543+0000 7fd162ffd640 1 -- 172.21.15.46:0/2359954390 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(47..47 src has 1..47) ==== 4346+0+0 (secure 0 0 0) 0x7fd16406b440 con 0x7fd16c121100 2024-09-16T04:39:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.543+0000 7fd1720cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.543+0000 7fd1720cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.543+0000 7fd1720cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.543+0000 7fd1718cc640 1 --2- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd1500383b0 0x7fd15003a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.543+0000 7fd1718cc640 10 monclient: get_auth_request con 0x7fd1500383b0 auth_method 0 2024-09-16T04:39:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.543+0000 7fd1720cd640 10 monclient(hunting): handle_auth_done global_id 4657 payload 824 2024-09-16T04:39:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.543+0000 7fd1720cd640 1 -- 172.21.15.46:0/2359954390 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd16c0a43d0 con 0x7fd16c09cbe0 2024-09-16T04:39:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.543+0000 7fd1720cd640 1 --2- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16c09cbe0 0x7fd16c09cfc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd16800af70 tx=0x7fd168004410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.543+0000 7fd1718cc640 1 --2- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd1500383b0 0x7fd15003a870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd16c121820 tx=0x7fd15c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.543+0000 7fd162ffd640 1 -- 172.21.15.46:0/2359954390 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd168018880 con 0x7fd16c09cbe0 2024-09-16T04:39:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.543+0000 7fd162ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.543+0000 7fd162ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.543+0000 7fd162ffd640 10 monclient: dump: 2024-09-16T04:39:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:47.574 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:47.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.544+0000 7fd162ffd640 1 -- 172.21.15.46:0/2359954390 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd16801d070 con 0x7fd16c09cbe0 2024-09-16T04:39:47.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.544+0000 7fd162ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:47.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.544+0000 7fd162ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:47.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.544+0000 7fd162ffd640 1 -- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16c09cbe0 msgr2=0x7fd16c09cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:47.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.544+0000 7fd162ffd640 1 --2- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16c09cbe0 0x7fd16c09cfc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd16800af70 tx=0x7fd168004410 comp rx=0 tx=0).stop 2024-09-16T04:39:47.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.544+0000 7fd162ffd640 1 -- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16c09cbe0 msgr2=0x7fd16c09cfc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:47.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.544+0000 7fd162ffd640 1 --2- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16c09cbe0 0x7fd16c09cfc0 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd16800af70 tx=0x7fd168004410 comp rx=0 tx=0).stop 2024-09-16T04:39:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.550+0000 7fd174331640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:39:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.550+0000 7fd174331640 1 --2- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16c09d500 0x7fd16c1d65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.550+0000 7fd174331640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.550+0000 7fd174331640 1 -- 172.21.15.46:0/2359954390 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c122500 con 0x7fd16c09d500 2024-09-16T04:39:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.550+0000 7fd1720cd640 1 --2- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16c09d500 0x7fd16c1d65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.550+0000 7fd1720cd640 10 monclient: get_auth_request con 0x7fd16c09d500 auth_method 0 2024-09-16T04:39:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.550+0000 7fd1720cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.550+0000 7fd1720cd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.550+0000 7fd1720cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.550+0000 7fd1720cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.550+0000 7fd1720cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.550+0000 7fd1720cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.550+0000 7fd1720cd640 10 monclient(hunting): handle_auth_done global_id 4660 payload 824 2024-09-16T04:39:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.550+0000 7fd1720cd640 1 -- 172.21.15.46:0/2359954390 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd16c067e20 con 0x7fd16c09d500 2024-09-16T04:39:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.551+0000 7fd1720cd640 1 --2- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16c09d500 0x7fd16c1d65c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd16800af70 tx=0x7fd168004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.551+0000 7fd162ffd640 1 -- 172.21.15.46:0/2359954390 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd168022c90 con 0x7fd16c09d500 2024-09-16T04:39:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.551+0000 7fd162ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.551+0000 7fd162ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.551+0000 7fd162ffd640 10 monclient: dump: 2024-09-16T04:39:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:47.579 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.551+0000 7fd162ffd640 1 -- 172.21.15.46:0/2359954390 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7fd16800a0c0 con 0x7fd16c09d500 2024-09-16T04:39:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.551+0000 7fd162ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:39:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.551+0000 7fd162ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.551+0000 7fd162ffd640 1 -- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16c09d500 msgr2=0x7fd16c1d65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.551+0000 7fd162ffd640 1 --2- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16c09d500 0x7fd16c1d65c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd16800af70 tx=0x7fd168004ab0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.551+0000 7fd162ffd640 1 -- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16c09d500 msgr2=0x7fd16c1d65c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.551+0000 7fd162ffd640 1 --2- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16c09d500 0x7fd16c1d65c0 secure :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd16800af70 tx=0x7fd168004ab0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.579 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:39:47.579 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:39:47.579 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:39:47.580 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:39:47.580 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 14, 2024-09-16T04:39:47.580 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 73454, 2024-09-16T04:39:47.580 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:39:47.580 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:39:47.580 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:39:47.580 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:39:47.580 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:47.580 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 5, 2024-09-16T04:39:47.580 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:47.580 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:39:47.580 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:39:47.580 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:47.581 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:47.581 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:47.581 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:47.581 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:47.581 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:47.581 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:47.581 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:47.581 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:47.581 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:47.581 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:47.581 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:47.581 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:39:47.581 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:39:47.581 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:47.582 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:47.582 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:47.582 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:47.582 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:47.582 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:47.582 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:47.582 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:47.582 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:47.582 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:47.583 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:47.583 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:47.583 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.583 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:39:47.583 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:39:47.583 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:39:47.583 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:39:47.583 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:39:47.583 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:39:47.583 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:47.583 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:47.584 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:39:47.584 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:39:47.584 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:39:47.584 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:39:47.584 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:39:47.584 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:39:47.584 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:39:47.584 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:47.584 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:39:47.584 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:47.584 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:47.584 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:47.584 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:47.585 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:47.585 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:47.585 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:47.585 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:47.585 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:47.585 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:47.585 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:47.585 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:47.585 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:39:47.585 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.585 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:39:47.585 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.585 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:39:47.586 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:39:47.586 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:47.586 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:47.586 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.586 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:47.586 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:39:47.586 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:47.586 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.586 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.586 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:47.586 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:39:47.586 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:47.586 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:47.586 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:47.587 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.587 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:39:47.587 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:39:47.587 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:47.587 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:47.587 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:47.587 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.587 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.587 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:39:47.587 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:39:47.587 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:47.587 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:47.587 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.587 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:47.588 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:39:47.588 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:47.588 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.588 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.588 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:47.588 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:39:47.588 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:47.588 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:47.588 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:47.588 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.588 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:39:47.588 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:39:47.588 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:47.588 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:47.589 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:47.589 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.589 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.589 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:39:47.589 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:39:47.589 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:47.589 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:47.589 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.589 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:47.589 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:39:47.589 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:47.589 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.589 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.589 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:47.589 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:39:47.590 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:47.590 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:47.590 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:47.590 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.590 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:39:47.590 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:39:47.590 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:47.590 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:47.590 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:47.590 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:47.591 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:47.591 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.591 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:39:47.591 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:39:47.591 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.591 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:47.591 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:47.591 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:39:47.591 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:47.591 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:47.591 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:39:47.591 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.591 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:47.592 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:47.592 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:39:47.592 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:47.592 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:47.592 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:39:47.592 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.592 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:47.592 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:47.592 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 7 2024-09-16T04:39:47.592 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:47.592 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:47.592 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:39:47.592 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:47.592 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:47.593 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:47.593 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:39:47.593 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:47.593 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:47.593 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:47.593 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:39:47.593 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:39:47.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.552+0000 7fd174331640 1 -- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd1500383b0 msgr2=0x7fd15003a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:47.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.552+0000 7fd174331640 1 --2- 172.21.15.46:0/2359954390 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd1500383b0 0x7fd15003a870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd16c121820 tx=0x7fd15c009040 comp rx=0 tx=0).stop 2024-09-16T04:39:47.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.552+0000 7fd174331640 10 monclient: shutdown 2024-09-16T04:39:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.552+0000 7fd174331640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.552+0000 7fd174331640 1 -- 172.21.15.46:0/2359954390 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd16c121100 msgr2=0x7fd16c1d2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.552+0000 7fd174331640 1 --2- 172.21.15.46:0/2359954390 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd16c121100 0x7fd16c1d2060 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd15800bbf0 tx=0x7fd1640019c0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.552+0000 7fd1728ce640 1 -- 172.21.15.46:0/2359954390 reap_dead start 2024-09-16T04:39:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.552+0000 7fd1728ce640 1 -- 172.21.15.46:0/2359954390 reap_dead start 2024-09-16T04:39:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.552+0000 7fd174331640 1 -- 172.21.15.46:0/2359954390 shutdown_connections 2024-09-16T04:39:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.552+0000 7fd174331640 1 -- 172.21.15.46:0/2359954390 >> 172.21.15.46:0/2359954390 conn(0x7fd16c0a6340 msgr2=0x7fd16c13b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.552+0000 7fd174331640 1 -- 172.21.15.46:0/2359954390 shutdown_connections 2024-09-16T04:39:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.552+0000 7fd174331640 1 -- 172.21.15.46:0/2359954390 wait complete. 2024-09-16T04:39:47.595 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:39:47.596 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:39:47.596 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:39:47.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.694+0000 7fb838778640 10 monclient: get_monmap_and_config 2024-09-16T04:39:47.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.694+0000 7fb838778640 10 monclient: build_initial_monmap 2024-09-16T04:39:47.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.694+0000 7fb838778640 10 monclient: monmap: 2024-09-16T04:39:47.697 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:47.696142+0000 2024-09-16T04:39:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:47.696142+0000 2024-09-16T04:39:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:47.699 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:47.699 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:47.699 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:47.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.695+0000 7fb838778640 1 Processor -- start 2024-09-16T04:39:47.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.695+0000 7fb838778640 1 -- start start 2024-09-16T04:39:47.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.695+0000 7fb838778640 10 monclient: init 2024-09-16T04:39:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.695+0000 7fb838778640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.695+0000 7fb838778640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-16T04:39:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.695+0000 7fb838778640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb83006e160 0x7fb8300a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.695+0000 7fb838778640 10 monclient(hunting): picked mon.noname-b con 0x7fb83006e160 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.695+0000 7fb838778640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb8300a2ee0 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.695+0000 7fb838778640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb830137050 0x7fb83013ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.695+0000 7fb838778640 10 monclient(hunting): picked mon.noname-c con 0x7fb830137050 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.695+0000 7fb838778640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8300a3fd0 con 0x7fb8300a2ee0 2024-09-16T04:39:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.695+0000 7fb838778640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.695+0000 7fb838778640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fb8300a3300 con 0x7fb830137050 2024-09-16T04:39:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.695+0000 7fb838778640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.695+0000 7fb838778640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fb83006e540 con 0x7fb83006e160 2024-09-16T04:39:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.695+0000 7fb838778640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.695+0000 7fb838778640 10 monclient(hunting): authenticate will time out at 786.111981s 2024-09-16T04:39:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb836514640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb83006e160 0x7fb8300a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb835d13640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb8300a2ee0 0x7fb830136940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43362/0 (socket says 172.21.15.46:43362) 2024-09-16T04:39:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb835d13640 1 -- 172.21.15.46:0/1197384501 learned_addr learned my addr 172.21.15.46:0/1197384501 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb836514640 10 monclient(hunting): get_auth_request con 0x7fb83006e160 auth_method 0 2024-09-16T04:39:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb836514640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb836514640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb836514640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb836514640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb836514640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb836514640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb836d15640 1 --2- 172.21.15.46:0/1197384501 >> v2:172.21.15.114:3300/0 conn(0x7fb830137050 0x7fb83013ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb836514640 10 monclient(hunting): handle_auth_done global_id 14108 payload 274 2024-09-16T04:39:47.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb836514640 1 -- 172.21.15.46:0/1197384501 >> v2:172.21.15.114:3300/0 conn(0x7fb830137050 msgr2=0x7fb83013ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:47.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb836514640 1 --2- 172.21.15.46:0/1197384501 >> v2:172.21.15.114:3300/0 conn(0x7fb830137050 0x7fb83013ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb836514640 1 -- 172.21.15.46:0/1197384501 >> v1:172.21.15.46:6789/0 conn(0x7fb8300a2ee0 legacy=0x7fb830136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:47.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb836514640 10 monclient: _finish_hunting 0 2024-09-16T04:39:47.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb836514640 1 monclient: found mon.noname-b 2024-09-16T04:39:47.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb835512640 1 -- 172.21.15.46:0/1197384501 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3061793729 0 0) 0x7fb820001110 con 0x7fb8300a2ee0 2024-09-16T04:39:47.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb836514640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:47.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb836514640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:39:47.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb836514640 1 -- 172.21.15.46:0/1197384501 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8300a0bd0 con 0x7fb83006e160 2024-09-16T04:39:47.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb836d15640 10 monclient: get_auth_request con 0x7fb830137050 auth_method 0 2024-09-16T04:39:47.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb836d15640 1 --2- 172.21.15.46:0/1197384501 >> v2:172.21.15.114:3300/0 conn(0x7fb830137050 0x7fb83013ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:39:47.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.696+0000 7fb835512640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:39:47.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb836514640 1 --2- 172.21.15.46:0/1197384501 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb83006e160 0x7fb8300a29a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0097a0 tx=0x7fb82c02a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c38d5e68de301cc8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:47.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb835512640 1 -- 172.21.15.46:0/1197384501 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82c037070 con 0x7fb83006e160 2024-09-16T04:39:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb835512640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb835512640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb835512640 10 monclient: dump: 2024-09-16T04:39:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:47.712 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb835512640 10 monclient: _finish_auth 0 2024-09-16T04:39:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb835512640 10 monclient: _check_auth_tickets 2024-09-16T04:39:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb835512640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb835512640 1 -- 172.21.15.46:0/1197384501 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb82c02adb0 con 0x7fb83006e160 2024-09-16T04:39:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb838778640 5 monclient: authenticate success, global_id 14108 2024-09-16T04:39:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb838778640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb835512640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb838778640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb835512640 1 -- 172.21.15.46:0/1197384501 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82c032660 con 0x7fb83006e160 2024-09-16T04:39:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb835512640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb838778640 4 set_mon_vals no callback set 2024-09-16T04:39:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb835512640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb835512640 10 monclient: dump: 2024-09-16T04:39:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:47.714 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb838778640 10 monclient: shutdown 2024-09-16T04:39:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb838778640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb838778640 1 -- 172.21.15.46:0/1197384501 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb83006e160 msgr2=0x7fb8300a29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb838778640 1 --2- 172.21.15.46:0/1197384501 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb83006e160 0x7fb8300a29a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0097a0 tx=0x7fb82c02a060 comp rx=0 tx=0).stop 2024-09-16T04:39:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb838778640 1 -- 172.21.15.46:0/1197384501 shutdown_connections 2024-09-16T04:39:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb838778640 1 --2- 172.21.15.46:0/1197384501 >> v2:172.21.15.114:3300/0 conn(0x7fb830137050 0x7fb83013ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb838778640 1 --2- 172.21.15.46:0/1197384501 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb83006e160 0x7fb8300a29a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb838778640 1 -- 172.21.15.46:0/1197384501 >> 172.21.15.46:0/1197384501 conn(0x7fb8300a5c10 msgr2=0x7fb8300a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.697+0000 7fb838778640 1 -- 172.21.15.46:0/1197384501 shutdown_connections 2024-09-16T04:39:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.698+0000 7fb838778640 1 -- 172.21.15.46:0/1197384501 wait complete. 2024-09-16T04:39:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.698+0000 7fb838778640 10 monclient: build_initial_monmap 2024-09-16T04:39:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.698+0000 7fb838778640 10 monclient: monmap: 2024-09-16T04:39:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:47.716 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.698+0000 7fb838778640 1 Processor -- start 2024-09-16T04:39:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.698+0000 7fb838778640 1 -- start start 2024-09-16T04:39:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.698+0000 7fb838778640 10 monclient: init 2024-09-16T04:39:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb838778640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb838778640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:39:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb838778640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb83006e160 0x7fb8301cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb838778640 10 monclient(hunting): picked mon.noname-b con 0x7fb83006e160 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb838778640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb8300a2ee0 0x7fb830135b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb838778640 10 monclient(hunting): picked mon.noname-c con 0x7fb8300a2ee0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb838778640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb830137050 0x7fb830136060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb838778640 10 monclient(hunting): picked mon.noname-a con 0x7fb830137050 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:47.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb838778640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:47.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb838778640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb8300a34a0 con 0x7fb830137050 2024-09-16T04:39:47.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb838778640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:47.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb838778640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb8300a5130 con 0x7fb83006e160 2024-09-16T04:39:47.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb838778640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:47.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb838778640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fb8300a8810 con 0x7fb8300a2ee0 2024-09-16T04:39:47.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb838778640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:47.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb838778640 10 monclient(hunting): authenticate will time out at 786.115352s 2024-09-16T04:39:47.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb835d13640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb8300a2ee0 0x7fb830135b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:47.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836d15640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb830137050 0x7fb830136060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:47.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb835d13640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb8300a2ee0 0x7fb830135b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:52330/0 (socket says 172.21.15.46:52330) 2024-09-16T04:39:47.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836d15640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb830137050 0x7fb830136060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38212/0 (socket says 172.21.15.46:38212) 2024-09-16T04:39:47.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb835d13640 1 -- 172.21.15.46:0/2963971612 learned_addr learned my addr 172.21.15.46:0/2963971612 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:47.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb835d13640 10 monclient(hunting): get_auth_request con 0x7fb8300a2ee0 auth_method 0 2024-09-16T04:39:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb835d13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb835d13640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb835d13640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836d15640 10 monclient(hunting): get_auth_request con 0x7fb830137050 auth_method 0 2024-09-16T04:39:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836d15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836d15640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836d15640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb835d13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb835d13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb835d13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836d15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836d15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836d15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836514640 1 --2- 172.21.15.46:0/2963971612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb83006e160 0x7fb8301cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836514640 10 monclient(hunting): get_auth_request con 0x7fb83006e160 auth_method 0 2024-09-16T04:39:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836514640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836514640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836514640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836514640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836514640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836514640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836d15640 10 monclient(hunting): handle_auth_done global_id 4689 payload 824 2024-09-16T04:39:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836d15640 1 -- 172.21.15.46:0/2963971612 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb8300a2ee0 msgr2=0x7fb830135b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836d15640 1 --2- 172.21.15.46:0/2963971612 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb8300a2ee0 0x7fb830135b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836d15640 1 -- 172.21.15.46:0/2963971612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb83006e160 msgr2=0x7fb8301cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836d15640 1 --2- 172.21.15.46:0/2963971612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb83006e160 0x7fb8301cbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836d15640 10 monclient: _finish_hunting 0 2024-09-16T04:39:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836d15640 1 monclient: found mon.noname-a 2024-09-16T04:39:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836d15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836d15640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836d15640 1 -- 172.21.15.46:0/2963971612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb830123220 con 0x7fb830137050 2024-09-16T04:39:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb835d13640 1 --2- 172.21.15.46:0/2963971612 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb8300a2ee0 0x7fb830135b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.699+0000 7fb836d15640 1 --2- 172.21.15.46:0/2963971612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb830137050 0x7fb830136060 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb82800ce20 tx=0x7fb82800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb826ffd640 1 -- 172.21.15.46:0/2963971612 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82801a070 con 0x7fb830137050 2024-09-16T04:39:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb826ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb826ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb826ffd640 10 monclient: dump: 2024-09-16T04:39:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.721 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.721 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:47.721 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:47.721 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:47.721 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:47.721 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:47.721 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:47.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb826ffd640 10 monclient: _finish_auth 0 2024-09-16T04:39:47.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb826ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:39:47.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb826ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:47.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb826ffd640 1 -- 172.21.15.46:0/2963971612 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb82800d840 con 0x7fb830137050 2024-09-16T04:39:47.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb838778640 5 monclient: authenticate success, global_id 4689 2024-09-16T04:39:47.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb826ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:47.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb838778640 10 monclient: _renew_subs 2024-09-16T04:39:47.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb826ffd640 1 -- 172.21.15.46:0/2963971612 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82800dc30 con 0x7fb830137050 2024-09-16T04:39:47.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb838778640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:47.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb838778640 1 -- 172.21.15.46:0/2963971612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb830122f40 con 0x7fb830137050 2024-09-16T04:39:47.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb826ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:47.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb826ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:47.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb826ffd640 10 monclient: dump: 2024-09-16T04:39:47.722 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:47.722 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:47.722 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.722 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:47.722 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:47.722 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:47.723 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:47.723 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:47.723 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:47.723 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:47.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb834d11640 4 set_mon_vals no callback set 2024-09-16T04:39:47.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb838778640 10 monclient: _renew_subs 2024-09-16T04:39:47.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb838778640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:47.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb838778640 1 -- 172.21.15.46:0/2963971612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb830121af0 con 0x7fb830137050 2024-09-16T04:39:47.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.700+0000 7fb826ffd640 1 -- 172.21.15.46:0/2963971612 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fb82801d070 con 0x7fb830137050 2024-09-16T04:39:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.701+0000 7fb838778640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.701+0000 7fb838778640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.701+0000 7fb838778640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.701+0000 7fb838778640 1 -- 172.21.15.46:0/2963971612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb830137500 con 0x7fb830137050 2024-09-16T04:39:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.701+0000 7fb826ffd640 1 --2- 172.21.15.46:0/2963971612 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb8140383e0 0x7fb81403a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.701+0000 7fb826ffd640 1 -- 172.21.15.46:0/2963971612 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4346+0+0 (secure 0 0 0) 0x7fb82804bd50 con 0x7fb830137050 2024-09-16T04:39:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.701+0000 7fb836514640 1 --2- 172.21.15.46:0/2963971612 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb8140383e0 0x7fb81403a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.701+0000 7fb836514640 10 monclient: get_auth_request con 0x7fb8140383e0 auth_method 0 2024-09-16T04:39:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.702+0000 7fb836514640 1 --2- 172.21.15.46:0/2963971612 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb8140383e0 0x7fb81403a8a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0097a0 tx=0x7fb82c02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.706+0000 7fb826ffd640 1 -- 172.21.15.46:0/2963971612 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb8280143a0 con 0x7fb830137050 2024-09-16T04:39:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.706+0000 7fb826ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.706+0000 7fb826ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:47.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.951+0000 7fb838778640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:39:47.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.951+0000 7fb838778640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:39:47.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.951+0000 7fb838778640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:47.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.951+0000 7fb838778640 1 -- 172.21.15.46:0/2963971612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fb8301366b0 con 0x7fb830137050 2024-09-16T04:39:47.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.952+0000 7fb826ffd640 1 -- 172.21.15.46:0/2963971612 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fb82800ddd0 con 0x7fb830137050 2024-09-16T04:39:47.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.952+0000 7fb826ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:39:47.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.952+0000 7fb826ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:47.956 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.953+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:39:47.956 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.953+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=2,health=11,kv=12,logm=68} crc {auth=1422133551,config=3541143443,health=305525474,kv=4017288519,logm=3176731768}) 2024-09-16T04:39:47.956 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.953+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=8,config=2,health=11,kv=12,logm=67} crc {auth=739670991,config=3541143443,health=4003814221,kv=4017288519,logm=78027837}) 2024-09-16T04:39:47.956 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.953+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:39:47.956 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.953+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=2,health=11,kv=12,logm=68} crc {auth=1422133551,config=3541143443,health=305525474,kv=4017288519,logm=3176731768}) 2024-09-16T04:39:47.956 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.953+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=7,config=2,health=11,kv=11,logm=69} crc {auth=1574318101,config=3541143443,health=24569004,kv=2534035223,logm=2077141374}) 2024-09-16T04:39:47.956 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.954+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:39:47.956 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.954+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3948763773}) 2024-09-16T04:39:47.957 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.954+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3703291095}) 2024-09-16T04:39:47.957 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.954+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:39:47.957 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.954+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3948763773}) 2024-09-16T04:39:47.957 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.954+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3570096572}) 2024-09-16T04:39:47.957 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.954+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:39:47.958 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.954+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3449329076}) 2024-09-16T04:39:47.958 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.954+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3434600503}) 2024-09-16T04:39:47.958 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.954+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:39:47.958 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.954+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3449329076}) 2024-09-16T04:39:47.958 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.954+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4215615263}) 2024-09-16T04:39:47.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.954+0000 7fb838778640 1 -- 172.21.15.46:0/2963971612 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb8140383e0 msgr2=0x7fb81403a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:47.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.954+0000 7fb838778640 1 --2- 172.21.15.46:0/2963971612 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb8140383e0 0x7fb81403a8a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0097a0 tx=0x7fb82c02e000 comp rx=0 tx=0).stop 2024-09-16T04:39:47.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.955+0000 7fb838778640 10 monclient: shutdown 2024-09-16T04:39:47.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.955+0000 7fb838778640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:47.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.955+0000 7fb838778640 1 -- 172.21.15.46:0/2963971612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb830137050 msgr2=0x7fb830136060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:47.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.955+0000 7fb838778640 1 --2- 172.21.15.46:0/2963971612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb830137050 0x7fb830136060 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb82800ce20 tx=0x7fb82800a710 comp rx=0 tx=0).stop 2024-09-16T04:39:47.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.955+0000 7fb838778640 1 -- 172.21.15.46:0/2963971612 shutdown_connections 2024-09-16T04:39:47.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.955+0000 7fb838778640 1 --2- 172.21.15.46:0/2963971612 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb8140383e0 0x7fb81403a8a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.955+0000 7fb838778640 1 --2- 172.21.15.46:0/2963971612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb830137050 0x7fb830136060 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.955+0000 7fb838778640 1 --2- 172.21.15.46:0/2963971612 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb8300a2ee0 0x7fb830135b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.955+0000 7fb838778640 1 --2- 172.21.15.46:0/2963971612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb83006e160 0x7fb8301cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:47.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.955+0000 7fb838778640 1 -- 172.21.15.46:0/2963971612 >> 172.21.15.46:0/2963971612 conn(0x7fb8300a5c10 msgr2=0x7fb83009f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:47.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.955+0000 7fb838778640 1 -- 172.21.15.46:0/2963971612 shutdown_connections 2024-09-16T04:39:47.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:47.955+0000 7fb838778640 1 -- 172.21.15.46:0/2963971612 wait complete. 2024-09-16T04:39:47.961 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.955+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:39:47.961 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.955+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3783722962}) 2024-09-16T04:39:47.961 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.955+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2484858237}) 2024-09-16T04:39:47.961 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.955+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:39:47.961 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.955+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3783722962}) 2024-09-16T04:39:47.961 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.955+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3174011849}) 2024-09-16T04:39:47.961 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.957+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:39:47.961 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.957+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=7,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=12,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=48} crc {logm=3276130558,mdsmap=3140514009,mgr=1081907435,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=3845019559,mon_config_key=1894172608,monmap=1243539061,osd_metadata=777518657,osd_pg_creating=983461158,osd_snap=723727393,osdmap=3847502899}) 2024-09-16T04:39:47.961 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.957+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=7,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=9,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=47} crc {logm=1459866608,mdsmap=3956149183,mgr=1081907435,mgr_command_descs=3961477881,mgrstat=3845019559,mon_config_key=2123209483,monmap=3088247691,osd_metadata=777518658,osd_pg_creating=983461158,osd_snap=723727393,osdmap=1842715568}) 2024-09-16T04:39:47.961 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.957+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:39:47.961 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.957+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=7,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=12,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=48} crc {logm=3276130558,mdsmap=3140514009,mgr=1081907435,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=3845019559,mon_config_key=1894172608,monmap=1243539061,osd_metadata=777518657,osd_pg_creating=983461158,osd_snap=723727393,osdmap=3847502899}) 2024-09-16T04:39:47.961 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.957+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=6,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=12,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=49} crc {logm=231767387,mdsmap=3956149183,mgr=1081907435,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=3845019559,mon_config_key=1987702823,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=983461158,osd_snap=2941457380,osdmap=2409383095}) 2024-09-16T04:39:47.962 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.959+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:39:47.962 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.959+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=35} crc {osdmap=3032487549}) 2024-09-16T04:39:47.962 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.959+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=35} crc {osdmap=3784324796}) 2024-09-16T04:39:47.962 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.959+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:39:47.962 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.959+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=35} crc {osdmap=3032487549}) 2024-09-16T04:39:47.962 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:39:47.959+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=37} crc {osdmap=3995096591}) 2024-09-16T04:39:47.969 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:39:48.970 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:39:48.971 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:39:49.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.087+0000 7f8953b9f640 10 monclient: get_monmap_and_config 2024-09-16T04:39:49.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.087+0000 7f8953b9f640 10 monclient: build_initial_monmap 2024-09-16T04:39:49.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.087+0000 7f8953b9f640 10 monclient: monmap: 2024-09-16T04:39:49.091 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:49.091 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.091 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:49.089440+0000 2024-09-16T04:39:49.091 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:49.089440+0000 2024-09-16T04:39:49.091 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:49.091 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:49.092 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.088+0000 7f8953b9f640 1 Processor -- start 2024-09-16T04:39:49.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.089+0000 7f8953b9f640 1 -- start start 2024-09-16T04:39:49.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.089+0000 7f8953b9f640 10 monclient: init 2024-09-16T04:39:49.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f8953b9f640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:49.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f8953b9f640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-16T04:39:49.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f8953b9f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f894c09f810 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:49.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f8953b9f640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f894c0a0300 0x7f894c138610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f8953b9f640 10 monclient(hunting): picked mon.noname-c con 0x7f894c0a0300 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:49.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f8953b9f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f894c138b50 0x7f894c13af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f8953b9f640 10 monclient(hunting): picked mon.noname-a con 0x7f894c138b50 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:49.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f8953b9f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f894c0738e0 con 0x7f894c09f810 2024-09-16T04:39:49.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f8953b9f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f8953b9f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f894c09d290 con 0x7f894c138b50 2024-09-16T04:39:49.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f895213c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f894c138b50 0x7f894c13af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f8953b9f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f895213c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f894c138b50 0x7f894c13af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38218/0 (socket says 172.21.15.46:38218) 2024-09-16T04:39:49.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f895213c640 1 -- 172.21.15.46:0/2322285502 learned_addr learned my addr 172.21.15.46:0/2322285502 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:49.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f8953b9f640 1 -- 172.21.15.46:0/2322285502 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f894c09d570 con 0x7f894c0a0300 2024-09-16T04:39:49.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f8953b9f640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:49.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f8953b9f640 10 monclient(hunting): authenticate will time out at 787.506569s 2024-09-16T04:39:49.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f895113a640 1 --2- 172.21.15.46:0/2322285502 >> v2:172.21.15.114:3300/0 conn(0x7f894c0a0300 0x7f894c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f895213c640 10 monclient(hunting): get_auth_request con 0x7f894c138b50 auth_method 0 2024-09-16T04:39:49.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f895213c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f895213c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f895213c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f895113a640 10 monclient(hunting): get_auth_request con 0x7f894c0a0300 auth_method 0 2024-09-16T04:39:49.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f895113a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f895113a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f895113a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f895213c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f895213c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f895213c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f895113a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f895113a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f895113a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f895213c640 10 monclient(hunting): handle_auth_done global_id 4692 payload 274 2024-09-16T04:39:49.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f895213c640 1 -- 172.21.15.46:0/2322285502 >> v2:172.21.15.114:3300/0 conn(0x7f894c0a0300 msgr2=0x7f894c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:49.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f895213c640 1 --2- 172.21.15.46:0/2322285502 >> v2:172.21.15.114:3300/0 conn(0x7f894c0a0300 0x7f894c138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.090+0000 7f8950939640 1 -- 172.21.15.46:0/2322285502 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1998668591 0 0) 0x7f894c0738e0 con 0x7f894c09f810 2024-09-16T04:39:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f895213c640 1 -- 172.21.15.46:0/2322285502 >> v1:172.21.15.114:6789/0 conn(0x7f894c09f810 legacy=0x7f894c09fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f895213c640 10 monclient: _finish_hunting 0 2024-09-16T04:39:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f895213c640 1 monclient: found mon.noname-a 2024-09-16T04:39:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f895213c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f895213c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f895213c640 1 -- 172.21.15.46:0/2322285502 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f894c09d850 con 0x7f894c138b50 2024-09-16T04:39:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f8950939640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:39:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f895213c640 1 --2- 172.21.15.46:0/2322285502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f894c138b50 0x7f894c13af40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f894c09edc0 tx=0x7f894402b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46c3151bc8327ba1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f8950939640 1 -- 172.21.15.46:0/2322285502 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8944037070 con 0x7f894c138b50 2024-09-16T04:39:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f8950939640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f8950939640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f8950939640 10 monclient: dump: 2024-09-16T04:39:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.119 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.119 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.119 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.119 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.119 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.119 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.119 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.119 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.119 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f8950939640 10 monclient: _finish_auth 0 2024-09-16T04:39:49.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f8950939640 10 monclient: _check_auth_tickets 2024-09-16T04:39:49.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f8950939640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:49.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f8950939640 1 -- 172.21.15.46:0/2322285502 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8944008500 con 0x7f894c138b50 2024-09-16T04:39:49.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f8953b9f640 5 monclient: authenticate success, global_id 4692 2024-09-16T04:39:49.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f8953b9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:49.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f8950939640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:49.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f8953b9f640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:49.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f8953b9f640 4 set_mon_vals no callback set 2024-09-16T04:39:49.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f8950939640 1 -- 172.21.15.46:0/2322285502 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89440088d0 con 0x7f894c138b50 2024-09-16T04:39:49.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f8950939640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f8950939640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:49.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.091+0000 7f8950939640 10 monclient: dump: 2024-09-16T04:39:49.121 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.121 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.121 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.121 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.121 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.121 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.121 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.121 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.121 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.121 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.092+0000 7f8953b9f640 10 monclient: shutdown 2024-09-16T04:39:49.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.092+0000 7f8953b9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:49.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.092+0000 7f8953b9f640 1 -- 172.21.15.46:0/2322285502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f894c138b50 msgr2=0x7f894c13af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.092+0000 7f8953b9f640 1 --2- 172.21.15.46:0/2322285502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f894c138b50 0x7f894c13af40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f894c09edc0 tx=0x7f894402b720 comp rx=0 tx=0).stop 2024-09-16T04:39:49.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.092+0000 7f8953b9f640 1 -- 172.21.15.46:0/2322285502 shutdown_connections 2024-09-16T04:39:49.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.092+0000 7f8953b9f640 1 --2- 172.21.15.46:0/2322285502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f894c138b50 0x7f894c13af40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.092+0000 7f8953b9f640 1 --2- 172.21.15.46:0/2322285502 >> v2:172.21.15.114:3300/0 conn(0x7f894c0a0300 0x7f894c138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.092+0000 7f8953b9f640 1 -- 172.21.15.46:0/2322285502 >> 172.21.15.46:0/2322285502 conn(0x7f894c0a5960 msgr2=0x7f894c0a1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:49.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.092+0000 7f8953b9f640 1 -- 172.21.15.46:0/2322285502 shutdown_connections 2024-09-16T04:39:49.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.092+0000 7f8953b9f640 1 -- 172.21.15.46:0/2322285502 wait complete. 2024-09-16T04:39:49.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.093+0000 7f8953b9f640 10 monclient: build_initial_monmap 2024-09-16T04:39:49.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.093+0000 7f8953b9f640 10 monclient: monmap: 2024-09-16T04:39:49.122 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:49.122 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:49.122 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:49.123 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:49.123 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:49.123 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.124 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:49.124 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:49.124 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:49.124 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.093+0000 7f8953b9f640 1 Processor -- start 2024-09-16T04:39:49.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.093+0000 7f8953b9f640 1 -- start start 2024-09-16T04:39:49.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.093+0000 7f8953b9f640 10 monclient: init 2024-09-16T04:39:49.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f8953b9f640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:49.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f8953b9f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:39:49.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f8953b9f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f894c09f810 0x7f894c1cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f8953b9f640 10 monclient(hunting): picked mon.noname-a con 0x7f894c09f810 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:49.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f8953b9f640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f894c0a0300 0x7f894c1cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f8953b9f640 10 monclient(hunting): picked mon.noname-c con 0x7f894c0a0300 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:49.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f8953b9f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f894c138b50 0x7f894c1241a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f8953b9f640 10 monclient(hunting): picked mon.noname-b con 0x7f894c138b50 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:49.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f8953b9f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f8953b9f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f894c09c490 con 0x7f894c09f810 2024-09-16T04:39:49.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f8953b9f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f8953b9f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f894c09c310 con 0x7f894c138b50 2024-09-16T04:39:49.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f8953b9f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f8953b9f640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f894c09f1b0 con 0x7f894c0a0300 2024-09-16T04:39:49.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895193b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f894c09f810 0x7f894c1cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f8953b9f640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:49.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895113a640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f894c0a0300 0x7f894c1cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895193b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f894c09f810 0x7f894c1cbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38232/0 (socket says 172.21.15.46:38232) 2024-09-16T04:39:49.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895193b640 1 -- 172.21.15.46:0/2235361703 learned_addr learned my addr 172.21.15.46:0/2235361703 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:49.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895213c640 1 --2- 172.21.15.46:0/2235361703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f894c138b50 0x7f894c1241a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f8953b9f640 10 monclient(hunting): authenticate will time out at 787.510760s 2024-09-16T04:39:49.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895113a640 10 monclient(hunting): get_auth_request con 0x7f894c0a0300 auth_method 0 2024-09-16T04:39:49.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895113a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895113a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895113a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895213c640 10 monclient(hunting): get_auth_request con 0x7f894c138b50 auth_method 0 2024-09-16T04:39:49.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895213c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895213c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895213c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895193b640 10 monclient(hunting): get_auth_request con 0x7f894c09f810 auth_method 0 2024-09-16T04:39:49.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895193b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895193b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895193b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895113a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895113a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895113a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895213c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895213c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895213c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895193b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895193b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.094+0000 7f895193b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f895213c640 10 monclient(hunting): handle_auth_done global_id 4672 payload 824 2024-09-16T04:39:49.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f895213c640 1 -- 172.21.15.46:0/2235361703 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f894c0a0300 msgr2=0x7f894c1cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f895213c640 1 --2- 172.21.15.46:0/2235361703 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f894c0a0300 0x7f894c1cc4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f895213c640 1 -- 172.21.15.46:0/2235361703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f894c09f810 msgr2=0x7f894c1cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f895213c640 1 --2- 172.21.15.46:0/2235361703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f894c09f810 0x7f894c1cbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f895213c640 10 monclient: _finish_hunting 0 2024-09-16T04:39:49.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f895213c640 1 monclient: found mon.noname-b 2024-09-16T04:39:49.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f895213c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:49.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f895213c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3300/0 2024-09-16T04:39:49.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f895213c640 1 -- 172.21.15.46:0/2235361703 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f894c127ae0 con 0x7f894c138b50 2024-09-16T04:39:49.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f895113a640 1 --2- 172.21.15.46:0/2235361703 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f894c0a0300 0x7f894c1cc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:49.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f895193b640 1 --2- 172.21.15.46:0/2235361703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f894c09f810 0x7f894c1cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:49.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f895213c640 1 --2- 172.21.15.46:0/2235361703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f894c138b50 0x7f894c1241a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8944031000 tx=0x7f8944004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:49.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f89427fc640 1 -- 172.21.15.46:0/2235361703 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8944041070 con 0x7f894c138b50 2024-09-16T04:39:49.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f89427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f89427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:49.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f89427fc640 10 monclient: dump: 2024-09-16T04:39:49.134 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.134 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.134 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.134 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.134 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.134 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.135 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.135 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.135 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.135 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f89427fc640 10 monclient: _finish_auth 0 2024-09-16T04:39:49.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f89427fc640 10 monclient: _check_auth_tickets 2024-09-16T04:39:49.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f89427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:49.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f8953b9f640 5 monclient: authenticate success, global_id 4672 2024-09-16T04:39:49.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f89427fc640 1 -- 172.21.15.46:0/2235361703 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8944008e50 con 0x7f894c138b50 2024-09-16T04:39:49.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f89427fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:49.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f89427fc640 1 -- 172.21.15.46:0/2235361703 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8944037040 con 0x7f894c138b50 2024-09-16T04:39:49.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f89427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f89427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:49.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f89427fc640 10 monclient: dump: 2024-09-16T04:39:49.136 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.136 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.136 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.136 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.136 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.136 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.136 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.136 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.136 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.136 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f8953b9f640 10 monclient: _renew_subs 2024-09-16T04:39:49.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f8953b9f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:49.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f8953b9f640 1 -- 172.21.15.46:0/2235361703 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f894c126910 con 0x7f894c138b50 2024-09-16T04:39:49.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f89437fe640 4 set_mon_vals no callback set 2024-09-16T04:39:49.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f8953b9f640 10 monclient: _renew_subs 2024-09-16T04:39:49.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f8953b9f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:49.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.095+0000 7f8953b9f640 1 -- 172.21.15.46:0/2235361703 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f894c1261f0 con 0x7f894c138b50 2024-09-16T04:39:49.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.096+0000 7f89427fc640 1 -- 172.21.15.46:0/2235361703 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f8944003bd0 con 0x7f894c138b50 2024-09-16T04:39:49.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.097+0000 7f8953b9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:49.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.097+0000 7f8953b9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:49.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.097+0000 7f8953b9f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:49.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.097+0000 7f8953b9f640 1 -- 172.21.15.46:0/2235361703 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f894c0724b0 con 0x7f894c138b50 2024-09-16T04:39:49.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.097+0000 7f89427fc640 1 --2- 172.21.15.46:0/2235361703 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f89300383b0 0x7f893003a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.097+0000 7f89427fc640 1 -- 172.21.15.46:0/2235361703 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4738+0+0 (secure 0 0 0) 0x7f894406c080 con 0x7f894c138b50 2024-09-16T04:39:49.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.097+0000 7f895193b640 1 --2- 172.21.15.46:0/2235361703 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f89300383b0 0x7f893003a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.097+0000 7f895193b640 10 monclient: get_auth_request con 0x7f89300383b0 auth_method 0 2024-09-16T04:39:49.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.097+0000 7f895193b640 1 --2- 172.21.15.46:0/2235361703 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f89300383b0 0x7f893003a870 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8948007fd0 tx=0x7f8948009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:49.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.107+0000 7f89427fc640 1 -- 172.21.15.46:0/2235361703 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f894404b020 con 0x7f894c138b50 2024-09-16T04:39:49.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.107+0000 7f89427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:49.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.107+0000 7f89427fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:49.170 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:49.170 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35778]: starting. 2024-09-16T04:39:49.170 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35778]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-35743 2024-09-16T04:39:49.170 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35778]: created object 0... 2024-09-16T04:39:49.171 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35778]: created object 25... 2024-09-16T04:39:49.171 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35778]: created object 49... 2024-09-16T04:39:49.171 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35778]: finishing. 2024-09-16T04:39:49.171 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35778]: shutting down. 2024-09-16T04:39:49.182 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:39:49.182 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35780]: starting. 2024-09-16T04:39:49.182 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35780]: listing objects. 2024-09-16T04:39:49.182 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35780]: listed object 0... 2024-09-16T04:39:49.182 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35780]: listed object 25... 2024-09-16T04:39:49.182 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35780]: saw 50 objects 2024-09-16T04:39:49.182 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35780]: shutting down. 2024-09-16T04:39:49.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.357+0000 7f8953b9f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:39:49.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.357+0000 7f8953b9f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:39:49.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.357+0000 7f8953b9f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:39:49.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.357+0000 7f8953b9f640 1 -- 172.21.15.46:0/2235361703 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f894c09e2c0 con 0x7f894c138b50 2024-09-16T04:39:49.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.357+0000 7f89427fc640 1 -- 172.21.15.46:0/2235361703 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f894403d820 con 0x7f894c138b50 2024-09-16T04:39:49.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.357+0000 7f89427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:39:49.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.357+0000 7f89427fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:49.360 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:39:49.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.359+0000 7f8953b9f640 1 -- 172.21.15.46:0/2235361703 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f89300383b0 msgr2=0x7f893003a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.359+0000 7f8953b9f640 1 --2- 172.21.15.46:0/2235361703 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f89300383b0 0x7f893003a870 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8948007fd0 tx=0x7f8948009040 comp rx=0 tx=0).stop 2024-09-16T04:39:49.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.359+0000 7f8953b9f640 10 monclient: shutdown 2024-09-16T04:39:49.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.359+0000 7f8953b9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:49.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.359+0000 7f8953b9f640 1 -- 172.21.15.46:0/2235361703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f894c138b50 msgr2=0x7f894c1241a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.359+0000 7f8953b9f640 1 --2- 172.21.15.46:0/2235361703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f894c138b50 0x7f894c1241a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8944031000 tx=0x7f8944004a40 comp rx=0 tx=0).stop 2024-09-16T04:39:49.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.359+0000 7f8953b9f640 1 -- 172.21.15.46:0/2235361703 shutdown_connections 2024-09-16T04:39:49.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.359+0000 7f8953b9f640 1 --2- 172.21.15.46:0/2235361703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f894c138b50 0x7f894c1241a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.359+0000 7f8953b9f640 1 --2- 172.21.15.46:0/2235361703 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f894c0a0300 0x7f894c1cc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.359+0000 7f8953b9f640 1 --2- 172.21.15.46:0/2235361703 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f89300383b0 0x7f893003a870 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.359+0000 7f8953b9f640 1 --2- 172.21.15.46:0/2235361703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f894c09f810 0x7f894c1cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.359+0000 7f8953b9f640 1 -- 172.21.15.46:0/2235361703 >> 172.21.15.46:0/2235361703 conn(0x7f894c0a5960 msgr2=0x7f894c070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:49.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.359+0000 7f8953b9f640 1 -- 172.21.15.46:0/2235361703 shutdown_connections 2024-09-16T04:39:49.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.359+0000 7f8953b9f640 1 -- 172.21.15.46:0/2235361703 wait complete. 2024-09-16T04:39:49.372 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:39:49.372 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:39:49.372 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:39:49.373 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:39:49.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.473+0000 7fd18e3d6640 10 monclient: get_monmap_and_config 2024-09-16T04:39:49.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.473+0000 7fd18e3d6640 10 monclient: build_initial_monmap 2024-09-16T04:39:49.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.473+0000 7fd18e3d6640 10 monclient: monmap: 2024-09-16T04:39:49.476 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:49.476 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.476 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:49.474897+0000 2024-09-16T04:39:49.476 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:49.474897+0000 2024-09-16T04:39:49.476 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:49.476 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.476 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:49.476 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:49.476 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:49.476 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:49.476 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:49.476 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18e3d6640 1 Processor -- start 2024-09-16T04:39:49.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18e3d6640 1 -- start start 2024-09-16T04:39:49.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18e3d6640 10 monclient: init 2024-09-16T04:39:49.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18e3d6640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:49.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18e3d6640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-16T04:39:49.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18e3d6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd1880a3c10 0x7fd18809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18e3d6640 10 monclient(hunting): picked mon.noname-b con 0x7fd1880a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18e3d6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd1880a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18e3d6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd18813be90 0x7fd18813e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18e3d6640 10 monclient(hunting): picked mon.noname-a con 0x7fd18813be90 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18e3d6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd18809b940 con 0x7fd1880a3ff0 2024-09-16T04:39:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18e3d6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18e3d6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd18809ab00 con 0x7fd18813be90 2024-09-16T04:39:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18e3d6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18e3d6640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd18809b030 con 0x7fd1880a3c10 2024-09-16T04:39:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18e3d6640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18e3d6640 10 monclient(hunting): authenticate will time out at 787.890700s 2024-09-16T04:39:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd187fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd1880a3c10 0x7fd18809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18c973640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd18813be90 0x7fd18813e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd187fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd1880a3c10 0x7fd18809bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:52340/0 (socket says 172.21.15.46:52340) 2024-09-16T04:39:49.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18c973640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd18813be90 0x7fd18813e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38234/0 (socket says 172.21.15.46:38234) 2024-09-16T04:39:49.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd187fff640 1 -- 172.21.15.46:0/2980103891 learned_addr learned my addr 172.21.15.46:0/2980103891 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:49.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd187fff640 10 monclient(hunting): get_auth_request con 0x7fd1880a3c10 auth_method 0 2024-09-16T04:39:49.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd187fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd187fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd187fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18c973640 10 monclient(hunting): get_auth_request con 0x7fd18813be90 auth_method 0 2024-09-16T04:39:49.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18c973640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18c973640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18c973640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd187fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd187fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd187fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18c973640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.474+0000 7fd18c973640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.475+0000 7fd18c973640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.475+0000 7fd187fff640 10 monclient(hunting): handle_auth_done global_id 14117 payload 274 2024-09-16T04:39:49.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.475+0000 7fd187fff640 1 -- 172.21.15.46:0/2980103891 >> v2:172.21.15.46:3300/0 conn(0x7fd18813be90 msgr2=0x7fd18813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:49.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.475+0000 7fd187fff640 1 --2- 172.21.15.46:0/2980103891 >> v2:172.21.15.46:3300/0 conn(0x7fd18813be90 0x7fd18813e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.475+0000 7fd187fff640 1 -- 172.21.15.46:0/2980103891 >> v1:172.21.15.114:6789/0 conn(0x7fd1880a3ff0 legacy=0x7fd1880a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:49.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.475+0000 7fd187fff640 10 monclient: _finish_hunting 0 2024-09-16T04:39:49.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.475+0000 7fd187fff640 1 monclient: found mon.noname-b 2024-09-16T04:39:49.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.475+0000 7fd187fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:49.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.475+0000 7fd187fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:39:49.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.475+0000 7fd187fff640 1 -- 172.21.15.46:0/2980103891 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd18809b310 con 0x7fd1880a3c10 2024-09-16T04:39:49.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.475+0000 7fd18c973640 1 --2- 172.21.15.46:0/2980103891 >> v2:172.21.15.46:3300/0 conn(0x7fd18813be90 0x7fd18813e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:49.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.475+0000 7fd187fff640 1 --2- 172.21.15.46:0/2980103891 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd1880a3c10 0x7fd18809bcf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd178009870 tx=0x7fd17802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86a41430e1c4fd2b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:49.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd186ffd640 1 -- 172.21.15.46:0/2980103891 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd178037070 con 0x7fd1880a3c10 2024-09-16T04:39:49.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd186ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd186ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:39:49.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd186ffd640 10 monclient: dump: 2024-09-16T04:39:49.490 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.490 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.490 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.490 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.490 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.491 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.491 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.491 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.491 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.491 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd186ffd640 10 monclient: _finish_auth 0 2024-09-16T04:39:49.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd186ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:39:49.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd186ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:49.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd186ffd640 1 -- 172.21.15.46:0/2980103891 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd17802ae10 con 0x7fd1880a3c10 2024-09-16T04:39:49.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd18e3d6640 5 monclient: authenticate success, global_id 14117 2024-09-16T04:39:49.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd18e3d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:49.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd186ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:49.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd18e3d6640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:49.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd186ffd640 1 -- 172.21.15.46:0/2980103891 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd178032660 con 0x7fd1880a3c10 2024-09-16T04:39:49.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd18e3d6640 4 set_mon_vals no callback set 2024-09-16T04:39:49.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd186ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd186ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:49.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd186ffd640 10 monclient: dump: 2024-09-16T04:39:49.492 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.492 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.493 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.493 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.493 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.493 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.493 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.493 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.493 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.493 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd18e3d6640 10 monclient: shutdown 2024-09-16T04:39:49.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd18e3d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:49.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd18e3d6640 1 -- 172.21.15.46:0/2980103891 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd1880a3c10 msgr2=0x7fd18809bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd18e3d6640 1 --2- 172.21.15.46:0/2980103891 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd1880a3c10 0x7fd18809bcf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd178009870 tx=0x7fd17802a100 comp rx=0 tx=0).stop 2024-09-16T04:39:49.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd18e3d6640 1 -- 172.21.15.46:0/2980103891 shutdown_connections 2024-09-16T04:39:49.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd18e3d6640 1 --2- 172.21.15.46:0/2980103891 >> v2:172.21.15.46:3300/0 conn(0x7fd18813be90 0x7fd18813e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd18e3d6640 1 --2- 172.21.15.46:0/2980103891 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd1880a3c10 0x7fd18809bcf0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd18e3d6640 1 -- 172.21.15.46:0/2980103891 >> 172.21.15.46:0/2980103891 conn(0x7fd1880a6340 msgr2=0x7fd18806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:49.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd18e3d6640 1 -- 172.21.15.46:0/2980103891 shutdown_connections 2024-09-16T04:39:49.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.476+0000 7fd18e3d6640 1 -- 172.21.15.46:0/2980103891 wait complete. 2024-09-16T04:39:49.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.477+0000 7fd18e3d6640 10 monclient: build_initial_monmap 2024-09-16T04:39:49.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.477+0000 7fd18e3d6640 10 monclient: monmap: 2024-09-16T04:39:49.494 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:49.494 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:49.494 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:49.494 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:49.494 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:49.495 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.495 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:49.495 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:49.495 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:49.495 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.477+0000 7fd18e3d6640 1 Processor -- start 2024-09-16T04:39:49.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.477+0000 7fd18e3d6640 1 -- start start 2024-09-16T04:39:49.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.477+0000 7fd18e3d6640 10 monclient: init 2024-09-16T04:39:49.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18e3d6640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:49.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18e3d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:39:49.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18e3d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1880a3c10 0x7fd1881cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18e3d6640 10 monclient(hunting): picked mon.noname-a con 0x7fd1880a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:49.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18e3d6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1880a3ff0 0x7fd1881cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18e3d6640 10 monclient(hunting): picked mon.noname-b con 0x7fd1880a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:49.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18e3d6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd18813be90 0x7fd1881cc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18e3d6640 10 monclient(hunting): picked mon.noname-c con 0x7fd18813be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:49.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18e3d6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18e3d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd1880a6b40 con 0x7fd1880a3c10 2024-09-16T04:39:49.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18e3d6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18e3d6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd1880a69c0 con 0x7fd1880a3ff0 2024-09-16T04:39:49.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18e3d6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18e3d6640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd18813e7c0 con 0x7fd18813be90 2024-09-16T04:39:49.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18e3d6640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:49.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18e3d6640 10 monclient(hunting): authenticate will time out at 787.894267s 2024-09-16T04:39:49.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd187fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1880a3c10 0x7fd1881cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd187fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1880a3c10 0x7fd1881cbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38242/0 (socket says 172.21.15.46:38242) 2024-09-16T04:39:49.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd187fff640 1 -- 172.21.15.46:0/954534861 learned_addr learned my addr 172.21.15.46:0/954534861 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:49.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd187fff640 10 monclient(hunting): get_auth_request con 0x7fd1880a3c10 auth_method 0 2024-09-16T04:39:49.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd187fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd187fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd187fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18c973640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd18813be90 0x7fd1881cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18c973640 10 monclient(hunting): get_auth_request con 0x7fd18813be90 auth_method 0 2024-09-16T04:39:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18c973640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18c973640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18c973640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd187fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd187fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd187fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18c973640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18c973640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18c973640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd1877fe640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1880a3ff0 0x7fd1881cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd1877fe640 10 monclient(hunting): get_auth_request con 0x7fd1880a3ff0 auth_method 0 2024-09-16T04:39:49.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd1877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd1877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd1877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18c973640 10 monclient(hunting): handle_auth_done global_id 14120 payload 824 2024-09-16T04:39:49.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18c973640 1 -- 172.21.15.46:0/954534861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1880a3ff0 msgr2=0x7fd1881cc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18c973640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1880a3ff0 0x7fd1881cc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18c973640 1 -- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1880a3c10 msgr2=0x7fd1881cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18c973640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1880a3c10 0x7fd1881cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18c973640 10 monclient: _finish_hunting 0 2024-09-16T04:39:49.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18c973640 1 monclient: found mon.noname-c 2024-09-16T04:39:49.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18c973640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:49.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18c973640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:39:49.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18c973640 1 -- 172.21.15.46:0/954534861 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1880a12e0 con 0x7fd18813be90 2024-09-16T04:39:49.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd187fff640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1880a3c10 0x7fd1881cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:49.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.478+0000 7fd18c973640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd18813be90 0x7fd1881cc620 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd18000ef90 tx=0x7fd18000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:49.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd184ff9640 1 -- 172.21.15.46:0/954534861 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd180018030 con 0x7fd18813be90 2024-09-16T04:39:49.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd184ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd184ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:39:49.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd184ff9640 10 monclient: dump: 2024-09-16T04:39:49.499 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.499 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.499 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.499 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.499 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.499 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.499 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.499 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.500 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.500 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd184ff9640 10 monclient: _finish_auth 0 2024-09-16T04:39:49.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd184ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:39:49.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd184ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:49.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd184ff9640 1 -- 172.21.15.46:0/954534861 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd18000cd30 con 0x7fd18813be90 2024-09-16T04:39:49.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd18e3d6640 5 monclient: authenticate success, global_id 14120 2024-09-16T04:39:49.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd18e3d6640 10 monclient: _renew_subs 2024-09-16T04:39:49.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd18e3d6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:49.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd18e3d6640 1 -- 172.21.15.46:0/954534861 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1880a2550 con 0x7fd18813be90 2024-09-16T04:39:49.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd184ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:49.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd184ff9640 1 -- 172.21.15.46:0/954534861 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd180003bb0 con 0x7fd18813be90 2024-09-16T04:39:49.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd184ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd184ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:49.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd184ff9640 10 monclient: dump: 2024-09-16T04:39:49.501 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.501 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.501 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.501 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.501 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.501 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.501 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.501 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.501 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.501 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd18e3d6640 10 monclient: _renew_subs 2024-09-16T04:39:49.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd18e3d6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:39:49.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd18e3d6640 1 -- 172.21.15.46:0/954534861 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1880a0e20 con 0x7fd18813be90 2024-09-16T04:39:49.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd1867fc640 4 set_mon_vals no callback set 2024-09-16T04:39:49.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd18e3d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:49.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd18e3d6640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18809bcf0 0x7fd1881339a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd18e3d6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd18e3d6640 1 -- 172.21.15.46:0/954534861 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd1880008d0 con 0x7fd18809bcf0 2024-09-16T04:39:49.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd184ff9640 1 -- 172.21.15.46:0/954534861 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fd180005d10 con 0x7fd18813be90 2024-09-16T04:39:49.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd187fff640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18809bcf0 0x7fd1881339a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd187fff640 10 monclient: get_auth_request con 0x7fd18809bcf0 auth_method 0 2024-09-16T04:39:49.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd187fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd187fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.479+0000 7fd187fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.480+0000 7fd187fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.480+0000 7fd187fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.480+0000 7fd187fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.480+0000 7fd184ff9640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd1740383b0 0x7fd17403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.480+0000 7fd184ff9640 1 -- 172.21.15.46:0/954534861 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(49..49 src has 1..49) ==== 4738+0+0 (secure 0 0 0) 0x7fd18004c040 con 0x7fd18813be90 2024-09-16T04:39:49.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.480+0000 7fd187fff640 10 monclient(hunting): handle_auth_done global_id 4704 payload 824 2024-09-16T04:39:49.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.480+0000 7fd187fff640 1 -- 172.21.15.46:0/954534861 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd1880a0400 con 0x7fd18809bcf0 2024-09-16T04:39:49.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.480+0000 7fd187fff640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18809bcf0 0x7fd1881339a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd178035040 tx=0x7fd1780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:49.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.480+0000 7fd1877fe640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd1740383b0 0x7fd17403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.480+0000 7fd184ff9640 1 -- 172.21.15.46:0/954534861 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd178037040 con 0x7fd18809bcf0 2024-09-16T04:39:49.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.480+0000 7fd184ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.480+0000 7fd184ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:49.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.480+0000 7fd184ff9640 10 monclient: dump: 2024-09-16T04:39:49.503 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.503 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.503 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.504 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.504 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.504 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.504 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.504 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.504 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.504 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.480+0000 7fd1877fe640 10 monclient: get_auth_request con 0x7fd1740383b0 auth_method 0 2024-09-16T04:39:49.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.480+0000 7fd184ff9640 1 -- 172.21.15.46:0/954534861 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd17803c8e0 con 0x7fd18809bcf0 2024-09-16T04:39:49.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.480+0000 7fd184ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:49.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.480+0000 7fd184ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:49.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.480+0000 7fd184ff9640 1 -- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18809bcf0 msgr2=0x7fd1881339a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.480+0000 7fd184ff9640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18809bcf0 0x7fd1881339a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd178035040 tx=0x7fd1780046c0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.480+0000 7fd184ff9640 1 -- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18809bcf0 msgr2=0x7fd1881339a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:49.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.480+0000 7fd184ff9640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18809bcf0 0x7fd1881339a0 secure :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd178035040 tx=0x7fd1780046c0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.481+0000 7fd1877fe640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd1740383b0 0x7fd17403a870 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd18809fae0 tx=0x7fd17c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.486+0000 7fd18e3d6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:39:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.486+0000 7fd18e3d6640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1880a2fc0 0x7fd188133ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd18e3d6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd18e3d6640 1 -- 172.21.15.46:0/954534861 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd1881d7100 con 0x7fd1880a2fc0 2024-09-16T04:39:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd187fff640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1880a2fc0 0x7fd188133ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd187fff640 10 monclient: get_auth_request con 0x7fd1880a2fc0 auth_method 0 2024-09-16T04:39:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd187fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd187fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd187fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd187fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd187fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd187fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd187fff640 10 monclient(hunting): handle_auth_done global_id 4707 payload 824 2024-09-16T04:39:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd187fff640 1 -- 172.21.15.46:0/954534861 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd1881d72b0 con 0x7fd1880a2fc0 2024-09-16T04:39:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd187fff640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1880a2fc0 0x7fd188133ee0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd1880a5910 tx=0x7fd178008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd184ff9640 1 -- 172.21.15.46:0/954534861 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd178037040 con 0x7fd1880a2fc0 2024-09-16T04:39:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd184ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd184ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd184ff9640 10 monclient: dump: 2024-09-16T04:39:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.509 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd184ff9640 1 -- 172.21.15.46:0/954534861 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7fd17804c2b0 con 0x7fd1880a2fc0 2024-09-16T04:39:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd184ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:39:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd184ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd184ff9640 1 -- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1880a2fc0 msgr2=0x7fd188133ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd184ff9640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1880a2fc0 0x7fd188133ee0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd1880a5910 tx=0x7fd178008030 comp rx=0 tx=0).stop 2024-09-16T04:39:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd184ff9640 1 -- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1880a2fc0 msgr2=0x7fd188133ee0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.487+0000 7fd184ff9640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1880a2fc0 0x7fd188133ee0 secure :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd1880a5910 tx=0x7fd178008030 comp rx=0 tx=0).stop 2024-09-16T04:39:49.510 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:39:49.510 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:39:49.510 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:39:49.510 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:39:49.510 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 14, 2024-09-16T04:39:49.510 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 75426, 2024-09-16T04:39:49.510 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:39:49.510 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:39:49.510 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:39:49.510 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:39:49.510 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:49.510 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 7, 2024-09-16T04:39:49.511 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:49.511 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:39:49.511 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:39:49.511 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:49.511 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:49.511 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:49.511 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:49.511 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:49.511 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:49.511 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:49.511 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:49.511 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:49.511 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:49.511 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:49.512 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:49.512 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:39:49.512 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:39:49.512 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:49.512 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:49.512 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:49.512 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:49.512 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:49.512 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:49.512 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:49.512 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:49.512 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:49.512 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:49.513 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:49.513 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:49.513 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.513 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:39:49.513 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:39:49.513 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:39:49.513 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:39:49.513 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:39:49.513 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:39:49.513 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:49.513 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:49.513 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:39:49.513 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:39:49.513 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:39:49.513 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:39:49.514 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:39:49.514 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:39:49.514 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:39:49.514 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:49.514 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:39:49.514 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:49.514 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:49.514 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:49.514 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:49.514 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:49.514 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:49.514 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:49.514 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:49.514 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:49.515 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:49.515 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:49.515 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:49.515 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:39:49.515 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.515 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:39:49.515 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.515 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:39:49.515 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:39:49.515 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:49.515 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:49.515 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.515 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:49.515 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:39:49.516 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:49.516 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.516 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.516 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:49.516 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:39:49.516 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:49.516 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:49.516 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:49.516 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.516 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:39:49.516 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:39:49.516 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:49.516 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:49.516 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:49.517 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.517 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.517 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:39:49.517 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:39:49.517 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:49.517 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:49.517 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.517 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:49.517 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:39:49.517 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:49.517 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.517 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.517 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:49.517 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:39:49.517 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:49.518 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:49.518 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:49.518 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.518 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:39:49.518 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:39:49.518 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:49.518 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:49.518 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:49.518 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.518 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.518 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:39:49.518 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:39:49.518 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:49.518 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:49.519 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.519 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:49.519 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:39:49.519 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:49.519 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.519 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.519 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:49.519 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:39:49.519 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:49.519 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:49.519 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:49.519 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.519 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:39:49.519 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:39:49.519 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:49.520 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:49.520 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:49.520 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:49.520 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:49.520 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.520 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:39:49.520 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:39:49.520 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.520 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:49.520 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:49.520 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:39:49.520 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:49.520 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:49.520 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:39:49.521 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.521 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:49.521 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:49.521 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:39:49.521 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:49.521 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:49.521 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:39:49.521 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.521 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:49.521 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:49.521 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:39:49.521 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:49.521 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:49.521 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.521 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:39:49.522 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:39:49.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.488+0000 7fd18e3d6640 1 -- 172.21.15.46:0/954534861 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd1740383b0 msgr2=0x7fd17403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.488+0000 7fd18e3d6640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd1740383b0 0x7fd17403a870 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd18809fae0 tx=0x7fd17c009040 comp rx=0 tx=0).stop 2024-09-16T04:39:49.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.488+0000 7fd18e3d6640 10 monclient: shutdown 2024-09-16T04:39:49.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.488+0000 7fd18e3d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:49.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.488+0000 7fd18e3d6640 1 -- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd18813be90 msgr2=0x7fd1881cc620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.488+0000 7fd18c973640 1 -- 172.21.15.46:0/954534861 reap_dead start 2024-09-16T04:39:49.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.488+0000 7fd18e3d6640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd18813be90 0x7fd1881cc620 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd18000ef90 tx=0x7fd18000c490 comp rx=0 tx=0).stop 2024-09-16T04:39:49.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.488+0000 7fd18e3d6640 1 -- 172.21.15.46:0/954534861 shutdown_connections 2024-09-16T04:39:49.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.488+0000 7fd18e3d6640 1 --2- 172.21.15.46:0/954534861 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd18813be90 0x7fd1881cc620 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.488+0000 7fd18e3d6640 1 -- 172.21.15.46:0/954534861 >> 172.21.15.46:0/954534861 conn(0x7fd1880a6340 msgr2=0x7fd1880a8720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:49.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.488+0000 7fd18e3d6640 1 -- 172.21.15.46:0/954534861 shutdown_connections 2024-09-16T04:39:49.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.488+0000 7fd18e3d6640 1 -- 172.21.15.46:0/954534861 wait complete. 2024-09-16T04:39:49.524 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:39:49.524 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:39:49.524 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:39:49.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.625+0000 7fad9ea3c640 10 monclient: get_monmap_and_config 2024-09-16T04:39:49.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.625+0000 7fad9ea3c640 10 monclient: build_initial_monmap 2024-09-16T04:39:49.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.625+0000 7fad9ea3c640 10 monclient: monmap: 2024-09-16T04:39:49.628 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:49.628 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.628 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:49.626944+0000 2024-09-16T04:39:49.628 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:49.626944+0000 2024-09-16T04:39:49.628 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:49.628 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.628 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:49.628 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:49.628 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:49.628 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:49.628 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:49.628 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9ea3c640 1 Processor -- start 2024-09-16T04:39:49.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9ea3c640 1 -- start start 2024-09-16T04:39:49.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9ea3c640 10 monclient: init 2024-09-16T04:39:49.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9ea3c640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:49.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9ea3c640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-16T04:39:49.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9ea3c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fad980a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:49.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9ea3c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fad980a3ff0 0x7fad980a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9ea3c640 10 monclient(hunting): picked mon.noname-a con 0x7fad980a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:49.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9ea3c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad9813be90 0x7fad9813e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9ea3c640 10 monclient(hunting): picked mon.noname-b con 0x7fad9813be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:49.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9ea3c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad9809b940 con 0x7fad980a3c10 2024-09-16T04:39:49.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9ea3c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9ea3c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fad9809ab00 con 0x7fad980a3ff0 2024-09-16T04:39:49.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9ea3c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9ea3c640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fad9809b030 con 0x7fad9813be90 2024-09-16T04:39:49.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9ea3c640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:49.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9ea3c640 10 monclient(hunting): authenticate will time out at 788.042764s 2024-09-16T04:39:49.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad977fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fad980a3ff0 0x7fad980a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad977fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fad980a3ff0 0x7fad980a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38282/0 (socket says 172.21.15.46:38282) 2024-09-16T04:39:49.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad977fe640 1 -- 172.21.15.46:0/997300463 learned_addr learned my addr 172.21.15.46:0/997300463 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:49.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad977fe640 10 monclient(hunting): get_auth_request con 0x7fad980a3ff0 auth_method 0 2024-09-16T04:39:49.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9cfd9640 1 --2- 172.21.15.46:0/997300463 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad9813be90 0x7fad9813e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9cfd9640 10 monclient(hunting): get_auth_request con 0x7fad9813be90 auth_method 0 2024-09-16T04:39:49.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9cfd9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9cfd9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad9cfd9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.626+0000 7fad977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.627+0000 7fad977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.627+0000 7fad977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.627+0000 7fad9cfd9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.627+0000 7fad9cfd9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.627+0000 7fad9cfd9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.627+0000 7fad977fe640 10 monclient(hunting): handle_auth_done global_id 4710 payload 274 2024-09-16T04:39:49.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.627+0000 7fad977fe640 1 -- 172.21.15.46:0/997300463 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad9813be90 msgr2=0x7fad9813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:49.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.627+0000 7fad977fe640 1 --2- 172.21.15.46:0/997300463 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad9813be90 0x7fad9813e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.627+0000 7fad977fe640 1 -- 172.21.15.46:0/997300463 >> v1:172.21.15.114:6789/0 conn(0x7fad980a3c10 legacy=0x7fad9809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.627+0000 7fad977fe640 10 monclient: _finish_hunting 0 2024-09-16T04:39:49.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.627+0000 7fad977fe640 1 monclient: found mon.noname-a 2024-09-16T04:39:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.627+0000 7fad977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.627+0000 7fad977fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.627+0000 7fad977fe640 1 -- 172.21.15.46:0/997300463 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad9809b310 con 0x7fad980a3ff0 2024-09-16T04:39:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.627+0000 7fad977fe640 1 --2- 172.21.15.46:0/997300463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad980a3ff0 0x7fad980a43d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fad8c009870 tx=0x7fad8c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11aac10d7c260a2d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad96ffd640 1 -- 172.21.15.46:0/997300463 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c037070 con 0x7fad980a3ff0 2024-09-16T04:39:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad96ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad96ffd640 10 monclient: dump: 2024-09-16T04:39:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.642 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.642 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.642 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.642 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad96ffd640 10 monclient: _finish_auth 0 2024-09-16T04:39:49.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad96ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:39:49.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:49.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad9ea3c640 5 monclient: authenticate success, global_id 4710 2024-09-16T04:39:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad96ffd640 1 -- 172.21.15.46:0/997300463 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad8c032440 con 0x7fad980a3ff0 2024-09-16T04:39:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad96ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad9ea3c640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad9ea3c640 4 set_mon_vals no callback set 2024-09-16T04:39:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad96ffd640 1 -- 172.21.15.46:0/997300463 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c032810 con 0x7fad980a3ff0 2024-09-16T04:39:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad96ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad96ffd640 10 monclient: dump: 2024-09-16T04:39:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.644 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.644 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.644 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.644 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad9ea3c640 10 monclient: shutdown 2024-09-16T04:39:49.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad9ea3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:49.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad9ea3c640 1 -- 172.21.15.46:0/997300463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad980a3ff0 msgr2=0x7fad980a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad9ea3c640 1 --2- 172.21.15.46:0/997300463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad980a3ff0 0x7fad980a43d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fad8c009870 tx=0x7fad8c02a1c0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad9ea3c640 1 -- 172.21.15.46:0/997300463 shutdown_connections 2024-09-16T04:39:49.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad9ea3c640 1 --2- 172.21.15.46:0/997300463 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad9813be90 0x7fad9813e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad9ea3c640 1 --2- 172.21.15.46:0/997300463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad980a3ff0 0x7fad980a43d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad9ea3c640 1 -- 172.21.15.46:0/997300463 >> 172.21.15.46:0/997300463 conn(0x7fad980a6340 msgr2=0x7fad9806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:49.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad9ea3c640 1 -- 172.21.15.46:0/997300463 shutdown_connections 2024-09-16T04:39:49.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.628+0000 7fad9ea3c640 1 -- 172.21.15.46:0/997300463 wait complete. 2024-09-16T04:39:49.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.629+0000 7fad9ea3c640 10 monclient: build_initial_monmap 2024-09-16T04:39:49.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.629+0000 7fad9ea3c640 10 monclient: monmap: 2024-09-16T04:39:49.645 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:49.645 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:49.645 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:49.645 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:49.645 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:49.645 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.645 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:49.645 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:49.645 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:49.645 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.629+0000 7fad9ea3c640 1 Processor -- start 2024-09-16T04:39:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.629+0000 7fad9ea3c640 1 -- start start 2024-09-16T04:39:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.629+0000 7fad9ea3c640 10 monclient: init 2024-09-16T04:39:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9ea3c640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9ea3c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:39:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9ea3c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad980a3c10 0x7fad98133cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9ea3c640 10 monclient(hunting): picked mon.noname-a con 0x7fad980a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9ea3c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad980a3ff0 0x7fad98134230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9ea3c640 10 monclient(hunting): picked mon.noname-b con 0x7fad980a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9ea3c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad9813be90 0x7fad98137e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9ea3c640 10 monclient(hunting): picked mon.noname-c con 0x7fad9813be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9ea3c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9ea3c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fad980a6b40 con 0x7fad980a3c10 2024-09-16T04:39:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9ea3c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9ea3c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fad980a69c0 con 0x7fad980a3ff0 2024-09-16T04:39:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9ea3c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9ea3c640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fad9813e7c0 con 0x7fad9813be90 2024-09-16T04:39:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9ea3c640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9ea3c640 10 monclient(hunting): authenticate will time out at 788.046329s 2024-09-16T04:39:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad97fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad980a3c10 0x7fad98133cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad97fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad980a3c10 0x7fad98133cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38296/0 (socket says 172.21.15.46:38296) 2024-09-16T04:39:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9cfd9640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad9813be90 0x7fad98137e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad97fff640 1 -- 172.21.15.46:0/341793874 learned_addr learned my addr 172.21.15.46:0/341793874 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad97fff640 10 monclient(hunting): get_auth_request con 0x7fad980a3c10 auth_method 0 2024-09-16T04:39:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad97fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9cfd9640 10 monclient(hunting): get_auth_request con 0x7fad9813be90 auth_method 0 2024-09-16T04:39:49.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9cfd9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9cfd9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9cfd9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad977fe640 1 --2- 172.21.15.46:0/341793874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad980a3ff0 0x7fad98134230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad977fe640 10 monclient(hunting): get_auth_request con 0x7fad980a3ff0 auth_method 0 2024-09-16T04:39:49.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9cfd9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9cfd9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad9cfd9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad97fff640 10 monclient(hunting): handle_auth_done global_id 4713 payload 824 2024-09-16T04:39:49.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad97fff640 1 -- 172.21.15.46:0/341793874 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad9813be90 msgr2=0x7fad98137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad97fff640 1 --2- 172.21.15.46:0/341793874 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad9813be90 0x7fad98137e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad97fff640 1 -- 172.21.15.46:0/341793874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad980a3ff0 msgr2=0x7fad98134230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad97fff640 1 --2- 172.21.15.46:0/341793874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad980a3ff0 0x7fad98134230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad97fff640 10 monclient: _finish_hunting 0 2024-09-16T04:39:49.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad97fff640 1 monclient: found mon.noname-a 2024-09-16T04:39:49.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:49.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad97fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:49.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad97fff640 1 -- 172.21.15.46:0/341793874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad9813a3b0 con 0x7fad980a3c10 2024-09-16T04:39:49.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.630+0000 7fad97fff640 1 --2- 172.21.15.46:0/341793874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad980a3c10 0x7fad98133cf0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fad8800c400 tx=0x7fad8800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:49.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.631+0000 7fad94ff9640 1 -- 172.21.15.46:0/341793874 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad880052d0 con 0x7fad980a3c10 2024-09-16T04:39:49.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.631+0000 7fad94ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.631+0000 7fad94ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:49.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.631+0000 7fad94ff9640 10 monclient: dump: 2024-09-16T04:39:49.650 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.650 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.650 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.651 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.651 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.651 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.651 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.651 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.651 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.651 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.631+0000 7fad94ff9640 10 monclient: _finish_auth 0 2024-09-16T04:39:49.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.631+0000 7fad94ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:39:49.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.631+0000 7fad94ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:49.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.631+0000 7fad9ea3c640 5 monclient: authenticate success, global_id 4713 2024-09-16T04:39:49.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.631+0000 7fad94ff9640 1 -- 172.21.15.46:0/341793874 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad88005490 con 0x7fad980a3c10 2024-09-16T04:39:49.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.631+0000 7fad94ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:49.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.631+0000 7fad94ff9640 1 -- 172.21.15.46:0/341793874 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad88007260 con 0x7fad980a3c10 2024-09-16T04:39:49.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.631+0000 7fad94ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.631+0000 7fad94ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:49.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.631+0000 7fad94ff9640 10 monclient: dump: 2024-09-16T04:39:49.652 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.652 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.652 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.652 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.652 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.652 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.652 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.652 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.652 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.652 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.631+0000 7fad9ea3c640 10 monclient: _renew_subs 2024-09-16T04:39:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.631+0000 7fad967fc640 4 set_mon_vals no callback set 2024-09-16T04:39:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.631+0000 7fad9ea3c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.631+0000 7fad9ea3c640 1 -- 172.21.15.46:0/341793874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad98139370 con 0x7fad980a3c10 2024-09-16T04:39:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.631+0000 7fad9ea3c640 10 monclient: _renew_subs 2024-09-16T04:39:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.631+0000 7fad9ea3c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.631+0000 7fad9ea3c640 1 -- 172.21.15.46:0/341793874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad9809fab0 con 0x7fad980a3c10 2024-09-16T04:39:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad94ff9640 1 -- 172.21.15.46:0/341793874 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fad88007400 con 0x7fad980a3c10 2024-09-16T04:39:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad9ea3c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad9ea3c640 1 --2- 172.21.15.46:0/341793874 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad9809f090 0x7fad981383b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad9ea3c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad9ea3c640 1 -- 172.21.15.46:0/341793874 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fad9813a0d0 con 0x7fad9809f090 2024-09-16T04:39:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad977fe640 1 --2- 172.21.15.46:0/341793874 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad9809f090 0x7fad981383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad977fe640 10 monclient: get_auth_request con 0x7fad9809f090 auth_method 0 2024-09-16T04:39:49.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad94ff9640 1 --2- 172.21.15.46:0/341793874 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fad840383b0 0x7fad8403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad94ff9640 1 -- 172.21.15.46:0/341793874 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4738+0+0 (secure 0 0 0) 0x7fad8804acc0 con 0x7fad980a3c10 2024-09-16T04:39:49.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad977fe640 10 monclient(hunting): handle_auth_done global_id 14129 payload 824 2024-09-16T04:39:49.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad977fe640 1 -- 172.21.15.46:0/341793874 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fad9809cbe0 con 0x7fad9809f090 2024-09-16T04:39:49.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad9cfd9640 1 --2- 172.21.15.46:0/341793874 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fad840383b0 0x7fad8403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad9cfd9640 10 monclient: get_auth_request con 0x7fad840383b0 auth_method 0 2024-09-16T04:39:49.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad977fe640 1 --2- 172.21.15.46:0/341793874 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad9809f090 0x7fad981383b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fad98068240 tx=0x7fad8c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:49.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad94ff9640 1 -- 172.21.15.46:0/341793874 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c043070 con 0x7fad9809f090 2024-09-16T04:39:49.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad94ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad94ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:49.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.632+0000 7fad94ff9640 10 monclient: dump: 2024-09-16T04:39:49.655 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.655 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.655 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.655 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.655 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.655 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.655 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.655 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.655 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.655 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.633+0000 7fad9cfd9640 1 --2- 172.21.15.46:0/341793874 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fad840383b0 0x7fad8403a870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fad9806eeb0 tx=0x7fad9000b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:49.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.633+0000 7fad94ff9640 1 -- 172.21.15.46:0/341793874 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fad8c03c640 con 0x7fad9809f090 2024-09-16T04:39:49.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.633+0000 7fad94ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:49.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.633+0000 7fad94ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:49.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.633+0000 7fad94ff9640 1 -- 172.21.15.46:0/341793874 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad9809f090 msgr2=0x7fad981383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.633+0000 7fad94ff9640 1 --2- 172.21.15.46:0/341793874 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad9809f090 0x7fad981383b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fad98068240 tx=0x7fad8c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.633+0000 7fad94ff9640 1 -- 172.21.15.46:0/341793874 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad9809f090 msgr2=0x7fad981383b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:49.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.633+0000 7fad94ff9640 1 --2- 172.21.15.46:0/341793874 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad9809f090 0x7fad981383b0 secure :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fad98068240 tx=0x7fad8c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.639+0000 7fad9ea3c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:39:49.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.639+0000 7fad9ea3c640 1 --2- 172.21.15.46:0/341793874 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad981388f0 0x7fad980a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.639+0000 7fad9ea3c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.639+0000 7fad9ea3c640 1 -- 172.21.15.46:0/341793874 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fad98138cd0 con 0x7fad981388f0 2024-09-16T04:39:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.639+0000 7fad977fe640 1 --2- 172.21.15.46:0/341793874 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad981388f0 0x7fad980a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.639+0000 7fad977fe640 10 monclient: get_auth_request con 0x7fad981388f0 auth_method 0 2024-09-16T04:39:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.639+0000 7fad977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.639+0000 7fad977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.639+0000 7fad977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.639+0000 7fad977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.639+0000 7fad977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.639+0000 7fad977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.639+0000 7fad977fe640 10 monclient(hunting): handle_auth_done global_id 14132 payload 824 2024-09-16T04:39:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.639+0000 7fad977fe640 1 -- 172.21.15.46:0/341793874 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fad980a43d0 con 0x7fad981388f0 2024-09-16T04:39:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.639+0000 7fad977fe640 1 --2- 172.21.15.46:0/341793874 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad981388f0 0x7fad980a05e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fad8c000c00 tx=0x7fad8c009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad94ff9640 1 -- 172.21.15.46:0/341793874 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c037040 con 0x7fad981388f0 2024-09-16T04:39:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad94ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad94ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:39:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad94ff9640 10 monclient: dump: 2024-09-16T04:39:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.660 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad94ff9640 1 -- 172.21.15.46:0/341793874 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7fad8c03ce50 con 0x7fad981388f0 2024-09-16T04:39:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad94ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:39:49.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad94ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:49.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad94ff9640 1 -- 172.21.15.46:0/341793874 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad981388f0 msgr2=0x7fad980a05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad94ff9640 1 --2- 172.21.15.46:0/341793874 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad981388f0 0x7fad980a05e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fad8c000c00 tx=0x7fad8c009240 comp rx=0 tx=0).stop 2024-09-16T04:39:49.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad94ff9640 1 -- 172.21.15.46:0/341793874 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad981388f0 msgr2=0x7fad980a05e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:49.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad94ff9640 1 --2- 172.21.15.46:0/341793874 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fad981388f0 0x7fad980a05e0 secure :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fad8c000c00 tx=0x7fad8c009240 comp rx=0 tx=0).stop 2024-09-16T04:39:49.661 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:39:49.661 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:39:49.661 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:39:49.661 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:39:49.661 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 14, 2024-09-16T04:39:49.661 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 7395, 2024-09-16T04:39:49.662 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:39:49.662 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:39:49.662 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:39:49.662 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:39:49.662 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:49.662 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 7, 2024-09-16T04:39:49.662 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:49.662 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:39:49.662 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:39:49.662 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:49.662 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:49.662 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:49.662 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:49.662 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:49.663 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:49.663 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:49.663 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:49.663 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:49.663 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:49.663 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:49.663 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:49.663 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:39:49.663 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:39:49.663 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:49.663 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:49.663 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:49.663 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:49.664 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:49.664 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:49.664 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:49.664 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:49.664 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:49.664 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:49.664 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:49.664 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:49.664 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.664 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:39:49.664 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:39:49.664 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:39:49.664 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:39:49.664 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:39:49.665 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:39:49.665 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:49.665 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:49.665 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:39:49.665 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:39:49.665 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:39:49.665 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:39:49.665 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:39:49.665 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:39:49.665 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:39:49.665 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:49.665 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:39:49.665 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:49.665 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:49.666 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:49.666 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:49.666 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:49.666 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:49.666 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:49.666 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:49.666 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:49.666 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:49.666 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:49.666 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:49.666 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:39:49.666 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.666 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:39:49.666 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.667 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:39:49.667 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:39:49.667 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:49.667 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:49.667 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.667 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:49.667 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:39:49.667 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:49.667 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.667 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.667 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:49.667 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:39:49.667 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:49.667 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:49.668 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:49.668 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.668 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:39:49.668 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:39:49.668 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:49.668 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:49.668 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:49.668 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.668 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.668 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:39:49.668 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:39:49.668 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:49.668 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:49.668 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.669 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:49.669 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:39:49.669 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:49.669 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.669 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.669 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:49.669 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:39:49.669 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:49.669 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:49.669 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:49.669 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.669 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:39:49.669 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:39:49.670 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:49.670 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:49.670 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:49.670 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.670 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.670 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:39:49.670 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:39:49.670 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:49.670 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:49.670 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.670 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:49.670 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:39:49.670 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:49.670 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.671 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.671 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:49.671 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:39:49.671 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:49.671 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:49.671 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:49.671 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.671 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:39:49.671 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:39:49.671 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:49.671 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:49.671 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:49.671 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:49.671 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:49.672 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.672 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:39:49.672 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:39:49.672 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.672 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:49.672 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:49.672 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:39:49.672 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:49.672 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:49.672 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:39:49.672 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.673 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:49.673 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:49.673 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:39:49.673 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:49.673 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:49.673 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.673 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:39:49.673 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:39:49.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad9ea3c640 1 -- 172.21.15.46:0/341793874 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fad840383b0 msgr2=0x7fad8403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad9ea3c640 1 --2- 172.21.15.46:0/341793874 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fad840383b0 0x7fad8403a870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fad9806eeb0 tx=0x7fad9000b040 comp rx=0 tx=0).stop 2024-09-16T04:39:49.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad9ea3c640 10 monclient: shutdown 2024-09-16T04:39:49.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad9ea3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:49.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad9ea3c640 1 -- 172.21.15.46:0/341793874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad980a3c10 msgr2=0x7fad98133cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad9ea3c640 1 --2- 172.21.15.46:0/341793874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad980a3c10 0x7fad98133cf0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fad8800c400 tx=0x7fad8800c8c0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad9cfd9640 1 -- 172.21.15.46:0/341793874 reap_dead start 2024-09-16T04:39:49.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad9cfd9640 1 -- 172.21.15.46:0/341793874 reap_dead start 2024-09-16T04:39:49.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad9ea3c640 1 -- 172.21.15.46:0/341793874 shutdown_connections 2024-09-16T04:39:49.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad9ea3c640 1 -- 172.21.15.46:0/341793874 >> 172.21.15.46:0/341793874 conn(0x7fad980a6340 msgr2=0x7fad9813db00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:49.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad9ea3c640 1 -- 172.21.15.46:0/341793874 shutdown_connections 2024-09-16T04:39:49.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.640+0000 7fad9ea3c640 1 -- 172.21.15.46:0/341793874 wait complete. 2024-09-16T04:39:49.676 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:39:49.676 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:39:49.676 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:39:49.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.776+0000 7ffab2ad0640 10 monclient: get_monmap_and_config 2024-09-16T04:39:49.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.776+0000 7ffab2ad0640 10 monclient: build_initial_monmap 2024-09-16T04:39:49.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.776+0000 7ffab2ad0640 10 monclient: monmap: 2024-09-16T04:39:49.778 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:49.778 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.778 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:49.777660+0000 2024-09-16T04:39:49.778 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:49.777660+0000 2024-09-16T04:39:49.779 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:49.779 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.779 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:49.779 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:49.779 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:49.779 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:49.779 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:49.779 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.776+0000 7ffab2ad0640 1 Processor -- start 2024-09-16T04:39:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.776+0000 7ffab2ad0640 1 -- start start 2024-09-16T04:39:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.776+0000 7ffab2ad0640 10 monclient: init 2024-09-16T04:39:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab2ad0640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab2ad0640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-16T04:39:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab2ad0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffaac0a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab2ad0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffaac0a3ff0 0x7ffaac0a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab2ad0640 10 monclient(hunting): picked mon.noname-a con 0x7ffaac0a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab2ad0640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ffaac13be90 0x7ffaac13e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab2ad0640 10 monclient(hunting): picked mon.noname-c con 0x7ffaac13be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab2ad0640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffaac09b940 con 0x7ffaac0a3c10 2024-09-16T04:39:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab2ad0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab2ad0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ffaac09ab00 con 0x7ffaac0a3ff0 2024-09-16T04:39:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab2ad0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab2ad0640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ffaac09b030 con 0x7ffaac13be90 2024-09-16T04:39:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab2ad0640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab2ad0640 10 monclient(hunting): authenticate will time out at 788.193550s 2024-09-16T04:39:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffaa3fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffaac0a3ff0 0x7ffaac0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffaa3fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffaac0a3ff0 0x7ffaac0a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38308/0 (socket says 172.21.15.46:38308) 2024-09-16T04:39:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffaa3fff640 1 -- 172.21.15.46:0/2043854097 learned_addr learned my addr 172.21.15.46:0/2043854097 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffaa3fff640 10 monclient(hunting): get_auth_request con 0x7ffaac0a3ff0 auth_method 0 2024-09-16T04:39:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffaa3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffaa3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffaa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab106d640 1 --2- 172.21.15.46:0/2043854097 >> v2:172.21.15.114:3300/0 conn(0x7ffaac13be90 0x7ffaac13e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab106d640 10 monclient(hunting): get_auth_request con 0x7ffaac13be90 auth_method 0 2024-09-16T04:39:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab106d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab106d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab106d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffaa3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffaa3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffaa3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab106d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab106d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffab106d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffaa3fff640 10 monclient(hunting): handle_auth_done global_id 4716 payload 274 2024-09-16T04:39:49.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffaa3fff640 1 -- 172.21.15.46:0/2043854097 >> v2:172.21.15.114:3300/0 conn(0x7ffaac13be90 msgr2=0x7ffaac13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:49.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffaa3fff640 1 --2- 172.21.15.46:0/2043854097 >> v2:172.21.15.114:3300/0 conn(0x7ffaac13be90 0x7ffaac13e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffaa3fff640 1 -- 172.21.15.46:0/2043854097 >> v1:172.21.15.114:6789/0 conn(0x7ffaac0a3c10 legacy=0x7ffaac09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffaa3fff640 10 monclient: _finish_hunting 0 2024-09-16T04:39:49.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffaa3fff640 1 monclient: found mon.noname-a 2024-09-16T04:39:49.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffaa3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:49.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffaa3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:49.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.777+0000 7ffaa3fff640 1 -- 172.21.15.46:0/2043854097 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaac09b310 con 0x7ffaac0a3ff0 2024-09-16T04:39:49.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.778+0000 7ffaa3fff640 1 --2- 172.21.15.46:0/2043854097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffaac0a3ff0 0x7ffaac0a43d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ffaac06ee70 tx=0x7ffa9c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc8e3ca00a26d8f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:49.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.778+0000 7ffaa37fe640 1 -- 172.21.15.46:0/2043854097 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c037070 con 0x7ffaac0a3ff0 2024-09-16T04:39:49.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.778+0000 7ffaa37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.778+0000 7ffaa37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:49.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.778+0000 7ffaa37fe640 10 monclient: dump: 2024-09-16T04:39:49.793 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.793 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.793 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.793 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.793 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.793 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.793 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.793 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.793 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.794 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.778+0000 7ffaa37fe640 10 monclient: _finish_auth 0 2024-09-16T04:39:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.778+0000 7ffaa37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:39:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.778+0000 7ffaa37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.778+0000 7ffaa37fe640 1 -- 172.21.15.46:0/2043854097 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa9c032440 con 0x7ffaac0a3ff0 2024-09-16T04:39:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.778+0000 7ffab2ad0640 5 monclient: authenticate success, global_id 4716 2024-09-16T04:39:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.778+0000 7ffab2ad0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.778+0000 7ffaa37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.778+0000 7ffab2ad0640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.778+0000 7ffaa37fe640 1 -- 172.21.15.46:0/2043854097 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c032810 con 0x7ffaac0a3ff0 2024-09-16T04:39:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.778+0000 7ffab2ad0640 4 set_mon_vals no callback set 2024-09-16T04:39:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.778+0000 7ffaa37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.778+0000 7ffaa37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.778+0000 7ffaa37fe640 10 monclient: dump: 2024-09-16T04:39:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.795 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.779+0000 7ffab2ad0640 10 monclient: shutdown 2024-09-16T04:39:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.779+0000 7ffab2ad0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.779+0000 7ffab2ad0640 1 -- 172.21.15.46:0/2043854097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffaac0a3ff0 msgr2=0x7ffaac0a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.779+0000 7ffab2ad0640 1 --2- 172.21.15.46:0/2043854097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffaac0a3ff0 0x7ffaac0a43d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ffaac06ee70 tx=0x7ffa9c02a1e0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.779+0000 7ffab2ad0640 1 -- 172.21.15.46:0/2043854097 shutdown_connections 2024-09-16T04:39:49.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.779+0000 7ffab2ad0640 1 --2- 172.21.15.46:0/2043854097 >> v2:172.21.15.114:3300/0 conn(0x7ffaac13be90 0x7ffaac13e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.779+0000 7ffab2ad0640 1 --2- 172.21.15.46:0/2043854097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffaac0a3ff0 0x7ffaac0a43d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.779+0000 7ffab2ad0640 1 -- 172.21.15.46:0/2043854097 >> 172.21.15.46:0/2043854097 conn(0x7ffaac0a6340 msgr2=0x7ffaac06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:49.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.779+0000 7ffab2ad0640 1 -- 172.21.15.46:0/2043854097 shutdown_connections 2024-09-16T04:39:49.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.779+0000 7ffab2ad0640 1 -- 172.21.15.46:0/2043854097 wait complete. 2024-09-16T04:39:49.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.779+0000 7ffab2ad0640 10 monclient: build_initial_monmap 2024-09-16T04:39:49.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.779+0000 7ffab2ad0640 10 monclient: monmap: 2024-09-16T04:39:49.796 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:49.796 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:49.796 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:49.796 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:49.796 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:49.796 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.797 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:49.797 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:49.797 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:49.797 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.780+0000 7ffab2ad0640 1 Processor -- start 2024-09-16T04:39:49.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.780+0000 7ffab2ad0640 1 -- start start 2024-09-16T04:39:49.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.780+0000 7ffab2ad0640 10 monclient: init 2024-09-16T04:39:49.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.780+0000 7ffab2ad0640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:49.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.780+0000 7ffab2ad0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:39:49.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.780+0000 7ffab2ad0640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ffaac0a3c10 0x7ffaac133d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.780+0000 7ffab2ad0640 10 monclient(hunting): picked mon.noname-c con 0x7ffaac0a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:49.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.780+0000 7ffab2ad0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffaac0a3ff0 0x7ffaac1342c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.780+0000 7ffab2ad0640 10 monclient(hunting): picked mon.noname-a con 0x7ffaac0a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:49.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.780+0000 7ffab2ad0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaac13be90 0x7ffaac1cb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab2ad0640 10 monclient(hunting): picked mon.noname-b con 0x7ffaac13be90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab2ad0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab2ad0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac0a6b40 con 0x7ffaac0a3ff0 2024-09-16T04:39:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab2ad0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab2ad0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac0a69c0 con 0x7ffaac13be90 2024-09-16T04:39:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab2ad0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab2ad0640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7ffaac13e7c0 con 0x7ffaac0a3c10 2024-09-16T04:39:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab2ad0640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab2ad0640 10 monclient(hunting): authenticate will time out at 788.197036s 2024-09-16T04:39:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab086c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ffaac0a3c10 0x7ffaac133d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffaac0a3ff0 0x7ffaac1342c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffaac0a3ff0 0x7ffaac1342c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38322/0 (socket says 172.21.15.46:38322) 2024-09-16T04:39:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab086c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ffaac0a3c10 0x7ffaac133d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:52378/0 (socket says 172.21.15.46:52378) 2024-09-16T04:39:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa3fff640 1 -- 172.21.15.46:0/4235080533 learned_addr learned my addr 172.21.15.46:0/4235080533 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa3fff640 10 monclient(hunting): get_auth_request con 0x7ffaac0a3ff0 auth_method 0 2024-09-16T04:39:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab086c640 10 monclient(hunting): get_auth_request con 0x7ffaac0a3c10 auth_method 0 2024-09-16T04:39:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab086c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab086c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab086c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab106d640 1 --2- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaac13be90 0x7ffaac1cb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab086c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab086c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab086c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab106d640 10 monclient(hunting): get_auth_request con 0x7ffaac13be90 auth_method 0 2024-09-16T04:39:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab106d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab106d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab106d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab106d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab106d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab106d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa3fff640 10 monclient(hunting): handle_auth_done global_id 4719 payload 824 2024-09-16T04:39:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa3fff640 1 -- 172.21.15.46:0/4235080533 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ffaac0a3c10 msgr2=0x7ffaac133d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa3fff640 1 --2- 172.21.15.46:0/4235080533 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ffaac0a3c10 0x7ffaac133d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa3fff640 1 -- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaac13be90 msgr2=0x7ffaac1cb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa3fff640 1 --2- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaac13be90 0x7ffaac1cb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa3fff640 10 monclient: _finish_hunting 0 2024-09-16T04:39:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa3fff640 1 monclient: found mon.noname-a 2024-09-16T04:39:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa3fff640 1 -- 172.21.15.46:0/4235080533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaac13a6d0 con 0x7ffaac0a3ff0 2024-09-16T04:39:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab086c640 1 --2- 172.21.15.46:0/4235080533 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ffaac0a3c10 0x7ffaac133d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:39:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa3fff640 1 --2- 172.21.15.46:0/4235080533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffaac0a3ff0 0x7ffaac1342c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c02a220 tx=0x7ffa9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa17fa640 1 -- 172.21.15.46:0/4235080533 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c037040 con 0x7ffaac0a3ff0 2024-09-16T04:39:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa17fa640 10 monclient: dump: 2024-09-16T04:39:49.802 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.802 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.802 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.802 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.802 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.802 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.802 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.802 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.802 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.803 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa17fa640 10 monclient: _finish_auth 0 2024-09-16T04:39:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa17fa640 10 monclient: _check_auth_tickets 2024-09-16T04:39:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab2ad0640 5 monclient: authenticate success, global_id 4719 2024-09-16T04:39:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa17fa640 1 -- 172.21.15.46:0/4235080533 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa9c042070 con 0x7ffaac0a3ff0 2024-09-16T04:39:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa17fa640 1 -- 172.21.15.46:0/4235080533 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c030880 con 0x7ffaac0a3ff0 2024-09-16T04:39:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab2ad0640 10 monclient: _renew_subs 2024-09-16T04:39:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab2ad0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab2ad0640 1 -- 172.21.15.46:0/4235080533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaac13c270 con 0x7ffaac0a3ff0 2024-09-16T04:39:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa17fa640 10 monclient: dump: 2024-09-16T04:39:49.804 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.804 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.804 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.804 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.804 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.804 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.804 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.804 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.804 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.804 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffaa2ffd640 4 set_mon_vals no callback set 2024-09-16T04:39:49.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab2ad0640 10 monclient: _renew_subs 2024-09-16T04:39:49.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab2ad0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:49.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.781+0000 7ffab2ad0640 1 -- 172.21.15.46:0/4235080533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaac13a110 con 0x7ffaac0a3ff0 2024-09-16T04:39:49.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.782+0000 7ffab2ad0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:49.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.782+0000 7ffaa17fa640 1 -- 172.21.15.46:0/4235080533 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7ffa9c030a20 con 0x7ffaac0a3ff0 2024-09-16T04:39:49.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.782+0000 7ffab2ad0640 1 --2- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaac1cbe10 0x7ffaac1cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.782+0000 7ffab2ad0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.782+0000 7ffab2ad0640 1 -- 172.21.15.46:0/4235080533 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac13c730 con 0x7ffaac1cbe10 2024-09-16T04:39:49.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.782+0000 7ffaa17fa640 1 --2- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ffa980383b0 0x7ffa9803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.783+0000 7ffaa17fa640 1 -- 172.21.15.46:0/4235080533 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4738+0+0 (secure 0 0 0) 0x7ffa9c06bfb0 con 0x7ffaac0a3ff0 2024-09-16T04:39:49.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.783+0000 7ffab086c640 1 --2- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaac1cbe10 0x7ffaac1cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.783+0000 7ffab086c640 10 monclient: get_auth_request con 0x7ffaac1cbe10 auth_method 0 2024-09-16T04:39:49.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.783+0000 7ffab086c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.783+0000 7ffab086c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.783+0000 7ffab086c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.783+0000 7ffab106d640 1 --2- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ffa980383b0 0x7ffa9803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.783+0000 7ffab106d640 10 monclient: get_auth_request con 0x7ffa980383b0 auth_method 0 2024-09-16T04:39:49.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.783+0000 7ffab086c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.783+0000 7ffab086c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.783+0000 7ffab086c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.783+0000 7ffab086c640 10 monclient(hunting): handle_auth_done global_id 4693 payload 824 2024-09-16T04:39:49.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.783+0000 7ffab086c640 1 -- 172.21.15.46:0/4235080533 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffaac134800 con 0x7ffaac1cbe10 2024-09-16T04:39:49.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.783+0000 7ffab086c640 1 --2- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaac1cbe10 0x7ffaac1cc1f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800fae0 tx=0x7ffaa8001d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:49.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.783+0000 7ffab106d640 1 --2- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ffa980383b0 0x7ffa9803a870 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffaac09c9d0 tx=0x7ffaa400b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:49.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.783+0000 7ffaa17fa640 1 -- 172.21.15.46:0/4235080533 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa80052e0 con 0x7ffaac1cbe10 2024-09-16T04:39:49.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.783+0000 7ffaa17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.783+0000 7ffaa17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:49.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.783+0000 7ffaa17fa640 10 monclient: dump: 2024-09-16T04:39:49.808 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.808 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.808 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.808 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.808 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.808 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.808 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.808 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.808 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.808 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.784+0000 7ffaa17fa640 1 -- 172.21.15.46:0/4235080533 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ffaa80052e0 con 0x7ffaac1cbe10 2024-09-16T04:39:49.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.784+0000 7ffaa17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:49.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.784+0000 7ffaa17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:49.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.784+0000 7ffaa17fa640 1 -- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaac1cbe10 msgr2=0x7ffaac1cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.784+0000 7ffaa17fa640 1 --2- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaac1cbe10 0x7ffaac1cc1f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800fae0 tx=0x7ffaa8001d70 comp rx=0 tx=0).stop 2024-09-16T04:39:49.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.784+0000 7ffaa17fa640 1 -- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaac1cbe10 msgr2=0x7ffaac1cc1f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:49.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.784+0000 7ffaa17fa640 1 --2- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaac1cbe10 0x7ffaac1cc1f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.790+0000 7ffab2ad0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:39:49.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.790+0000 7ffab2ad0640 1 --2- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaac0a05e0 0x7ffaac0a09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.790+0000 7ffab2ad0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.790+0000 7ffab2ad0640 1 -- 172.21.15.46:0/4235080533 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac1cc730 con 0x7ffaac0a05e0 2024-09-16T04:39:49.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.790+0000 7ffab086c640 1 --2- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaac0a05e0 0x7ffaac0a09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.790+0000 7ffab086c640 10 monclient: get_auth_request con 0x7ffaac0a05e0 auth_method 0 2024-09-16T04:39:49.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.790+0000 7ffab086c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.790+0000 7ffab086c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.790+0000 7ffab086c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.790+0000 7ffab086c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.790+0000 7ffab086c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.790+0000 7ffab086c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.790+0000 7ffab086c640 10 monclient(hunting): handle_auth_done global_id 4696 payload 824 2024-09-16T04:39:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.790+0000 7ffab086c640 1 -- 172.21.15.46:0/4235080533 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ffaac0a43d0 con 0x7ffaac0a05e0 2024-09-16T04:39:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.791+0000 7ffab086c640 1 --2- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaac0a05e0 0x7ffaac0a09f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800fae0 tx=0x7ffaa8001120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.791+0000 7ffaa17fa640 1 -- 172.21.15.46:0/4235080533 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8007680 con 0x7ffaac0a05e0 2024-09-16T04:39:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.791+0000 7ffaa17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.791+0000 7ffaa17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:39:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.791+0000 7ffaa17fa640 10 monclient: dump: 2024-09-16T04:39:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.813 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.813 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.813 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.813 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.813 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.813 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.791+0000 7ffaa17fa640 1 -- 172.21.15.46:0/4235080533 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7ffaa8007680 con 0x7ffaac0a05e0 2024-09-16T04:39:49.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.791+0000 7ffaa17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:39:49.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.791+0000 7ffaa17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:49.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.791+0000 7ffaa17fa640 1 -- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaac0a05e0 msgr2=0x7ffaac0a09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.791+0000 7ffaa17fa640 1 --2- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaac0a05e0 0x7ffaac0a09f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800fae0 tx=0x7ffaa8001120 comp rx=0 tx=0).stop 2024-09-16T04:39:49.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.791+0000 7ffaa17fa640 1 -- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaac0a05e0 msgr2=0x7ffaac0a09f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:39:49.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.791+0000 7ffaa17fa640 1 --2- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaac0a05e0 0x7ffaac0a09f0 secure :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800fae0 tx=0x7ffaa8001120 comp rx=0 tx=0).stop 2024-09-16T04:39:49.814 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:39:49.814 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:39:49.814 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:39:49.814 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:39:49.814 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 14, 2024-09-16T04:39:49.814 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 75694, 2024-09-16T04:39:49.814 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:39:49.814 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:39:49.814 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:39:49.814 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:39:49.814 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:49.815 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 7, 2024-09-16T04:39:49.815 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:49.815 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:39:49.815 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:39:49.815 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:49.815 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:49.815 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:49.815 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:49.815 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:49.815 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:49.815 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:49.815 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:49.815 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:49.815 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:49.816 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:49.816 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:49.816 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:39:49.816 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:39:49.816 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:49.816 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:49.816 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:49.816 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:49.816 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:49.816 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:49.816 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:49.816 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:49.816 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:49.816 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:49.817 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:49.817 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:49.817 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.817 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:39:49.817 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:39:49.817 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:39:49.817 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:39:49.817 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:39:49.817 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:39:49.817 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:49.817 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:39:49.817 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:39:49.817 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:39:49.817 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:39:49.818 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:39:49.818 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:39:49.818 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:39:49.818 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:39:49.818 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:39:49.818 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:39:49.818 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:39:49.818 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:39:49.818 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:39:49.818 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:39:49.818 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:39:49.818 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:39:49.818 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:39:49.818 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:39:49.819 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:39:49.819 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:39:49.819 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:39:49.819 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:49.819 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:39:49.819 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.819 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:39:49.819 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.819 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:39:49.819 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:39:49.819 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:49.819 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:49.819 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.819 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:49.820 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:39:49.820 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:49.820 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.820 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.820 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:49.820 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:39:49.820 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:49.820 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:49.820 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:49.820 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.820 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:39:49.820 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:39:49.820 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:49.821 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:49.821 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:49.821 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.821 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.821 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:39:49.821 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:39:49.821 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:49.821 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:49.821 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.821 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:49.821 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:39:49.821 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:49.821 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.821 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.822 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:49.822 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:39:49.822 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:49.822 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:49.822 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:49.822 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.822 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:39:49.822 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:39:49.822 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:49.822 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:49.822 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:49.822 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.822 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.822 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:39:49.823 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:39:49.823 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:39:49.823 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:39:49.823 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.823 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:39:49.823 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:39:49.823 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:49.823 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.823 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.823 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:39:49.823 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:39:49.823 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:39:49.823 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:49.823 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:49.824 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.824 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:39:49.824 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:39:49.824 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:39:49.824 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:39:49.824 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:39:49.824 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:49.824 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:49.824 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.824 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:39:49.825 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:39:49.825 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.825 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:49.825 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:49.825 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:39:49.825 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:49.825 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:49.825 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:39:49.825 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.825 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:49.825 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:49.825 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:39:49.825 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:49.826 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:49.826 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:39:49.826 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.826 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:49.826 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:49.826 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6 2024-09-16T04:39:49.826 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:49.826 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:39:49.826 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:39:49.827 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:39:49.827 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:39:49.827 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:39:49.827 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:39:49.827 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:39:49.827 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:39:49.827 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:39:49.828 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:39:49.828 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:39:49.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.791+0000 7ffab2ad0640 1 -- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ffa980383b0 msgr2=0x7ffa9803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.791+0000 7ffab2ad0640 1 --2- 172.21.15.46:0/4235080533 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ffa980383b0 0x7ffa9803a870 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffaac09c9d0 tx=0x7ffaa400b040 comp rx=0 tx=0).stop 2024-09-16T04:39:49.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.791+0000 7ffab2ad0640 10 monclient: shutdown 2024-09-16T04:39:49.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.791+0000 7ffab2ad0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:49.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.791+0000 7ffab2ad0640 1 -- 172.21.15.46:0/4235080533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffaac0a3ff0 msgr2=0x7ffaac1342c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.791+0000 7ffab2ad0640 1 --2- 172.21.15.46:0/4235080533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffaac0a3ff0 0x7ffaac1342c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c02a220 tx=0x7ffa9c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.791+0000 7ffab106d640 1 -- 172.21.15.46:0/4235080533 reap_dead start 2024-09-16T04:39:49.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.791+0000 7ffab106d640 1 -- 172.21.15.46:0/4235080533 reap_dead start 2024-09-16T04:39:49.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.791+0000 7ffab2ad0640 1 -- 172.21.15.46:0/4235080533 shutdown_connections 2024-09-16T04:39:49.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.792+0000 7ffab2ad0640 1 -- 172.21.15.46:0/4235080533 >> 172.21.15.46:0/4235080533 conn(0x7ffaac0a6340 msgr2=0x7ffaac13da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:49.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.792+0000 7ffab2ad0640 1 -- 172.21.15.46:0/4235080533 shutdown_connections 2024-09-16T04:39:49.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.792+0000 7ffab2ad0640 1 -- 172.21.15.46:0/4235080533 wait complete. 2024-09-16T04:39:49.832 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:39:49.832 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T04:39:49.832 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:39:49.832 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T04:39:49.833 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:39:49.833 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:39:49.833 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:39:49.842 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:39:49.843 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:39:49.843 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:39:49.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.981+0000 7fa2669fd640 10 monclient: get_monmap_and_config 2024-09-16T04:39:49.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.981+0000 7fa2669fd640 10 monclient: build_initial_monmap 2024-09-16T04:39:49.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.981+0000 7fa2669fd640 10 monclient: monmap: 2024-09-16T04:39:49.984 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:49.984 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.984 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:49.983166+0000 2024-09-16T04:39:49.984 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:49.983166+0000 2024-09-16T04:39:49.984 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:49.984 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.984 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:49.984 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:49.984 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:49.984 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:49.984 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:49.985 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.982+0000 7fa2669fd640 1 Processor -- start 2024-09-16T04:39:49.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.982+0000 7fa2669fd640 1 -- start start 2024-09-16T04:39:49.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.982+0000 7fa2669fd640 10 monclient: init 2024-09-16T04:39:49.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.982+0000 7fa2669fd640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:49.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.982+0000 7fa2669fd640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-16T04:39:49.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.982+0000 7fa2669fd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa2600a3c10 0x7fa26009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.982+0000 7fa2669fd640 10 monclient(hunting): picked mon.noname-a con 0x7fa2600a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:49.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.982+0000 7fa2669fd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa2600a3ff0 0x7fa2600a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.982+0000 7fa2669fd640 10 monclient(hunting): picked mon.noname-b con 0x7fa2600a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:49.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa2669fd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa2601369d0 0x7fa260136e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:49.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa2669fd640 10 monclient(hunting): picked mon.noname-c con 0x7fa2601369d0 addr v2:172.21.15.114:3300/0 2024-09-16T04:39:49.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa2669fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa2669fd640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa26009b940 con 0x7fa2600a3c10 2024-09-16T04:39:49.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa2669fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa2669fd640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fa26009ab00 con 0x7fa2601369d0 2024-09-16T04:39:49.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa2669fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:49.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa2669fd640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fa26009b030 con 0x7fa2600a3ff0 2024-09-16T04:39:49.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa2669fd640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:49.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25f7fe640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa2600a3ff0 msgr2=0x7fa2600a43d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:39:49.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa2669fd640 10 monclient(hunting): authenticate will time out at 788.399066s 2024-09-16T04:39:49.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25f7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa2600a3ff0 0x7fa2600a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:49.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa2600a3c10 0x7fa26009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa2600a3c10 0x7fa26009bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38324/0 (socket says 172.21.15.46:38324) 2024-09-16T04:39:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25ffff640 1 -- 172.21.15.46:0/1755068004 learned_addr learned my addr 172.21.15.46:0/1755068004 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25ffff640 10 monclient(hunting): get_auth_request con 0x7fa2600a3c10 auth_method 0 2024-09-16T04:39:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25ffff640 10 monclient(hunting): handle_auth_done global_id 4722 payload 274 2024-09-16T04:39:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25ffff640 1 -- 172.21.15.46:0/1755068004 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa2600a3ff0 msgr2=0x7fa2600a43d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:39:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa264f9a640 1 --2- 172.21.15.46:0/1755068004 >> v2:172.21.15.114:3300/0 conn(0x7fa2601369d0 0x7fa260136e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25ffff640 1 --2- 172.21.15.46:0/1755068004 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa2600a3ff0 0x7fa2600a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25ffff640 1 -- 172.21.15.46:0/1755068004 >> v2:172.21.15.114:3300/0 conn(0x7fa2601369d0 msgr2=0x7fa260136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:39:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25ffff640 1 --2- 172.21.15.46:0/1755068004 >> v2:172.21.15.114:3300/0 conn(0x7fa2601369d0 0x7fa260136e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:39:49.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25ffff640 1 monclient: found mon.noname-a 2024-09-16T04:39:49.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:49.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:49.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa25ffff640 1 -- 172.21.15.46:0/1755068004 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa26009b310 con 0x7fa2600a3c10 2024-09-16T04:39:49.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa264f9a640 10 monclient: get_auth_request con 0x7fa2601369d0 auth_method 0 2024-09-16T04:39:49.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.983+0000 7fa264f9a640 1 --2- 172.21.15.46:0/1755068004 >> v2:172.21.15.114:3300/0 conn(0x7fa2601369d0 0x7fa260136e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:39:49.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa25ffff640 1 --2- 172.21.15.46:0/1755068004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2600a3c10 0x7fa26009bcf0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa260068160 tx=0x7fa25002a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c40e7bae15a18e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:49.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa25effd640 1 -- 172.21.15.46:0/1755068004 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa250037070 con 0x7fa2600a3c10 2024-09-16T04:39:49.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa25effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa25effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:49.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa25effd640 10 monclient: dump: 2024-09-16T04:39:49.996 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.996 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.997 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.997 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.997 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.997 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.997 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.997 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.997 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.997 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa25effd640 10 monclient: _finish_auth 0 2024-09-16T04:39:49.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa25effd640 10 monclient: _check_auth_tickets 2024-09-16T04:39:49.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa25effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:49.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa25effd640 1 -- 172.21.15.46:0/1755068004 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa25002ae20 con 0x7fa2600a3c10 2024-09-16T04:39:49.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa2669fd640 5 monclient: authenticate success, global_id 4722 2024-09-16T04:39:49.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa2669fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:49.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa25effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:49.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa2669fd640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:49.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa25effd640 1 -- 172.21.15.46:0/1755068004 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa250032660 con 0x7fa2600a3c10 2024-09-16T04:39:49.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa25effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:49.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa2669fd640 4 set_mon_vals no callback set 2024-09-16T04:39:49.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa25effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:49.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa25effd640 10 monclient: dump: 2024-09-16T04:39:49.998 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:49.998 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:49.998 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.998 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:49.998 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:49.998 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:49.998 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:49.999 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:49.999 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:49.999 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:49.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa2669fd640 10 monclient: shutdown 2024-09-16T04:39:49.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa2669fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:49.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa2669fd640 1 -- 172.21.15.46:0/1755068004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2600a3c10 msgr2=0x7fa26009bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:49.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa2669fd640 1 --2- 172.21.15.46:0/1755068004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2600a3c10 0x7fa26009bcf0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa260068160 tx=0x7fa25002a0d0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa2669fd640 1 -- 172.21.15.46:0/1755068004 shutdown_connections 2024-09-16T04:39:49.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa2669fd640 1 --2- 172.21.15.46:0/1755068004 >> v2:172.21.15.114:3300/0 conn(0x7fa2601369d0 0x7fa260136e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa2669fd640 1 --2- 172.21.15.46:0/1755068004 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa2600a3ff0 0x7fa2600a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa2669fd640 1 --2- 172.21.15.46:0/1755068004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2600a3c10 0x7fa26009bcf0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:49.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa2669fd640 1 -- 172.21.15.46:0/1755068004 >> 172.21.15.46:0/1755068004 conn(0x7fa2600a6340 msgr2=0x7fa26006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:49.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.984+0000 7fa2669fd640 1 -- 172.21.15.46:0/1755068004 shutdown_connections 2024-09-16T04:39:49.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.985+0000 7fa2669fd640 1 -- 172.21.15.46:0/1755068004 wait complete. 2024-09-16T04:39:50.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.985+0000 7fa2669fd640 10 monclient: build_initial_monmap 2024-09-16T04:39:50.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.985+0000 7fa2669fd640 10 monclient: monmap: 2024-09-16T04:39:50.000 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:50.000 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:50.000 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:50.000 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:50.000 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:50.000 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:50.000 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:50.000 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:50.000 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:50.000 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:50.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.985+0000 7fa2669fd640 1 Processor -- start 2024-09-16T04:39:50.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa2669fd640 1 -- start start 2024-09-16T04:39:50.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa2669fd640 10 monclient: init 2024-09-16T04:39:50.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa2669fd640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:50.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa2669fd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:39:50.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa2669fd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2600a3c10 0x7fa2601cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:50.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa2669fd640 10 monclient(hunting): picked mon.noname-a con 0x7fa2600a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:50.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa2669fd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2600a3ff0 0x7fa2601cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:50.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa2669fd640 10 monclient(hunting): picked mon.noname-b con 0x7fa2600a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:50.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa2669fd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa2601369d0 0x7fa2601231f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:50.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa2669fd640 10 monclient(hunting): picked mon.noname-c con 0x7fa2601369d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:50.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa2669fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:50.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa2669fd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa2600a4910 con 0x7fa2600a3c10 2024-09-16T04:39:50.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa2669fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:50.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa2669fd640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa2600a69c0 con 0x7fa2600a3ff0 2024-09-16T04:39:50.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa2669fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:50.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa2669fd640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fa2600a8a10 con 0x7fa2601369d0 2024-09-16T04:39:50.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa2669fd640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:50.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa2669fd640 10 monclient(hunting): authenticate will time out at 788.402602s 2024-09-16T04:39:50.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa264f9a640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa2601369d0 msgr2=0x7fa2601231f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:39:50.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa264f9a640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa2601369d0 0x7fa2601231f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:50.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa25ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2600a3c10 0x7fa2601cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:50.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa25ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2600a3c10 0x7fa2601cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38330/0 (socket says 172.21.15.46:38330) 2024-09-16T04:39:50.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa25ffff640 1 -- 172.21.15.46:0/3790236778 learned_addr learned my addr 172.21.15.46:0/3790236778 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:50.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa25ffff640 10 monclient(hunting): get_auth_request con 0x7fa2600a3c10 auth_method 0 2024-09-16T04:39:50.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa25ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:50.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa25ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:50.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa25ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:50.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa25ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:50.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa25ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:50.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa25ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:50.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa25ffff640 10 monclient(hunting): handle_auth_done global_id 4725 payload 824 2024-09-16T04:39:50.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa25f7fe640 1 --2- 172.21.15.46:0/3790236778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2600a3ff0 0x7fa2601cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:50.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa25ffff640 1 -- 172.21.15.46:0/3790236778 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa2601369d0 msgr2=0x7fa2601231f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:39:50.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa25ffff640 1 --2- 172.21.15.46:0/3790236778 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa2601369d0 0x7fa2601231f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:50.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa25ffff640 1 -- 172.21.15.46:0/3790236778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2600a3ff0 msgr2=0x7fa2601cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:50.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa25ffff640 1 --2- 172.21.15.46:0/3790236778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2600a3ff0 0x7fa2601cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:50.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa25ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:39:50.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa25ffff640 1 monclient: found mon.noname-a 2024-09-16T04:39:50.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa25ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:50.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa25ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:50.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.986+0000 7fa25ffff640 1 -- 172.21.15.46:0/3790236778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa260126b40 con 0x7fa2600a3c10 2024-09-16T04:39:50.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa25ffff640 1 --2- 172.21.15.46:0/3790236778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2600a3c10 0x7fa2601cbee0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa250000c00 tx=0x7fa2500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa25cff9640 1 -- 172.21.15.46:0/3790236778 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa250041070 con 0x7fa2600a3c10 2024-09-16T04:39:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa25cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa25cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa25cff9640 10 monclient: dump: 2024-09-16T04:39:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:50.004 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:50.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa25cff9640 10 monclient: _finish_auth 0 2024-09-16T04:39:50.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa25cff9640 10 monclient: _check_auth_tickets 2024-09-16T04:39:50.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa25cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:50.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa25cff9640 1 -- 172.21.15.46:0/3790236778 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa25003c680 con 0x7fa2600a3c10 2024-09-16T04:39:50.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa25cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:50.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa25cff9640 1 -- 172.21.15.46:0/3790236778 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa250037040 con 0x7fa2600a3c10 2024-09-16T04:39:50.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa2669fd640 5 monclient: authenticate success, global_id 4725 2024-09-16T04:39:50.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa2669fd640 10 monclient: _renew_subs 2024-09-16T04:39:50.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa2669fd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:50.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa2669fd640 1 -- 172.21.15.46:0/3790236778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2601252c0 con 0x7fa2600a3c10 2024-09-16T04:39:50.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa25cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:50.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa25cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:50.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa25cff9640 10 monclient: dump: 2024-09-16T04:39:50.005 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:50.006 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:50.006 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:50.006 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:50.006 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:50.006 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:50.006 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:50.006 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:50.006 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:50.006 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:50.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa25e7fc640 4 set_mon_vals no callback set 2024-09-16T04:39:50.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa2669fd640 10 monclient: _renew_subs 2024-09-16T04:39:50.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa2669fd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:50.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.987+0000 7fa2669fd640 1 -- 172.21.15.46:0/3790236778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa260127100 con 0x7fa2600a3c10 2024-09-16T04:39:50.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.988+0000 7fa25cff9640 1 -- 172.21.15.46:0/3790236778 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fa250044070 con 0x7fa2600a3c10 2024-09-16T04:39:50.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.988+0000 7fa2669fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:50.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.988+0000 7fa2669fd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:50.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.988+0000 7fa2669fd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:50.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.988+0000 7fa2669fd640 1 -- 172.21.15.46:0/3790236778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2600a43d0 con 0x7fa2600a3c10 2024-09-16T04:39:50.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.988+0000 7fa25cff9640 1 --2- 172.21.15.46:0/3790236778 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa24c038360 0x7fa24c03a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:50.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.988+0000 7fa25cff9640 1 -- 172.21.15.46:0/3790236778 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4738+0+0 (secure 0 0 0) 0x7fa25006c070 con 0x7fa2600a3c10 2024-09-16T04:39:50.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.988+0000 7fa25f7fe640 1 --2- 172.21.15.46:0/3790236778 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa24c038360 0x7fa24c03a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:50.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.989+0000 7fa25f7fe640 10 monclient: get_auth_request con 0x7fa24c038360 auth_method 0 2024-09-16T04:39:50.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.989+0000 7fa25f7fe640 1 --2- 172.21.15.46:0/3790236778 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa24c038360 0x7fa24c03a820 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa2600a38d0 tx=0x7fa254008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:50.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.993+0000 7fa25cff9640 1 -- 172.21.15.46:0/3790236778 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa25003d770 con 0x7fa2600a3c10 2024-09-16T04:39:50.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.993+0000 7fa25cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:50.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:49.993+0000 7fa25cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:50.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:50.236+0000 7fa2669fd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:39:50.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:50.236+0000 7fa2669fd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:39:50.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:50.236+0000 7fa2669fd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:50.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:50.236+0000 7fa2669fd640 1 -- 172.21.15.46:0/3790236778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa2601268c0 con 0x7fa2600a3c10 2024-09-16T04:39:50.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:50.237+0000 7fa25cff9640 1 -- 172.21.15.46:0/3790236778 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fa25002ae20 con 0x7fa2600a3c10 2024-09-16T04:39:50.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:50.237+0000 7fa25cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:39:50.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:50.237+0000 7fa25cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:50.240 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:39:50.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:50.239+0000 7fa2669fd640 1 -- 172.21.15.46:0/3790236778 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa24c038360 msgr2=0x7fa24c03a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:50.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:50.239+0000 7fa2669fd640 1 --2- 172.21.15.46:0/3790236778 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa24c038360 0x7fa24c03a820 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa2600a38d0 tx=0x7fa254008040 comp rx=0 tx=0).stop 2024-09-16T04:39:50.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:50.239+0000 7fa2669fd640 10 monclient: shutdown 2024-09-16T04:39:50.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:50.239+0000 7fa2669fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:50.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:50.239+0000 7fa2669fd640 1 -- 172.21.15.46:0/3790236778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2600a3c10 msgr2=0x7fa2601cbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:50.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:50.239+0000 7fa2669fd640 1 --2- 172.21.15.46:0/3790236778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2600a3c10 0x7fa2601cbee0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa250000c00 tx=0x7fa2500046c0 comp rx=0 tx=0).stop 2024-09-16T04:39:50.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:50.239+0000 7fa2669fd640 1 -- 172.21.15.46:0/3790236778 shutdown_connections 2024-09-16T04:39:50.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:50.239+0000 7fa2669fd640 1 --2- 172.21.15.46:0/3790236778 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa2601369d0 0x7fa2601231f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:50.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:50.239+0000 7fa2669fd640 1 --2- 172.21.15.46:0/3790236778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2600a3ff0 0x7fa2601cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:50.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:50.239+0000 7fa2669fd640 1 --2- 172.21.15.46:0/3790236778 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa24c038360 0x7fa24c03a820 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:50.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:50.239+0000 7fa2669fd640 1 --2- 172.21.15.46:0/3790236778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2600a3c10 0x7fa2601cbee0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:50.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:50.239+0000 7fa2669fd640 1 -- 172.21.15.46:0/3790236778 >> 172.21.15.46:0/3790236778 conn(0x7fa2600a6340 msgr2=0x7fa2600a30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:50.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:50.239+0000 7fa2669fd640 1 -- 172.21.15.46:0/3790236778 shutdown_connections 2024-09-16T04:39:50.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:50.239+0000 7fa2669fd640 1 -- 172.21.15.46:0/3790236778 wait complete. 2024-09-16T04:39:53.253 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:39:53.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.364+0000 7f57ceaaf640 10 monclient: get_monmap_and_config 2024-09-16T04:39:53.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.364+0000 7f57ceaaf640 10 monclient: build_initial_monmap 2024-09-16T04:39:53.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.364+0000 7f57ceaaf640 10 monclient: monmap: 2024-09-16T04:39:53.367 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:53.367 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:53.367 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:53.366146+0000 2024-09-16T04:39:53.367 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:53.366146+0000 2024-09-16T04:39:53.368 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:53.368 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:53.368 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:53.368 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:53.368 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:53.368 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:53.368 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:53.369 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.365+0000 7f57ceaaf640 1 Processor -- start 2024-09-16T04:39:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.365+0000 7f57ceaaf640 1 -- start start 2024-09-16T04:39:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.365+0000 7f57ceaaf640 10 monclient: init 2024-09-16T04:39:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57ceaaf640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57ceaaf640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-16T04:39:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57ceaaf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f57c80a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:39:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57ceaaf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f57c80a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57ceaaf640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f57c813cea0 0x7f57c813f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57ceaaf640 10 monclient(hunting): picked mon.noname-a con 0x7f57c813cea0 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57ceaaf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57c809b940 con 0x7f57c80a3c10 2024-09-16T04:39:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57ceaaf640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57c809ab00 con 0x7f57c80a3ff0 2024-09-16T04:39:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57ceaaf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57ceaaf640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f57c809b030 con 0x7f57c813cea0 2024-09-16T04:39:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57ceaaf640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57ceaaf640 10 monclient(hunting): authenticate will time out at 791.782252s 2024-09-16T04:39:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57cc84b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f57c80a3c10 0x7f57c809bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43368/0 (socket says 172.21.15.46:43368) 2024-09-16T04:39:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57cc84b640 1 -- 172.21.15.46:0/1961508140 learned_addr learned my addr 172.21.15.46:0/1961508140 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:39:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57cd04c640 1 --2- 172.21.15.46:0/1961508140 >> v2:172.21.15.46:3300/0 conn(0x7f57c813cea0 0x7f57c813f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57cd04c640 10 monclient(hunting): get_auth_request con 0x7f57c813cea0 auth_method 0 2024-09-16T04:39:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57cd04c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57cd04c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57cd04c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57cd04c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57cd04c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57cd04c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57cd04c640 10 monclient(hunting): handle_auth_done global_id 4728 payload 274 2024-09-16T04:39:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57cd04c640 1 -- 172.21.15.46:0/1961508140 >> v1:172.21.15.114:6789/0 conn(0x7f57c80a3ff0 legacy=0x7f57c80a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57cd04c640 1 -- 172.21.15.46:0/1961508140 >> v1:172.21.15.46:6789/0 conn(0x7f57c80a3c10 legacy=0x7f57c809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57cd04c640 10 monclient: _finish_hunting 0 2024-09-16T04:39:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57cd04c640 1 monclient: found mon.noname-a 2024-09-16T04:39:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57cd04c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57cd04c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.366+0000 7f57cd04c640 1 -- 172.21.15.46:0/1961508140 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c809b310 con 0x7f57c813cea0 2024-09-16T04:39:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.367+0000 7f57cd04c640 1 --2- 172.21.15.46:0/1961508140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57c813cea0 0x7f57c813f290 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f57c0009d90 tx=0x7f57c002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b34ecd1827bd560d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.367+0000 7f57c77fe640 1 -- 172.21.15.46:0/1961508140 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c0007740 con 0x7f57c813cea0 2024-09-16T04:39:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.367+0000 7f57c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.367+0000 7f57c77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.367+0000 7f57c77fe640 10 monclient: dump: 2024-09-16T04:39:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:53.385 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.367+0000 7f57c77fe640 10 monclient: _finish_auth 0 2024-09-16T04:39:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.367+0000 7f57c77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:39:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.367+0000 7f57c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.367+0000 7f57c77fe640 1 -- 172.21.15.46:0/1961508140 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57c00078e0 con 0x7f57c813cea0 2024-09-16T04:39:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.367+0000 7f57ceaaf640 5 monclient: authenticate success, global_id 4728 2024-09-16T04:39:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.367+0000 7f57ceaaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:39:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.367+0000 7f57c77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.367+0000 7f57c77fe640 1 -- 172.21.15.46:0/1961508140 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c0007cb0 con 0x7f57c813cea0 2024-09-16T04:39:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.367+0000 7f57ceaaf640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.367+0000 7f57ceaaf640 4 set_mon_vals no callback set 2024-09-16T04:39:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.367+0000 7f57c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.367+0000 7f57c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.367+0000 7f57c77fe640 10 monclient: dump: 2024-09-16T04:39:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:53.386 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.368+0000 7f57ceaaf640 10 monclient: shutdown 2024-09-16T04:39:53.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.368+0000 7f57ceaaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:53.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.368+0000 7f57ceaaf640 1 -- 172.21.15.46:0/1961508140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57c813cea0 msgr2=0x7f57c813f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:53.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.368+0000 7f57ceaaf640 1 --2- 172.21.15.46:0/1961508140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57c813cea0 0x7f57c813f290 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f57c0009d90 tx=0x7f57c002a660 comp rx=0 tx=0).stop 2024-09-16T04:39:53.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.368+0000 7f57ceaaf640 1 -- 172.21.15.46:0/1961508140 shutdown_connections 2024-09-16T04:39:53.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.368+0000 7f57ceaaf640 1 --2- 172.21.15.46:0/1961508140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57c813cea0 0x7f57c813f290 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:53.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.368+0000 7f57ceaaf640 1 -- 172.21.15.46:0/1961508140 >> 172.21.15.46:0/1961508140 conn(0x7f57c80a6340 msgr2=0x7f57c806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:53.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.368+0000 7f57ceaaf640 1 -- 172.21.15.46:0/1961508140 shutdown_connections 2024-09-16T04:39:53.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.368+0000 7f57ceaaf640 1 -- 172.21.15.46:0/1961508140 wait complete. 2024-09-16T04:39:53.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.368+0000 7f57ceaaf640 10 monclient: build_initial_monmap 2024-09-16T04:39:53.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.368+0000 7f57ceaaf640 10 monclient: monmap: 2024-09-16T04:39:53.387 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:53.387 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:53.387 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:53.388 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57ceaaf640 1 Processor -- start 2024-09-16T04:39:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57ceaaf640 1 -- start start 2024-09-16T04:39:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57ceaaf640 10 monclient: init 2024-09-16T04:39:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57ceaaf640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57ceaaf640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:39:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57ceaaf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57c80a3c10 0x7f57c81cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57ceaaf640 10 monclient(hunting): picked mon.noname-b con 0x7f57c80a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57ceaaf640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f57c80a3ff0 0x7f57c8136cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57ceaaf640 10 monclient(hunting): picked mon.noname-c con 0x7f57c80a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57ceaaf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57c813cea0 0x7f57c8139e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57ceaaf640 10 monclient(hunting): picked mon.noname-a con 0x7f57c813cea0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57ceaaf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57ceaaf640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f57c813f950 con 0x7f57c813cea0 2024-09-16T04:39:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57ceaaf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57ceaaf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f57c813f7d0 con 0x7f57c80a3c10 2024-09-16T04:39:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57ceaaf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57ceaaf640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f57c813fad0 con 0x7f57c80a3ff0 2024-09-16T04:39:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57ceaaf640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57ceaaf640 10 monclient(hunting): authenticate will time out at 791.785736s 2024-09-16T04:39:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57c7fff640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f57c80a3ff0 msgr2=0x7f57c8136cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:39:53.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57c7fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f57c80a3ff0 0x7f57c8136cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:53.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57cd04c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57c813cea0 0x7f57c8139e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:53.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57cd04c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57c813cea0 0x7f57c8139e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38350/0 (socket says 172.21.15.46:38350) 2024-09-16T04:39:53.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57cd04c640 1 -- 172.21.15.46:0/1720610716 learned_addr learned my addr 172.21.15.46:0/1720610716 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:53.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57cd04c640 10 monclient(hunting): get_auth_request con 0x7f57c813cea0 auth_method 0 2024-09-16T04:39:53.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57cd04c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:53.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57cd04c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:53.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57cd04c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:53.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57cc84b640 1 --2- 172.21.15.46:0/1720610716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57c80a3c10 0x7f57c81cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:53.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57cd04c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:53.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57cd04c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:53.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57cd04c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:53.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.369+0000 7f57cc84b640 10 monclient(hunting): get_auth_request con 0x7f57c80a3c10 auth_method 0 2024-09-16T04:39:53.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57cc84b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:53.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57cc84b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:53.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57cc84b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:53.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57cc84b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:53.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57cc84b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:53.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57cc84b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:53.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57cd04c640 10 monclient(hunting): handle_auth_done global_id 4734 payload 824 2024-09-16T04:39:53.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57cd04c640 1 -- 172.21.15.46:0/1720610716 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f57c80a3ff0 msgr2=0x7f57c8136cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:39:53.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57cd04c640 1 --2- 172.21.15.46:0/1720610716 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f57c80a3ff0 0x7f57c8136cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:53.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57cd04c640 1 -- 172.21.15.46:0/1720610716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57c80a3c10 msgr2=0x7f57c81cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:53.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57cd04c640 1 --2- 172.21.15.46:0/1720610716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57c80a3c10 0x7f57c81cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:53.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57cd04c640 10 monclient: _finish_hunting 0 2024-09-16T04:39:53.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57cd04c640 1 monclient: found mon.noname-a 2024-09-16T04:39:53.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57cd04c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:53.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57cd04c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:53.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57cd04c640 1 -- 172.21.15.46:0/1720610716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c81cc960 con 0x7f57c813cea0 2024-09-16T04:39:53.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57cd04c640 1 --2- 172.21.15.46:0/1720610716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57c813cea0 0x7f57c8139e40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f57c0009eb0 tx=0x7f57c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:53.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57c57fa640 1 -- 172.21.15.46:0/1720610716 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c0040070 con 0x7f57c813cea0 2024-09-16T04:39:53.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57c57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:53.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57c57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:53.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57c57fa640 10 monclient: dump: 2024-09-16T04:39:53.393 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:53.393 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:53.393 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:53.393 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:53.393 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:53.393 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:53.393 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:53.393 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:53.393 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:53.393 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:53.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57c57fa640 10 monclient: _finish_auth 0 2024-09-16T04:39:53.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57c57fa640 10 monclient: _check_auth_tickets 2024-09-16T04:39:53.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57c57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:53.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57c57fa640 1 -- 172.21.15.46:0/1720610716 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57c0031c40 con 0x7f57c813cea0 2024-09-16T04:39:53.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57c57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:53.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57c57fa640 1 -- 172.21.15.46:0/1720610716 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c0009450 con 0x7f57c813cea0 2024-09-16T04:39:53.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57ceaaf640 5 monclient: authenticate success, global_id 4734 2024-09-16T04:39:53.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57c57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:53.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57c57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:53.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57c57fa640 10 monclient: dump: 2024-09-16T04:39:53.394 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:53.394 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:53.394 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:53.394 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:53.394 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:53.394 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:53.395 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:53.395 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:53.395 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:53.395 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:53.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57ceaaf640 10 monclient: _renew_subs 2024-09-16T04:39:53.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57c6ffd640 4 set_mon_vals no callback set 2024-09-16T04:39:53.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57ceaaf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:53.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57ceaaf640 1 -- 172.21.15.46:0/1720610716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57c813be70 con 0x7f57c813cea0 2024-09-16T04:39:53.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57ceaaf640 10 monclient: _renew_subs 2024-09-16T04:39:53.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57ceaaf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:53.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.370+0000 7f57ceaaf640 1 -- 172.21.15.46:0/1720610716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57c8120670 con 0x7f57c813cea0 2024-09-16T04:39:53.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.371+0000 7f57c57fa640 1 -- 172.21.15.46:0/1720610716 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f57c0039040 con 0x7f57c813cea0 2024-09-16T04:39:53.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.371+0000 7f57ceaaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:53.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.371+0000 7f57ceaaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:53.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.371+0000 7f57ceaaf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:53.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.371+0000 7f57ceaaf640 1 -- 172.21.15.46:0/1720610716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57c813f330 con 0x7f57c813cea0 2024-09-16T04:39:53.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.371+0000 7f57c57fa640 1 --2- 172.21.15.46:0/1720610716 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f57ac038360 0x7f57ac03a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:53.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.371+0000 7f57c57fa640 1 -- 172.21.15.46:0/1720610716 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4738+0+0 (secure 0 0 0) 0x7f57c006c920 con 0x7f57c813cea0 2024-09-16T04:39:53.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.372+0000 7f57cc84b640 1 --2- 172.21.15.46:0/1720610716 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f57ac038360 0x7f57ac03a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:53.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.372+0000 7f57cc84b640 10 monclient: get_auth_request con 0x7f57ac038360 auth_method 0 2024-09-16T04:39:53.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.372+0000 7f57cc84b640 1 --2- 172.21.15.46:0/1720610716 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f57ac038360 0x7f57ac03a820 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f57bc005880 tx=0x7f57bc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:53.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.376+0000 7f57c57fa640 1 -- 172.21.15.46:0/1720610716 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f57c003acb0 con 0x7f57c813cea0 2024-09-16T04:39:53.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.376+0000 7f57c57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:53.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.376+0000 7f57c57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:53.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.620+0000 7f57ceaaf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:39:53.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.620+0000 7f57ceaaf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:39:53.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.621+0000 7f57ceaaf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:53.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.621+0000 7f57ceaaf640 1 -- 172.21.15.46:0/1720610716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f57c813c2a0 con 0x7f57c813cea0 2024-09-16T04:39:53.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.621+0000 7f57c57fa640 1 -- 172.21.15.46:0/1720610716 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f57c00309f0 con 0x7f57c813cea0 2024-09-16T04:39:53.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.621+0000 7f57c57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:39:53.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.621+0000 7f57c57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:53.624 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:39:53.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.623+0000 7f57ceaaf640 1 -- 172.21.15.46:0/1720610716 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f57ac038360 msgr2=0x7f57ac03a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:53.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.623+0000 7f57ceaaf640 1 --2- 172.21.15.46:0/1720610716 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f57ac038360 0x7f57ac03a820 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f57bc005880 tx=0x7f57bc009040 comp rx=0 tx=0).stop 2024-09-16T04:39:53.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.623+0000 7f57ceaaf640 10 monclient: shutdown 2024-09-16T04:39:53.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.623+0000 7f57ceaaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:53.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.623+0000 7f57ceaaf640 1 -- 172.21.15.46:0/1720610716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57c813cea0 msgr2=0x7f57c8139e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:53.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.623+0000 7f57ceaaf640 1 --2- 172.21.15.46:0/1720610716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57c813cea0 0x7f57c8139e40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f57c0009eb0 tx=0x7f57c0004a40 comp rx=0 tx=0).stop 2024-09-16T04:39:53.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.623+0000 7f57ceaaf640 1 -- 172.21.15.46:0/1720610716 shutdown_connections 2024-09-16T04:39:53.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.623+0000 7f57ceaaf640 1 --2- 172.21.15.46:0/1720610716 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f57ac038360 0x7f57ac03a820 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:53.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.623+0000 7f57ceaaf640 1 --2- 172.21.15.46:0/1720610716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57c813cea0 0x7f57c8139e40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:53.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.623+0000 7f57ceaaf640 1 --2- 172.21.15.46:0/1720610716 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f57c80a3ff0 0x7f57c8136cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:53.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.623+0000 7f57ceaaf640 1 --2- 172.21.15.46:0/1720610716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57c80a3c10 0x7f57c81cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:53.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.623+0000 7f57ceaaf640 1 -- 172.21.15.46:0/1720610716 >> 172.21.15.46:0/1720610716 conn(0x7f57c80a6340 msgr2=0x7f57c813e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:53.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.623+0000 7f57ceaaf640 1 -- 172.21.15.46:0/1720610716 shutdown_connections 2024-09-16T04:39:53.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:53.623+0000 7f57ceaaf640 1 -- 172.21.15.46:0/1720610716 wait complete. 2024-09-16T04:39:56.637 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:39:56.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.749+0000 7fe910118640 10 monclient: get_monmap_and_config 2024-09-16T04:39:56.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.749+0000 7fe910118640 10 monclient: build_initial_monmap 2024-09-16T04:39:56.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.749+0000 7fe910118640 10 monclient: monmap: 2024-09-16T04:39:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:39:56.750812+0000 2024-09-16T04:39:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:39:56.750812+0000 2024-09-16T04:39:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:39:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:39:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:39:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:39:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:39:56.752 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.749+0000 7fe910118640 1 Processor -- start 2024-09-16T04:39:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe910118640 1 -- start start 2024-09-16T04:39:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe910118640 10 monclient: init 2024-09-16T04:39:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe910118640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe910118640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-16T04:39:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe910118640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe9080a3c10 0x7fe90809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe910118640 10 monclient(hunting): picked mon.noname-a con 0x7fe9080a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:39:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe910118640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9080a3ff0 0x7fe9080a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe910118640 10 monclient(hunting): picked mon.noname-b con 0x7fe9080a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe910118640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9081369d0 addr v1:172.21.15.114:6789/0 2024-09-16T04:39:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe910118640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe90809ab70 con 0x7fe9081369d0 2024-09-16T04:39:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe910118640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe910118640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe9080a4d60 con 0x7fe9080a3c10 2024-09-16T04:39:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe910118640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe910118640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe90809b030 con 0x7fe9080a3ff0 2024-09-16T04:39:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe910118640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe910118640 10 monclient(hunting): authenticate will time out at 795.166792s 2024-09-16T04:39:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe90d6b3640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9080a3ff0 msgr2=0x7fe9080a43d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:39:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe90d6b3640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9080a3ff0 0x7fe9080a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe90deb4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe9080a3c10 0x7fe90809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe90deb4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe9080a3c10 0x7fe90809bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54528/0 (socket says 172.21.15.46:54528) 2024-09-16T04:39:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe90deb4640 1 -- 172.21.15.46:0/2779273221 learned_addr learned my addr 172.21.15.46:0/2779273221 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe90deb4640 10 monclient(hunting): get_auth_request con 0x7fe9080a3c10 auth_method 0 2024-09-16T04:39:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe90deb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe90deb4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.750+0000 7fe90deb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.751+0000 7fe90deb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.751+0000 7fe90deb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.751+0000 7fe90deb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.751+0000 7fe90deb4640 10 monclient(hunting): handle_auth_done global_id 4737 payload 274 2024-09-16T04:39:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.751+0000 7fe90deb4640 1 -- 172.21.15.46:0/2779273221 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9080a3ff0 msgr2=0x7fe9080a43d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:39:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.751+0000 7fe90deb4640 1 --2- 172.21.15.46:0/2779273221 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9080a3ff0 0x7fe9080a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.751+0000 7fe90deb4640 1 -- 172.21.15.46:0/2779273221 >> v1:172.21.15.114:6789/0 conn(0x7fe9081369d0 legacy=0x7fe908136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.751+0000 7fe90deb4640 10 monclient: _finish_hunting 0 2024-09-16T04:39:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.751+0000 7fe90deb4640 1 monclient: found mon.noname-a 2024-09-16T04:39:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.751+0000 7fe90deb4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.751+0000 7fe90deb4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.751+0000 7fe90deb4640 1 -- 172.21.15.46:0/2779273221 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe90809b310 con 0x7fe9080a3c10 2024-09-16T04:39:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.751+0000 7fe90deb4640 1 --2- 172.21.15.46:0/2779273221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9080a3c10 0x7fe90809bcf0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe904009e50 tx=0x7fe90402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b38d8307523ef6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe90ceb2640 1 -- 172.21.15.46:0/2779273221 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe904036070 con 0x7fe9080a3c10 2024-09-16T04:39:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe90ceb2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe90ceb2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe90ceb2640 10 monclient: dump: 2024-09-16T04:39:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:56.766 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe90ceb2640 10 monclient: _finish_auth 0 2024-09-16T04:39:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe90ceb2640 10 monclient: _check_auth_tickets 2024-09-16T04:39:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe90ceb2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe90ceb2640 1 -- 172.21.15.46:0/2779273221 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe904031600 con 0x7fe9080a3c10 2024-09-16T04:39:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe90ceb2640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe90ceb2640 1 -- 172.21.15.46:0/2779273221 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9040319d0 con 0x7fe9080a3c10 2024-09-16T04:39:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe910118640 5 monclient: authenticate success, global_id 4737 2024-09-16T04:39:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe910118640 10 monclient: get_monmap_and_config success 2024-09-16T04:39:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe90ceb2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe910118640 4 set_mon_vals no callback set 2024-09-16T04:39:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe90ceb2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe90ceb2640 10 monclient: dump: 2024-09-16T04:39:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:56.768 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:56.768 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:56.768 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:56.768 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:56.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe910118640 10 monclient: shutdown 2024-09-16T04:39:56.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe910118640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:56.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe910118640 1 -- 172.21.15.46:0/2779273221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9080a3c10 msgr2=0x7fe90809bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:56.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe910118640 1 --2- 172.21.15.46:0/2779273221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9080a3c10 0x7fe90809bcf0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe904009e50 tx=0x7fe90402a6a0 comp rx=0 tx=0).stop 2024-09-16T04:39:56.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe910118640 1 -- 172.21.15.46:0/2779273221 shutdown_connections 2024-09-16T04:39:56.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe910118640 1 --2- 172.21.15.46:0/2779273221 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9080a3ff0 0x7fe9080a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:56.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe910118640 1 --2- 172.21.15.46:0/2779273221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9080a3c10 0x7fe90809bcf0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:56.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe910118640 1 -- 172.21.15.46:0/2779273221 >> 172.21.15.46:0/2779273221 conn(0x7fe9080a5d10 msgr2=0x7fe9080a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:56.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe910118640 1 -- 172.21.15.46:0/2779273221 shutdown_connections 2024-09-16T04:39:56.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.752+0000 7fe910118640 1 -- 172.21.15.46:0/2779273221 wait complete. 2024-09-16T04:39:56.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.753+0000 7fe910118640 10 monclient: build_initial_monmap 2024-09-16T04:39:56.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.753+0000 7fe910118640 10 monclient: monmap: 2024-09-16T04:39:56.769 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:39:56.769 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:39:56.769 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:39:56.769 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:39:56.769 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:39:56.769 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:56.769 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:39:56.769 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:39:56.769 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:39:56.769 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:56.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.753+0000 7fe910118640 1 Processor -- start 2024-09-16T04:39:56.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.753+0000 7fe910118640 1 -- start start 2024-09-16T04:39:56.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.753+0000 7fe910118640 10 monclient: init 2024-09-16T04:39:56.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe910118640 10 monclient: _reopen_session rank -1 2024-09-16T04:39:56.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe910118640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:39:56.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe910118640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9080a3c10 0x7fe9081cc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:56.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe910118640 10 monclient(hunting): picked mon.noname-a con 0x7fe9080a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:39:56.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe910118640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9080a3ff0 0x7fe908123340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:56.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe910118640 10 monclient(hunting): picked mon.noname-b con 0x7fe9080a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:39:56.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe910118640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9081369d0 0x7fe908123880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:56.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe910118640 10 monclient(hunting): picked mon.noname-c con 0x7fe9081369d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:39:56.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe910118640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:56.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe910118640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe9080a4910 con 0x7fe9080a3c10 2024-09-16T04:39:56.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe910118640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:56.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe910118640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe90806e3d0 con 0x7fe9080a3ff0 2024-09-16T04:39:56.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe910118640 10 monclient(hunting): start opening mon connection 2024-09-16T04:39:56.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe910118640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe908127cb0 con 0x7fe9081369d0 2024-09-16T04:39:56.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe910118640 10 monclient(hunting): _renew_subs 2024-09-16T04:39:56.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe910118640 10 monclient(hunting): authenticate will time out at 795.170464s 2024-09-16T04:39:56.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90e6b5640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9081369d0 msgr2=0x7fe908123880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:39:56.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90e6b5640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9081369d0 0x7fe908123880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:39:56.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90deb4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9080a3c10 0x7fe9081cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:56.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90deb4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9080a3c10 0x7fe9081cc230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54544/0 (socket says 172.21.15.46:54544) 2024-09-16T04:39:56.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90deb4640 1 -- 172.21.15.46:0/1142563604 learned_addr learned my addr 172.21.15.46:0/1142563604 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:39:56.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90deb4640 10 monclient(hunting): get_auth_request con 0x7fe9080a3c10 auth_method 0 2024-09-16T04:39:56.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90deb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:56.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90deb4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:56.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90deb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:56.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90d6b3640 1 --2- 172.21.15.46:0/1142563604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9080a3ff0 0x7fe908123340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:56.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90deb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:56.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90deb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:56.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90deb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:56.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90d6b3640 10 monclient(hunting): get_auth_request con 0x7fe9080a3ff0 auth_method 0 2024-09-16T04:39:56.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90d6b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:39:56.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90d6b3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:39:56.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90d6b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:39:56.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90d6b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:39:56.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90d6b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:39:56.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90d6b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:39:56.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90deb4640 10 monclient(hunting): handle_auth_done global_id 4740 payload 824 2024-09-16T04:39:56.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90deb4640 1 -- 172.21.15.46:0/1142563604 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9081369d0 msgr2=0x7fe908123880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:39:56.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90deb4640 1 --2- 172.21.15.46:0/1142563604 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9081369d0 0x7fe908123880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:56.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90deb4640 1 -- 172.21.15.46:0/1142563604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9080a3ff0 msgr2=0x7fe908123340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:56.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90deb4640 1 --2- 172.21.15.46:0/1142563604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9080a3ff0 0x7fe908123340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:56.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90deb4640 10 monclient: _finish_hunting 0 2024-09-16T04:39:56.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90deb4640 1 monclient: found mon.noname-a 2024-09-16T04:39:56.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90deb4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:39:56.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90deb4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:39:56.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.754+0000 7fe90deb4640 1 -- 172.21.15.46:0/1142563604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe908125b40 con 0x7fe9080a3c10 2024-09-16T04:39:56.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe90deb4640 1 --2- 172.21.15.46:0/1142563604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9080a3c10 0x7fe9081cc230 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe904009e50 tx=0x7fe904004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:56.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe8fa7fc640 1 -- 172.21.15.46:0/1142563604 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe904041020 con 0x7fe9080a3c10 2024-09-16T04:39:56.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe8fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:56.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe8fa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:39:56.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe8fa7fc640 10 monclient: dump: 2024-09-16T04:39:56.773 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:56.773 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:56.774 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:56.774 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:56.774 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:56.774 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:56.774 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:56.774 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:56.774 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:56.774 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:56.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe8fa7fc640 10 monclient: _finish_auth 0 2024-09-16T04:39:56.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe8fa7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:39:56.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe8fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:39:56.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe8fa7fc640 1 -- 172.21.15.46:0/1142563604 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe904007800 con 0x7fe9080a3c10 2024-09-16T04:39:56.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe8fa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:39:56.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe8fa7fc640 1 -- 172.21.15.46:0/1142563604 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe904036040 con 0x7fe9080a3c10 2024-09-16T04:39:56.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe8fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:39:56.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe8fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:39:56.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe8fa7fc640 10 monclient: dump: 2024-09-16T04:39:56.775 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:39:56.775 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:39:56.775 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:56.775 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:39:56.775 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:39:56.775 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:39:56.775 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:39:56.775 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:39:56.775 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:39:56.775 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:39:56.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe910118640 5 monclient: authenticate success, global_id 4740 2024-09-16T04:39:56.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe910118640 10 monclient: _renew_subs 2024-09-16T04:39:56.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe910118640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:56.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe910118640 1 -- 172.21.15.46:0/1142563604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9081272d0 con 0x7fe9080a3c10 2024-09-16T04:39:56.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe8fbfff640 4 set_mon_vals no callback set 2024-09-16T04:39:56.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe910118640 10 monclient: _renew_subs 2024-09-16T04:39:56.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe910118640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:56.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.755+0000 7fe910118640 1 -- 172.21.15.46:0/1142563604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9080a2e50 con 0x7fe9080a3c10 2024-09-16T04:39:56.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.756+0000 7fe8fa7fc640 1 -- 172.21.15.46:0/1142563604 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fe904044020 con 0x7fe9080a3c10 2024-09-16T04:39:56.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.756+0000 7fe910118640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:56.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.756+0000 7fe910118640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:56.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.756+0000 7fe910118640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:56.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.756+0000 7fe910118640 1 -- 172.21.15.46:0/1142563604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe908137580 con 0x7fe9080a3c10 2024-09-16T04:39:56.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.756+0000 7fe8fa7fc640 1 --2- 172.21.15.46:0/1142563604 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe8ec038360 0x7fe8ec03a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:39:56.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.756+0000 7fe8fa7fc640 1 -- 172.21.15.46:0/1142563604 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4738+0+0 (secure 0 0 0) 0x7fe90406c390 con 0x7fe9080a3c10 2024-09-16T04:39:56.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.756+0000 7fe90d6b3640 1 --2- 172.21.15.46:0/1142563604 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe8ec038360 0x7fe8ec03a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:39:56.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.756+0000 7fe90d6b3640 10 monclient: get_auth_request con 0x7fe8ec038360 auth_method 0 2024-09-16T04:39:56.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.757+0000 7fe90d6b3640 1 --2- 172.21.15.46:0/1142563604 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe8ec038360 0x7fe8ec03a820 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe908136410 tx=0x7fe8fc003b90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:39:56.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.761+0000 7fe8fa7fc640 1 -- 172.21.15.46:0/1142563604 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe90404b070 con 0x7fe9080a3c10 2024-09-16T04:39:56.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.761+0000 7fe8fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:39:56.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:56.761+0000 7fe8fa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:39:57.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:57.006+0000 7fe910118640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:39:57.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:57.006+0000 7fe910118640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:39:57.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:57.006+0000 7fe910118640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:39:57.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:57.006+0000 7fe910118640 1 -- 172.21.15.46:0/1142563604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe908067e40 con 0x7fe9080a3c10 2024-09-16T04:39:57.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:57.007+0000 7fe8fa7fc640 1 -- 172.21.15.46:0/1142563604 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fe90406c0d0 con 0x7fe9080a3c10 2024-09-16T04:39:57.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:57.007+0000 7fe8fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:39:57.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:57.007+0000 7fe8fa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:39:57.010 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:39:57.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:57.009+0000 7fe910118640 1 -- 172.21.15.46:0/1142563604 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe8ec038360 msgr2=0x7fe8ec03a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:57.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:57.009+0000 7fe910118640 1 --2- 172.21.15.46:0/1142563604 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe8ec038360 0x7fe8ec03a820 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe908136410 tx=0x7fe8fc003b90 comp rx=0 tx=0).stop 2024-09-16T04:39:57.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:57.009+0000 7fe910118640 10 monclient: shutdown 2024-09-16T04:39:57.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:57.009+0000 7fe910118640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:39:57.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:57.009+0000 7fe910118640 1 -- 172.21.15.46:0/1142563604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9080a3c10 msgr2=0x7fe9081cc230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:39:57.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:57.009+0000 7fe910118640 1 --2- 172.21.15.46:0/1142563604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9080a3c10 0x7fe9081cc230 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe904009e50 tx=0x7fe904004a40 comp rx=0 tx=0).stop 2024-09-16T04:39:57.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:57.009+0000 7fe910118640 1 -- 172.21.15.46:0/1142563604 shutdown_connections 2024-09-16T04:39:57.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:57.009+0000 7fe910118640 1 --2- 172.21.15.46:0/1142563604 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9081369d0 0x7fe908123880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:57.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:57.009+0000 7fe910118640 1 --2- 172.21.15.46:0/1142563604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9080a3ff0 0x7fe908123340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:57.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:57.009+0000 7fe910118640 1 --2- 172.21.15.46:0/1142563604 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe8ec038360 0x7fe8ec03a820 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:57.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:57.009+0000 7fe910118640 1 --2- 172.21.15.46:0/1142563604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9080a3c10 0x7fe9081cc230 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:39:57.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:57.009+0000 7fe910118640 1 -- 172.21.15.46:0/1142563604 >> 172.21.15.46:0/1142563604 conn(0x7fe9080a5d10 msgr2=0x7fe9080a7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:39:57.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:57.009+0000 7fe910118640 1 -- 172.21.15.46:0/1142563604 shutdown_connections 2024-09-16T04:39:57.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:39:57.009+0000 7fe910118640 1 -- 172.21.15.46:0/1142563604 wait complete. 2024-09-16T04:40:00.023 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:40:00.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.136+0000 7f7e259c0640 10 monclient: get_monmap_and_config 2024-09-16T04:40:00.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.136+0000 7f7e259c0640 10 monclient: build_initial_monmap 2024-09-16T04:40:00.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.136+0000 7f7e259c0640 10 monclient: monmap: 2024-09-16T04:40:00.139 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:00.139 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:00.140 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:40:00.138174+0000 2024-09-16T04:40:00.140 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:40:00.138174+0000 2024-09-16T04:40:00.140 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:00.140 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:00.140 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:40:00.140 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:40:00.140 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:40:00.140 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:40:00.141 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:40:00.141 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:00.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.137+0000 7f7e259c0640 1 Processor -- start 2024-09-16T04:40:00.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.137+0000 7f7e259c0640 1 -- start start 2024-09-16T04:40:00.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.137+0000 7f7e259c0640 10 monclient: init 2024-09-16T04:40:00.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.137+0000 7f7e259c0640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:00.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.137+0000 7f7e259c0640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-16T04:40:00.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.137+0000 7f7e259c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7e2006e160 addr v1:172.21.15.46:6789/0 2024-09-16T04:40:00.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.137+0000 7f7e259c0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 0x7f7e2013c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:00.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.137+0000 7f7e259c0640 10 monclient(hunting): picked mon.noname-a con 0x7f7e20139c50 addr v2:172.21.15.46:3300/0 2024-09-16T04:40:00.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.137+0000 7f7e259c0640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e2013c580 0x7f7e2013e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:00.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.137+0000 7f7e259c0640 10 monclient(hunting): picked mon.noname-b con 0x7f7e2013c580 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:00.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.137+0000 7f7e259c0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e200a3fd0 con 0x7f7e2006e160 2024-09-16T04:40:00.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.137+0000 7f7e259c0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:00.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.137+0000 7f7e259c0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f7e2006e540 con 0x7f7e20139c50 2024-09-16T04:40:00.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.137+0000 7f7e259c0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:00.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.137+0000 7f7e259c0640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f7e200a0bd0 con 0x7f7e2013c580 2024-09-16T04:40:00.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.137+0000 7f7e259c0640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:00.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.137+0000 7f7e259c0640 10 monclient(hunting): authenticate will time out at 798.553949s 2024-09-16T04:40:00.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.137+0000 7f7e1f7fe640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e2013c580 msgr2=0x7f7e2013e970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:40:00.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.138+0000 7f7e1f7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e2013c580 0x7f7e2013e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:00.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.138+0000 7f7e1e7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 0x7f7e2013c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:00.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.138+0000 7f7e1e7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 0x7f7e2013c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54558/0 (socket says 172.21.15.46:54558) 2024-09-16T04:40:00.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.138+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/1624135766 learned_addr learned my addr 172.21.15.46:0/1624135766 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:00.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.138+0000 7f7e1e7fc640 10 monclient(hunting): get_auth_request con 0x7f7e20139c50 auth_method 0 2024-09-16T04:40:00.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.138+0000 7f7e1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:00.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.138+0000 7f7e1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:00.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.138+0000 7f7e1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:00.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.138+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/1624135766 >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 msgr2=0x7f7e2013c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:40:00.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.138+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/1624135766 >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 msgr2=0x7f7e2013c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:40:00.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.138+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 0x7f7e2013c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:00.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.138+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 0x7f7e2013c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:00.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.339+0000 7f7e1f7fe640 1 -- 172.21.15.46:0/1624135766 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e2013c580 msgr2=0x7f7e2013e970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:40:00.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.339+0000 7f7e1f7fe640 1 --2- 172.21.15.46:0/1624135766 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e2013c580 0x7f7e2013e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:40:00.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.339+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 0x7f7e2013c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:00.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.339+0000 7f7e1e7fc640 10 monclient(hunting): get_auth_request con 0x7f7e20139c50 auth_method 0 2024-09-16T04:40:00.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.339+0000 7f7e1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:00.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.339+0000 7f7e1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:00.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.339+0000 7f7e1e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:00.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.339+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/1624135766 >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 msgr2=0x7f7e2013c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:40:00.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.339+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/1624135766 >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 msgr2=0x7f7e2013c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:40:00.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.339+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 0x7f7e2013c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:00.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.339+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 0x7f7e2013c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:40:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.740+0000 7f7e1f7fe640 1 -- 172.21.15.46:0/1624135766 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e2013c580 msgr2=0x7f7e2013e970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:40:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.740+0000 7f7e1f7fe640 1 --2- 172.21.15.46:0/1624135766 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e2013c580 0x7f7e2013e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:40:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.740+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 0x7f7e2013c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.740+0000 7f7e1e7fc640 10 monclient(hunting): get_auth_request con 0x7f7e20139c50 auth_method 0 2024-09-16T04:40:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.740+0000 7f7e1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.740+0000 7f7e1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.740+0000 7f7e1e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.740+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/1624135766 >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 msgr2=0x7f7e2013c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:40:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.740+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/1624135766 >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 msgr2=0x7f7e2013c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:40:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.740+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 0x7f7e2013c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:00.740+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 0x7f7e2013c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:40:01.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:01.541+0000 7f7e1f7fe640 1 -- 172.21.15.46:0/1624135766 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e2013c580 msgr2=0x7f7e2013e970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:40:01.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:01.541+0000 7f7e1f7fe640 1 --2- 172.21.15.46:0/1624135766 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e2013c580 0x7f7e2013e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:40:01.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:01.541+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 0x7f7e2013c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:01.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:01.541+0000 7f7e1e7fc640 10 monclient(hunting): get_auth_request con 0x7f7e20139c50 auth_method 0 2024-09-16T04:40:01.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:01.541+0000 7f7e1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:01.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:01.541+0000 7f7e1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:01.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:01.541+0000 7f7e1e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:01.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:01.541+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/1624135766 >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 msgr2=0x7f7e2013c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:40:01.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:01.541+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/1624135766 >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 msgr2=0x7f7e2013c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:40:01.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:01.541+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 0x7f7e2013c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:01.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:01.541+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 0x7f7e2013c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:40:03.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1cff9640 10 monclient(hunting): tick 2024-09-16T04:40:03.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1cff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:40:03.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1cff9640 1 monclient(hunting): continuing hunt 2024-09-16T04:40:03.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1cff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:40:03.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1cff9640 1 -- 172.21.15.46:0/1624135766 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e2013c580 msgr2=0x7f7e2013e970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:40:03.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1cff9640 1 --2- 172.21.15.46:0/1624135766 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e2013c580 0x7f7e2013e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:03.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1cff9640 1 -- 172.21.15.46:0/1624135766 >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 msgr2=0x7f7e2013c040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:40:03.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1cff9640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.46:3300/0 conn(0x7f7e20139c50 0x7f7e2013c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:03.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1cff9640 1 -- 172.21.15.46:0/1624135766 >> v1:172.21.15.46:6789/0 conn(0x7f7e2006e160 legacy=0x7f7e200a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:03.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1cff9640 10 monclient: _add_conns ranks=[1,4] 2024-09-16T04:40:03.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1cff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e0c0036e0 addr v1:172.21.15.114:6789/0 2024-09-16T04:40:03.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1cff9640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.114:3300/0 conn(0x7f7e0c007410 0x7f7e0c0098d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:03.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1cff9640 10 monclient(hunting): picked mon.noname-c con 0x7f7e0c007410 addr v2:172.21.15.114:3300/0 2024-09-16T04:40:03.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1cff9640 1 -- 172.21.15.46:0/1624135766 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e200a0eb0 con 0x7f7e0c0036e0 2024-09-16T04:40:03.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1cff9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:03.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1cff9640 1 -- 172.21.15.46:0/1624135766 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f7e2006e540 con 0x7f7e0c007410 2024-09-16T04:40:03.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1cff9640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:03.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.114:3300/0 conn(0x7f7e0c007410 0x7f7e0c0098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:03.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1e7fc640 10 monclient(hunting): get_auth_request con 0x7f7e0c007410 auth_method 0 2024-09-16T04:40:03.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:03.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:03.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:03.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/1624135766 >> v2:172.21.15.114:3300/0 conn(0x7f7e0c007410 msgr2=0x7f7e0c0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:40:03.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/1624135766 >> v2:172.21.15.114:3300/0 conn(0x7f7e0c007410 msgr2=0x7f7e0c0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:40:03.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.114:3300/0 conn(0x7f7e0c007410 0x7f7e0c0098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:03.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.138+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.114:3300/0 conn(0x7f7e0c007410 0x7f7e0c0098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:03.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.340+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.114:3300/0 conn(0x7f7e0c007410 0x7f7e0c0098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:03.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.340+0000 7f7e1e7fc640 10 monclient(hunting): get_auth_request con 0x7f7e0c007410 auth_method 0 2024-09-16T04:40:03.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.340+0000 7f7e1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:03.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.340+0000 7f7e1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:03.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.340+0000 7f7e1e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:03.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.340+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/1624135766 >> v2:172.21.15.114:3300/0 conn(0x7f7e0c007410 msgr2=0x7f7e0c0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:40:03.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.340+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/1624135766 >> v2:172.21.15.114:3300/0 conn(0x7f7e0c007410 msgr2=0x7f7e0c0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:40:03.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.340+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.114:3300/0 conn(0x7f7e0c007410 0x7f7e0c0098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:03.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.340+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.114:3300/0 conn(0x7f7e0c007410 0x7f7e0c0098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:40:03.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.741+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.114:3300/0 conn(0x7f7e0c007410 0x7f7e0c0098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:03.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.741+0000 7f7e1e7fc640 10 monclient(hunting): get_auth_request con 0x7f7e0c007410 auth_method 0 2024-09-16T04:40:03.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.741+0000 7f7e1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:03.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.741+0000 7f7e1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:03.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.741+0000 7f7e1e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:03.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.741+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/1624135766 >> v2:172.21.15.114:3300/0 conn(0x7f7e0c007410 msgr2=0x7f7e0c0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:40:03.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.741+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/1624135766 >> v2:172.21.15.114:3300/0 conn(0x7f7e0c007410 msgr2=0x7f7e0c0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:40:03.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.741+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.114:3300/0 conn(0x7f7e0c007410 0x7f7e0c0098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:03.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:03.741+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.114:3300/0 conn(0x7f7e0c007410 0x7f7e0c0098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:40:04.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.114+0000 7f7e1dffb640 1 -- 172.21.15.46:0/1624135766 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 223218385 0 0) 0x7f7e200a0eb0 con 0x7f7e0c0036e0 2024-09-16T04:40:04.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.114+0000 7f7e1dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:40:04.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.114+0000 7f7e1dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:04.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.114+0000 7f7e1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:04.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.114+0000 7f7e1dffb640 10 monclient(hunting): my global_id is 4705 2024-09-16T04:40:04.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.114+0000 7f7e1dffb640 1 -- 172.21.15.46:0/1624135766 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e08003520 con 0x7f7e0c0036e0 2024-09-16T04:40:04.119 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:04.119 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[35779]: starting. 2024-09-16T04:40:04.119 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[35779]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-35743 2024-09-16T04:40:04.119 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[35779]: shutting down. 2024-09-16T04:40:04.120 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:04.120 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:04.121 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:40:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 1 -- 172.21.15.46:0/1624135766 <== mon.1 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4230490565 0 0) 0x7f7e08003520 con 0x7f7e0c0036e0 2024-09-16T04:40:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 1 -- 172.21.15.46:0/1624135766 >> v2:172.21.15.114:3300/0 conn(0x7f7e0c007410 msgr2=0x7f7e0c0098d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:40:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 1 --2- 172.21.15.46:0/1624135766 >> v2:172.21.15.114:3300/0 conn(0x7f7e0c007410 0x7f7e0c0098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:40:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 1 monclient: found mon.noname-c-legacy 2024-09-16T04:40:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.114:6789/0 2024-09-16T04:40:04.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 1 -- 172.21.15.46:0/1624135766 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e0c00b020 con 0x7f7e0c0036e0 2024-09-16T04:40:04.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 10 monclient: _finish_auth 0 2024-09-16T04:40:04.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:40:04.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:04.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 1 -- 172.21.15.46:0/1624135766 <== mon.1 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f7e10001670 con 0x7f7e0c0036e0 2024-09-16T04:40:04.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:04.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T04:40:04.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 10 monclient: dump: 2024-09-16T04:40:04.125 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:04.125 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:04.126 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T04:40:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 1 -- 172.21.15.46:0/1624135766 >> v1:172.21.15.114:6789/0 conn(0x7f7e0c0036e0 legacy=0x7f7e0c005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:04.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1f7fe640 1 -- 172.21.15.46:0/1624135766 reap_dead start 2024-09-16T04:40:04.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:40:04.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 1 --2- 172.21.15.46:0/1624135766 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e080039b0 0x7f7e08003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 10 monclient(hunting): picked mon.c con 0x7f7e080039b0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:04.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 1 --2- 172.21.15.46:0/1624135766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e08004800 0x7f7e08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 10 monclient(hunting): picked mon.a con 0x7f7e08004800 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:40:04.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 1 --2- 172.21.15.46:0/1624135766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e08005210 0x7f7e0800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 10 monclient(hunting): picked mon.b con 0x7f7e08005210 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:40:04.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:04.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 1 -- 172.21.15.46:0/1624135766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2006e540 con 0x7f7e08004800 2024-09-16T04:40:04.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:04.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 1 -- 172.21.15.46:0/1624135766 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0800c160 con 0x7f7e08005210 2024-09-16T04:40:04.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:04.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 1 -- 172.21.15.46:0/1624135766 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f7e0800c360 con 0x7f7e080039b0 2024-09-16T04:40:04.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:04.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:40:04.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1effd640 1 -- 172.21.15.46:0/1624135766 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e080039b0 msgr2=0x7f7e08003dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:40:04.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1effd640 1 --2- 172.21.15.46:0/1624135766 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e080039b0 0x7f7e08003dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:04.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e08004800 0x7f7e08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:04.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1e7fc640 10 monclient(hunting): get_auth_request con 0x7f7e08004800 auth_method 0 2024-09-16T04:40:04.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:04.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:04.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:04.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1f7fe640 1 --2- 172.21.15.46:0/1624135766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e08005210 0x7f7e0800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:04.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1f7fe640 10 monclient(hunting): get_auth_request con 0x7f7e08005210 auth_method 0 2024-09-16T04:40:04.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:04.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:04.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:04.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:04.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:04.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:40:04.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:04.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:04.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:40:04.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1e7fc640 10 monclient(hunting): handle_auth_done global_id 4705 payload 293 2024-09-16T04:40:04.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/1624135766 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e080039b0 msgr2=0x7f7e08003dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:40:04.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e080039b0 0x7f7e08003dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.120+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/1624135766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e08005210 msgr2=0x7f7e0800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:40:04.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.121+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e08005210 0x7f7e0800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.121+0000 7f7e1e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:40:04.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.121+0000 7f7e1e7fc640 1 monclient: found mon.a 2024-09-16T04:40:04.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.121+0000 7f7e1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:04.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.121+0000 7f7e1e7fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:04.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.121+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/1624135766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e08003520 con 0x7f7e08004800 2024-09-16T04:40:04.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.121+0000 7f7e1e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:40:04.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.121+0000 7f7e1e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:40:04.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.121+0000 7f7e1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:04.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.121+0000 7f7e259c0640 5 monclient: authenticate success, global_id 4705 2024-09-16T04:40:04.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.121+0000 7f7e259c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:40:04.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.121+0000 7f7e1f7fe640 1 --2- 172.21.15.46:0/1624135766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e08005210 0x7f7e0800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:40:04.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.121+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/1624135766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e08004800 0x7f7e08004c80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7e14018ae0 tx=0x7f7e08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fd95fb710d20adf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:04.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.122+0000 7f7e1dffb640 1 -- 172.21.15.46:0/1624135766 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1400f040 con 0x7f7e08004800 2024-09-16T04:40:04.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.122+0000 7f7e1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:04.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.122+0000 7f7e1dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:04.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.122+0000 7f7e1dffb640 10 monclient: dump: 2024-09-16T04:40:04.131 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:04.131 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:04.131 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.132 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.132 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:04.132 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.132 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:04.132 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:04.132 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:04.132 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.122+0000 7f7e1dffb640 1 -- 172.21.15.46:0/1624135766 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e1402d6c0 con 0x7f7e08004800 2024-09-16T04:40:04.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.122+0000 7f7e1dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:04.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.122+0000 7f7e1dffb640 1 -- 172.21.15.46:0/1624135766 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1402dab0 con 0x7f7e08004800 2024-09-16T04:40:04.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.122+0000 7f7e259c0640 10 monclient: get_monmap_and_config success 2024-09-16T04:40:04.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.122+0000 7f7e259c0640 4 set_mon_vals no callback set 2024-09-16T04:40:04.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.122+0000 7f7e1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:04.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.122+0000 7f7e1dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:04.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.122+0000 7f7e1dffb640 10 monclient: dump: 2024-09-16T04:40:04.133 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:04.133 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:04.133 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.133 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.133 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:04.133 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.133 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:04.133 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:04.133 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:04.133 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.135 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:40:04.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.122+0000 7f7e259c0640 10 monclient: shutdown 2024-09-16T04:40:04.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.122+0000 7f7e259c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:04.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.122+0000 7f7e259c0640 1 -- 172.21.15.46:0/1624135766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e08004800 msgr2=0x7f7e08004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:04.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.122+0000 7f7e259c0640 1 --2- 172.21.15.46:0/1624135766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e08004800 0x7f7e08004c80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7e14018ae0 tx=0x7f7e08005830 comp rx=0 tx=0).stop 2024-09-16T04:40:04.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.123+0000 7f7e259c0640 1 -- 172.21.15.46:0/1624135766 shutdown_connections 2024-09-16T04:40:04.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.123+0000 7f7e259c0640 1 --2- 172.21.15.46:0/1624135766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e08005210 0x7f7e0800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.123+0000 7f7e259c0640 1 --2- 172.21.15.46:0/1624135766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e08004800 0x7f7e08004c80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.123+0000 7f7e259c0640 1 --2- 172.21.15.46:0/1624135766 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e080039b0 0x7f7e08003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.123+0000 7f7e259c0640 1 -- 172.21.15.46:0/1624135766 >> 172.21.15.46:0/1624135766 conn(0x7f7e200a5c10 msgr2=0x7f7e200a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:04.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.123+0000 7f7e259c0640 1 -- 172.21.15.46:0/1624135766 shutdown_connections 2024-09-16T04:40:04.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.123+0000 7f7e259c0640 1 -- 172.21.15.46:0/1624135766 wait complete. 2024-09-16T04:40:04.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.124+0000 7f7e259c0640 10 monclient: build_initial_monmap 2024-09-16T04:40:04.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.124+0000 7f7e259c0640 10 monclient: monmap: 2024-09-16T04:40:04.143 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:04.143 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:40:04.143 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:40:04.143 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:40:04.143 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:04.143 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.147 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:40:04.148 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:40:04.148 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:40:04.148 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.124+0000 7f7e259c0640 1 Processor -- start 2024-09-16T04:40:04.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.124+0000 7f7e259c0640 1 -- start start 2024-09-16T04:40:04.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.124+0000 7f7e259c0640 10 monclient: init 2024-09-16T04:40:04.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e259c0640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:04.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e259c0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:40:04.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e259c0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e080039b0 0x7f7e20136a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e259c0640 10 monclient(hunting): picked mon.noname-b con 0x7f7e080039b0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:40:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e259c0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e08004800 0x7f7e20136f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e259c0640 10 monclient(hunting): picked mon.noname-a con 0x7f7e08004800 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:40:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e259c0640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e08005210 0x7f7e2011f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e259c0640 10 monclient(hunting): picked mon.noname-c con 0x7f7e08005210 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e259c0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e259c0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7e200a34a0 con 0x7f7e08004800 2024-09-16T04:40:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e259c0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e259c0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7e200a5130 con 0x7f7e080039b0 2024-09-16T04:40:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e259c0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e259c0640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f7e20122de0 con 0x7f7e08005210 2024-09-16T04:40:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e259c0640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e259c0640 10 monclient(hunting): authenticate will time out at 802.541365s 2024-09-16T04:40:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1f7fe640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e08005210 msgr2=0x7f7e2011f760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:40:04.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1f7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e08005210 0x7f7e2011f760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:04.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e08004800 0x7f7e20136f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:04.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e080039b0 0x7f7e20136a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:04.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e08004800 0x7f7e20136f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54610/0 (socket says 172.21.15.46:54610) 2024-09-16T04:40:04.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e080039b0 0x7f7e20136a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.46:58458/0 (socket says 172.21.15.46:58458) 2024-09-16T04:40:04.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/431330415 learned_addr learned my addr 172.21.15.46:0/431330415 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:04.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1e7fc640 10 monclient(hunting): get_auth_request con 0x7f7e08004800 auth_method 0 2024-09-16T04:40:04.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:04.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:04.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:04.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1effd640 10 monclient(hunting): get_auth_request con 0x7f7e080039b0 auth_method 0 2024-09-16T04:40:04.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:04.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:04.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:04.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:04.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:04.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:04.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:04.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:04.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:04.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1e7fc640 10 monclient(hunting): handle_auth_done global_id 4743 payload 824 2024-09-16T04:40:04.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/431330415 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e08005210 msgr2=0x7f7e2011f760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:40:04.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/431330415 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e08005210 0x7f7e2011f760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/431330415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e080039b0 msgr2=0x7f7e20136a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:04.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/431330415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e080039b0 0x7f7e20136a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:40:04.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1e7fc640 1 monclient: found mon.noname-a 2024-09-16T04:40:04.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:04.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:40:04.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1e7fc640 1 -- 172.21.15.46:0/431330415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e20121570 con 0x7f7e08004800 2024-09-16T04:40:04.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.125+0000 7f7e1effd640 1 --2- 172.21.15.46:0/431330415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e080039b0 0x7f7e20136a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:40:04.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e1e7fc640 1 --2- 172.21.15.46:0/431330415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e08004800 0x7f7e20136f80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7e1401b090 tx=0x7f7e14004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:04.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e07fff640 1 -- 172.21.15.46:0/431330415 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e14047070 con 0x7f7e08004800 2024-09-16T04:40:04.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:04.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e07fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:40:04.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e07fff640 10 monclient: dump: 2024-09-16T04:40:04.155 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:04.155 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:04.155 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.155 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.155 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:04.155 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.155 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:04.155 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:04.155 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:04.155 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e07fff640 10 monclient: _finish_auth 0 2024-09-16T04:40:04.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e07fff640 10 monclient: _check_auth_tickets 2024-09-16T04:40:04.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:04.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e07fff640 1 -- 172.21.15.46:0/431330415 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e14037440 con 0x7f7e08004800 2024-09-16T04:40:04.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e259c0640 5 monclient: authenticate success, global_id 4743 2024-09-16T04:40:04.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e259c0640 10 monclient: _renew_subs 2024-09-16T04:40:04.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e259c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:04.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e259c0640 1 -- 172.21.15.46:0/431330415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e201230c0 con 0x7f7e08004800 2024-09-16T04:40:04.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e07fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:04.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e07fff640 1 -- 172.21.15.46:0/431330415 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1403c040 con 0x7f7e08004800 2024-09-16T04:40:04.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:04.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e07fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:04.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e07fff640 10 monclient: dump: 2024-09-16T04:40:04.157 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:04.157 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:04.157 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.157 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.157 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:04.157 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.157 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:04.157 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:04.157 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:04.157 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e259c0640 10 monclient: _renew_subs 2024-09-16T04:40:04.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e259c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:04.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e259c0640 1 -- 172.21.15.46:0/431330415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e201210b0 con 0x7f7e08004800 2024-09-16T04:40:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.126+0000 7f7e1d7fa640 4 set_mon_vals no callback set 2024-09-16T04:40:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.127+0000 7f7e07fff640 1 -- 172.21.15.46:0/431330415 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f7e1404a070 con 0x7f7e08004800 2024-09-16T04:40:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.127+0000 7f7e07fff640 1 --2- 172.21.15.46:0/431330415 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7e0c038c80 0x7f7e0c03b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.127+0000 7f7e259c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.127+0000 7f7e259c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.127+0000 7f7e259c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.127+0000 7f7e07fff640 1 -- 172.21.15.46:0/431330415 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 4346+0+0 (secure 0 0 0) 0x7f7e1406ceb0 con 0x7f7e08004800 2024-09-16T04:40:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.127+0000 7f7e259c0640 1 -- 172.21.15.46:0/431330415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e200a29a0 con 0x7f7e08004800 2024-09-16T04:40:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.128+0000 7f7e1effd640 1 --2- 172.21.15.46:0/431330415 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7e0c038c80 0x7f7e0c03b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.128+0000 7f7e1effd640 10 monclient: get_auth_request con 0x7f7e0c038c80 auth_method 0 2024-09-16T04:40:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.128+0000 7f7e1effd640 1 --2- 172.21.15.46:0/431330415 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7e0c038c80 0x7f7e0c03b140 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7e10000f60 tx=0x7f7e10009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.133+0000 7f7e07fff640 1 -- 172.21.15.46:0/431330415 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7e20121570 con 0x7f7e08004800 2024-09-16T04:40:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.133+0000 7f7e07fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.133+0000 7f7e07fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:40:04.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.383+0000 7f7e259c0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:40:04.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.383+0000 7f7e259c0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:40:04.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.383+0000 7f7e259c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:04.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.383+0000 7f7e259c0640 1 -- 172.21.15.46:0/431330415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7e20122b00 con 0x7f7e08004800 2024-09-16T04:40:04.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.383+0000 7f7e07fff640 1 -- 172.21.15.46:0/431330415 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f7e14051070 con 0x7f7e08004800 2024-09-16T04:40:04.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.383+0000 7f7e07fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:40:04.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.383+0000 7f7e07fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:40:04.386 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":18,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:40:04.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.385+0000 7f7e259c0640 1 -- 172.21.15.46:0/431330415 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7e0c038c80 msgr2=0x7f7e0c03b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:04.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.385+0000 7f7e259c0640 1 --2- 172.21.15.46:0/431330415 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7e0c038c80 0x7f7e0c03b140 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7e10000f60 tx=0x7f7e10009040 comp rx=0 tx=0).stop 2024-09-16T04:40:04.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.385+0000 7f7e259c0640 10 monclient: shutdown 2024-09-16T04:40:04.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.385+0000 7f7e259c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:04.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.385+0000 7f7e259c0640 1 -- 172.21.15.46:0/431330415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e08004800 msgr2=0x7f7e20136f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:04.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.385+0000 7f7e259c0640 1 --2- 172.21.15.46:0/431330415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e08004800 0x7f7e20136f80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7e1401b090 tx=0x7f7e14004730 comp rx=0 tx=0).stop 2024-09-16T04:40:04.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.385+0000 7f7e259c0640 1 -- 172.21.15.46:0/431330415 shutdown_connections 2024-09-16T04:40:04.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.385+0000 7f7e259c0640 1 --2- 172.21.15.46:0/431330415 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7e08005210 0x7f7e2011f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.385+0000 7f7e259c0640 1 --2- 172.21.15.46:0/431330415 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7e0c038c80 0x7f7e0c03b140 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.385+0000 7f7e259c0640 1 --2- 172.21.15.46:0/431330415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e08004800 0x7f7e20136f80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.385+0000 7f7e259c0640 1 --2- 172.21.15.46:0/431330415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e080039b0 0x7f7e20136a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.385+0000 7f7e259c0640 1 -- 172.21.15.46:0/431330415 >> 172.21.15.46:0/431330415 conn(0x7f7e200a5c10 msgr2=0x7f7e201389c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:04.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.385+0000 7f7e259c0640 1 -- 172.21.15.46:0/431330415 shutdown_connections 2024-09-16T04:40:04.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.385+0000 7f7e259c0640 1 -- 172.21.15.46:0/431330415 wait complete. 2024-09-16T04:40:04.399 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:40:04.399 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:40:04.399 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:40:04.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.500+0000 7f30528e6640 10 monclient: get_monmap_and_config 2024-09-16T04:40:04.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.500+0000 7f30528e6640 10 monclient: build_initial_monmap 2024-09-16T04:40:04.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.500+0000 7f30528e6640 10 monclient: monmap: 2024-09-16T04:40:04.502 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:04.502 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:04.502 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:40:04.501124+0000 2024-09-16T04:40:04.502 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:40:04.501124+0000 2024-09-16T04:40:04.502 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:04.502 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.502 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:40:04.502 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:40:04.502 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:40:04.503 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:40:04.503 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:40:04.503 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.500+0000 7f30528e6640 1 Processor -- start 2024-09-16T04:40:04.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.500+0000 7f30528e6640 1 -- start start 2024-09-16T04:40:04.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.500+0000 7f30528e6640 10 monclient: init 2024-09-16T04:40:04.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.500+0000 7f30528e6640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:04.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.500+0000 7f30528e6640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-16T04:40:04.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.500+0000 7f30528e6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f304c0736b0 0x7f304c09faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.500+0000 7f30528e6640 10 monclient(hunting): picked mon.noname-a con 0x7f304c0736b0 addr v2:172.21.15.46:3300/0 2024-09-16T04:40:04.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.500+0000 7f30528e6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f304c073a90 0x7f304c073e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.500+0000 7f30528e6640 10 monclient(hunting): picked mon.noname-b con 0x7f304c073a90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:04.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.500+0000 7f30528e6640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f304c06cd20 0x7f304c06d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.500+0000 7f30528e6640 10 monclient(hunting): picked mon.noname-c con 0x7f304c06cd20 addr v2:172.21.15.114:3300/0 2024-09-16T04:40:04.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.500+0000 7f30528e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:04.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f30528e6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f304c09edf0 con 0x7f304c0736b0 2024-09-16T04:40:04.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f30528e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:04.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f30528e6640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f304c09e8a0 con 0x7f304c06cd20 2024-09-16T04:40:04.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f30528e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:04.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f30528e6640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f304c09eb10 con 0x7f304c073a90 2024-09-16T04:40:04.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f30528e6640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:04.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f30528e6640 10 monclient(hunting): authenticate will time out at 802.917032s 2024-09-16T04:40:04.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304b7fe640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f304c073a90 msgr2=0x7f304c073e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:40:04.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304b7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f304c073a90 0x7f304c073e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:04.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f304c0736b0 0x7f304c09faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:04.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f304c0736b0 0x7f304c09faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48324/0 (socket says 172.21.15.46:48324) 2024-09-16T04:40:04.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304bfff640 1 -- 172.21.15.46:0/2420118443 learned_addr learned my addr 172.21.15.46:0/2420118443 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304bfff640 10 monclient(hunting): get_auth_request con 0x7f304c0736b0 auth_method 0 2024-09-16T04:40:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f3050e83640 1 --2- 172.21.15.46:0/2420118443 >> v2:172.21.15.114:3300/0 conn(0x7f304c06cd20 0x7f304c06d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f3050e83640 10 monclient(hunting): get_auth_request con 0x7f304c06cd20 auth_method 0 2024-09-16T04:40:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f3050e83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f3050e83640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f3050e83640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f3050e83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f3050e83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f3050e83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304bfff640 10 monclient(hunting): handle_auth_done global_id 4761 payload 274 2024-09-16T04:40:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304bfff640 1 -- 172.21.15.46:0/2420118443 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f304c073a90 msgr2=0x7f304c073e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:40:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304bfff640 1 --2- 172.21.15.46:0/2420118443 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f304c073a90 0x7f304c073e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304bfff640 1 -- 172.21.15.46:0/2420118443 >> v2:172.21.15.114:3300/0 conn(0x7f304c06cd20 msgr2=0x7f304c06d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:40:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304bfff640 1 --2- 172.21.15.46:0/2420118443 >> v2:172.21.15.114:3300/0 conn(0x7f304c06cd20 0x7f304c06d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:40:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304bfff640 1 monclient: found mon.noname-a 2024-09-16T04:40:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:40:04.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.501+0000 7f304bfff640 1 -- 172.21.15.46:0/2420118443 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f304c0a2d00 con 0x7f304c0736b0 2024-09-16T04:40:04.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f304bfff640 1 --2- 172.21.15.46:0/2420118443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c0736b0 0x7f304c09faa0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f304c0a3490 tx=0x7f303c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66073737af590d71 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:04.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f304affd640 1 -- 172.21.15.46:0/2420118443 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c037070 con 0x7f304c0736b0 2024-09-16T04:40:04.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f304affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:04.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f304affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:40:04.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f304affd640 10 monclient: dump: 2024-09-16T04:40:04.518 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:04.518 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:04.518 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.518 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.518 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:04.518 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.518 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:04.518 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:04.518 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:04.519 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f304affd640 10 monclient: _finish_auth 0 2024-09-16T04:40:04.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f304affd640 10 monclient: _check_auth_tickets 2024-09-16T04:40:04.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f304affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:04.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f30528e6640 5 monclient: authenticate success, global_id 4761 2024-09-16T04:40:04.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f30528e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:40:04.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f304affd640 1 -- 172.21.15.46:0/2420118443 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f303c02ae20 con 0x7f304c0736b0 2024-09-16T04:40:04.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f304affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:04.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f30528e6640 10 monclient: get_monmap_and_config success 2024-09-16T04:40:04.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f30528e6640 4 set_mon_vals no callback set 2024-09-16T04:40:04.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f304affd640 1 -- 172.21.15.46:0/2420118443 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c032660 con 0x7f304c0736b0 2024-09-16T04:40:04.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f304affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:04.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f304affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:04.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f304affd640 10 monclient: dump: 2024-09-16T04:40:04.520 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:04.520 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:04.520 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.520 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.520 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:04.520 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.520 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:04.520 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:04.520 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:04.521 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f30528e6640 10 monclient: shutdown 2024-09-16T04:40:04.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f30528e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:04.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f30528e6640 1 -- 172.21.15.46:0/2420118443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c0736b0 msgr2=0x7f304c09faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:04.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f30528e6640 1 --2- 172.21.15.46:0/2420118443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c0736b0 0x7f304c09faa0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f304c0a3490 tx=0x7f303c02a110 comp rx=0 tx=0).stop 2024-09-16T04:40:04.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f30528e6640 1 -- 172.21.15.46:0/2420118443 shutdown_connections 2024-09-16T04:40:04.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f30528e6640 1 --2- 172.21.15.46:0/2420118443 >> v2:172.21.15.114:3300/0 conn(0x7f304c06cd20 0x7f304c06d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f30528e6640 1 --2- 172.21.15.46:0/2420118443 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f304c073a90 0x7f304c073e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f30528e6640 1 --2- 172.21.15.46:0/2420118443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c0736b0 0x7f304c09faa0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f30528e6640 1 -- 172.21.15.46:0/2420118443 >> 172.21.15.46:0/2420118443 conn(0x7f304c0a3d60 msgr2=0x7f304c0a4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:04.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.502+0000 7f30528e6640 1 -- 172.21.15.46:0/2420118443 shutdown_connections 2024-09-16T04:40:04.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 1 -- 172.21.15.46:0/2420118443 wait complete. 2024-09-16T04:40:04.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 10 monclient: build_initial_monmap 2024-09-16T04:40:04.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 10 monclient: monmap: 2024-09-16T04:40:04.522 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:04.522 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:40:04.522 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:40:04.522 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:40:04.522 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:04.522 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.522 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:40:04.522 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:40:04.522 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:40:04.522 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 1 Processor -- start 2024-09-16T04:40:04.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 1 -- start start 2024-09-16T04:40:04.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 10 monclient: init 2024-09-16T04:40:04.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:04.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:40:04.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f304c06cd20 0x7f304c1c8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 10 monclient(hunting): picked mon.noname-c con 0x7f304c06cd20 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:04.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c0736b0 0x7f304c1c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 10 monclient(hunting): picked mon.noname-a con 0x7f304c0736b0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:40:04.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f304c073a90 0x7f304c1cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 10 monclient(hunting): picked mon.noname-b con 0x7f304c073a90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:40:04.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:04.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f304c06da40 con 0x7f304c0736b0 2024-09-16T04:40:04.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:04.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f304c06d8c0 con 0x7f304c073a90 2024-09-16T04:40:04.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:04.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f304c06dbc0 con 0x7f304c06cd20 2024-09-16T04:40:04.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:04.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f30528e6640 10 monclient(hunting): authenticate will time out at 802.920557s 2024-09-16T04:40:04.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f304bfff640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f304c06cd20 msgr2=0x7f304c1c8910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:40:04.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f304bfff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f304c06cd20 0x7f304c1c8910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:04.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f304b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c0736b0 0x7f304c1c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:04.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f304b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c0736b0 0x7f304c1c5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48332/0 (socket says 172.21.15.46:48332) 2024-09-16T04:40:04.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f304b7fe640 1 -- 172.21.15.46:0/768755472 learned_addr learned my addr 172.21.15.46:0/768755472 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:04.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f304b7fe640 10 monclient(hunting): get_auth_request con 0x7f304c0736b0 auth_method 0 2024-09-16T04:40:04.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f304b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:04.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f304b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:04.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f304b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:04.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f304b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:04.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f304b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:04.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f304b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:04.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f3050e83640 1 --2- 172.21.15.46:0/768755472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f304c073a90 0x7f304c1cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:04.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f3050e83640 10 monclient(hunting): get_auth_request con 0x7f304c073a90 auth_method 0 2024-09-16T04:40:04.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f3050e83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:04.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f3050e83640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:04.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f3050e83640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:04.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f3050e83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:04.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f3050e83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:04.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f3050e83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:04.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f304b7fe640 10 monclient(hunting): handle_auth_done global_id 4764 payload 824 2024-09-16T04:40:04.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f304b7fe640 1 -- 172.21.15.46:0/768755472 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f304c06cd20 msgr2=0x7f304c1c8910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:40:04.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f304b7fe640 1 --2- 172.21.15.46:0/768755472 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f304c06cd20 0x7f304c1c8910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f304b7fe640 1 -- 172.21.15.46:0/768755472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f304c073a90 msgr2=0x7f304c1cbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.504+0000 7f304b7fe640 1 --2- 172.21.15.46:0/768755472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f304c073a90 0x7f304c1cbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f304b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:40:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f304b7fe640 1 monclient: found mon.noname-a 2024-09-16T04:40:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f304b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f304b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:40:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f304b7fe640 1 -- 172.21.15.46:0/768755472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f304c1aad40 con 0x7f304c0736b0 2024-09-16T04:40:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f304b7fe640 1 --2- 172.21.15.46:0/768755472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c0736b0 0x7f304c1c5ea0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f304000b550 tx=0x7f304000ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f3048ff9640 1 -- 172.21.15.46:0/768755472 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3040003bc0 con 0x7f304c0736b0 2024-09-16T04:40:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f3048ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f3048ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:40:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f3048ff9640 10 monclient: dump: 2024-09-16T04:40:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:04.527 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f3048ff9640 10 monclient: _finish_auth 0 2024-09-16T04:40:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f3048ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:40:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f3048ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f30528e6640 5 monclient: authenticate success, global_id 4764 2024-09-16T04:40:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f3048ff9640 1 -- 172.21.15.46:0/768755472 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3040015070 con 0x7f304c0736b0 2024-09-16T04:40:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f3048ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f30528e6640 10 monclient: _renew_subs 2024-09-16T04:40:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f3048ff9640 1 -- 172.21.15.46:0/768755472 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3040005070 con 0x7f304c0736b0 2024-09-16T04:40:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f30528e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f30528e6640 1 -- 172.21.15.46:0/768755472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f304c1ad800 con 0x7f304c0736b0 2024-09-16T04:40:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f3048ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f3048ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f304a7fc640 4 set_mon_vals no callback set 2024-09-16T04:40:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f3048ff9640 10 monclient: dump: 2024-09-16T04:40:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:04.529 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f30528e6640 10 monclient: _renew_subs 2024-09-16T04:40:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f30528e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.505+0000 7f30528e6640 1 -- 172.21.15.46:0/768755472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f304c1ab530 con 0x7f304c0736b0 2024-09-16T04:40:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f30528e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f30528e6640 1 --2- 172.21.15.46:0/768755472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c02c640 0x7f304c02ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f30528e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f30528e6640 1 -- 172.21.15.46:0/768755472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f304c0a2240 con 0x7f304c02c640 2024-09-16T04:40:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f304bfff640 1 --2- 172.21.15.46:0/768755472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c02c640 0x7f304c02ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f304bfff640 10 monclient: get_auth_request con 0x7f304c02c640 auth_method 0 2024-09-16T04:40:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f304bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f304bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f304bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f304bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f304bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f304bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f3048ff9640 1 -- 172.21.15.46:0/768755472 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f3040005210 con 0x7f304c0736b0 2024-09-16T04:40:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f304bfff640 10 monclient(hunting): handle_auth_done global_id 4767 payload 824 2024-09-16T04:40:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f304bfff640 1 -- 172.21.15.46:0/768755472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f304c09faa0 con 0x7f304c02c640 2024-09-16T04:40:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f3048ff9640 1 --2- 172.21.15.46:0/768755472 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3038038360 0x7f303803a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f3048ff9640 1 -- 172.21.15.46:0/768755472 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 4346+0+0 (secure 0 0 0) 0x7f304004aee0 con 0x7f304c0736b0 2024-09-16T04:40:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f304bfff640 1 --2- 172.21.15.46:0/768755472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c02c640 0x7f304c02ca20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f304c06cac0 tx=0x7f303c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f3050e83640 1 --2- 172.21.15.46:0/768755472 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3038038360 0x7f303803a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f3050e83640 10 monclient: get_auth_request con 0x7f3038038360 auth_method 0 2024-09-16T04:40:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f3048ff9640 1 -- 172.21.15.46:0/768755472 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c041070 con 0x7f304c02c640 2024-09-16T04:40:04.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f3048ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:04.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f3048ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:04.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f3048ff9640 10 monclient: dump: 2024-09-16T04:40:04.531 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:04.531 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:04.531 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.531 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.531 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:04.531 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.531 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:04.531 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:04.531 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:04.531 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f3048ff9640 1 -- 172.21.15.46:0/768755472 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f303c03c460 con 0x7f304c02c640 2024-09-16T04:40:04.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f3048ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:04.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f3048ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:40:04.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f3048ff9640 1 -- 172.21.15.46:0/768755472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c02c640 msgr2=0x7f304c02ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:04.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f3048ff9640 1 --2- 172.21.15.46:0/768755472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c02c640 0x7f304c02ca20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f304c06cac0 tx=0x7f303c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f3048ff9640 1 -- 172.21.15.46:0/768755472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c02c640 msgr2=0x7f304c02ca20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:40:04.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f3048ff9640 1 --2- 172.21.15.46:0/768755472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c02c640 0x7f304c02ca20 secure :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f304c06cac0 tx=0x7f303c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.507+0000 7f3050e83640 1 --2- 172.21.15.46:0/768755472 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3038038360 0x7f303803a820 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f304c1ac020 tx=0x7f304400b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:04.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.513+0000 7f30528e6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:40:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.513+0000 7f30528e6640 1 --2- 172.21.15.46:0/768755472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c02cf60 0x7f304c02d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.513+0000 7f30528e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.513+0000 7f30528e6640 1 -- 172.21.15.46:0/768755472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f304c06ed10 con 0x7f304c02cf60 2024-09-16T04:40:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.513+0000 7f304bfff640 1 --2- 172.21.15.46:0/768755472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c02cf60 0x7f304c02d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.513+0000 7f304bfff640 10 monclient: get_auth_request con 0x7f304c02cf60 auth_method 0 2024-09-16T04:40:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.513+0000 7f304bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.513+0000 7f304bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.513+0000 7f304bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.513+0000 7f304bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.513+0000 7f304bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.513+0000 7f304bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.513+0000 7f304bfff640 10 monclient(hunting): handle_auth_done global_id 4770 payload 824 2024-09-16T04:40:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.513+0000 7f304bfff640 1 -- 172.21.15.46:0/768755472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f304c09d190 con 0x7f304c02cf60 2024-09-16T04:40:04.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f304bfff640 1 --2- 172.21.15.46:0/768755472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c02cf60 0x7f304c02d340 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f303c000c00 tx=0x7f303c009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:04.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f3048ff9640 1 -- 172.21.15.46:0/768755472 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c04d070 con 0x7f304c02cf60 2024-09-16T04:40:04.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f3048ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:04.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f3048ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:04.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f3048ff9640 10 monclient: dump: 2024-09-16T04:40:04.536 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:04.536 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:04.536 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.536 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.536 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:04.536 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.536 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:04.536 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:04.536 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:04.537 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f3048ff9640 1 -- 172.21.15.46:0/768755472 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4702 (secure 0 0 0) 0x7f303c03d4f0 con 0x7f304c02cf60 2024-09-16T04:40:04.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f3048ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:40:04.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f3048ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:40:04.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f3048ff9640 1 -- 172.21.15.46:0/768755472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c02cf60 msgr2=0x7f304c02d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:04.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f3048ff9640 1 --2- 172.21.15.46:0/768755472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c02cf60 0x7f304c02d340 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f303c000c00 tx=0x7f303c009240 comp rx=0 tx=0).stop 2024-09-16T04:40:04.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f3048ff9640 1 -- 172.21.15.46:0/768755472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c02cf60 msgr2=0x7f304c02d340 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:40:04.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f3048ff9640 1 --2- 172.21.15.46:0/768755472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c02cf60 0x7f304c02d340 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.537 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:40:04.538 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:40:04.538 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:40:04.538 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:40:04.538 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 18, 2024-09-16T04:40:04.538 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 90454, 2024-09-16T04:40:04.538 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:40:04.538 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:40:04.538 INFO:teuthology.orchestra.run.smithi046.stdout: 1 2024-09-16T04:40:04.538 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:04.538 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:40:04.538 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:40:04.538 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:40:04.538 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:40:04.539 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:04.539 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:04.539 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:04.539 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:04.539 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:04.539 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:04.539 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:04.539 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:04.539 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:04.539 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:04.539 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:04.539 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:04.539 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:40:04.539 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:40:04.540 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:04.540 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:04.540 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:04.540 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:04.540 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:04.540 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:04.540 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:04.540 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:04.540 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:04.540 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:04.540 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:04.540 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:04.540 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.541 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:40:04.541 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:40:04.541 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:40:04.541 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:40:04.541 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:40:04.541 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:40:04.541 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:40:04.541 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:40:04.541 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:40:04.541 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:40:04.541 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:40:04.541 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:40:04.541 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:40:04.541 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:40:04.542 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:40:04.542 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:40:04.542 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:40:04.542 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:04.542 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:04.542 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:04.542 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:04.542 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:04.542 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:04.542 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:04.542 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:04.542 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:04.542 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:04.542 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:04.543 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:04.543 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:40:04.543 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.543 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:40:04.543 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.543 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:40:04.543 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:40:04.543 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:04.543 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:04.543 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.543 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:04.543 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:40:04.543 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:04.543 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.544 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.544 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:04.544 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:40:04.544 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:04.544 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:04.544 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:04.544 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.544 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:40:04.544 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:40:04.544 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:04.544 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:04.544 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:04.544 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.544 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.545 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:40:04.545 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:40:04.545 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:04.545 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:04.545 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.545 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:04.545 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:40:04.545 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:04.545 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.545 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.545 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:04.545 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:40:04.545 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:04.545 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:04.546 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:04.546 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.546 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:40:04.546 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:40:04.546 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:04.546 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:04.546 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:04.546 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.546 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.546 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:40:04.546 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:40:04.546 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:04.546 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:04.546 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.547 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:04.547 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:40:04.547 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:04.547 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.547 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.547 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:04.547 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:40:04.547 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:04.547 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:04.547 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:04.547 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.547 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:40:04.547 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:40:04.547 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:04.547 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:04.548 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:04.548 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:04.548 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:04.548 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.548 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:40:04.548 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:40:04.548 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.548 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:04.548 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:04.548 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:40:04.548 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:04.548 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:04.548 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:40:04.548 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.549 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:04.549 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:04.549 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:40:04.549 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:04.549 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:04.549 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:40:04.549 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.549 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:04.549 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:04.549 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:40:04.549 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:04.549 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:04.549 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.549 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:40:04.550 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:40:04.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f30528e6640 1 -- 172.21.15.46:0/768755472 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3038038360 msgr2=0x7f303803a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:04.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f30528e6640 1 --2- 172.21.15.46:0/768755472 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3038038360 0x7f303803a820 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f304c1ac020 tx=0x7f304400b040 comp rx=0 tx=0).stop 2024-09-16T04:40:04.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f30528e6640 10 monclient: shutdown 2024-09-16T04:40:04.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f30528e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:04.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f30528e6640 1 -- 172.21.15.46:0/768755472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c0736b0 msgr2=0x7f304c1c5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:04.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f30528e6640 1 --2- 172.21.15.46:0/768755472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c0736b0 0x7f304c1c5ea0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f304000b550 tx=0x7f304000ba10 comp rx=0 tx=0).stop 2024-09-16T04:40:04.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f3050e83640 1 -- 172.21.15.46:0/768755472 reap_dead start 2024-09-16T04:40:04.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f30528e6640 1 -- 172.21.15.46:0/768755472 shutdown_connections 2024-09-16T04:40:04.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f30528e6640 1 --2- 172.21.15.46:0/768755472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f304c0736b0 0x7f304c1c5ea0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f30528e6640 1 -- 172.21.15.46:0/768755472 >> 172.21.15.46:0/768755472 conn(0x7f304c0a3d60 msgr2=0x7f304c0a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:04.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f30528e6640 1 -- 172.21.15.46:0/768755472 shutdown_connections 2024-09-16T04:40:04.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.515+0000 7f30528e6640 1 -- 172.21.15.46:0/768755472 wait complete. 2024-09-16T04:40:04.552 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:40:04.552 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:40:04.552 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:40:04.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.656+0000 7f5ef4ad0640 10 monclient: get_monmap_and_config 2024-09-16T04:40:04.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.656+0000 7f5ef4ad0640 10 monclient: build_initial_monmap 2024-09-16T04:40:04.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.656+0000 7f5ef4ad0640 10 monclient: monmap: 2024-09-16T04:40:04.657 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:04.657 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:04.657 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:40:04.656715+0000 2024-09-16T04:40:04.657 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:40:04.656715+0000 2024-09-16T04:40:04.658 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:04.658 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.658 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:40:04.658 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:40:04.658 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:40:04.658 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:40:04.658 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:40:04.658 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.656+0000 7f5ef4ad0640 1 Processor -- start 2024-09-16T04:40:04.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.656+0000 7f5ef4ad0640 1 -- start start 2024-09-16T04:40:04.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.656+0000 7f5ef4ad0640 10 monclient: init 2024-09-16T04:40:04.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.657+0000 7f5ef4ad0640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:04.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.657+0000 7f5ef4ad0640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-16T04:40:04.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.657+0000 7f5ef4ad0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5ef00a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:40:04.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.657+0000 7f5ef4ad0640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5ef00a3ff0 0x7f5ef00a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.657+0000 7f5ef4ad0640 10 monclient(hunting): picked mon.noname-c con 0x7f5ef00a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:40:04.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.657+0000 7f5ef4ad0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5ef013be90 addr v1:172.21.15.46:6789/0 2024-09-16T04:40:04.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.657+0000 7f5ef4ad0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef009b940 con 0x7f5ef013be90 2024-09-16T04:40:04.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.657+0000 7f5ef4ad0640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef009ab00 con 0x7f5ef00a3c10 2024-09-16T04:40:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.657+0000 7f5ef4ad0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.657+0000 7f5ef4ad0640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f5ef009b030 con 0x7f5ef00a3ff0 2024-09-16T04:40:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.657+0000 7f5ef4ad0640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.657+0000 7f5ef4ad0640 10 monclient(hunting): authenticate will time out at 803.072659s 2024-09-16T04:40:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.657+0000 7f5eeed9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5ef013be90 0x7f5ef013e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37102/0 (socket says 172.21.15.46:37102) 2024-09-16T04:40:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.657+0000 7f5eeed9d640 1 -- 172.21.15.46:0/3221844331 learned_addr learned my addr 172.21.15.46:0/3221844331 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:40:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.657+0000 7f5eedd9b640 1 --2- 172.21.15.46:0/3221844331 >> v2:172.21.15.114:3300/0 conn(0x7f5ef00a3ff0 0x7f5ef00a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.657+0000 7f5eedd9b640 10 monclient(hunting): get_auth_request con 0x7f5ef00a3ff0 auth_method 0 2024-09-16T04:40:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.657+0000 7f5eedd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.657+0000 7f5eedd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.657+0000 7f5eedd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eedd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eedd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eedd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:04.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eed59a640 1 -- 172.21.15.46:0/3221844331 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1749965926 0 0) 0x7f5ef009b940 con 0x7f5ef013be90 2024-09-16T04:40:04.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eed59a640 10 monclient(hunting): _negotiate 2024-09-16T04:40:04.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eed59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:04.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eed59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:04.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eed59a640 10 monclient(hunting): my global_id is 4773 2024-09-16T04:40:04.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eed59a640 1 -- 172.21.15.46:0/3221844331 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5edc003520 con 0x7f5ef013be90 2024-09-16T04:40:04.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eed59a640 1 -- 172.21.15.46:0/3221844331 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3342132052 0 0) 0x7f5ef009ab00 con 0x7f5ef00a3c10 2024-09-16T04:40:04.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eed59a640 10 monclient(hunting): _negotiate 2024-09-16T04:40:04.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eed59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:04.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eed59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:04.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eed59a640 10 monclient(hunting): my global_id is 4735 2024-09-16T04:40:04.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eed59a640 1 -- 172.21.15.46:0/3221844331 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ef009b940 con 0x7f5ef00a3c10 2024-09-16T04:40:04.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eedd9b640 10 monclient(hunting): handle_auth_done global_id 4732 payload 274 2024-09-16T04:40:04.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eedd9b640 1 -- 172.21.15.46:0/3221844331 >> v1:172.21.15.114:6789/0 conn(0x7f5ef00a3c10 legacy=0x7f5ef009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:04.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eedd9b640 1 -- 172.21.15.46:0/3221844331 >> v1:172.21.15.46:6789/0 conn(0x7f5ef013be90 legacy=0x7f5ef013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:04.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eedd9b640 10 monclient: _finish_hunting 0 2024-09-16T04:40:04.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eedd9b640 1 monclient: found mon.noname-c 2024-09-16T04:40:04.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eedd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:04.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eedd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:40:04.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eedd9b640 1 -- 172.21.15.46:0/3221844331 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ef009b310 con 0x7f5ef00a3ff0 2024-09-16T04:40:04.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.658+0000 7f5eedd9b640 1 --2- 172.21.15.46:0/3221844331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef00a3ff0 0x7f5ef00a43d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5ed80097a0 tx=0x7f5ed802a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b59950c1f2b8d5b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:04.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5eed59a640 1 -- 172.21.15.46:0/3221844331 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed8037070 con 0x7f5ef00a3ff0 2024-09-16T04:40:04.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5eed59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:04.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5eed59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:40:04.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5eed59a640 10 monclient: dump: 2024-09-16T04:40:04.670 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:04.670 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:04.670 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.671 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.671 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:04.671 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.671 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:04.671 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:04.671 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:04.671 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5eed59a640 10 monclient: _finish_auth 0 2024-09-16T04:40:04.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5eed59a640 10 monclient: _check_auth_tickets 2024-09-16T04:40:04.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5eed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:04.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5ef4ad0640 5 monclient: authenticate success, global_id 4732 2024-09-16T04:40:04.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5eed59a640 1 -- 172.21.15.46:0/3221844331 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ed8032440 con 0x7f5ef00a3ff0 2024-09-16T04:40:04.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5eed59a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:04.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5ef4ad0640 10 monclient: get_monmap_and_config success 2024-09-16T04:40:04.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5ef4ad0640 4 set_mon_vals no callback set 2024-09-16T04:40:04.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5eed59a640 1 -- 172.21.15.46:0/3221844331 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed8032810 con 0x7f5ef00a3ff0 2024-09-16T04:40:04.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5eed59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:04.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5eed59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:40:04.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5eed59a640 10 monclient: dump: 2024-09-16T04:40:04.672 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:04.672 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:04.672 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.672 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.672 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:04.672 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.672 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:04.672 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:04.672 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:04.672 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5ef4ad0640 10 monclient: shutdown 2024-09-16T04:40:04.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5ef4ad0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:04.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5ef4ad0640 1 -- 172.21.15.46:0/3221844331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef00a3ff0 msgr2=0x7f5ef00a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:04.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5ef4ad0640 1 --2- 172.21.15.46:0/3221844331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef00a3ff0 0x7f5ef00a43d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5ed80097a0 tx=0x7f5ed802a1a0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5ef4ad0640 1 -- 172.21.15.46:0/3221844331 shutdown_connections 2024-09-16T04:40:04.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5ef4ad0640 1 --2- 172.21.15.46:0/3221844331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef00a3ff0 0x7f5ef00a43d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5ef4ad0640 1 -- 172.21.15.46:0/3221844331 >> 172.21.15.46:0/3221844331 conn(0x7f5ef00a6340 msgr2=0x7f5ef006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:04.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5ef4ad0640 1 -- 172.21.15.46:0/3221844331 shutdown_connections 2024-09-16T04:40:04.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.659+0000 7f5ef4ad0640 1 -- 172.21.15.46:0/3221844331 wait complete. 2024-09-16T04:40:04.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.660+0000 7f5ef4ad0640 10 monclient: build_initial_monmap 2024-09-16T04:40:04.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.660+0000 7f5ef4ad0640 10 monclient: monmap: 2024-09-16T04:40:04.673 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:04.673 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:40:04.673 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:40:04.674 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:40:04.674 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:04.674 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.674 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:40:04.674 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:40:04.674 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:40:04.674 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.660+0000 7f5ef4ad0640 1 Processor -- start 2024-09-16T04:40:04.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.660+0000 7f5ef4ad0640 1 -- start start 2024-09-16T04:40:04.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.660+0000 7f5ef4ad0640 10 monclient: init 2024-09-16T04:40:04.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5ef4ad0640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:04.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5ef4ad0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:40:04.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5ef4ad0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef00a3c10 0x7f5ef01368a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5ef4ad0640 10 monclient(hunting): picked mon.noname-b con 0x7f5ef00a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:40:04.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5ef4ad0640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5ef00a3ff0 0x7f5ef0136de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5ef4ad0640 10 monclient(hunting): picked mon.noname-c con 0x7f5ef00a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:04.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5ef4ad0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ef013be90 0x7f5ef0138d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5ef4ad0640 10 monclient(hunting): picked mon.noname-a con 0x7f5ef013be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:40:04.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5ef4ad0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:04.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5ef4ad0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef00a6b40 con 0x7f5ef013be90 2024-09-16T04:40:04.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5ef4ad0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:04.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5ef4ad0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef00a69c0 con 0x7f5ef00a3c10 2024-09-16T04:40:04.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5ef4ad0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:04.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5ef4ad0640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f5ef013f9a0 con 0x7f5ef00a3ff0 2024-09-16T04:40:04.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5ef4ad0640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:04.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5ef4ad0640 10 monclient(hunting): authenticate will time out at 803.076383s 2024-09-16T04:40:04.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eedd9b640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5ef00a3ff0 msgr2=0x7f5ef0136de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:40:04.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eedd9b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5ef00a3ff0 0x7f5ef0136de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:04.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eeed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ef013be90 0x7f5ef0138d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:04.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eeed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ef013be90 0x7f5ef0138d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48360/0 (socket says 172.21.15.46:48360) 2024-09-16T04:40:04.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eeed9d640 1 -- 172.21.15.46:0/1021853765 learned_addr learned my addr 172.21.15.46:0/1021853765 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:04.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eeed9d640 10 monclient(hunting): get_auth_request con 0x7f5ef013be90 auth_method 0 2024-09-16T04:40:04.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eeed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:04.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eeed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:04.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eeed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:04.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eee59c640 1 --2- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef00a3c10 0x7f5ef01368a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:04.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eee59c640 10 monclient(hunting): get_auth_request con 0x7f5ef00a3c10 auth_method 0 2024-09-16T04:40:04.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:04.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eee59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:04.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:04.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eeed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:04.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eeed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:04.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eeed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:04.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:04.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:04.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:04.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eeed9d640 10 monclient(hunting): handle_auth_done global_id 4776 payload 824 2024-09-16T04:40:04.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eeed9d640 1 -- 172.21.15.46:0/1021853765 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5ef00a3ff0 msgr2=0x7f5ef0136de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:40:04.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eeed9d640 1 --2- 172.21.15.46:0/1021853765 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5ef00a3ff0 0x7f5ef0136de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eeed9d640 1 -- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef00a3c10 msgr2=0x7f5ef01368a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:04.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eeed9d640 1 --2- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef00a3c10 0x7f5ef01368a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:04.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eeed9d640 10 monclient: _finish_hunting 0 2024-09-16T04:40:04.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eeed9d640 1 monclient: found mon.noname-a 2024-09-16T04:40:04.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eeed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:04.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eeed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:40:04.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.661+0000 7f5eeed9d640 1 -- 172.21.15.46:0/1021853765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ef01200f0 con 0x7f5ef013be90 2024-09-16T04:40:04.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5eeed9d640 1 --2- 172.21.15.46:0/1021853765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ef013be90 0x7f5ef0138d90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5ee800ca70 tx=0x7f5ee800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:04.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5ee6ffd640 1 -- 172.21.15.46:0/1021853765 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8017070 con 0x7f5ef013be90 2024-09-16T04:40:04.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5ee6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:04.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5ee6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:40:04.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5ee6ffd640 10 monclient: dump: 2024-09-16T04:40:04.678 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:04.678 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:04.678 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.678 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.678 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:04.678 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.678 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:04.678 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:04.678 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:04.678 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5ee6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:40:04.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5ee6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:40:04.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5ee6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:04.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5ee6ffd640 1 -- 172.21.15.46:0/1021853765 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ee800abd0 con 0x7f5ef013be90 2024-09-16T04:40:04.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5ee6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:04.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5ee6ffd640 1 -- 172.21.15.46:0/1021853765 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8012400 con 0x7f5ef013be90 2024-09-16T04:40:04.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5ef4ad0640 5 monclient: authenticate success, global_id 4776 2024-09-16T04:40:04.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5ef4ad0640 10 monclient: _renew_subs 2024-09-16T04:40:04.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5ef4ad0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:04.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5ef4ad0640 1 -- 172.21.15.46:0/1021853765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ef01203d0 con 0x7f5ef013be90 2024-09-16T04:40:04.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5ee6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:04.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5ee7fff640 4 set_mon_vals no callback set 2024-09-16T04:40:04.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5ee6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:04.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5ee6ffd640 10 monclient: dump: 2024-09-16T04:40:04.680 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:04.680 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:04.680 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.680 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.680 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:04.680 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.680 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:04.680 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:04.680 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:04.680 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5ef4ad0640 10 monclient: _renew_subs 2024-09-16T04:40:04.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5ef4ad0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:04.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.662+0000 7f5ef4ad0640 1 -- 172.21.15.46:0/1021853765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ef013b4d0 con 0x7f5ef013be90 2024-09-16T04:40:04.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.663+0000 7f5ee6ffd640 1 -- 172.21.15.46:0/1021853765 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f5ee801a070 con 0x7f5ef013be90 2024-09-16T04:40:04.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.663+0000 7f5ef4ad0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:04.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.663+0000 7f5ef4ad0640 1 --2- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef01392d0 0x7f5ef01396b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.663+0000 7f5ef4ad0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:04.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.663+0000 7f5ef4ad0640 1 -- 172.21.15.46:0/1021853765 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef00a4f80 con 0x7f5ef01392d0 2024-09-16T04:40:04.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.663+0000 7f5ee6ffd640 1 --2- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5ed0038360 0x7f5ed003a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.663+0000 7f5ee6ffd640 1 -- 172.21.15.46:0/1021853765 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 4346+0+0 (secure 0 0 0) 0x7f5ee804b260 con 0x7f5ef013be90 2024-09-16T04:40:04.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.663+0000 7f5eee59c640 1 --2- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef01392d0 0x7f5ef01396b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:04.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.663+0000 7f5eee59c640 10 monclient: get_auth_request con 0x7f5ef01392d0 auth_method 0 2024-09-16T04:40:04.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.663+0000 7f5eee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:04.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.663+0000 7f5eee59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:04.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.663+0000 7f5eee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:04.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.663+0000 7f5eedd9b640 1 --2- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5ed0038360 0x7f5ed003a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:04.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.663+0000 7f5eedd9b640 10 monclient: get_auth_request con 0x7f5ed0038360 auth_method 0 2024-09-16T04:40:04.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.663+0000 7f5eee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:04.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.663+0000 7f5eee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:04.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.663+0000 7f5eee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:04.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.663+0000 7f5eee59c640 10 monclient(hunting): handle_auth_done global_id 4741 payload 824 2024-09-16T04:40:04.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.663+0000 7f5eee59c640 1 -- 172.21.15.46:0/1021853765 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5ef00a4450 con 0x7f5ef01392d0 2024-09-16T04:40:04.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.663+0000 7f5eee59c640 1 --2- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef01392d0 0x7f5ef01396b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5ee000f770 tx=0x7f5ee000fc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:04.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.663+0000 7f5eedd9b640 1 --2- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5ed0038360 0x7f5ed003a820 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8034000 tx=0x7f5ed802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:04.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.664+0000 7f5ee6ffd640 1 -- 172.21.15.46:0/1021853765 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee00050d0 con 0x7f5ef01392d0 2024-09-16T04:40:04.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.664+0000 7f5ee6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:04.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.664+0000 7f5ee6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:40:04.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.664+0000 7f5ee6ffd640 10 monclient: dump: 2024-09-16T04:40:04.682 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:04.682 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:04.682 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.682 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.682 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:04.683 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.683 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:04.683 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:04.683 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:04.683 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.664+0000 7f5ee6ffd640 1 -- 172.21.15.46:0/1021853765 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5ee0018070 con 0x7f5ef01392d0 2024-09-16T04:40:04.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.664+0000 7f5ee6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:04.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.664+0000 7f5ee6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:40:04.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.664+0000 7f5ee6ffd640 1 -- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef01392d0 msgr2=0x7f5ef01396b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:04.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.664+0000 7f5ee6ffd640 1 --2- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef01392d0 0x7f5ef01396b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5ee000f770 tx=0x7f5ee000fc30 comp rx=0 tx=0).stop 2024-09-16T04:40:04.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.664+0000 7f5ee6ffd640 1 -- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef01392d0 msgr2=0x7f5ef01396b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:40:04.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.664+0000 7f5ee6ffd640 1 --2- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef01392d0 0x7f5ef01396b0 secure :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5ee000f770 tx=0x7f5ee000fc30 comp rx=0 tx=0).stop 2024-09-16T04:40:04.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.670+0000 7f5ef4ad0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:40:04.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.670+0000 7f5ef4ad0640 1 --2- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef0139bf0 0x7f5ef009f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.670+0000 7f5ef4ad0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.670+0000 7f5ef4ad0640 1 -- 172.21.15.46:0/1021853765 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef0067e20 con 0x7f5ef0139bf0 2024-09-16T04:40:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.670+0000 7f5eee59c640 1 --2- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef0139bf0 0x7f5ef009f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.670+0000 7f5eee59c640 10 monclient: get_auth_request con 0x7f5ef0139bf0 auth_method 0 2024-09-16T04:40:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.670+0000 7f5eee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.670+0000 7f5eee59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.670+0000 7f5eee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.671+0000 7f5eee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.671+0000 7f5eee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.671+0000 7f5eee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.671+0000 7f5eee59c640 10 monclient(hunting): handle_auth_done global_id 4744 payload 824 2024-09-16T04:40:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.671+0000 7f5eee59c640 1 -- 172.21.15.46:0/1021853765 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5ef011fe10 con 0x7f5ef0139bf0 2024-09-16T04:40:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.671+0000 7f5eee59c640 1 --2- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef0139bf0 0x7f5ef009f120 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0017000 tx=0x7f5ee0001150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5ee6ffd640 1 -- 172.21.15.46:0/1021853765 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee001ee30 con 0x7f5ef0139bf0 2024-09-16T04:40:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5ee6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5ee6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:40:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5ee6ffd640 10 monclient: dump: 2024-09-16T04:40:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:04.687 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5ee6ffd640 1 -- 172.21.15.46:0/1021853765 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4867 (secure 0 0 0) 0x7f5ee000d960 con 0x7f5ef0139bf0 2024-09-16T04:40:04.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5ee6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:40:04.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5ee6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:40:04.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5ee6ffd640 1 -- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef0139bf0 msgr2=0x7f5ef009f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:04.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5ee6ffd640 1 --2- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef0139bf0 0x7f5ef009f120 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0017000 tx=0x7f5ee0001150 comp rx=0 tx=0).stop 2024-09-16T04:40:04.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5ee6ffd640 1 -- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef0139bf0 msgr2=0x7f5ef009f120 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:40:04.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5ee6ffd640 1 --2- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ef0139bf0 0x7f5ef009f120 secure :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0017000 tx=0x7f5ee0001150 comp rx=0 tx=0).stop 2024-09-16T04:40:04.688 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:40:04.688 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:40:04.688 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:40:04.688 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:40:04.689 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 18, 2024-09-16T04:40:04.689 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 90573, 2024-09-16T04:40:04.689 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:40:04.689 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:40:04.689 INFO:teuthology.orchestra.run.smithi046.stdout: 1 2024-09-16T04:40:04.689 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:04.689 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:40:04.689 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:40:04.689 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:40:04.689 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:40:04.689 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:04.689 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:04.689 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:04.689 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:04.690 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:04.690 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:04.690 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:04.690 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:04.690 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:04.690 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:04.690 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:04.690 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:04.690 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:40:04.690 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:40:04.691 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:04.691 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:04.691 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:04.691 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:04.691 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:04.691 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:04.691 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:04.691 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:04.691 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:04.692 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:04.692 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:04.692 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:04.692 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.692 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:40:04.692 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:40:04.692 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:40:04.692 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:40:04.692 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:40:04.692 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:40:04.693 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:40:04.693 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:40:04.693 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:40:04.693 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:40:04.693 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:40:04.693 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:40:04.693 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:40:04.693 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:40:04.693 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:40:04.693 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:40:04.693 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:40:04.693 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:04.693 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:04.694 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:04.694 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:04.694 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:04.694 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:04.694 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:04.694 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:04.694 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:04.694 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:04.694 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:04.694 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:04.694 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:40:04.694 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.694 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:40:04.694 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.695 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:40:04.695 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:40:04.695 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:04.695 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:04.695 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.695 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:04.695 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:40:04.695 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:04.695 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.695 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.695 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:04.695 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:40:04.695 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:04.695 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:04.696 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:04.696 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.696 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:40:04.696 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:40:04.696 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:04.696 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:04.696 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:04.696 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.696 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.696 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:40:04.696 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:40:04.696 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:04.696 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:04.696 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.697 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:04.697 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:40:04.697 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:04.697 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.697 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.697 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:04.697 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:40:04.697 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:04.697 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:04.697 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:04.697 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.697 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:40:04.697 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:40:04.697 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:04.697 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:04.698 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:04.698 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.698 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.698 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:40:04.698 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:40:04.698 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:04.698 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:04.698 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.698 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:04.698 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:40:04.698 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:04.698 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.698 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.698 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:04.699 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:40:04.699 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:04.699 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:04.699 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:04.699 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.699 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:40:04.699 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:40:04.699 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:04.699 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:04.699 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:04.699 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:04.699 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:04.699 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.699 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:40:04.699 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:40:04.700 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.700 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:04.700 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:04.700 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:40:04.700 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:04.700 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:04.700 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:40:04.700 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.700 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:04.700 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:04.700 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:40:04.700 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:04.700 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:04.700 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:40:04.701 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.701 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:04.701 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:04.701 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6 2024-09-16T04:40:04.701 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:04.701 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:04.701 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:40:04.701 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:04.701 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:04.701 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:04.701 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:40:04.701 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:04.701 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:04.701 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:04.702 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:40:04.702 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:40:04.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5ef4ad0640 1 -- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5ed0038360 msgr2=0x7f5ed003a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:04.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5ef4ad0640 1 --2- 172.21.15.46:0/1021853765 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5ed0038360 0x7f5ed003a820 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8034000 tx=0x7f5ed802e000 comp rx=0 tx=0).stop 2024-09-16T04:40:04.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5ef4ad0640 10 monclient: shutdown 2024-09-16T04:40:04.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5ef4ad0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:04.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5ef4ad0640 1 -- 172.21.15.46:0/1021853765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ef013be90 msgr2=0x7f5ef0138d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:04.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5ef4ad0640 1 --2- 172.21.15.46:0/1021853765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ef013be90 0x7f5ef0138d90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5ee800ca70 tx=0x7f5ee800cf30 comp rx=0 tx=0).stop 2024-09-16T04:40:04.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5eeed9d640 1 -- 172.21.15.46:0/1021853765 reap_dead start 2024-09-16T04:40:04.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5eeed9d640 1 -- 172.21.15.46:0/1021853765 reap_dead start 2024-09-16T04:40:04.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5ef4ad0640 1 -- 172.21.15.46:0/1021853765 shutdown_connections 2024-09-16T04:40:04.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5ef4ad0640 1 -- 172.21.15.46:0/1021853765 >> 172.21.15.46:0/1021853765 conn(0x7f5ef00a6340 msgr2=0x7f5ef00a1d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:04.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5ef4ad0640 1 -- 172.21.15.46:0/1021853765 shutdown_connections 2024-09-16T04:40:04.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:04.672+0000 7f5ef4ad0640 1 -- 172.21.15.46:0/1021853765 wait complete. 2024-09-16T04:40:04.705 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:40:04.705 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:40:06.221 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35848]: starting. 2024-09-16T04:40:06.221 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35848]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-35846 2024-09-16T04:40:06.221 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35848]: created object 0... 2024-09-16T04:40:06.221 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35848]: created object 25... 2024-09-16T04:40:06.222 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35848]: created object 49... 2024-09-16T04:40:06.222 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35848]: finishing. 2024-09-16T04:40:06.222 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35848]: shutting down. 2024-09-16T04:40:07.132 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[35849]: starting. 2024-09-16T04:40:07.132 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[35849]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-35846 2024-09-16T04:40:07.132 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[35849]: shutting down. 2024-09-16T04:40:09.174 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:09.174 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35882]: starting. 2024-09-16T04:40:09.174 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35882]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-35846 2024-09-16T04:40:09.174 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35882]: created object 0... 2024-09-16T04:40:09.174 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35882]: created object 25... 2024-09-16T04:40:09.174 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35882]: created object 49... 2024-09-16T04:40:09.174 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35882]: finishing. 2024-09-16T04:40:09.174 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35882]: shutting down. 2024-09-16T04:40:09.186 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:09.187 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35884]: starting. 2024-09-16T04:40:09.187 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35884]: listing objects. 2024-09-16T04:40:09.187 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35884]: listed object 0... 2024-09-16T04:40:09.187 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35884]: listed object 25... 2024-09-16T04:40:09.187 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35884]: saw 50 objects 2024-09-16T04:40:09.187 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35884]: shutting down. 2024-09-16T04:40:10.091 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:10.091 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[35883]: starting. 2024-09-16T04:40:10.091 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[35883]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-35846 2024-09-16T04:40:10.091 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[35883]: shutting down. 2024-09-16T04:40:10.092 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:10.092 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:10.092 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:40:10.095 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:40:12.232 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35951]: starting. 2024-09-16T04:40:12.232 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35951]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-35950 2024-09-16T04:40:12.233 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35951]: created object 0... 2024-09-16T04:40:12.233 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35951]: created object 25... 2024-09-16T04:40:12.233 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35951]: created object 49... 2024-09-16T04:40:12.233 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35951]: finishing. 2024-09-16T04:40:12.233 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[35951]: shutting down. 2024-09-16T04:40:13.153 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[35952]: starting. 2024-09-16T04:40:13.154 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[35952]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-35950 2024-09-16T04:40:13.154 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[35952]: shutting down. 2024-09-16T04:40:15.253 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:15.253 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35985]: starting. 2024-09-16T04:40:15.253 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35985]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-35950 2024-09-16T04:40:15.254 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35985]: created object 0... 2024-09-16T04:40:15.254 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35985]: created object 25... 2024-09-16T04:40:15.254 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35985]: created object 49... 2024-09-16T04:40:15.254 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35985]: finishing. 2024-09-16T04:40:15.254 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[35985]: shutting down. 2024-09-16T04:40:15.266 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:15.266 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35987]: starting. 2024-09-16T04:40:15.266 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35987]: listing objects. 2024-09-16T04:40:15.267 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35987]: listed object 0... 2024-09-16T04:40:15.267 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35987]: listed object 25... 2024-09-16T04:40:15.267 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35987]: saw 50 objects 2024-09-16T04:40:15.267 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[35987]: shutting down. 2024-09-16T04:40:16.173 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:16.173 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[35986]: starting. 2024-09-16T04:40:16.173 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[35986]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-35950 2024-09-16T04:40:16.173 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[35986]: shutting down. 2024-09-16T04:40:16.174 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:16.174 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:16.174 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:40:16.177 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:40:18.272 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36054]: starting. 2024-09-16T04:40:18.272 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36054]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-36053 2024-09-16T04:40:18.272 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36054]: created object 0... 2024-09-16T04:40:18.272 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36054]: created object 25... 2024-09-16T04:40:18.272 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36054]: created object 49... 2024-09-16T04:40:18.272 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36054]: finishing. 2024-09-16T04:40:18.272 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36054]: shutting down. 2024-09-16T04:40:19.089 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36055]: starting. 2024-09-16T04:40:19.089 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36055]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-36053 2024-09-16T04:40:19.089 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36055]: shutting down. 2024-09-16T04:40:21.181 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:21.181 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36088]: starting. 2024-09-16T04:40:21.181 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36088]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-36053 2024-09-16T04:40:21.181 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36088]: created object 0... 2024-09-16T04:40:21.181 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36088]: created object 25... 2024-09-16T04:40:21.181 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36088]: created object 49... 2024-09-16T04:40:21.181 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36088]: finishing. 2024-09-16T04:40:21.181 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36088]: shutting down. 2024-09-16T04:40:21.194 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:21.194 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36090]: starting. 2024-09-16T04:40:21.194 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36090]: listing objects. 2024-09-16T04:40:21.194 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36090]: listed object 0... 2024-09-16T04:40:21.194 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36090]: listed object 25... 2024-09-16T04:40:21.194 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36090]: saw 50 objects 2024-09-16T04:40:21.194 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36090]: shutting down. 2024-09-16T04:40:22.105 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:22.105 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36089]: starting. 2024-09-16T04:40:22.105 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36089]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-36053 2024-09-16T04:40:22.105 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36089]: shutting down. 2024-09-16T04:40:22.106 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:22.106 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:22.107 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:40:22.109 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:40:24.198 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36157]: starting. 2024-09-16T04:40:24.198 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36157]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-36156 2024-09-16T04:40:24.198 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36157]: created object 0... 2024-09-16T04:40:24.198 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36157]: created object 25... 2024-09-16T04:40:24.199 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36157]: created object 49... 2024-09-16T04:40:24.199 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36157]: finishing. 2024-09-16T04:40:24.199 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36157]: shutting down. 2024-09-16T04:40:24.706 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:40:24.707 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T04:40:24.707 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:40:24.707 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:40:24.710 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:40:24.710 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:40:24.710 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:40:24.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.821+0000 7ff4ae86e640 10 monclient: get_monmap_and_config 2024-09-16T04:40:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.821+0000 7ff4ae86e640 10 monclient: build_initial_monmap 2024-09-16T04:40:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.821+0000 7ff4ae86e640 10 monclient: monmap: 2024-09-16T04:40:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:40:24.822191+0000 2024-09-16T04:40:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:40:24.822191+0000 2024-09-16T04:40:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:40:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:40:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:40:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:40:24.824 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:40:24.824 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:24.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.822+0000 7ff4ae86e640 1 Processor -- start 2024-09-16T04:40:24.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.822+0000 7ff4ae86e640 1 -- start start 2024-09-16T04:40:24.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.822+0000 7ff4ae86e640 10 monclient: init 2024-09-16T04:40:24.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4ae86e640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:24.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4ae86e640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-16T04:40:24.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4ae86e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4a80a3c10 0x7ff4a809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:24.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4ae86e640 10 monclient(hunting): picked mon.noname-b con 0x7ff4a80a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:24.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4ae86e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff4a80a3ff0 0x7ff4a80a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:24.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4ae86e640 10 monclient(hunting): picked mon.noname-a con 0x7ff4a80a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:40:24.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4ae86e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff4a81369d0 0x7ff4a8136e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:24.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4ae86e640 10 monclient(hunting): picked mon.noname-c con 0x7ff4a81369d0 addr v2:172.21.15.114:3300/0 2024-09-16T04:40:24.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4ae86e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:24.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4ae86e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff4a809ab70 con 0x7ff4a80a3ff0 2024-09-16T04:40:24.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4ae86e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:24.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4ae86e640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ff4a80a4d60 con 0x7ff4a81369d0 2024-09-16T04:40:24.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4ae86e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:24.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4ae86e640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7ff4a809b030 con 0x7ff4a80a3c10 2024-09-16T04:40:24.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4ae86e640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:24.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4ae86e640 10 monclient(hunting): authenticate will time out at 823.238124s 2024-09-16T04:40:24.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a7fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4a80a3c10 0x7ff4a809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:24.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff4a80a3ff0 0x7ff4a80a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:24.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a7fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4a80a3c10 0x7ff4a809bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:43926/0 (socket says 172.21.15.46:43926) 2024-09-16T04:40:24.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff4a80a3ff0 0x7ff4a80a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58396/0 (socket says 172.21.15.46:58396) 2024-09-16T04:40:24.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a7fff640 1 -- 172.21.15.46:0/3671546032 learned_addr learned my addr 172.21.15.46:0/3671546032 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:24.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a7fff640 10 monclient(hunting): get_auth_request con 0x7ff4a80a3c10 auth_method 0 2024-09-16T04:40:24.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:24.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:24.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:24.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a77fe640 10 monclient(hunting): get_auth_request con 0x7ff4a80a3ff0 auth_method 0 2024-09-16T04:40:24.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:24.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:24.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:24.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:24.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:24.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a7fff640 1 -- 172.21.15.46:0/3671546032 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4a80a3c10 msgr2=0x7ff4a809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:40:24.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a7fff640 1 -- 172.21.15.46:0/3671546032 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4a80a3c10 msgr2=0x7ff4a809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:40:24.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a7fff640 1 --2- 172.21.15.46:0/3671546032 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4a80a3c10 0x7ff4a809bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:24.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:24.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a7fff640 1 --2- 172.21.15.46:0/3671546032 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4a80a3c10 0x7ff4a809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:24.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4ace0b640 1 --2- 172.21.15.46:0/3671546032 >> v2:172.21.15.114:3300/0 conn(0x7ff4a81369d0 0x7ff4a8136e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:24.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4ace0b640 10 monclient(hunting): get_auth_request con 0x7ff4a81369d0 auth_method 0 2024-09-16T04:40:24.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4ace0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:24.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4ace0b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:24.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4ace0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:24.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a77fe640 10 monclient(hunting): handle_auth_done global_id 4872 payload 274 2024-09-16T04:40:24.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a77fe640 1 -- 172.21.15.46:0/3671546032 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4a80a3c10 msgr2=0x7ff4a809bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:40:24.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a77fe640 1 --2- 172.21.15.46:0/3671546032 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4a80a3c10 0x7ff4a809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:24.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a77fe640 1 -- 172.21.15.46:0/3671546032 >> v2:172.21.15.114:3300/0 conn(0x7ff4a81369d0 msgr2=0x7ff4a8136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:40:24.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a77fe640 1 --2- 172.21.15.46:0/3671546032 >> v2:172.21.15.114:3300/0 conn(0x7ff4a81369d0 0x7ff4a8136e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:24.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:40:24.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a77fe640 1 monclient: found mon.noname-a 2024-09-16T04:40:24.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:24.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:40:24.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.823+0000 7ff4a77fe640 1 -- 172.21.15.46:0/3671546032 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a809b310 con 0x7ff4a80a3ff0 2024-09-16T04:40:24.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.824+0000 7ff4a77fe640 1 --2- 172.21.15.46:0/3671546032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4a80a3ff0 0x7ff4a80a43d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff4a809d0c0 tx=0x7ff49c02a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32612f444b486e47 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:24.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.824+0000 7ff4a6ffd640 1 -- 172.21.15.46:0/3671546032 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff49c037070 con 0x7ff4a80a3ff0 2024-09-16T04:40:24.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.824+0000 7ff4a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:24.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.824+0000 7ff4a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:40:24.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.824+0000 7ff4a6ffd640 10 monclient: dump: 2024-09-16T04:40:24.839 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:24.839 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:24.839 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:24.839 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:24.839 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:24.839 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:24.839 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:24.839 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:24.839 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:24.839 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:24.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.824+0000 7ff4a6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:40:24.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.824+0000 7ff4a6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:40:24.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.824+0000 7ff4a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.824+0000 7ff4a6ffd640 1 -- 172.21.15.46:0/3671546032 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff49c02add0 con 0x7ff4a80a3ff0 2024-09-16T04:40:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.824+0000 7ff4ae86e640 5 monclient: authenticate success, global_id 4872 2024-09-16T04:40:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.824+0000 7ff4ae86e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:40:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.824+0000 7ff4a6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.824+0000 7ff4ae86e640 10 monclient: get_monmap_and_config success 2024-09-16T04:40:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.824+0000 7ff4a6ffd640 1 -- 172.21.15.46:0/3671546032 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff49c032660 con 0x7ff4a80a3ff0 2024-09-16T04:40:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.824+0000 7ff4ae86e640 4 set_mon_vals no callback set 2024-09-16T04:40:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.824+0000 7ff4a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.824+0000 7ff4a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:24.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.824+0000 7ff4a6ffd640 10 monclient: dump: 2024-09-16T04:40:24.841 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:24.841 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:24.841 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:24.841 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:24.841 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:24.841 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:24.841 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:24.841 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:24.841 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:24.841 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:24.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.825+0000 7ff4ae86e640 10 monclient: shutdown 2024-09-16T04:40:24.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.825+0000 7ff4ae86e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:24.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.825+0000 7ff4ae86e640 1 -- 172.21.15.46:0/3671546032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4a80a3ff0 msgr2=0x7ff4a80a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:24.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.825+0000 7ff4ae86e640 1 --2- 172.21.15.46:0/3671546032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4a80a3ff0 0x7ff4a80a43d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff4a809d0c0 tx=0x7ff49c02a080 comp rx=0 tx=0).stop 2024-09-16T04:40:24.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.825+0000 7ff4ae86e640 1 -- 172.21.15.46:0/3671546032 shutdown_connections 2024-09-16T04:40:24.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.825+0000 7ff4ae86e640 1 --2- 172.21.15.46:0/3671546032 >> v2:172.21.15.114:3300/0 conn(0x7ff4a81369d0 0x7ff4a8136e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:24.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.825+0000 7ff4ae86e640 1 --2- 172.21.15.46:0/3671546032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4a80a3ff0 0x7ff4a80a43d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:24.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.825+0000 7ff4ae86e640 1 --2- 172.21.15.46:0/3671546032 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4a80a3c10 0x7ff4a809bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:24.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.825+0000 7ff4ae86e640 1 -- 172.21.15.46:0/3671546032 >> 172.21.15.46:0/3671546032 conn(0x7ff4a80a5d10 msgr2=0x7ff4a80a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:24.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.825+0000 7ff4ae86e640 1 -- 172.21.15.46:0/3671546032 shutdown_connections 2024-09-16T04:40:24.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.825+0000 7ff4ae86e640 1 -- 172.21.15.46:0/3671546032 wait complete. 2024-09-16T04:40:24.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.825+0000 7ff4ae86e640 10 monclient: build_initial_monmap 2024-09-16T04:40:24.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.825+0000 7ff4ae86e640 10 monclient: monmap: 2024-09-16T04:40:24.844 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:24.844 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:40:24.845 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:40:24.845 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:40:24.845 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:24.845 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:24.845 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:40:24.845 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:40:24.845 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:40:24.845 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:24.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.826+0000 7ff4ae86e640 1 Processor -- start 2024-09-16T04:40:24.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.826+0000 7ff4ae86e640 1 -- start start 2024-09-16T04:40:24.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.826+0000 7ff4ae86e640 10 monclient: init 2024-09-16T04:40:24.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.826+0000 7ff4ae86e640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:24.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.826+0000 7ff4ae86e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:40:24.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.826+0000 7ff4ae86e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4a80a3c10 0x7ff4a81cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:24.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.826+0000 7ff4ae86e640 10 monclient(hunting): picked mon.noname-a con 0x7ff4a80a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:40:24.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.826+0000 7ff4ae86e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4a80a3ff0 0x7ff4a81cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:24.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.826+0000 7ff4ae86e640 10 monclient(hunting): picked mon.noname-b con 0x7ff4a80a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:40:24.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.826+0000 7ff4ae86e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4a81369d0 0x7ff4a8123280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:24.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.826+0000 7ff4ae86e640 10 monclient(hunting): picked mon.noname-c con 0x7ff4a81369d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:24.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.826+0000 7ff4ae86e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:24.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.826+0000 7ff4ae86e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a80a4910 con 0x7ff4a80a3c10 2024-09-16T04:40:24.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.826+0000 7ff4ae86e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:24.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.826+0000 7ff4ae86e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a806e3d0 con 0x7ff4a80a3ff0 2024-09-16T04:40:24.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.826+0000 7ff4ae86e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:24.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.826+0000 7ff4ae86e640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7ff4a8127210 con 0x7ff4a81369d0 2024-09-16T04:40:24.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.826+0000 7ff4ae86e640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:24.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.826+0000 7ff4ae86e640 10 monclient(hunting): authenticate will time out at 823.241994s 2024-09-16T04:40:24.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4a80a3c10 0x7ff4a81cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:24.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4a80a3c10 0x7ff4a81cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58402/0 (socket says 172.21.15.46:58402) 2024-09-16T04:40:24.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a7fff640 1 -- 172.21.15.46:0/2949720447 learned_addr learned my addr 172.21.15.46:0/2949720447 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:24.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a7fff640 10 monclient(hunting): get_auth_request con 0x7ff4a80a3c10 auth_method 0 2024-09-16T04:40:24.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:24.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:24.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:24.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4ace0b640 1 --2- 172.21.15.46:0/2949720447 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4a81369d0 0x7ff4a8123280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:24.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4ace0b640 10 monclient(hunting): get_auth_request con 0x7ff4a81369d0 auth_method 0 2024-09-16T04:40:24.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4ace0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:24.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4ace0b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:24.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4ace0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:24.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:24.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:24.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:24.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4ace0b640 1 -- 172.21.15.46:0/2949720447 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4a81369d0 msgr2=0x7ff4a8123280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:40:24.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4ace0b640 1 -- 172.21.15.46:0/2949720447 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4a81369d0 msgr2=0x7ff4a8123280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:40:24.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4ace0b640 1 --2- 172.21.15.46:0/2949720447 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4a81369d0 0x7ff4a8123280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:24.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4ace0b640 1 --2- 172.21.15.46:0/2949720447 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4a81369d0 0x7ff4a8123280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:24.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a77fe640 1 --2- 172.21.15.46:0/2949720447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4a80a3ff0 0x7ff4a81cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:24.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a77fe640 10 monclient(hunting): get_auth_request con 0x7ff4a80a3ff0 auth_method 0 2024-09-16T04:40:24.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:24.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:24.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:24.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a7fff640 10 monclient(hunting): handle_auth_done global_id 4875 payload 824 2024-09-16T04:40:24.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a7fff640 1 -- 172.21.15.46:0/2949720447 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4a81369d0 msgr2=0x7ff4a8123280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:40:24.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a7fff640 1 --2- 172.21.15.46:0/2949720447 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4a81369d0 0x7ff4a8123280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:24.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a7fff640 1 -- 172.21.15.46:0/2949720447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4a80a3ff0 msgr2=0x7ff4a81cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:24.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a7fff640 1 --2- 172.21.15.46:0/2949720447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4a80a3ff0 0x7ff4a81cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:24.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:40:24.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a7fff640 1 monclient: found mon.noname-a 2024-09-16T04:40:24.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:24.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:40:24.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a7fff640 1 -- 172.21.15.46:0/2949720447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a8126c50 con 0x7ff4a80a3c10 2024-09-16T04:40:24.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a7fff640 1 --2- 172.21.15.46:0/2949720447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4a80a3c10 0x7ff4a81cbee0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff49800e810 tx=0x7ff49800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:24.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a4ff9640 1 -- 172.21.15.46:0/2949720447 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff498017070 con 0x7ff4a80a3c10 2024-09-16T04:40:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:40:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a4ff9640 10 monclient: dump: 2024-09-16T04:40:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:24.851 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:24.851 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:24.851 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:24.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a4ff9640 10 monclient: _finish_auth 0 2024-09-16T04:40:24.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:40:24.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:24.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4ae86e640 5 monclient: authenticate success, global_id 4875 2024-09-16T04:40:24.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a4ff9640 1 -- 172.21.15.46:0/2949720447 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff498004020 con 0x7ff4a80a3c10 2024-09-16T04:40:24.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4ae86e640 10 monclient: _renew_subs 2024-09-16T04:40:24.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4ae86e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:24.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4ae86e640 1 -- 172.21.15.46:0/2949720447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4a8125a80 con 0x7ff4a80a3c10 2024-09-16T04:40:24.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:24.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a4ff9640 1 -- 172.21.15.46:0/2949720447 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff49800f040 con 0x7ff4a80a3c10 2024-09-16T04:40:24.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:24.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:24.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a4ff9640 10 monclient: dump: 2024-09-16T04:40:24.852 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:24.852 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:24.852 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:24.852 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:24.852 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:24.852 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:24.852 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:24.852 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:24.852 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:24.852 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:24.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4ae86e640 10 monclient: _renew_subs 2024-09-16T04:40:24.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4ae86e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:24.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4ae86e640 1 -- 172.21.15.46:0/2949720447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4a80a8f30 con 0x7ff4a80a3c10 2024-09-16T04:40:24.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.827+0000 7ff4a67fc640 4 set_mon_vals no callback set 2024-09-16T04:40:24.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.828+0000 7ff4a4ff9640 1 -- 172.21.15.46:0/2949720447 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7ff49800c6d0 con 0x7ff4a80a3c10 2024-09-16T04:40:24.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.828+0000 7ff4ae86e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:24.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.828+0000 7ff4ae86e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:24.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.828+0000 7ff4ae86e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:24.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.828+0000 7ff4ae86e640 1 -- 172.21.15.46:0/2949720447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4a80a64f0 con 0x7ff4a80a3c10 2024-09-16T04:40:24.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.828+0000 7ff4a4ff9640 1 --2- 172.21.15.46:0/2949720447 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff4940383b0 0x7ff49403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:24.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.829+0000 7ff4a4ff9640 1 -- 172.21.15.46:0/2949720447 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 4738+0+0 (secure 0 0 0) 0x7ff49804b2d0 con 0x7ff4a80a3c10 2024-09-16T04:40:24.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.829+0000 7ff4a77fe640 1 --2- 172.21.15.46:0/2949720447 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff4940383b0 0x7ff49403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:24.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.829+0000 7ff4a77fe640 10 monclient: get_auth_request con 0x7ff4940383b0 auth_method 0 2024-09-16T04:40:24.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.829+0000 7ff4a77fe640 1 --2- 172.21.15.46:0/2949720447 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff4940383b0 0x7ff49403a870 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff49c034000 tx=0x7ff49c02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:24.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.834+0000 7ff4a4ff9640 1 -- 172.21.15.46:0/2949720447 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff4a80a64f0 con 0x7ff4a80a3c10 2024-09-16T04:40:24.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.834+0000 7ff4a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:24.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:24.834+0000 7ff4a4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:40:25.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.084+0000 7ff4ae86e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:40:25.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.084+0000 7ff4ae86e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:40:25.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.084+0000 7ff4ae86e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:25.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.084+0000 7ff4ae86e640 1 -- 172.21.15.46:0/2949720447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff4a809bcf0 con 0x7ff4a80a3c10 2024-09-16T04:40:25.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.084+0000 7ff4a4ff9640 1 -- 172.21.15.46:0/2949720447 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7ff49804b0d0 con 0x7ff4a80a3c10 2024-09-16T04:40:25.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.084+0000 7ff4a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:40:25.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.084+0000 7ff4a4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:40:25.086 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:40:25.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.086+0000 7ff4ae86e640 1 -- 172.21.15.46:0/2949720447 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff4940383b0 msgr2=0x7ff49403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:25.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.086+0000 7ff4ae86e640 1 --2- 172.21.15.46:0/2949720447 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff4940383b0 0x7ff49403a870 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff49c034000 tx=0x7ff49c02e000 comp rx=0 tx=0).stop 2024-09-16T04:40:25.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.086+0000 7ff4ae86e640 10 monclient: shutdown 2024-09-16T04:40:25.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.086+0000 7ff4ae86e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:25.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.086+0000 7ff4ae86e640 1 -- 172.21.15.46:0/2949720447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4a80a3c10 msgr2=0x7ff4a81cbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:25.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.086+0000 7ff4ae86e640 1 --2- 172.21.15.46:0/2949720447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4a80a3c10 0x7ff4a81cbee0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff49800e810 tx=0x7ff49800ecd0 comp rx=0 tx=0).stop 2024-09-16T04:40:25.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.086+0000 7ff4ae86e640 1 -- 172.21.15.46:0/2949720447 shutdown_connections 2024-09-16T04:40:25.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.086+0000 7ff4ae86e640 1 --2- 172.21.15.46:0/2949720447 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4a81369d0 0x7ff4a8123280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:25.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.086+0000 7ff4ae86e640 1 --2- 172.21.15.46:0/2949720447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4a80a3ff0 0x7ff4a81cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:25.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.086+0000 7ff4ae86e640 1 --2- 172.21.15.46:0/2949720447 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff4940383b0 0x7ff49403a870 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:25.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.086+0000 7ff4ae86e640 1 --2- 172.21.15.46:0/2949720447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4a80a3c10 0x7ff4a81cbee0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:25.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.086+0000 7ff4ae86e640 1 -- 172.21.15.46:0/2949720447 >> 172.21.15.46:0/2949720447 conn(0x7ff4a80a5d10 msgr2=0x7ff4a809e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:25.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.086+0000 7ff4ae86e640 1 -- 172.21.15.46:0/2949720447 shutdown_connections 2024-09-16T04:40:25.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.086+0000 7ff4ae86e640 1 -- 172.21.15.46:0/2949720447 wait complete. 2024-09-16T04:40:25.098 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:40:25.099 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:40:25.099 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:40:25.123 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36158]: starting. 2024-09-16T04:40:25.123 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36158]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-36156 2024-09-16T04:40:25.123 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36158]: shutting down. 2024-09-16T04:40:25.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.202+0000 7f53f42ac640 10 monclient: get_monmap_and_config 2024-09-16T04:40:25.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.202+0000 7f53f42ac640 10 monclient: build_initial_monmap 2024-09-16T04:40:25.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.202+0000 7f53f42ac640 10 monclient: monmap: 2024-09-16T04:40:25.204 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:25.204 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:25.204 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:40:25.202873+0000 2024-09-16T04:40:25.204 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:40:25.202873+0000 2024-09-16T04:40:25.204 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:25.204 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:25.204 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:40:25.204 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:40:25.204 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:40:25.204 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:40:25.204 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:40:25.205 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:25.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.202+0000 7f53f42ac640 1 Processor -- start 2024-09-16T04:40:25.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f42ac640 1 -- start start 2024-09-16T04:40:25.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f42ac640 10 monclient: init 2024-09-16T04:40:25.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f42ac640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f42ac640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-16T04:40:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f42ac640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f53ec0a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:40:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f42ac640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53ec0a3ff0 0x7f53ec0a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f42ac640 10 monclient(hunting): picked mon.noname-a con 0x7f53ec0a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:40:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f42ac640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53ec13be90 0x7f53ec13e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f42ac640 10 monclient(hunting): picked mon.noname-b con 0x7f53ec13be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f42ac640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53ec09b940 con 0x7f53ec0a3c10 2024-09-16T04:40:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f42ac640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f42ac640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f53ec09ab00 con 0x7f53ec0a3ff0 2024-09-16T04:40:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f42ac640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f42ac640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f53ec09b030 con 0x7f53ec13be90 2024-09-16T04:40:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f42ac640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f42ac640 10 monclient(hunting): authenticate will time out at 823.618735s 2024-09-16T04:40:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f1847640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53ec0a3ff0 0x7f53ec0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f2849640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53ec13be90 0x7f53ec13e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f2849640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53ec13be90 0x7f53ec13e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:43954/0 (socket says 172.21.15.46:43954) 2024-09-16T04:40:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f2849640 1 -- 172.21.15.46:0/2527114243 learned_addr learned my addr 172.21.15.46:0/2527114243 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f1847640 10 monclient(hunting): get_auth_request con 0x7f53ec0a3ff0 auth_method 0 2024-09-16T04:40:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f1847640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f1847640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f1847640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f2849640 10 monclient(hunting): get_auth_request con 0x7f53ec13be90 auth_method 0 2024-09-16T04:40:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f2849640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f2849640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.203+0000 7f53f2849640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.204+0000 7f53f2849640 1 -- 172.21.15.46:0/2527114243 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53ec13be90 msgr2=0x7f53ec13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:40:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.204+0000 7f53f2849640 1 -- 172.21.15.46:0/2527114243 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53ec13be90 msgr2=0x7f53ec13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:40:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.204+0000 7f53f2849640 1 --2- 172.21.15.46:0/2527114243 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53ec13be90 0x7f53ec13e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:25.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.204+0000 7f53f1847640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:25.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.204+0000 7f53f1847640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:25.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.204+0000 7f53f1847640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:25.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.204+0000 7f53f2849640 1 --2- 172.21.15.46:0/2527114243 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53ec13be90 0x7f53ec13e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:25.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.204+0000 7f53f1847640 10 monclient(hunting): handle_auth_done global_id 4902 payload 274 2024-09-16T04:40:25.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.204+0000 7f53f1847640 1 -- 172.21.15.46:0/2527114243 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53ec13be90 msgr2=0x7f53ec13e280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:40:25.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.204+0000 7f53f1847640 1 --2- 172.21.15.46:0/2527114243 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53ec13be90 0x7f53ec13e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:25.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.204+0000 7f53f1847640 1 -- 172.21.15.46:0/2527114243 >> v1:172.21.15.46:6789/0 conn(0x7f53ec0a3c10 legacy=0x7f53ec09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:25.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.204+0000 7f53f1847640 10 monclient: _finish_hunting 0 2024-09-16T04:40:25.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.204+0000 7f53f1847640 1 monclient: found mon.noname-a 2024-09-16T04:40:25.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.204+0000 7f53f1847640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:25.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.204+0000 7f53f1847640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:40:25.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.204+0000 7f53f1046640 1 -- 172.21.15.46:0/2527114243 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4191858275 0 0) 0x7f53ec09b940 con 0x7f53ec0a3c10 2024-09-16T04:40:25.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.204+0000 7f53f1847640 1 -- 172.21.15.46:0/2527114243 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ec09b310 con 0x7f53ec0a3ff0 2024-09-16T04:40:25.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.204+0000 7f53f1046640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:40:25.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.204+0000 7f53f1847640 1 --2- 172.21.15.46:0/2527114243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec0a3ff0 0x7f53ec0a43d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f53dc0097a0 tx=0x7f53dc02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19a9f47e633dc40b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:25.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f1046640 1 -- 172.21.15.46:0/2527114243 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53dc037070 con 0x7f53ec0a3ff0 2024-09-16T04:40:25.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f1046640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:25.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f1046640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:40:25.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f1046640 10 monclient: dump: 2024-09-16T04:40:25.217 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:25.217 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:25.217 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:25.217 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:25.217 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:25.217 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:25.217 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:25.217 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:25.218 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:25.218 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:25.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f1046640 10 monclient: _finish_auth 0 2024-09-16T04:40:25.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f1046640 10 monclient: _check_auth_tickets 2024-09-16T04:40:25.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f1046640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:25.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f1046640 1 -- 172.21.15.46:0/2527114243 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53dc02ae50 con 0x7f53ec0a3ff0 2024-09-16T04:40:25.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f42ac640 5 monclient: authenticate success, global_id 4902 2024-09-16T04:40:25.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f42ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:40:25.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f1046640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:25.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f42ac640 10 monclient: get_monmap_and_config success 2024-09-16T04:40:25.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f1046640 1 -- 172.21.15.46:0/2527114243 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53dc032660 con 0x7f53ec0a3ff0 2024-09-16T04:40:25.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f1046640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:25.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f42ac640 4 set_mon_vals no callback set 2024-09-16T04:40:25.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f1046640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:25.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f1046640 10 monclient: dump: 2024-09-16T04:40:25.219 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:25.219 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:25.219 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:25.219 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:25.219 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:25.219 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:25.219 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:25.219 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:25.219 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:25.219 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:25.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f42ac640 10 monclient: shutdown 2024-09-16T04:40:25.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f42ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:25.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f42ac640 1 -- 172.21.15.46:0/2527114243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec0a3ff0 msgr2=0x7f53ec0a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:25.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f42ac640 1 --2- 172.21.15.46:0/2527114243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec0a3ff0 0x7f53ec0a43d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f53dc0097a0 tx=0x7f53dc02a100 comp rx=0 tx=0).stop 2024-09-16T04:40:25.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f42ac640 1 -- 172.21.15.46:0/2527114243 shutdown_connections 2024-09-16T04:40:25.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f42ac640 1 --2- 172.21.15.46:0/2527114243 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53ec13be90 0x7f53ec13e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:25.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f42ac640 1 --2- 172.21.15.46:0/2527114243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec0a3ff0 0x7f53ec0a43d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:25.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f42ac640 1 -- 172.21.15.46:0/2527114243 >> 172.21.15.46:0/2527114243 conn(0x7f53ec0a6340 msgr2=0x7f53ec06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:25.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f42ac640 1 -- 172.21.15.46:0/2527114243 shutdown_connections 2024-09-16T04:40:25.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.205+0000 7f53f42ac640 1 -- 172.21.15.46:0/2527114243 wait complete. 2024-09-16T04:40:25.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.206+0000 7f53f42ac640 10 monclient: build_initial_monmap 2024-09-16T04:40:25.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.206+0000 7f53f42ac640 10 monclient: monmap: 2024-09-16T04:40:25.220 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:25.220 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:40:25.220 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:40:25.220 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:40:25.221 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:25.221 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:25.221 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:40:25.221 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:40:25.221 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:40:25.221 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:25.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.206+0000 7f53f42ac640 1 Processor -- start 2024-09-16T04:40:25.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.206+0000 7f53f42ac640 1 -- start start 2024-09-16T04:40:25.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.206+0000 7f53f42ac640 10 monclient: init 2024-09-16T04:40:25.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f42ac640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:25.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f42ac640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:40:25.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f42ac640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53ec0a3c10 0x7f53ec133cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:25.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f42ac640 10 monclient(hunting): picked mon.noname-c con 0x7f53ec0a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:25.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f42ac640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec0a3ff0 0x7f53ec134230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:25.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f42ac640 10 monclient(hunting): picked mon.noname-a con 0x7f53ec0a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:40:25.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f42ac640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ec13be90 0x7f53ec1cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:25.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f42ac640 10 monclient(hunting): picked mon.noname-b con 0x7f53ec13be90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:40:25.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f42ac640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:25.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f42ac640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f53ec0a6b40 con 0x7f53ec0a3ff0 2024-09-16T04:40:25.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f42ac640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:25.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f42ac640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f53ec0a69c0 con 0x7f53ec13be90 2024-09-16T04:40:25.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f42ac640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:25.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f42ac640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f53ec13e7c0 con 0x7f53ec0a3c10 2024-09-16T04:40:25.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f42ac640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:25.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f42ac640 10 monclient(hunting): authenticate will time out at 823.622288s 2024-09-16T04:40:25.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f2048640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53ec0a3c10 0x7f53ec133cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:25.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f1847640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec0a3ff0 0x7f53ec134230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:25.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f1847640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec0a3ff0 0x7f53ec134230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58426/0 (socket says 172.21.15.46:58426) 2024-09-16T04:40:25.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f1847640 1 -- 172.21.15.46:0/1157185587 learned_addr learned my addr 172.21.15.46:0/1157185587 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:25.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f1847640 10 monclient(hunting): get_auth_request con 0x7f53ec0a3ff0 auth_method 0 2024-09-16T04:40:25.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f1847640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:25.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f1847640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:25.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f1847640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:25.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f2048640 10 monclient(hunting): get_auth_request con 0x7f53ec0a3c10 auth_method 0 2024-09-16T04:40:25.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f2048640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:25.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f2048640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:25.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f2048640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:25.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f1847640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:25.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f1847640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:25.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f1847640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:25.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f2048640 1 -- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53ec0a3c10 msgr2=0x7f53ec133cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:40:25.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f2048640 1 -- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53ec0a3c10 msgr2=0x7f53ec133cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:40:25.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f2048640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53ec0a3c10 0x7f53ec133cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:25.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f2048640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53ec0a3c10 0x7f53ec133cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:25.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f2849640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ec13be90 0x7f53ec1cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:25.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f2849640 10 monclient(hunting): get_auth_request con 0x7f53ec13be90 auth_method 0 2024-09-16T04:40:25.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f2849640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:25.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f2849640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:25.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f2849640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:25.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f1847640 10 monclient(hunting): handle_auth_done global_id 4908 payload 824 2024-09-16T04:40:25.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f1847640 1 -- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53ec0a3c10 msgr2=0x7f53ec133cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:40:25.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f1847640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53ec0a3c10 0x7f53ec133cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:25.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f1847640 1 -- 172.21.15.46:0/1157185587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ec13be90 msgr2=0x7f53ec1cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:25.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f1847640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ec13be90 0x7f53ec1cb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:25.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f1847640 10 monclient: _finish_hunting 0 2024-09-16T04:40:25.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f1847640 1 monclient: found mon.noname-a 2024-09-16T04:40:25.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f1847640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:25.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f1847640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:40:25.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f1847640 1 -- 172.21.15.46:0/1157185587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ec139cf0 con 0x7f53ec0a3ff0 2024-09-16T04:40:25.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53f1847640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec0a3ff0 0x7f53ec134230 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f53dc000c00 tx=0x7f53dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:25.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.207+0000 7f53e2ffd640 1 -- 172.21.15.46:0/1157185587 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53dc041070 con 0x7f53ec0a3ff0 2024-09-16T04:40:25.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:25.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53e2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:40:25.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53e2ffd640 10 monclient: dump: 2024-09-16T04:40:25.225 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:25.225 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:25.226 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:25.226 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:25.226 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:25.226 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:25.226 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:25.226 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:25.226 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:25.226 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:25.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53e2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:40:25.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53e2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:40:25.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:25.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53e2ffd640 1 -- 172.21.15.46:0/1157185587 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53dc03c670 con 0x7f53ec0a3ff0 2024-09-16T04:40:25.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53f42ac640 5 monclient: authenticate success, global_id 4908 2024-09-16T04:40:25.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53f42ac640 10 monclient: _renew_subs 2024-09-16T04:40:25.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53f42ac640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:25.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53f42ac640 1 -- 172.21.15.46:0/1157185587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53ec13a590 con 0x7f53ec0a3ff0 2024-09-16T04:40:25.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53e2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:25.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53e2ffd640 1 -- 172.21.15.46:0/1157185587 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53dc037040 con 0x7f53ec0a3ff0 2024-09-16T04:40:25.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:25.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:25.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53e2ffd640 10 monclient: dump: 2024-09-16T04:40:25.227 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:25.227 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:25.227 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:25.227 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:25.227 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:25.227 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:25.227 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:25.228 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:25.228 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:25.228 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:25.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53f42ac640 10 monclient: _renew_subs 2024-09-16T04:40:25.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53e3fff640 4 set_mon_vals no callback set 2024-09-16T04:40:25.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53f42ac640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:25.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53f42ac640 1 -- 172.21.15.46:0/1157185587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53ec13a2b0 con 0x7f53ec0a3ff0 2024-09-16T04:40:25.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53f42ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:25.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53f42ac640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec1cbdd0 0x7f53ec1cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:25.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53f42ac640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:25.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53f42ac640 1 -- 172.21.15.46:0/1157185587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f53ec0a0400 con 0x7f53ec1cbdd0 2024-09-16T04:40:25.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.208+0000 7f53f2048640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec1cbdd0 0x7f53ec1cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:25.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.209+0000 7f53f2048640 10 monclient: get_auth_request con 0x7f53ec1cbdd0 auth_method 0 2024-09-16T04:40:25.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.209+0000 7f53f2048640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:25.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.209+0000 7f53f2048640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:25.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.209+0000 7f53f2048640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:25.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.209+0000 7f53f2048640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:25.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.209+0000 7f53f2048640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:25.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.209+0000 7f53e2ffd640 1 -- 172.21.15.46:0/1157185587 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f53dc044070 con 0x7f53ec0a3ff0 2024-09-16T04:40:25.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.209+0000 7f53f2048640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:25.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.209+0000 7f53f2048640 10 monclient(hunting): handle_auth_done global_id 4911 payload 824 2024-09-16T04:40:25.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.209+0000 7f53f2048640 1 -- 172.21.15.46:0/1157185587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53ec1cc6f0 con 0x7f53ec1cbdd0 2024-09-16T04:40:25.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.209+0000 7f53e2ffd640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f53d00383b0 0x7f53d003a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:25.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.209+0000 7f53e2ffd640 1 -- 172.21.15.46:0/1157185587 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 4346+0+0 (secure 0 0 0) 0x7f53dc06bd00 con 0x7f53ec0a3ff0 2024-09-16T04:40:25.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.209+0000 7f53f2048640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec1cbdd0 0x7f53ec1cc1b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f53e8000f60 tx=0x7f53e800e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:25.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.209+0000 7f53f2849640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f53d00383b0 0x7f53d003a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:25.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.209+0000 7f53e2ffd640 1 -- 172.21.15.46:0/1157185587 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e80052e0 con 0x7f53ec1cbdd0 2024-09-16T04:40:25.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.209+0000 7f53e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:25.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.209+0000 7f53e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:25.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.209+0000 7f53e2ffd640 10 monclient: dump: 2024-09-16T04:40:25.230 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:25.230 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:25.230 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:25.230 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:25.230 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:25.230 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:25.230 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:25.230 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:25.230 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:25.230 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:25.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.209+0000 7f53f2849640 10 monclient: get_auth_request con 0x7f53d00383b0 auth_method 0 2024-09-16T04:40:25.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.210+0000 7f53e2ffd640 1 -- 172.21.15.46:0/1157185587 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f53e8018070 con 0x7f53ec1cbdd0 2024-09-16T04:40:25.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.210+0000 7f53e2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:25.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.210+0000 7f53e2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:40:25.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.210+0000 7f53e2ffd640 1 -- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec1cbdd0 msgr2=0x7f53ec1cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:25.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.210+0000 7f53e2ffd640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec1cbdd0 0x7f53ec1cc1b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f53e8000f60 tx=0x7f53e800e960 comp rx=0 tx=0).stop 2024-09-16T04:40:25.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.210+0000 7f53e2ffd640 1 -- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec1cbdd0 msgr2=0x7f53ec1cc1b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:40:25.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.210+0000 7f53e2ffd640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec1cbdd0 0x7f53ec1cc1b0 secure :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f53e8000f60 tx=0x7f53e800e960 comp rx=0 tx=0).stop 2024-09-16T04:40:25.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.210+0000 7f53f2849640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f53d00383b0 0x7f53d003a870 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f53e4009440 tx=0x7f53e400b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.216+0000 7f53f42ac640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:40:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.216+0000 7f53f42ac640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec0a05b0 0x7f53ec0a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.216+0000 7f53f42ac640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.216+0000 7f53f42ac640 1 -- 172.21.15.46:0/1157185587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f53ec0a43d0 con 0x7f53ec0a05b0 2024-09-16T04:40:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.216+0000 7f53f2048640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec0a05b0 0x7f53ec0a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.216+0000 7f53f2048640 10 monclient: get_auth_request con 0x7f53ec0a05b0 auth_method 0 2024-09-16T04:40:25.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.216+0000 7f53f2048640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:25.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.216+0000 7f53f2048640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:25.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.216+0000 7f53f2048640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:25.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.216+0000 7f53f2048640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:25.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.216+0000 7f53f2048640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:25.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.216+0000 7f53f2048640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:25.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.216+0000 7f53f2048640 10 monclient(hunting): handle_auth_done global_id 4914 payload 824 2024-09-16T04:40:25.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.216+0000 7f53f2048640 1 -- 172.21.15.46:0/1157185587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f53ec138b10 con 0x7f53ec0a05b0 2024-09-16T04:40:25.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.216+0000 7f53f2048640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec0a05b0 0x7f53ec0a09c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f53e8000f60 tx=0x7f53e8001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:25.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.216+0000 7f53e2ffd640 1 -- 172.21.15.46:0/1157185587 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e80035f0 con 0x7f53ec0a05b0 2024-09-16T04:40:25.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.216+0000 7f53e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:25.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.216+0000 7f53e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:25.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.216+0000 7f53e2ffd640 10 monclient: dump: 2024-09-16T04:40:25.235 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:25.235 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:25.235 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:25.235 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:25.235 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:25.235 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:25.235 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:25.235 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:25.235 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:25.235 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:25.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.217+0000 7f53e2ffd640 1 -- 172.21.15.46:0/1157185587 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7f53e80035f0 con 0x7f53ec0a05b0 2024-09-16T04:40:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.217+0000 7f53e2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:40:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.217+0000 7f53e2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:40:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.217+0000 7f53e2ffd640 1 -- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec0a05b0 msgr2=0x7f53ec0a09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.217+0000 7f53e2ffd640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec0a05b0 0x7f53ec0a09c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f53e8000f60 tx=0x7f53e8001080 comp rx=0 tx=0).stop 2024-09-16T04:40:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.217+0000 7f53e2ffd640 1 -- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec0a05b0 msgr2=0x7f53ec0a09c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:40:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.217+0000 7f53e2ffd640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec0a05b0 0x7f53ec0a09c0 secure :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f53e8000f60 tx=0x7f53e8001080 comp rx=0 tx=0).stop 2024-09-16T04:40:25.236 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:40:25.236 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:40:25.237 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:40:25.237 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:40:25.237 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 20, 2024-09-16T04:40:25.237 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 111154, 2024-09-16T04:40:25.237 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:40:25.237 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:40:25.237 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:40:25.237 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:40:25.237 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:25.237 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:40:25.237 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:40:25.237 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:40:25.237 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:40:25.238 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:25.238 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:25.238 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:25.238 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:25.238 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:25.238 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:25.238 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:25.238 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:25.238 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:25.238 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:25.238 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:25.238 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:25.239 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:40:25.239 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:40:25.239 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:25.239 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:25.239 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:25.239 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:25.239 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:25.239 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:25.239 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:25.239 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:25.239 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:25.240 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:25.240 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:25.240 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:25.240 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:25.240 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:40:25.240 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:40:25.240 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:40:25.240 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:40:25.240 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:40:25.240 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:40:25.240 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:40:25.241 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:40:25.241 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:40:25.241 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:40:25.241 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:40:25.241 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:40:25.241 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:40:25.241 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:40:25.241 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:40:25.241 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:40:25.241 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:40:25.241 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:25.241 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:25.241 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:25.242 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:25.242 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:25.242 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:25.242 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:25.242 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:25.242 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:25.242 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:25.242 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:25.242 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:25.242 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:40:25.242 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:25.242 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:40:25.243 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:25.243 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:40:25.243 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:40:25.243 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:25.243 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:25.243 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:25.243 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:25.243 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:40:25.243 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:25.243 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:25.243 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:25.244 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:25.244 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:40:25.244 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:25.244 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:25.244 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:25.244 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:25.244 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:40:25.244 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:40:25.244 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:25.244 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:25.245 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:25.245 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:25.245 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:25.245 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:40:25.245 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:40:25.245 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:25.245 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:25.245 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:25.245 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:25.245 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:40:25.245 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:25.245 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:25.246 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:25.246 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:25.246 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:40:25.246 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:25.246 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:25.246 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:25.246 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:25.246 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:40:25.246 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:40:25.246 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:25.246 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:25.246 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:25.246 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:25.246 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:25.247 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:40:25.247 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:40:25.247 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:25.247 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:25.247 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:25.247 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:25.247 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:40:25.247 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:25.247 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:25.247 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:25.247 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:25.247 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:40:25.247 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:25.248 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:25.248 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:25.248 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:25.248 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:40:25.248 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:40:25.248 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:25.248 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:25.248 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:25.248 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:25.248 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:25.248 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:25.248 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:40:25.248 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:40:25.249 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:25.249 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:25.249 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:25.249 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:40:25.249 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:25.249 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:25.249 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:40:25.249 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:25.249 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:25.249 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:25.249 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:40:25.249 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:25.249 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:25.249 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:40:25.250 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:25.253 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:25.253 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:25.253 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4 2024-09-16T04:40:25.253 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:25.253 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:25.253 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:25.253 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:40:25.253 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:40:25.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.217+0000 7f53f42ac640 1 -- 172.21.15.46:0/1157185587 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f53d00383b0 msgr2=0x7f53d003a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:25.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.217+0000 7f53f42ac640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f53d00383b0 0x7f53d003a870 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f53e4009440 tx=0x7f53e400b040 comp rx=0 tx=0).stop 2024-09-16T04:40:25.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.217+0000 7f53f42ac640 10 monclient: shutdown 2024-09-16T04:40:25.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.217+0000 7f53f42ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:25.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.217+0000 7f53f42ac640 1 -- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec0a3ff0 msgr2=0x7f53ec134230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:25.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.217+0000 7f53f42ac640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec0a3ff0 0x7f53ec134230 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f53dc000c00 tx=0x7f53dc0046c0 comp rx=0 tx=0).stop 2024-09-16T04:40:25.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.217+0000 7f53f2849640 1 -- 172.21.15.46:0/1157185587 reap_dead start 2024-09-16T04:40:25.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.217+0000 7f53f42ac640 1 -- 172.21.15.46:0/1157185587 shutdown_connections 2024-09-16T04:40:25.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.217+0000 7f53f42ac640 1 --2- 172.21.15.46:0/1157185587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec0a3ff0 0x7f53ec134230 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:25.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.217+0000 7f53f42ac640 1 -- 172.21.15.46:0/1157185587 >> 172.21.15.46:0/1157185587 conn(0x7f53ec0a6340 msgr2=0x7f53ec0a7110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:25.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.217+0000 7f53f42ac640 1 -- 172.21.15.46:0/1157185587 shutdown_connections 2024-09-16T04:40:25.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.217+0000 7f53f42ac640 1 -- 172.21.15.46:0/1157185587 wait complete. 2024-09-16T04:40:25.255 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:40:25.256 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:40:25.256 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:40:25.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.357+0000 7fa59bc27640 10 monclient: get_monmap_and_config 2024-09-16T04:40:25.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.357+0000 7fa59bc27640 10 monclient: build_initial_monmap 2024-09-16T04:40:25.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.357+0000 7fa59bc27640 10 monclient: monmap: 2024-09-16T04:40:25.360 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:25.360 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:25.360 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:40:25.357510+0000 2024-09-16T04:40:25.360 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:40:25.357510+0000 2024-09-16T04:40:25.360 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:25.360 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:25.360 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:40:25.360 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:40:25.360 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:40:25.360 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:40:25.360 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:40:25.361 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:25.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.357+0000 7fa59bc27640 1 Processor -- start 2024-09-16T04:40:25.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.357+0000 7fa59bc27640 1 -- start start 2024-09-16T04:40:25.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.357+0000 7fa59bc27640 10 monclient: init 2024-09-16T04:40:25.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa59bc27640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:25.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa59bc27640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-16T04:40:25.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa59bc27640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa5940a3c10 0x7fa59409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:25.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa59bc27640 10 monclient(hunting): picked mon.noname-b con 0x7fa5940a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:25.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa59bc27640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa5940a3ff0 0x7fa5940a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:25.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa59bc27640 10 monclient(hunting): picked mon.noname-c con 0x7fa5940a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:40:25.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa59bc27640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa5941369d0 addr v1:172.21.15.114:6789/0 2024-09-16T04:40:25.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa59bc27640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa59409b940 con 0x7fa5941369d0 2024-09-16T04:40:25.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa59bc27640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:25.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa59bc27640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fa59409ab00 con 0x7fa5940a3ff0 2024-09-16T04:40:25.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa59bc27640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:25.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa59bc27640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fa59409b030 con 0x7fa5940a3c10 2024-09-16T04:40:25.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa59bc27640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:25.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa59bc27640 10 monclient(hunting): authenticate will time out at 823.773458s 2024-09-16T04:40:25.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa5999c3640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa5940a3c10 0x7fa59409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:25.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa5999c3640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa5940a3c10 0x7fa59409bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:43962/0 (socket says 172.21.15.46:43962) 2024-09-16T04:40:25.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa5999c3640 1 -- 172.21.15.46:0/1834876958 learned_addr learned my addr 172.21.15.46:0/1834876958 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:25.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa5999c3640 10 monclient(hunting): get_auth_request con 0x7fa5940a3c10 auth_method 0 2024-09-16T04:40:25.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa5999c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:25.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa5999c3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:25.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa5999c3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:25.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa5999c3640 1 -- 172.21.15.46:0/1834876958 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa5940a3c10 msgr2=0x7fa59409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:40:25.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa5999c3640 1 -- 172.21.15.46:0/1834876958 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa5940a3c10 msgr2=0x7fa59409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:40:25.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa5999c3640 1 --2- 172.21.15.46:0/1834876958 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa5940a3c10 0x7fa59409bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:25.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa5991c2640 1 --2- 172.21.15.46:0/1834876958 >> v2:172.21.15.114:3300/0 conn(0x7fa5940a3ff0 0x7fa5940a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:25.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa5999c3640 1 --2- 172.21.15.46:0/1834876958 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa5940a3c10 0x7fa59409bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:25.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa5991c2640 10 monclient(hunting): get_auth_request con 0x7fa5940a3ff0 auth_method 0 2024-09-16T04:40:25.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa5991c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:25.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa5991c2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:25.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa5991c2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:25.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa5991c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:25.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa5991c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:25.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.358+0000 7fa5991c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:25.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.359+0000 7fa5991c2640 10 monclient(hunting): handle_auth_done global_id 4864 payload 274 2024-09-16T04:40:25.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.359+0000 7fa5991c2640 1 -- 172.21.15.46:0/1834876958 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa5940a3c10 msgr2=0x7fa59409bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:40:25.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.359+0000 7fa5991c2640 1 --2- 172.21.15.46:0/1834876958 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa5940a3c10 0x7fa59409bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:25.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.359+0000 7fa5991c2640 1 -- 172.21.15.46:0/1834876958 >> v1:172.21.15.114:6789/0 conn(0x7fa5941369d0 legacy=0x7fa594136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:25.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.359+0000 7fa5991c2640 10 monclient: _finish_hunting 0 2024-09-16T04:40:25.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.359+0000 7fa5991c2640 1 monclient: found mon.noname-c 2024-09-16T04:40:25.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.359+0000 7fa5991c2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:25.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.359+0000 7fa5991c2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:40:25.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.359+0000 7fa5991c2640 1 -- 172.21.15.46:0/1834876958 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa59409b310 con 0x7fa5940a3ff0 2024-09-16T04:40:25.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.359+0000 7fa5989c1640 1 -- 172.21.15.46:0/1834876958 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 92056443 0 0) 0x7fa59409b940 con 0x7fa5941369d0 2024-09-16T04:40:25.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.359+0000 7fa5989c1640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:40:25.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.359+0000 7fa5991c2640 1 --2- 172.21.15.46:0/1834876958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5940a3ff0 0x7fa5940a43d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa5840097a0 tx=0x7fa58402a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60e111092cc83338 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:25.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa5989c1640 1 -- 172.21.15.46:0/1834876958 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa584037070 con 0x7fa5940a3ff0 2024-09-16T04:40:25.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa5989c1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:25.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa5989c1640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:40:25.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa5989c1640 10 monclient: dump: 2024-09-16T04:40:25.378 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:25.378 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:25.378 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:25.378 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:25.378 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:25.378 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:25.378 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:25.378 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:25.378 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:25.378 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:25.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa5989c1640 10 monclient: _finish_auth 0 2024-09-16T04:40:25.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa5989c1640 10 monclient: _check_auth_tickets 2024-09-16T04:40:25.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa5989c1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:25.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa5989c1640 1 -- 172.21.15.46:0/1834876958 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa58402adb0 con 0x7fa5940a3ff0 2024-09-16T04:40:25.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa59bc27640 5 monclient: authenticate success, global_id 4864 2024-09-16T04:40:25.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa59bc27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:40:25.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa5989c1640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:25.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa5989c1640 1 -- 172.21.15.46:0/1834876958 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa584032660 con 0x7fa5940a3ff0 2024-09-16T04:40:25.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa59bc27640 10 monclient: get_monmap_and_config success 2024-09-16T04:40:25.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa59bc27640 4 set_mon_vals no callback set 2024-09-16T04:40:25.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa5989c1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:25.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa5989c1640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:40:25.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa5989c1640 10 monclient: dump: 2024-09-16T04:40:25.379 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:25.380 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:25.380 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:25.380 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:25.380 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:25.380 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:25.380 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:25.380 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:25.380 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:25.380 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:25.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa59bc27640 10 monclient: shutdown 2024-09-16T04:40:25.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa59bc27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:25.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa59bc27640 1 -- 172.21.15.46:0/1834876958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5940a3ff0 msgr2=0x7fa5940a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:25.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa59bc27640 1 --2- 172.21.15.46:0/1834876958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5940a3ff0 0x7fa5940a43d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa5840097a0 tx=0x7fa58402a060 comp rx=0 tx=0).stop 2024-09-16T04:40:25.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa59bc27640 1 -- 172.21.15.46:0/1834876958 shutdown_connections 2024-09-16T04:40:25.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa59bc27640 1 --2- 172.21.15.46:0/1834876958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5940a3ff0 0x7fa5940a43d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:25.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa59bc27640 1 --2- 172.21.15.46:0/1834876958 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa5940a3c10 0x7fa59409bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:25.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa59bc27640 1 -- 172.21.15.46:0/1834876958 >> 172.21.15.46:0/1834876958 conn(0x7fa5940a6340 msgr2=0x7fa59406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:25.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa59bc27640 1 -- 172.21.15.46:0/1834876958 shutdown_connections 2024-09-16T04:40:25.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.360+0000 7fa59bc27640 1 -- 172.21.15.46:0/1834876958 wait complete. 2024-09-16T04:40:25.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.361+0000 7fa59bc27640 10 monclient: build_initial_monmap 2024-09-16T04:40:25.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.361+0000 7fa59bc27640 10 monclient: monmap: 2024-09-16T04:40:25.381 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:25.381 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:40:25.382 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:40:25.382 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:40:25.382 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:25.382 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:25.382 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:40:25.382 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:40:25.382 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:40:25.382 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:25.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.361+0000 7fa59bc27640 1 Processor -- start 2024-09-16T04:40:25.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.361+0000 7fa59bc27640 1 -- start start 2024-09-16T04:40:25.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.361+0000 7fa59bc27640 10 monclient: init 2024-09-16T04:40:25.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59bc27640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:25.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59bc27640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:40:25.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59bc27640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa5940a3c10 0x7fa5941cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:25.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59bc27640 10 monclient(hunting): picked mon.noname-c con 0x7fa5940a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:25.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59bc27640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940a3ff0 0x7fa5941cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:25.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59bc27640 10 monclient(hunting): picked mon.noname-a con 0x7fa5940a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:40:25.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59bc27640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5941369d0 0x7fa594122ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:25.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59bc27640 10 monclient(hunting): picked mon.noname-b con 0x7fa5941369d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:40:25.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59bc27640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:25.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59bc27640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa5940a4910 con 0x7fa5940a3ff0 2024-09-16T04:40:25.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59bc27640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:25.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59bc27640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa5940a69c0 con 0x7fa5941369d0 2024-09-16T04:40:25.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59bc27640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:25.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59bc27640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fa5940a29b0 con 0x7fa5940a3c10 2024-09-16T04:40:25.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59bc27640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:25.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59bc27640 10 monclient(hunting): authenticate will time out at 823.777172s 2024-09-16T04:40:25.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5999c3640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa5940a3c10 0x7fa5941cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:25.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5991c2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940a3ff0 0x7fa5941cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:25.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5999c3640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa5940a3c10 0x7fa5941cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:43964/0 (socket says 172.21.15.46:43964) 2024-09-16T04:40:25.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5999c3640 1 -- 172.21.15.46:0/3182096672 learned_addr learned my addr 172.21.15.46:0/3182096672 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:25.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5991c2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940a3ff0 0x7fa5941cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58452/0 (socket says 172.21.15.46:58452) 2024-09-16T04:40:25.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5999c3640 10 monclient(hunting): get_auth_request con 0x7fa5940a3c10 auth_method 0 2024-09-16T04:40:25.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5999c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:25.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5999c3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:25.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5999c3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:25.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5991c2640 10 monclient(hunting): get_auth_request con 0x7fa5940a3ff0 auth_method 0 2024-09-16T04:40:25.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5991c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:25.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5991c2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:25.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5991c2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:25.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5999c3640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa5940a3c10 msgr2=0x7fa5941cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:40:25.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5991c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:25.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5999c3640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa5940a3c10 msgr2=0x7fa5941cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:40:25.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5991c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:25.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa5940a3c10 0x7fa5941cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:25.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5991c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:25.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa5940a3c10 0x7fa5941cbee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:25.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59a1c4640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5941369d0 0x7fa594122ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:25.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59a1c4640 10 monclient(hunting): get_auth_request con 0x7fa5941369d0 auth_method 0 2024-09-16T04:40:25.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59a1c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:25.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59a1c4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:25.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59a1c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:25.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59a1c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:25.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59a1c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:25.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59a1c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:25.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5991c2640 10 monclient(hunting): handle_auth_done global_id 4917 payload 824 2024-09-16T04:40:25.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5991c2640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa5940a3c10 msgr2=0x7fa5941cbee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:40:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5991c2640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa5940a3c10 0x7fa5941cbee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5991c2640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5941369d0 msgr2=0x7fa594122ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5991c2640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5941369d0 0x7fa594122ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5991c2640 10 monclient: _finish_hunting 0 2024-09-16T04:40:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5991c2640 1 monclient: found mon.noname-a 2024-09-16T04:40:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5991c2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5991c2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:40:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5991c2640 1 -- 172.21.15.46:0/3182096672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa594125130 con 0x7fa5940a3ff0 2024-09-16T04:40:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa5991c2640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940a3ff0 0x7fa5941cc420 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa584031000 tx=0x7fa5840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa58a7fc640 1 -- 172.21.15.46:0/3182096672 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa584041070 con 0x7fa5940a3ff0 2024-09-16T04:40:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa58a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa58a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:40:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa58a7fc640 10 monclient: dump: 2024-09-16T04:40:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:25.388 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa58a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:40:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa58a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:40:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa58a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59bc27640 5 monclient: authenticate success, global_id 4917 2024-09-16T04:40:25.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa58a7fc640 1 -- 172.21.15.46:0/3182096672 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa58403c670 con 0x7fa5940a3ff0 2024-09-16T04:40:25.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa58a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:25.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa58a7fc640 1 -- 172.21.15.46:0/3182096672 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa584037040 con 0x7fa5940a3ff0 2024-09-16T04:40:25.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59bc27640 10 monclient: _renew_subs 2024-09-16T04:40:25.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59bc27640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:25.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa59bc27640 1 -- 172.21.15.46:0/3182096672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5941257f0 con 0x7fa5940a3ff0 2024-09-16T04:40:25.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.363+0000 7fa58a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:25.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.363+0000 7fa58a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:25.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.363+0000 7fa58a7fc640 10 monclient: dump: 2024-09-16T04:40:25.389 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:25.389 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:25.389 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:25.389 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:25.389 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:25.390 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:25.390 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:25.390 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:25.390 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:25.390 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:25.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.362+0000 7fa58bfff640 4 set_mon_vals no callback set 2024-09-16T04:40:25.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.363+0000 7fa59bc27640 10 monclient: _renew_subs 2024-09-16T04:40:25.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.363+0000 7fa59bc27640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:25.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.363+0000 7fa59bc27640 1 -- 172.21.15.46:0/3182096672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5941269c0 con 0x7fa5940a3ff0 2024-09-16T04:40:25.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.363+0000 7fa59bc27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:25.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.363+0000 7fa59bc27640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 0x7fa594123530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:25.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.363+0000 7fa59bc27640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:25.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.363+0000 7fa59bc27640 1 -- 172.21.15.46:0/3182096672 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fa5940a4a90 con 0x7fa594126c00 2024-09-16T04:40:25.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.363+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 0x7fa594123530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:25.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.363+0000 7fa5999c3640 10 monclient: get_auth_request con 0x7fa594126c00 auth_method 0 2024-09-16T04:40:25.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.363+0000 7fa5999c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:25.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.363+0000 7fa5999c3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:25.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.363+0000 7fa5999c3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:25.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.363+0000 7fa5999c3640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 msgr2=0x7fa594123530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:40:25.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.363+0000 7fa5999c3640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 msgr2=0x7fa594123530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:40:25.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.363+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 0x7fa594123530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:25.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.363+0000 7fa58a7fc640 1 -- 172.21.15.46:0/3182096672 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fa584044070 con 0x7fa5940a3ff0 2024-09-16T04:40:25.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.363+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 0x7fa594123530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:25.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.364+0000 7fa58a7fc640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa5780383b0 0x7fa57803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:25.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.364+0000 7fa58a7fc640 1 -- 172.21.15.46:0/3182096672 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 4346+0+0 (secure 0 0 0) 0x7fa58406bd50 con 0x7fa5940a3ff0 2024-09-16T04:40:25.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.364+0000 7fa59a1c4640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa5780383b0 0x7fa57803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:25.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.364+0000 7fa59a1c4640 10 monclient: get_auth_request con 0x7fa5780383b0 auth_method 0 2024-09-16T04:40:25.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.364+0000 7fa59a1c4640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa5780383b0 0x7fa57803a870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa58c0012e0 tx=0x7fa58c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:25.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.564+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 0x7fa594123530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:25.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.564+0000 7fa5999c3640 10 monclient: get_auth_request con 0x7fa594126c00 auth_method 0 2024-09-16T04:40:25.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.564+0000 7fa5999c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:25.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.564+0000 7fa5999c3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:25.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.564+0000 7fa5999c3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:25.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.564+0000 7fa5999c3640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 msgr2=0x7fa594123530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:40:25.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.564+0000 7fa5999c3640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 msgr2=0x7fa594123530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:40:25.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.564+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 0x7fa594123530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:25.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.564+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 0x7fa594123530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:40:25.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.965+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 0x7fa594123530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:25.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.965+0000 7fa5999c3640 10 monclient: get_auth_request con 0x7fa594126c00 auth_method 0 2024-09-16T04:40:25.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.965+0000 7fa5999c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:25.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.965+0000 7fa5999c3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:25.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.965+0000 7fa5999c3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:25.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.965+0000 7fa5999c3640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 msgr2=0x7fa594123530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:40:25.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.965+0000 7fa5999c3640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 msgr2=0x7fa594123530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:40:25.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.965+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 0x7fa594123530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:25.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:25.965+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 0x7fa594123530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:40:26.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:26.766+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 0x7fa594123530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:26.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:26.766+0000 7fa5999c3640 10 monclient: get_auth_request con 0x7fa594126c00 auth_method 0 2024-09-16T04:40:26.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:26.766+0000 7fa5999c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:26.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:26.766+0000 7fa5999c3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:26.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:26.766+0000 7fa5999c3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:26.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:26.766+0000 7fa5999c3640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 msgr2=0x7fa594123530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:40:26.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:26.766+0000 7fa5999c3640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 msgr2=0x7fa594123530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:40:26.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:26.766+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 0x7fa594123530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:26.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:26.766+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 0x7fa594123530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:40:27.215 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:27.216 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36191]: starting. 2024-09-16T04:40:27.216 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36191]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-36156 2024-09-16T04:40:27.216 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36191]: created object 0... 2024-09-16T04:40:27.216 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36191]: created object 25... 2024-09-16T04:40:27.216 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36191]: created object 49... 2024-09-16T04:40:27.216 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36191]: finishing. 2024-09-16T04:40:27.216 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36191]: shutting down. 2024-09-16T04:40:27.228 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:27.228 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36193]: starting. 2024-09-16T04:40:27.228 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36193]: listing objects. 2024-09-16T04:40:27.228 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36193]: listed object 0... 2024-09-16T04:40:27.228 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36193]: listed object 25... 2024-09-16T04:40:27.228 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36193]: saw 50 objects 2024-09-16T04:40:27.228 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36193]: shutting down. 2024-09-16T04:40:28.134 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:28.134 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36192]: starting. 2024-09-16T04:40:28.134 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36192]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-36156 2024-09-16T04:40:28.134 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36192]: shutting down. 2024-09-16T04:40:28.135 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:28.136 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:28.136 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:40:28.138 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:40:28.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:28.362+0000 7fa5897fa640 10 monclient: tick 2024-09-16T04:40:28.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:28.362+0000 7fa5897fa640 10 monclient: _check_auth_tickets 2024-09-16T04:40:28.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:28.362+0000 7fa5897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:28.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:28.362+0000 7fa5897fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:28.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:28.368+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 0x7fa594123530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:28.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:28.368+0000 7fa5999c3640 10 monclient: get_auth_request con 0x7fa594126c00 auth_method 0 2024-09-16T04:40:28.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:28.368+0000 7fa5999c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:28.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:28.368+0000 7fa5999c3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:28.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:28.368+0000 7fa5999c3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:28.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:28.368+0000 7fa5999c3640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 msgr2=0x7fa594123530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:40:28.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:28.368+0000 7fa5999c3640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 msgr2=0x7fa594123530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:40:28.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:28.368+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 0x7fa594123530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:28.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:28.368+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 0x7fa594123530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-09-16T04:40:29.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.362+0000 7fa5897fa640 10 monclient: tick 2024-09-16T04:40:29.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.362+0000 7fa5897fa640 10 monclient: _check_auth_tickets 2024-09-16T04:40:29.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.362+0000 7fa5897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:29.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.362+0000 7fa5897fa640 5 monclient: _check_tell_commands timeout tell command 1 2024-09-16T04:40:29.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.362+0000 7fa5897fa640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 msgr2=0x7fa594123530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:40:29.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.362+0000 7fa5897fa640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 0x7fa594123530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:29.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.362+0000 7fa5897fa640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 0x7fa5700059d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:29.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.362+0000 7fa5897fa640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 msgr2=0x7fa594123530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:40:29.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.362+0000 7fa5897fa640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594126c00 0x7fa594123530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:29.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.362+0000 7fa5897fa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:29.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.362+0000 7fa5897fa640 1 -- 172.21.15.46:0/3182096672 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fa570006080 con 0x7fa570003540 2024-09-16T04:40:29.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.362+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 0x7fa5700059d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:29.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.363+0000 7fa5999c3640 10 monclient: get_auth_request con 0x7fa570003540 auth_method 0 2024-09-16T04:40:29.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.363+0000 7fa5999c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:29.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.363+0000 7fa5999c3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:29.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.363+0000 7fa5999c3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:29.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.363+0000 7fa5999c3640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 msgr2=0x7fa5700059d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:40:29.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.363+0000 7fa5999c3640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 msgr2=0x7fa5700059d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:40:29.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.363+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 0x7fa5700059d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:29.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.363+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 0x7fa5700059d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:29.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.564+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 0x7fa5700059d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:29.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.564+0000 7fa5999c3640 10 monclient: get_auth_request con 0x7fa570003540 auth_method 0 2024-09-16T04:40:29.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.564+0000 7fa5999c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:29.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.564+0000 7fa5999c3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:29.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.564+0000 7fa5999c3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:29.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.564+0000 7fa5999c3640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 msgr2=0x7fa5700059d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:40:29.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.564+0000 7fa5999c3640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 msgr2=0x7fa5700059d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:40:29.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.564+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 0x7fa5700059d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:29.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.564+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 0x7fa5700059d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:40:29.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.965+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 0x7fa5700059d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:29.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.965+0000 7fa5999c3640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 msgr2=0x7fa5700059d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:40:29.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.965+0000 7fa5999c3640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 msgr2=0x7fa5700059d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:40:29.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.965+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 0x7fa5700059d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:29.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:29.965+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 0x7fa5700059d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:40:30.183 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36260]: starting. 2024-09-16T04:40:30.184 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36260]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-36259 2024-09-16T04:40:30.184 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36260]: created object 0... 2024-09-16T04:40:30.184 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36260]: created object 25... 2024-09-16T04:40:30.184 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36260]: created object 49... 2024-09-16T04:40:30.184 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36260]: finishing. 2024-09-16T04:40:30.184 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36260]: shutting down. 2024-09-16T04:40:30.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.363+0000 7fa5897fa640 10 monclient: tick 2024-09-16T04:40:30.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.363+0000 7fa5897fa640 10 monclient: _check_auth_tickets 2024-09-16T04:40:30.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.363+0000 7fa5897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.766+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 0x7fa5700059d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.766+0000 7fa5999c3640 10 monclient: get_auth_request con 0x7fa570003540 auth_method 0 2024-09-16T04:40:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.766+0000 7fa5999c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.766+0000 7fa5999c3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.766+0000 7fa5999c3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.766+0000 7fa5999c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.766+0000 7fa5999c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.766+0000 7fa5999c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.766+0000 7fa5999c3640 10 monclient(hunting): handle_auth_done global_id 24101 payload 824 2024-09-16T04:40:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.766+0000 7fa5999c3640 1 -- 172.21.15.46:0/3182096672 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa570006230 con 0x7fa570003540 2024-09-16T04:40:30.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.767+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 0x7fa5700059d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa5900189f0 tx=0x7fa590018df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:30.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.767+0000 7fa58a7fc640 1 -- 172.21.15.46:0/3182096672 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa590028070 con 0x7fa570003540 2024-09-16T04:40:30.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.767+0000 7fa58a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:30.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.767+0000 7fa58a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.767+0000 7fa58a7fc640 10 monclient: dump: 2024-09-16T04:40:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:30.770 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.767+0000 7fa58a7fc640 1 -- 172.21.15.46:0/3182096672 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa590023440 con 0x7fa570003540 2024-09-16T04:40:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.767+0000 7fa58a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.767+0000 7fa58a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:40:30.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.768+0000 7fa58a7fc640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 msgr2=0x7fa5700059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:30.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.768+0000 7fa58a7fc640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 0x7fa5700059d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa5900189f0 tx=0x7fa590018df0 comp rx=0 tx=0).stop 2024-09-16T04:40:30.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.768+0000 7fa58a7fc640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 msgr2=0x7fa5700059d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:40:30.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.768+0000 7fa58a7fc640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa570003540 0x7fa5700059d0 secure :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa5900189f0 tx=0x7fa590018df0 comp rx=0 tx=0).stop 2024-09-16T04:40:30.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.774+0000 7fa59bc27640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:40:30.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.774+0000 7fa59bc27640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594123a70 0x7fa594123e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:30.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.774+0000 7fa59bc27640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:30.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.774+0000 7fa59bc27640 1 -- 172.21.15.46:0/3182096672 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fa5941243c0 con 0x7fa594123a70 2024-09-16T04:40:30.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.775+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594123a70 0x7fa594123e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:30.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.775+0000 7fa5999c3640 10 monclient: get_auth_request con 0x7fa594123a70 auth_method 0 2024-09-16T04:40:30.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.775+0000 7fa5999c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:30.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.775+0000 7fa5999c3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:30.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.775+0000 7fa5999c3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:30.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.775+0000 7fa5999c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:30.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.775+0000 7fa5999c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:30.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.775+0000 7fa5999c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:30.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.775+0000 7fa5999c3640 10 monclient(hunting): handle_auth_done global_id 24104 payload 824 2024-09-16T04:40:30.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.775+0000 7fa5999c3640 1 -- 172.21.15.46:0/3182096672 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa594067e20 con 0x7fa594123a70 2024-09-16T04:40:30.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.775+0000 7fa5999c3640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594123a70 0x7fa594123e80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa5900189f0 tx=0x7fa5900046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:30.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.775+0000 7fa58a7fc640 1 -- 172.21.15.46:0/3182096672 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa590028040 con 0x7fa594123a70 2024-09-16T04:40:30.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.775+0000 7fa58a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:30.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.775+0000 7fa58a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:30.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.775+0000 7fa58a7fc640 10 monclient: dump: 2024-09-16T04:40:30.779 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:30.779 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:30.779 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:30.779 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:30.779 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:30.779 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:30.779 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:30.779 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:30.779 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:30.779 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:30.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.776+0000 7fa58a7fc640 1 -- 172.21.15.46:0/3182096672 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7fa5900235f0 con 0x7fa594123a70 2024-09-16T04:40:30.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.776+0000 7fa58a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:40:30.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.776+0000 7fa58a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:40:30.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.776+0000 7fa58a7fc640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594123a70 msgr2=0x7fa594123e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:30.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.776+0000 7fa58a7fc640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594123a70 0x7fa594123e80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa5900189f0 tx=0x7fa5900046c0 comp rx=0 tx=0).stop 2024-09-16T04:40:30.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.776+0000 7fa58a7fc640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594123a70 msgr2=0x7fa594123e80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:40:30.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.776+0000 7fa58a7fc640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa594123a70 0x7fa594123e80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:30.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.776+0000 7fa59a1c4640 1 -- 172.21.15.46:0/3182096672 reap_dead start 2024-09-16T04:40:30.780 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:40:30.780 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:40:30.780 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:40:30.781 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:40:30.781 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 20, 2024-09-16T04:40:30.781 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 5974, 2024-09-16T04:40:30.781 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:40:30.781 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:40:30.782 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:40:30.782 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:40:30.782 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:30.782 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 5, 2024-09-16T04:40:30.782 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:40:30.782 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:40:30.782 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:40:30.782 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:30.782 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:30.782 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:30.782 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:30.782 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:30.782 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:30.783 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:30.783 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:30.783 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:30.783 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:30.783 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:30.783 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:30.783 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:40:30.783 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:40:30.783 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:30.783 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:30.783 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:30.783 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:30.783 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:30.783 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:30.784 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:30.784 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:30.784 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:30.784 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:30.784 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:30.784 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:30.784 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.784 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:40:30.784 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:40:30.784 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:40:30.784 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:40:30.784 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:40:30.784 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:40:30.784 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:40:30.785 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:40:30.785 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:40:30.785 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:40:30.785 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:40:30.785 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:40:30.785 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:40:30.785 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:40:30.785 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:40:30.786 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:40:30.786 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:40:30.786 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:30.786 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:30.786 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:30.786 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:30.786 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:30.786 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:30.786 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:30.786 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:30.786 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:30.786 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:30.786 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:30.786 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:30.787 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:40:30.787 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.787 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:40:30.787 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.787 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:40:30.787 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:40:30.787 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:30.787 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:30.787 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.787 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:30.787 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:40:30.787 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:30.787 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.787 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.787 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:30.788 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:40:30.788 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:30.788 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:30.788 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:30.788 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.788 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:40:30.788 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:40:30.788 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:30.788 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:30.788 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:30.788 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.788 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.788 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:40:30.789 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:40:30.789 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:30.789 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:30.789 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.789 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:30.789 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:40:30.789 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:30.789 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.789 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.789 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:30.789 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:40:30.789 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:30.790 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:30.790 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:30.790 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.790 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:40:30.790 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:40:30.790 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:30.790 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:30.791 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:30.791 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.791 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.791 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:40:30.791 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:40:30.791 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:30.792 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:30.792 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.792 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:30.792 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:40:30.792 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:30.792 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.792 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.792 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:30.792 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:40:30.792 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:30.793 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:30.793 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:30.793 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.793 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:40:30.793 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:40:30.793 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:30.793 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:30.793 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:30.793 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:30.793 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:30.793 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.793 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:40:30.793 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:40:30.793 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.794 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:30.794 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:30.794 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:40:30.794 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:30.794 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:30.794 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:40:30.794 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.794 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:30.794 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:30.794 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:40:30.794 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:30.794 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:30.794 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.794 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:40:30.794 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:40:30.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.776+0000 7fa59bc27640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa5780383b0 msgr2=0x7fa57803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:30.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.776+0000 7fa59bc27640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa5780383b0 0x7fa57803a870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa58c0012e0 tx=0x7fa58c009040 comp rx=0 tx=0).stop 2024-09-16T04:40:30.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.776+0000 7fa59bc27640 10 monclient: shutdown 2024-09-16T04:40:30.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.776+0000 7fa59bc27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:30.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.776+0000 7fa59bc27640 1 -- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940a3ff0 msgr2=0x7fa5941cc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:30.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.776+0000 7fa59bc27640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940a3ff0 0x7fa5941cc420 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa584031000 tx=0x7fa5840046c0 comp rx=0 tx=0).stop 2024-09-16T04:40:30.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.776+0000 7fa59bc27640 1 -- 172.21.15.46:0/3182096672 shutdown_connections 2024-09-16T04:40:30.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.776+0000 7fa59bc27640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa5780383b0 0x7fa57803a870 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:30.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.776+0000 7fa59bc27640 1 --2- 172.21.15.46:0/3182096672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940a3ff0 0x7fa5941cc420 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:30.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.776+0000 7fa59bc27640 1 -- 172.21.15.46:0/3182096672 >> 172.21.15.46:0/3182096672 conn(0x7fa5940a6340 msgr2=0x7fa5940a2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:30.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.776+0000 7fa59bc27640 1 -- 172.21.15.46:0/3182096672 shutdown_connections 2024-09-16T04:40:30.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.776+0000 7fa59bc27640 1 -- 172.21.15.46:0/3182096672 wait complete. 2024-09-16T04:40:30.797 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:40:30.797 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:40:30.798 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:40:30.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.908+0000 7fcb5ad49640 10 monclient: get_monmap_and_config 2024-09-16T04:40:30.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.908+0000 7fcb5ad49640 10 monclient: build_initial_monmap 2024-09-16T04:40:30.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.908+0000 7fcb5ad49640 10 monclient: monmap: 2024-09-16T04:40:30.910 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:30.910 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:30.910 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:40:30.909132+0000 2024-09-16T04:40:30.910 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:40:30.909132+0000 2024-09-16T04:40:30.910 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:30.910 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:30.910 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:40:30.910 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:40:30.910 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:40:30.910 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:40:30.911 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:40:30.911 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:30.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.909+0000 7fcb5ad49640 1 Processor -- start 2024-09-16T04:40:30.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.909+0000 7fcb5ad49640 1 -- start start 2024-09-16T04:40:30.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.909+0000 7fcb5ad49640 10 monclient: init 2024-09-16T04:40:30.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.909+0000 7fcb5ad49640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:30.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.909+0000 7fcb5ad49640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-16T04:40:30.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.909+0000 7fcb5ad49640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb540a3c10 0x7fcb5409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:30.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.909+0000 7fcb5ad49640 10 monclient(hunting): picked mon.noname-b con 0x7fcb540a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:30.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb5ad49640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb540a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:40:30.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb5ad49640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb5413be90 addr v1:172.21.15.46:6789/0 2024-09-16T04:40:30.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb5ad49640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb5409b940 con 0x7fcb5413be90 2024-09-16T04:40:30.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb5ad49640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb5409ab00 con 0x7fcb540a3ff0 2024-09-16T04:40:30.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb5ad49640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:30.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb5ad49640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fcb5409b030 con 0x7fcb540a3c10 2024-09-16T04:40:30.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb5ad49640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb5ad49640 10 monclient(hunting): authenticate will time out at 829.325101s 2024-09-16T04:40:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb58ae5640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb540a3c10 0x7fcb5409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb58ae5640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb540a3c10 0x7fcb5409bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:44068/0 (socket says 172.21.15.46:44068) 2024-09-16T04:40:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb592e6640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcb5413be90 0x7fcb5413e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60962/0 (socket says 172.21.15.46:60962) 2024-09-16T04:40:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb58ae5640 1 -- 172.21.15.46:0/988673653 learned_addr learned my addr 172.21.15.46:0/988673653 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb58ae5640 10 monclient(hunting): get_auth_request con 0x7fcb540a3c10 auth_method 0 2024-09-16T04:40:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb58ae5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb58ae5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb58ae5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb58ae5640 1 -- 172.21.15.46:0/988673653 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb540a3c10 msgr2=0x7fcb5409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:40:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb58ae5640 1 -- 172.21.15.46:0/988673653 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb540a3c10 msgr2=0x7fcb5409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:40:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb58ae5640 1 --2- 172.21.15.46:0/988673653 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb540a3c10 0x7fcb5409bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb58ae5640 1 --2- 172.21.15.46:0/988673653 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb540a3c10 0x7fcb5409bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:30.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 1 -- 172.21.15.46:0/988673653 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2695354973 0 0) 0x7fcb5409b940 con 0x7fcb5413be90 2024-09-16T04:40:30.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 10 monclient(hunting): _negotiate 2024-09-16T04:40:30.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:30.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:30.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 10 monclient(hunting): my global_id is 4932 2024-09-16T04:40:30.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 1 -- 172.21.15.46:0/988673653 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb44003520 con 0x7fcb5413be90 2024-09-16T04:40:30.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 1 -- 172.21.15.46:0/988673653 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3095445853 0 0) 0x7fcb5409ab00 con 0x7fcb540a3ff0 2024-09-16T04:40:30.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 10 monclient(hunting): _negotiate 2024-09-16T04:40:30.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:30.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:30.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 10 monclient(hunting): my global_id is 4888 2024-09-16T04:40:30.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 1 -- 172.21.15.46:0/988673653 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb5409b940 con 0x7fcb540a3ff0 2024-09-16T04:40:30.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 1 -- 172.21.15.46:0/988673653 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 346332050 0 0) 0x7fcb44003520 con 0x7fcb5413be90 2024-09-16T04:40:30.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 1 -- 172.21.15.46:0/988673653 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb540a3c10 msgr2=0x7fcb5409bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:40:30.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 1 --2- 172.21.15.46:0/988673653 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb540a3c10 0x7fcb5409bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:30.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 1 -- 172.21.15.46:0/988673653 >> v1:172.21.15.114:6789/0 conn(0x7fcb540a3ff0 legacy=0x7fcb540a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:30.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 10 monclient: _finish_hunting 0 2024-09-16T04:40:30.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:40:30.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:30.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:40:30.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 1 -- 172.21.15.46:0/988673653 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb5409b310 con 0x7fcb5413be90 2024-09-16T04:40:30.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 10 monclient: _finish_auth 0 2024-09-16T04:40:30.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 10 monclient: _check_auth_tickets 2024-09-16T04:40:30.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:30.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 1 -- 172.21.15.46:0/988673653 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7fcb4c0019f0 con 0x7fcb5413be90 2024-09-16T04:40:30.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb5ad49640 5 monclient: authenticate success, global_id 4932 2024-09-16T04:40:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb5ad49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:40:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:40:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 10 monclient: dump: 2024-09-16T04:40:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:30.928 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:40:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 1 -- 172.21.15.46:0/988673653 >> v1:172.21.15.46:6789/0 conn(0x7fcb5413be90 legacy=0x7fcb5413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:40:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 1 --2- 172.21.15.46:0/988673653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb44003b70 0x7fcb44003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.910+0000 7fcb537fe640 10 monclient(hunting): picked mon.b con 0x7fcb44003b70 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:40:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb537fe640 1 --2- 172.21.15.46:0/988673653 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb540a3c10 0x7fcb44004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb537fe640 10 monclient(hunting): picked mon.c con 0x7fcb540a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb537fe640 1 --2- 172.21.15.46:0/988673653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb44004d90 0x7fcb44005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb537fe640 10 monclient(hunting): picked mon.a con 0x7fcb44004d90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:40:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb537fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb537fe640 1 -- 172.21.15.46:0/988673653 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5409b030 con 0x7fcb44004d90 2024-09-16T04:40:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb537fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb537fe640 1 -- 172.21.15.46:0/988673653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4400bc20 con 0x7fcb44003b70 2024-09-16T04:40:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb537fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb537fe640 1 -- 172.21.15.46:0/988673653 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fcb4400be20 con 0x7fcb540a3c10 2024-09-16T04:40:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb537fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb537fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:40:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb5ad49640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:40:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb53fff640 1 --2- 172.21.15.46:0/988673653 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb540a3c10 0x7fcb44004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb592e6640 1 --2- 172.21.15.46:0/988673653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb44004d90 0x7fcb44005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb53fff640 10 monclient(hunting): get_auth_request con 0x7fcb540a3c10 auth_method 0 2024-09-16T04:40:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb53fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb53fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:30.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb592e6640 10 monclient(hunting): get_auth_request con 0x7fcb44004d90 auth_method 0 2024-09-16T04:40:30.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb592e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:30.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb592e6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:30.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb592e6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:30.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb58ae5640 1 --2- 172.21.15.46:0/988673653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb44003b70 0x7fcb44003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:30.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb58ae5640 10 monclient(hunting): get_auth_request con 0x7fcb44003b70 auth_method 0 2024-09-16T04:40:30.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb58ae5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:30.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb58ae5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:30.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb58ae5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:30.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:30.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:30.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb53fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:40:30.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb592e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:30.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb592e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:30.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb592e6640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:40:30.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb58ae5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:30.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb58ae5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:30.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb58ae5640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:40:30.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb53fff640 10 monclient(hunting): handle_auth_done global_id 4932 payload 293 2024-09-16T04:40:30.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb53fff640 1 -- 172.21.15.46:0/988673653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb44003b70 msgr2=0x7fcb44003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:40:30.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb53fff640 1 --2- 172.21.15.46:0/988673653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb44003b70 0x7fcb44003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:30.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb53fff640 1 -- 172.21.15.46:0/988673653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb44004d90 msgr2=0x7fcb44005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:40:30.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb53fff640 1 --2- 172.21.15.46:0/988673653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb44004d90 0x7fcb44005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:30.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb53fff640 10 monclient: _finish_hunting 0 2024-09-16T04:40:30.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb53fff640 1 monclient: found mon.c 2024-09-16T04:40:30.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:30.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb53fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:40:30.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb53fff640 1 -- 172.21.15.46:0/988673653 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb44003520 con 0x7fcb540a3c10 2024-09-16T04:40:30.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb53fff640 10 monclient: _finish_auth 0 2024-09-16T04:40:30.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb53fff640 10 monclient: _check_auth_tickets 2024-09-16T04:40:30.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:30.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.911+0000 7fcb592e6640 1 --2- 172.21.15.46:0/988673653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb44004d90 0x7fcb44005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:40:30.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.912+0000 7fcb53fff640 1 --2- 172.21.15.46:0/988673653 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb540a3c10 0x7fcb44004800 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcb540a38d0 tx=0x7fcb44005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94cc5b5c15d4e0c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:30.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.912+0000 7fcb537fe640 1 -- 172.21.15.46:0/988673653 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb400014f0 con 0x7fcb540a3c10 2024-09-16T04:40:30.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.912+0000 7fcb537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:30.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.912+0000 7fcb537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:30.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.912+0000 7fcb537fe640 10 monclient: dump: 2024-09-16T04:40:30.932 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:30.932 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:30.932 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:30.932 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:30.932 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:30.932 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:30.933 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.912+0000 7fcb537fe640 1 -- 172.21.15.46:0/988673653 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb400052d0 con 0x7fcb540a3c10 2024-09-16T04:40:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.912+0000 7fcb5ad49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:40:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.912+0000 7fcb537fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.912+0000 7fcb5ad49640 10 monclient: get_monmap_and_config success 2024-09-16T04:40:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.912+0000 7fcb537fe640 1 -- 172.21.15.46:0/988673653 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4002ac30 con 0x7fcb540a3c10 2024-09-16T04:40:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.912+0000 7fcb537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.912+0000 7fcb5ad49640 4 set_mon_vals no callback set 2024-09-16T04:40:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.912+0000 7fcb537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.912+0000 7fcb537fe640 10 monclient: dump: 2024-09-16T04:40:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:30.934 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.912+0000 7fcb5ad49640 10 monclient: shutdown 2024-09-16T04:40:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.912+0000 7fcb5ad49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.912+0000 7fcb5ad49640 1 -- 172.21.15.46:0/988673653 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb540a3c10 msgr2=0x7fcb44004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.912+0000 7fcb5ad49640 1 --2- 172.21.15.46:0/988673653 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb540a3c10 0x7fcb44004800 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcb540a38d0 tx=0x7fcb44005830 comp rx=0 tx=0).stop 2024-09-16T04:40:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.912+0000 7fcb592e6640 1 -- 172.21.15.46:0/988673653 reap_dead start 2024-09-16T04:40:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.913+0000 7fcb5ad49640 1 -- 172.21.15.46:0/988673653 shutdown_connections 2024-09-16T04:40:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.913+0000 7fcb5ad49640 1 -- 172.21.15.46:0/988673653 >> 172.21.15.46:0/988673653 conn(0x7fcb540a6340 msgr2=0x7fcb5406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.913+0000 7fcb5ad49640 1 -- 172.21.15.46:0/988673653 shutdown_connections 2024-09-16T04:40:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.913+0000 7fcb5ad49640 1 -- 172.21.15.46:0/988673653 wait complete. 2024-09-16T04:40:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.913+0000 7fcb5ad49640 10 monclient: build_initial_monmap 2024-09-16T04:40:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.913+0000 7fcb5ad49640 10 monclient: monmap: 2024-09-16T04:40:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:40:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:40:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:40:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:40:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:40:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:40:30.936 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.913+0000 7fcb5ad49640 1 Processor -- start 2024-09-16T04:40:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb5ad49640 1 -- start start 2024-09-16T04:40:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb5ad49640 10 monclient: init 2024-09-16T04:40:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb5ad49640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb5ad49640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:40:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb5ad49640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb540a3c10 0x7fcb54137460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb5ad49640 10 monclient(hunting): picked mon.noname-b con 0x7fcb540a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:40:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb5ad49640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb5411fcc0 0x7fcb541200e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb5ad49640 10 monclient(hunting): picked mon.noname-c con 0x7fcb5411fcc0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb5ad49640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb54120620 0x7fcb541d2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb5ad49640 10 monclient(hunting): picked mon.noname-a con 0x7fcb54120620 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:40:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb5ad49640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb5ad49640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5409be70 con 0x7fcb54120620 2024-09-16T04:40:30.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb5ad49640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:30.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb5ad49640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5409bcf0 con 0x7fcb540a3c10 2024-09-16T04:40:30.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb5ad49640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:30.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb5ad49640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fcb5409bff0 con 0x7fcb5411fcc0 2024-09-16T04:40:30.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb5ad49640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:30.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb5ad49640 10 monclient(hunting): authenticate will time out at 829.329679s 2024-09-16T04:40:30.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb53fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb5411fcc0 0x7fcb541200e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:30.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb592e6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb54120620 0x7fcb541d2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:30.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb592e6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb54120620 0x7fcb541d2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58472/0 (socket says 172.21.15.46:58472) 2024-09-16T04:40:30.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb53fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb5411fcc0 0x7fcb541200e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:44090/0 (socket says 172.21.15.46:44090) 2024-09-16T04:40:30.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb592e6640 1 -- 172.21.15.46:0/1319422946 learned_addr learned my addr 172.21.15.46:0/1319422946 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:30.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb592e6640 10 monclient(hunting): get_auth_request con 0x7fcb54120620 auth_method 0 2024-09-16T04:40:30.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb592e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:30.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb592e6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:30.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb592e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:30.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb53fff640 10 monclient(hunting): get_auth_request con 0x7fcb5411fcc0 auth_method 0 2024-09-16T04:40:30.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:30.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb53fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:30.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:30.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb592e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:30.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb592e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:30.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb592e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:30.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.914+0000 7fcb58ae5640 1 --2- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb540a3c10 0x7fcb54137460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb58ae5640 10 monclient(hunting): get_auth_request con 0x7fcb540a3c10 auth_method 0 2024-09-16T04:40:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb58ae5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb58ae5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb58ae5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb592e6640 10 monclient(hunting): handle_auth_done global_id 4935 payload 824 2024-09-16T04:40:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb592e6640 1 -- 172.21.15.46:0/1319422946 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb5411fcc0 msgr2=0x7fcb541200e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb592e6640 1 --2- 172.21.15.46:0/1319422946 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb5411fcc0 0x7fcb541200e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb592e6640 1 -- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb540a3c10 msgr2=0x7fcb54137460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb592e6640 1 --2- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb540a3c10 0x7fcb54137460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb592e6640 10 monclient: _finish_hunting 0 2024-09-16T04:40:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb592e6640 1 monclient: found mon.noname-a 2024-09-16T04:40:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb592e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb592e6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:40:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb592e6640 1 -- 172.21.15.46:0/1319422946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb541223c0 con 0x7fcb54120620 2024-09-16T04:40:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb53fff640 1 --2- 172.21.15.46:0/1319422946 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcb5411fcc0 0x7fcb541200e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:40:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb592e6640 1 --2- 172.21.15.46:0/1319422946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb54120620 0x7fcb541d2060 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcb5409af30 tx=0x7fcb4c007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb517fa640 1 -- 172.21.15.46:0/1319422946 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c00c0e0 con 0x7fcb54120620 2024-09-16T04:40:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb517fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:40:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb517fa640 10 monclient: dump: 2024-09-16T04:40:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:30.941 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:30.941 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:30.941 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:30.941 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:30.941 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:30.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb517fa640 10 monclient: _finish_auth 0 2024-09-16T04:40:30.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb517fa640 10 monclient: _check_auth_tickets 2024-09-16T04:40:30.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:30.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb517fa640 1 -- 172.21.15.46:0/1319422946 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb4c018070 con 0x7fcb54120620 2024-09-16T04:40:30.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb517fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:30.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb517fa640 1 -- 172.21.15.46:0/1319422946 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c00cc00 con 0x7fcb54120620 2024-09-16T04:40:30.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:30.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:30.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb517fa640 10 monclient: dump: 2024-09-16T04:40:30.941 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:30.942 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb5ad49640 5 monclient: authenticate success, global_id 4935 2024-09-16T04:40:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb5ad49640 10 monclient: _renew_subs 2024-09-16T04:40:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb5ad49640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb5ad49640 1 -- 172.21.15.46:0/1319422946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb54120ad0 con 0x7fcb54120620 2024-09-16T04:40:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb52ffd640 4 set_mon_vals no callback set 2024-09-16T04:40:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb5ad49640 10 monclient: _renew_subs 2024-09-16T04:40:30.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb5ad49640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:30.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.915+0000 7fcb5ad49640 1 -- 172.21.15.46:0/1319422946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb540a2810 con 0x7fcb54120620 2024-09-16T04:40:30.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.916+0000 7fcb5ad49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:30.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.916+0000 7fcb5ad49640 1 --2- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb5409d110 0x7fcb5409d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:30.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.916+0000 7fcb5ad49640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:30.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.916+0000 7fcb5ad49640 1 -- 172.21.15.46:0/1319422946 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcb541232b0 con 0x7fcb5409d110 2024-09-16T04:40:30.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.916+0000 7fcb517fa640 1 -- 172.21.15.46:0/1319422946 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fcb4c01d460 con 0x7fcb54120620 2024-09-16T04:40:30.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.916+0000 7fcb58ae5640 1 --2- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb5409d110 0x7fcb5409d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:30.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.916+0000 7fcb517fa640 1 --2- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcb380383b0 0x7fcb3803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:30.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.916+0000 7fcb58ae5640 10 monclient: get_auth_request con 0x7fcb5409d110 auth_method 0 2024-09-16T04:40:30.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.916+0000 7fcb58ae5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:30.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.916+0000 7fcb58ae5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:30.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.916+0000 7fcb58ae5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:30.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.916+0000 7fcb517fa640 1 -- 172.21.15.46:0/1319422946 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 4738+0+0 (secure 0 0 0) 0x7fcb4c04b8a0 con 0x7fcb54120620 2024-09-16T04:40:30.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.916+0000 7fcb58ae5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:30.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.916+0000 7fcb58ae5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:30.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.916+0000 7fcb58ae5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:30.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.916+0000 7fcb53fff640 1 --2- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcb380383b0 0x7fcb3803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:30.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.916+0000 7fcb53fff640 10 monclient: get_auth_request con 0x7fcb380383b0 auth_method 0 2024-09-16T04:40:30.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.917+0000 7fcb58ae5640 10 monclient(hunting): handle_auth_done global_id 4894 payload 824 2024-09-16T04:40:30.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.917+0000 7fcb58ae5640 1 -- 172.21.15.46:0/1319422946 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb540a43d0 con 0x7fcb5409d110 2024-09-16T04:40:30.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.917+0000 7fcb58ae5640 1 --2- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb5409d110 0x7fcb5409d4f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fcb48013580 tx=0x7fcb48013a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:30.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.917+0000 7fcb53fff640 1 --2- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcb380383b0 0x7fcb3803a870 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcb40032000 tx=0x7fcb40009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:30.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.917+0000 7fcb517fa640 1 -- 172.21.15.46:0/1319422946 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4801c070 con 0x7fcb5409d110 2024-09-16T04:40:30.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.917+0000 7fcb517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:30.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.917+0000 7fcb517fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:40:30.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.917+0000 7fcb517fa640 10 monclient: dump: 2024-09-16T04:40:30.944 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:30.945 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:30.945 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:30.945 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:30.945 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:30.945 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:30.945 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:30.945 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:30.945 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:30.945 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:30.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.917+0000 7fcb517fa640 1 -- 172.21.15.46:0/1319422946 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcb48011990 con 0x7fcb5409d110 2024-09-16T04:40:30.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.917+0000 7fcb517fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:30.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.917+0000 7fcb517fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:40:30.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.917+0000 7fcb517fa640 1 -- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb5409d110 msgr2=0x7fcb5409d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:30.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.917+0000 7fcb517fa640 1 --2- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb5409d110 0x7fcb5409d4f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fcb48013580 tx=0x7fcb48013a40 comp rx=0 tx=0).stop 2024-09-16T04:40:30.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.917+0000 7fcb517fa640 1 -- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb5409d110 msgr2=0x7fcb5409d4f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:40:30.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.917+0000 7fcb517fa640 1 --2- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb5409d110 0x7fcb5409d4f0 secure :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fcb48013580 tx=0x7fcb48013a40 comp rx=0 tx=0).stop 2024-09-16T04:40:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.923+0000 7fcb5ad49640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:40:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.923+0000 7fcb5ad49640 1 --2- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb5409da30 0x7fcb541d65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.923+0000 7fcb5ad49640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.923+0000 7fcb5ad49640 1 -- 172.21.15.46:0/1319422946 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcb540a0b50 con 0x7fcb5409da30 2024-09-16T04:40:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.924+0000 7fcb58ae5640 1 --2- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb5409da30 0x7fcb541d65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.924+0000 7fcb58ae5640 10 monclient: get_auth_request con 0x7fcb5409da30 auth_method 0 2024-09-16T04:40:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.924+0000 7fcb58ae5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.924+0000 7fcb58ae5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.924+0000 7fcb58ae5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.924+0000 7fcb58ae5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.924+0000 7fcb58ae5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.924+0000 7fcb58ae5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.924+0000 7fcb58ae5640 10 monclient(hunting): handle_auth_done global_id 4897 payload 824 2024-09-16T04:40:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.924+0000 7fcb58ae5640 1 -- 172.21.15.46:0/1319422946 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcb54122140 con 0x7fcb5409da30 2024-09-16T04:40:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.925+0000 7fcb58ae5640 1 --2- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb5409da30 0x7fcb541d65c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcb48013580 tx=0x7fcb480046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:30.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.925+0000 7fcb517fa640 1 -- 172.21.15.46:0/1319422946 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4802f020 con 0x7fcb5409da30 2024-09-16T04:40:30.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.925+0000 7fcb517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:30.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.925+0000 7fcb517fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:40:30.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.925+0000 7fcb517fa640 10 monclient: dump: 2024-09-16T04:40:30.949 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:30.949 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:30.949 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:30.949 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:30.949 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:30.949 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:30.949 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:30.950 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:30.950 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:30.950 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:30.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.925+0000 7fcb517fa640 1 -- 172.21.15.46:0/1319422946 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7fcb48016e20 con 0x7fcb5409da30 2024-09-16T04:40:30.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.925+0000 7fcb517fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:40:30.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.925+0000 7fcb517fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:40:30.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.925+0000 7fcb517fa640 1 -- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb5409da30 msgr2=0x7fcb541d65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:30.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.925+0000 7fcb517fa640 1 --2- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb5409da30 0x7fcb541d65c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcb48013580 tx=0x7fcb480046c0 comp rx=0 tx=0).stop 2024-09-16T04:40:30.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.925+0000 7fcb517fa640 1 -- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb5409da30 msgr2=0x7fcb541d65c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:40:30.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.925+0000 7fcb517fa640 1 --2- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb5409da30 0x7fcb541d65c0 secure :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcb48013580 tx=0x7fcb480046c0 comp rx=0 tx=0).stop 2024-09-16T04:40:30.950 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:40:30.951 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:40:30.951 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:40:30.951 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:40:30.951 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 20, 2024-09-16T04:40:30.951 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 116827, 2024-09-16T04:40:30.951 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:40:30.951 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:40:30.951 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:40:30.951 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:40:30.951 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:30.951 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 5, 2024-09-16T04:40:30.951 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:40:30.951 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:40:30.952 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:40:30.952 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:30.952 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:30.952 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:30.952 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:30.952 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:30.952 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:30.952 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:30.952 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:30.952 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:30.952 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:30.952 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:30.952 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:30.952 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:40:30.953 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:40:30.953 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:30.953 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:30.953 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:30.953 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:30.953 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:30.953 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:30.953 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:30.953 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:30.953 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:30.953 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:30.953 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:30.954 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:30.954 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.954 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:40:30.954 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:40:30.954 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:40:30.954 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:40:30.954 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:40:30.954 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:40:30.954 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:40:30.954 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:40:30.954 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:40:30.954 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:40:30.954 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:40:30.955 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:40:30.955 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:40:30.955 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:40:30.955 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:40:30.955 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:40:30.955 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:40:30.955 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:30.955 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:30.955 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:30.955 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:30.955 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:30.955 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:30.955 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:30.955 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:30.956 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:30.956 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:30.956 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:30.956 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:30.956 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:40:30.956 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.956 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:40:30.956 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.956 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:40:30.956 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:40:30.956 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:30.956 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:30.956 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.956 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:30.957 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:40:30.957 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:30.957 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.957 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.957 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:30.957 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:40:30.957 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:30.957 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:30.957 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:30.957 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.957 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:40:30.957 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:40:30.957 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:30.957 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:30.958 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:30.958 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.958 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.958 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:40:30.958 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:40:30.958 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:30.958 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:30.958 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.958 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:30.958 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:40:30.958 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:30.958 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.958 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.958 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:30.959 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:40:30.959 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:30.959 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:30.959 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:30.959 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.959 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:40:30.959 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:40:30.959 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:30.959 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:30.959 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:30.959 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.959 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.959 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:40:30.959 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:40:30.960 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:30.960 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:30.960 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.960 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:30.960 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:40:30.960 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:30.960 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.960 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.960 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:30.960 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:40:30.960 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:30.960 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:30.960 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:30.961 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.961 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:40:30.961 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:40:30.961 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:30.961 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:30.961 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:30.961 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:30.961 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:30.961 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.961 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:40:30.961 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:40:30.961 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.962 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:30.962 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:30.962 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:40:30.962 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:30.962 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:30.962 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:40:30.962 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.962 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:30.962 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:30.962 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:40:30.962 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:30.962 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:30.962 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:40:30.962 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.963 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:30.963 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:30.963 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6 2024-09-16T04:40:30.963 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:30.963 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:30.963 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:40:30.963 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:30.963 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:30.963 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:30.963 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:40:30.963 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:30.963 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:30.963 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:30.963 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:40:30.964 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:40:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.926+0000 7fcb5ad49640 1 -- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcb380383b0 msgr2=0x7fcb3803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.926+0000 7fcb5ad49640 1 --2- 172.21.15.46:0/1319422946 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcb380383b0 0x7fcb3803a870 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcb40032000 tx=0x7fcb40009030 comp rx=0 tx=0).stop 2024-09-16T04:40:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.926+0000 7fcb5ad49640 10 monclient: shutdown 2024-09-16T04:40:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.926+0000 7fcb5ad49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.926+0000 7fcb5ad49640 1 -- 172.21.15.46:0/1319422946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb54120620 msgr2=0x7fcb541d2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.926+0000 7fcb5ad49640 1 --2- 172.21.15.46:0/1319422946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb54120620 0x7fcb541d2060 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcb5409af30 tx=0x7fcb4c007a90 comp rx=0 tx=0).stop 2024-09-16T04:40:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.926+0000 7fcb592e6640 1 -- 172.21.15.46:0/1319422946 reap_dead start 2024-09-16T04:40:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.926+0000 7fcb5ad49640 1 -- 172.21.15.46:0/1319422946 shutdown_connections 2024-09-16T04:40:30.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.926+0000 7fcb5ad49640 1 --2- 172.21.15.46:0/1319422946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb54120620 0x7fcb541d2060 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:30.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.926+0000 7fcb5ad49640 1 -- 172.21.15.46:0/1319422946 >> 172.21.15.46:0/1319422946 conn(0x7fcb540a6340 msgr2=0x7fcb540a2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:30.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.926+0000 7fcb5ad49640 1 -- 172.21.15.46:0/1319422946 shutdown_connections 2024-09-16T04:40:30.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:30.926+0000 7fcb5ad49640 1 -- 172.21.15.46:0/1319422946 wait complete. 2024-09-16T04:40:30.966 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:40:30.966 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:40:30.966 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:40:31.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.067+0000 7f6284ed9640 10 monclient: get_monmap_and_config 2024-09-16T04:40:31.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.067+0000 7f6284ed9640 10 monclient: build_initial_monmap 2024-09-16T04:40:31.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.067+0000 7f6284ed9640 10 monclient: monmap: 2024-09-16T04:40:31.069 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:31.069 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:31.069 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:40:31.067545+0000 2024-09-16T04:40:31.069 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:40:31.067545+0000 2024-09-16T04:40:31.069 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:31.069 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:31.069 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:40:31.069 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:40:31.069 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:40:31.069 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:40:31.069 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:40:31.069 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:31.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.067+0000 7f6284ed9640 1 Processor -- start 2024-09-16T04:40:31.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.067+0000 7f6284ed9640 1 -- start start 2024-09-16T04:40:31.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.067+0000 7f6284ed9640 10 monclient: init 2024-09-16T04:40:31.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6284ed9640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:31.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6284ed9640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T04:40:31.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6284ed9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f62800a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:40:31.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6284ed9640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f62800a3ff0 0x7f62800a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:31.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6284ed9640 10 monclient(hunting): picked mon.noname-b con 0x7f62800a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:31.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6284ed9640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f628013be90 0x7f628013e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:31.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6284ed9640 10 monclient(hunting): picked mon.noname-c con 0x7f628013be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:40:31.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6284ed9640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f628009ab70 con 0x7f62800a3c10 2024-09-16T04:40:31.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6284ed9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:31.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6284ed9640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f62800a4d60 con 0x7f628013be90 2024-09-16T04:40:31.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6284ed9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:31.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6284ed9640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f628009b030 con 0x7f62800a3ff0 2024-09-16T04:40:31.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6284ed9640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:31.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6284ed9640 10 monclient(hunting): authenticate will time out at 829.483354s 2024-09-16T04:40:31.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6275d9b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f62800a3ff0 0x7f62800a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:31.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6275d9b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f62800a3ff0 0x7f62800a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:44094/0 (socket says 172.21.15.46:44094) 2024-09-16T04:40:31.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6275d9b640 1 -- 172.21.15.46:0/823163470 learned_addr learned my addr 172.21.15.46:0/823163470 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:31.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6275d9b640 10 monclient(hunting): get_auth_request con 0x7f62800a3ff0 auth_method 0 2024-09-16T04:40:31.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6275d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:31.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6275d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:31.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6275d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:31.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6275d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:31.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6275d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:31.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6275d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:31.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f627ed9d640 1 --2- 172.21.15.46:0/823163470 >> v2:172.21.15.114:3300/0 conn(0x7f628013be90 0x7f628013e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:31.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f627ed9d640 10 monclient(hunting): get_auth_request con 0x7f628013be90 auth_method 0 2024-09-16T04:40:31.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f627ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:31.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f627ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:31.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f627ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:31.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f627ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:31.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f627ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:31.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f627ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:31.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6275d9b640 10 monclient(hunting): handle_auth_done global_id 24113 payload 274 2024-09-16T04:40:31.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6275d9b640 1 -- 172.21.15.46:0/823163470 >> v2:172.21.15.114:3300/0 conn(0x7f628013be90 msgr2=0x7f628013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:40:31.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6275d9b640 1 --2- 172.21.15.46:0/823163470 >> v2:172.21.15.114:3300/0 conn(0x7f628013be90 0x7f628013e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:31.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6275d9b640 1 -- 172.21.15.46:0/823163470 >> v1:172.21.15.114:6789/0 conn(0x7f62800a3c10 legacy=0x7f628009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:31.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6275d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:40:31.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6275d9b640 1 monclient: found mon.noname-b 2024-09-16T04:40:31.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6275d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:31.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6275d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:40:31.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.068+0000 7f6275d9b640 1 -- 172.21.15.46:0/823163470 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f628009b310 con 0x7f62800a3ff0 2024-09-16T04:40:31.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.069+0000 7f6275d9b640 1 --2- 172.21.15.46:0/823163470 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f62800a3ff0 0x7f62800a43d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6268009870 tx=0x7f626802a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d168c3d524957f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:31.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.069+0000 7f627dd9b640 1 -- 172.21.15.46:0/823163470 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6268037070 con 0x7f62800a3ff0 2024-09-16T04:40:31.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.069+0000 7f627dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:31.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.069+0000 7f627dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:40:31.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.069+0000 7f627dd9b640 10 monclient: dump: 2024-09-16T04:40:31.083 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:31.083 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:31.083 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:31.083 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:31.083 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:31.083 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:31.083 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:31.084 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:31.084 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:31.084 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:31.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.069+0000 7f627dd9b640 10 monclient: _finish_auth 0 2024-09-16T04:40:31.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.069+0000 7f627dd9b640 10 monclient: _check_auth_tickets 2024-09-16T04:40:31.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.069+0000 7f627dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:31.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.069+0000 7f627dd9b640 1 -- 172.21.15.46:0/823163470 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6268032440 con 0x7f62800a3ff0 2024-09-16T04:40:31.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.069+0000 7f6284ed9640 5 monclient: authenticate success, global_id 24113 2024-09-16T04:40:31.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.069+0000 7f6284ed9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:40:31.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.069+0000 7f627dd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:31.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.069+0000 7f6284ed9640 10 monclient: get_monmap_and_config success 2024-09-16T04:40:31.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.069+0000 7f627dd9b640 1 -- 172.21.15.46:0/823163470 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6268032810 con 0x7f62800a3ff0 2024-09-16T04:40:31.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.069+0000 7f6284ed9640 4 set_mon_vals no callback set 2024-09-16T04:40:31.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.069+0000 7f627dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:31.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.069+0000 7f627dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:31.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.069+0000 7f627dd9b640 10 monclient: dump: 2024-09-16T04:40:31.085 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:31.085 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:31.085 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:31.085 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:31.085 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:31.085 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:31.085 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:31.085 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:31.085 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:31.085 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:31.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.070+0000 7f6284ed9640 10 monclient: shutdown 2024-09-16T04:40:31.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.070+0000 7f6284ed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.070+0000 7f6284ed9640 1 -- 172.21.15.46:0/823163470 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f62800a3ff0 msgr2=0x7f62800a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.070+0000 7f6284ed9640 1 --2- 172.21.15.46:0/823163470 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f62800a3ff0 0x7f62800a43d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6268009870 tx=0x7f626802a130 comp rx=0 tx=0).stop 2024-09-16T04:40:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.070+0000 7f6284ed9640 1 -- 172.21.15.46:0/823163470 shutdown_connections 2024-09-16T04:40:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.070+0000 7f6284ed9640 1 --2- 172.21.15.46:0/823163470 >> v2:172.21.15.114:3300/0 conn(0x7f628013be90 0x7f628013e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.070+0000 7f6284ed9640 1 --2- 172.21.15.46:0/823163470 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f62800a3ff0 0x7f62800a43d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.070+0000 7f6284ed9640 1 -- 172.21.15.46:0/823163470 >> 172.21.15.46:0/823163470 conn(0x7f62800a5d10 msgr2=0x7f62800a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.070+0000 7f6284ed9640 1 -- 172.21.15.46:0/823163470 shutdown_connections 2024-09-16T04:40:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.070+0000 7f6284ed9640 1 -- 172.21.15.46:0/823163470 wait complete. 2024-09-16T04:40:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.070+0000 7f6284ed9640 10 monclient: build_initial_monmap 2024-09-16T04:40:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.070+0000 7f6284ed9640 10 monclient: monmap: 2024-09-16T04:40:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:40:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:40:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:40:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:40:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:40:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:40:31.087 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.071+0000 7f6284ed9640 1 Processor -- start 2024-09-16T04:40:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.071+0000 7f6284ed9640 1 -- start start 2024-09-16T04:40:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.071+0000 7f6284ed9640 10 monclient: init 2024-09-16T04:40:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.071+0000 7f6284ed9640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.071+0000 7f6284ed9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:40:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.071+0000 7f6284ed9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62800a3c10 0x7f62801370b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.071+0000 7f6284ed9640 10 monclient(hunting): picked mon.noname-b con 0x7f62800a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:40:31.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.071+0000 7f6284ed9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62800a3ff0 0x7f628009e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:31.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.071+0000 7f6284ed9640 10 monclient(hunting): picked mon.noname-a con 0x7f62800a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:40:31.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.071+0000 7f6284ed9640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f628013be90 0x7f628009e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:31.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.071+0000 7f6284ed9640 10 monclient(hunting): picked mon.noname-c con 0x7f628013be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:31.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.071+0000 7f6284ed9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:31.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.071+0000 7f6284ed9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f628006e550 con 0x7f62800a3ff0 2024-09-16T04:40:31.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.071+0000 7f6284ed9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:31.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.071+0000 7f6284ed9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f628006e3d0 con 0x7f62800a3c10 2024-09-16T04:40:31.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.071+0000 7f6284ed9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:31.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.071+0000 7f6284ed9640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f628013e7c0 con 0x7f628013be90 2024-09-16T04:40:31.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.071+0000 7f6284ed9640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:31.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.071+0000 7f6284ed9640 10 monclient(hunting): authenticate will time out at 829.486993s 2024-09-16T04:40:31.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f627ed9d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f628013be90 0x7f628009e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:31.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6275d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62800a3ff0 0x7f628009e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:31.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f627ed9d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f628013be90 0x7f628009e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:44104/0 (socket says 172.21.15.46:44104) 2024-09-16T04:40:31.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6275d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62800a3ff0 0x7f628009e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58484/0 (socket says 172.21.15.46:58484) 2024-09-16T04:40:31.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f627e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62800a3c10 0x7f62801370b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:31.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f627ed9d640 1 -- 172.21.15.46:0/3606257090 learned_addr learned my addr 172.21.15.46:0/3606257090 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:31.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f627ed9d640 10 monclient(hunting): get_auth_request con 0x7f628013be90 auth_method 0 2024-09-16T04:40:31.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f627ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:31.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f627ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:31.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f627ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:31.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6275d9b640 10 monclient(hunting): get_auth_request con 0x7f62800a3ff0 auth_method 0 2024-09-16T04:40:31.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6275d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:31.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6275d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:31.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6275d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:31.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f627e59c640 10 monclient(hunting): get_auth_request con 0x7f62800a3c10 auth_method 0 2024-09-16T04:40:31.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f627e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:31.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f627e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:31.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f627e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:31.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f627ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:31.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f627ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:31.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f627ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:31.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6275d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:31.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6275d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:31.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6275d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:31.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f627e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:31.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f627e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:31.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f627e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:31.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6275d9b640 10 monclient(hunting): handle_auth_done global_id 4938 payload 824 2024-09-16T04:40:31.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6275d9b640 1 -- 172.21.15.46:0/3606257090 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f628013be90 msgr2=0x7f628009e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:31.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6275d9b640 1 --2- 172.21.15.46:0/3606257090 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f628013be90 0x7f628009e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:31.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6275d9b640 1 -- 172.21.15.46:0/3606257090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62800a3c10 msgr2=0x7f62801370b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:31.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6275d9b640 1 --2- 172.21.15.46:0/3606257090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62800a3c10 0x7f62801370b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:31.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6275d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:40:31.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6275d9b640 1 monclient: found mon.noname-a 2024-09-16T04:40:31.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6275d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:31.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6275d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:40:31.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6275d9b640 1 -- 172.21.15.46:0/3606257090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f628013cd40 con 0x7f62800a3ff0 2024-09-16T04:40:31.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f627ed9d640 1 --2- 172.21.15.46:0/3606257090 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f628013be90 0x7f628009e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:40:31.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f627e59c640 1 --2- 172.21.15.46:0/3606257090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62800a3c10 0x7f62801370b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:40:31.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6275d9b640 1 --2- 172.21.15.46:0/3606257090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62800a3ff0 0x7f628009e030 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6268031000 tx=0x7f62680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:31.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f62777fe640 1 -- 172.21.15.46:0/3606257090 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6268041020 con 0x7f62800a3ff0 2024-09-16T04:40:31.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f62777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:31.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f62777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:40:31.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f62777fe640 10 monclient: dump: 2024-09-16T04:40:31.093 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:31.093 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:31.093 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:31.093 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:31.093 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:31.093 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:31.093 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:31.093 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:31.093 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:31.093 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:31.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f62777fe640 10 monclient: _finish_auth 0 2024-09-16T04:40:31.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f62777fe640 10 monclient: _check_auth_tickets 2024-09-16T04:40:31.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f62777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:31.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f62777fe640 1 -- 172.21.15.46:0/3606257090 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6268030660 con 0x7f62800a3ff0 2024-09-16T04:40:31.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6284ed9640 5 monclient: authenticate success, global_id 4938 2024-09-16T04:40:31.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6284ed9640 10 monclient: _renew_subs 2024-09-16T04:40:31.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6284ed9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:31.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6284ed9640 1 -- 172.21.15.46:0/3606257090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f628013c810 con 0x7f62800a3ff0 2024-09-16T04:40:31.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f62777fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:31.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f62777fe640 1 -- 172.21.15.46:0/3606257090 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6268037040 con 0x7f62800a3ff0 2024-09-16T04:40:31.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f62777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:31.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f62777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:31.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f62777fe640 10 monclient: dump: 2024-09-16T04:40:31.094 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:31.094 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:31.095 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:31.095 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:31.095 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:31.095 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:31.095 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:31.095 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:31.095 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:31.095 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:31.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f627d59a640 4 set_mon_vals no callback set 2024-09-16T04:40:31.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6284ed9640 10 monclient: _renew_subs 2024-09-16T04:40:31.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.072+0000 7f6284ed9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:31.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.073+0000 7f6284ed9640 1 -- 172.21.15.46:0/3606257090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6280121a40 con 0x7f62800a3ff0 2024-09-16T04:40:31.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.073+0000 7f62777fe640 1 -- 172.21.15.46:0/3606257090 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f6268044020 con 0x7f62800a3ff0 2024-09-16T04:40:31.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.073+0000 7f6284ed9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:31.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.073+0000 7f6284ed9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:31.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.073+0000 7f6284ed9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:31.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.073+0000 7f6284ed9640 1 -- 172.21.15.46:0/3606257090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62800a43d0 con 0x7f62800a3ff0 2024-09-16T04:40:31.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.074+0000 7f62777fe640 1 --2- 172.21.15.46:0/3606257090 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f62600383b0 0x7f626003a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:31.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.074+0000 7f62777fe640 1 -- 172.21.15.46:0/3606257090 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 4738+0+0 (secure 0 0 0) 0x7f626806bef0 con 0x7f62800a3ff0 2024-09-16T04:40:31.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.074+0000 7f627e59c640 1 --2- 172.21.15.46:0/3606257090 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f62600383b0 0x7f626003a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:31.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.074+0000 7f627e59c640 10 monclient: get_auth_request con 0x7f62600383b0 auth_method 0 2024-09-16T04:40:31.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.074+0000 7f627e59c640 1 --2- 172.21.15.46:0/3606257090 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f62600383b0 0x7f626003a870 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6270000f60 tx=0x7f6270009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:31.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.079+0000 7f62777fe640 1 -- 172.21.15.46:0/3606257090 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f626804b030 con 0x7f62800a3ff0 2024-09-16T04:40:31.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.079+0000 7f62777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:31.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.079+0000 7f62777fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:40:31.097 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36261]: starting. 2024-09-16T04:40:31.097 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36261]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-36259 2024-09-16T04:40:31.097 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36261]: shutting down. 2024-09-16T04:40:31.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.334+0000 7f6284ed9640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:40:31.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.334+0000 7f6284ed9640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:40:31.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.334+0000 7f6284ed9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:31.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.334+0000 7f6284ed9640 1 -- 172.21.15.46:0/3606257090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f6280069150 con 0x7f62800a3ff0 2024-09-16T04:40:31.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.335+0000 7f62777fe640 1 -- 172.21.15.46:0/3606257090 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f626803d7a0 con 0x7f62800a3ff0 2024-09-16T04:40:31.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.335+0000 7f62777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:40:31.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.335+0000 7f62777fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:40:31.337 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.335+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:40:31.337 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.335+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=2,health=12,kv=12,logm=66} crc {auth=3595534027,config=3541143443,health=1083265261,kv=4017288519,logm=4073505531}) 2024-09-16T04:40:31.337 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.335+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=12,kv=12,logm=65} crc {auth=1219556220,config=3541143443,health=1083265261,kv=4017288519,logm=4266968728}) 2024-09-16T04:40:31.337 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.335+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:40:31.337 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.335+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=2,health=12,kv=12,logm=66} crc {auth=3595534027,config=3541143443,health=1083265261,kv=4017288519,logm=4073505531}) 2024-09-16T04:40:31.337 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.335+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=9,config=2,health=12,kv=12,logm=65} crc {auth=3746612619,config=3541143443,health=1083265261,kv=4017288519,logm=2448209148}) 2024-09-16T04:40:31.339 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.336+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:40:31.339 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.336+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=75691532}) 2024-09-16T04:40:31.339 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.336+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3912520000}) 2024-09-16T04:40:31.339 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.336+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:40:31.339 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.336+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=75691532}) 2024-09-16T04:40:31.339 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.336+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1289920236}) 2024-09-16T04:40:31.339 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.337+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:40:31.339 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.337+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2565400940}) 2024-09-16T04:40:31.339 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.337+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3881422994}) 2024-09-16T04:40:31.339 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.337+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:40:31.339 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.337+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2565400940}) 2024-09-16T04:40:31.340 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.337+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2023299839}) 2024-09-16T04:40:31.340 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.338+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:40:31.341 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.338+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=635655599}) 2024-09-16T04:40:31.341 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.338+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3881073413}) 2024-09-16T04:40:31.341 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.338+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:40:31.341 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.338+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=635655599}) 2024-09-16T04:40:31.341 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.338+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=27671431}) 2024-09-16T04:40:31.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.338+0000 7f6284ed9640 1 -- 172.21.15.46:0/3606257090 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f62600383b0 msgr2=0x7f626003a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:31.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.338+0000 7f6284ed9640 1 --2- 172.21.15.46:0/3606257090 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f62600383b0 0x7f626003a870 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6270000f60 tx=0x7f6270009040 comp rx=0 tx=0).stop 2024-09-16T04:40:31.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.338+0000 7f6284ed9640 10 monclient: shutdown 2024-09-16T04:40:31.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.338+0000 7f6284ed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:31.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.338+0000 7f6284ed9640 1 -- 172.21.15.46:0/3606257090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62800a3ff0 msgr2=0x7f628009e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:31.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.338+0000 7f6284ed9640 1 --2- 172.21.15.46:0/3606257090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62800a3ff0 0x7f628009e030 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6268031000 tx=0x7f62680046c0 comp rx=0 tx=0).stop 2024-09-16T04:40:31.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.339+0000 7f6284ed9640 1 -- 172.21.15.46:0/3606257090 shutdown_connections 2024-09-16T04:40:31.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.339+0000 7f6284ed9640 1 --2- 172.21.15.46:0/3606257090 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f628013be90 0x7f628009e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:31.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.339+0000 7f6284ed9640 1 --2- 172.21.15.46:0/3606257090 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f62600383b0 0x7f626003a870 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:31.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.339+0000 7f6284ed9640 1 --2- 172.21.15.46:0/3606257090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62800a3ff0 0x7f628009e030 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:31.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.339+0000 7f6284ed9640 1 --2- 172.21.15.46:0/3606257090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62800a3c10 0x7f62801370b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:31.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.339+0000 7f6284ed9640 1 -- 172.21.15.46:0/3606257090 >> 172.21.15.46:0/3606257090 conn(0x7f62800a5d10 msgr2=0x7f628013a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:31.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.339+0000 7f6284ed9640 1 -- 172.21.15.46:0/3606257090 shutdown_connections 2024-09-16T04:40:31.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:31.339+0000 7f6284ed9640 1 -- 172.21.15.46:0/3606257090 wait complete. 2024-09-16T04:40:31.345 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.339+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:40:31.345 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.339+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1746021623}) 2024-09-16T04:40:31.345 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.339+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2339046412}) 2024-09-16T04:40:31.346 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.339+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:40:31.346 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.339+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1746021623}) 2024-09-16T04:40:31.346 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.339+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2043567860}) 2024-09-16T04:40:31.346 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.340+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:40:31.346 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.340+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=78,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9} crc {logm=4092296190,mdsmap=3956149183,mgr=1081907435,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=2237398787}) 2024-09-16T04:40:31.346 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.340+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=79,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9} crc {logm=1354477078,mdsmap=3956149183,mgr=2343726890,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=2237398787}) 2024-09-16T04:40:31.346 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.340+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:40:31.346 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.340+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=78,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9} crc {logm=4092296190,mdsmap=3956149183,mgr=1081907435,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=2237398787}) 2024-09-16T04:40:31.346 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.340+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=82,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=3026143136,mdsmap=571966421,mgr=1081907435,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=77798386}) 2024-09-16T04:40:31.346 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.341+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:40:31.346 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.341+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=74} crc {mon_config_key=1385660123,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=681133776,osd_snap=723727394,osdmap=1827250413}) 2024-09-16T04:40:31.346 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.341+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=74} crc {mon_config_key=1385660123,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=681133776,osd_snap=723727393,osdmap=4022574312}) 2024-09-16T04:40:31.346 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.341+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:40:31.346 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.341+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=74} crc {mon_config_key=1385660123,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=681133776,osd_snap=723727394,osdmap=1827250413}) 2024-09-16T04:40:31.347 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.341+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=74} crc {mon_config_key=1385660123,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=681133776,osd_snap=723727393,osdmap=715250757}) 2024-09-16T04:40:31.347 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.343+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:40:31.347 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.343+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=51} crc {osdmap=939206429}) 2024-09-16T04:40:31.347 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.343+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=49} crc {osdmap=3814998607}) 2024-09-16T04:40:31.347 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.343+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:40:31.347 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.343+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=51} crc {osdmap=939206429}) 2024-09-16T04:40:31.347 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:40:31.343+0000 7fc4c4045640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=52} crc {osdmap=910929289}) 2024-09-16T04:40:31.352 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:40:32.353 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:40:32.353 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:40:32.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.465+0000 7fe4cfc65640 10 monclient: get_monmap_and_config 2024-09-16T04:40:32.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.465+0000 7fe4cfc65640 10 monclient: build_initial_monmap 2024-09-16T04:40:32.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.465+0000 7fe4cfc65640 10 monclient: monmap: 2024-09-16T04:40:32.467 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:32.467 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:32.467 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:40:32.466384+0000 2024-09-16T04:40:32.467 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:40:32.466384+0000 2024-09-16T04:40:32.467 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:32.467 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:32.467 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:40:32.467 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:40:32.468 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:40:32.468 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:40:32.468 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:40:32.468 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:32.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.466+0000 7fe4cfc65640 1 Processor -- start 2024-09-16T04:40:32.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.467+0000 7fe4cfc65640 1 -- start start 2024-09-16T04:40:32.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.467+0000 7fe4cfc65640 10 monclient: init 2024-09-16T04:40:32.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.467+0000 7fe4cfc65640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:32.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.467+0000 7fe4cfc65640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-16T04:40:32.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.467+0000 7fe4cfc65640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4c806e160 0x7fe4c80a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:32.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.467+0000 7fe4cfc65640 10 monclient(hunting): picked mon.noname-a con 0x7fe4c806e160 addr v2:172.21.15.46:3300/0 2024-09-16T04:40:32.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.467+0000 7fe4cfc65640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe4c80a2ee0 0x7fe4c8136940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:32.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.467+0000 7fe4cfc65640 10 monclient(hunting): picked mon.noname-c con 0x7fe4c80a2ee0 addr v2:172.21.15.114:3300/0 2024-09-16T04:40:32.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.467+0000 7fe4cfc65640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8136e80 0x7fe4c81372f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:32.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.467+0000 7fe4cfc65640 10 monclient(hunting): picked mon.noname-b con 0x7fe4c8136e80 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:32.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.467+0000 7fe4cfc65640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:32.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.467+0000 7fe4cfc65640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe4c80a3300 con 0x7fe4c806e160 2024-09-16T04:40:32.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.467+0000 7fe4cfc65640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:32.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.467+0000 7fe4cfc65640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fe4c806e540 con 0x7fe4c80a2ee0 2024-09-16T04:40:32.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.467+0000 7fe4cfc65640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:32.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.467+0000 7fe4cfc65640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe4c80a3fd0 con 0x7fe4c8136e80 2024-09-16T04:40:32.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.467+0000 7fe4cfc65640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:32.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.467+0000 7fe4cfc65640 10 monclient(hunting): authenticate will time out at 830.882768s 2024-09-16T04:40:32.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cda01640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4c806e160 0x7fe4c80a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:32.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cda01640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4c806e160 0x7fe4c80a29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58494/0 (socket says 172.21.15.46:58494) 2024-09-16T04:40:32.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4ce202640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8136e80 0x7fe4c81372f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:32.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cda01640 1 -- 172.21.15.46:0/3346102484 learned_addr learned my addr 172.21.15.46:0/3346102484 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:32.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cda01640 10 monclient(hunting): get_auth_request con 0x7fe4c806e160 auth_method 0 2024-09-16T04:40:32.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cda01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cda01640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cda01640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4ce202640 10 monclient(hunting): get_auth_request con 0x7fe4c8136e80 auth_method 0 2024-09-16T04:40:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4ce202640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4ce202640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4ce202640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cda01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cda01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cda01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cd200640 1 --2- 172.21.15.46:0/3346102484 >> v2:172.21.15.114:3300/0 conn(0x7fe4c80a2ee0 0x7fe4c8136940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4ce202640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4ce202640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4ce202640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cd200640 10 monclient(hunting): get_auth_request con 0x7fe4c80a2ee0 auth_method 0 2024-09-16T04:40:32.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cd200640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:32.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cd200640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:32.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cd200640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:32.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cda01640 10 monclient(hunting): handle_auth_done global_id 4959 payload 274 2024-09-16T04:40:32.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cda01640 1 -- 172.21.15.46:0/3346102484 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8136e80 msgr2=0x7fe4c81372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:40:32.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cda01640 1 --2- 172.21.15.46:0/3346102484 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8136e80 0x7fe4c81372f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:32.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cda01640 1 -- 172.21.15.46:0/3346102484 >> v2:172.21.15.114:3300/0 conn(0x7fe4c80a2ee0 msgr2=0x7fe4c8136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:40:32.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cda01640 1 --2- 172.21.15.46:0/3346102484 >> v2:172.21.15.114:3300/0 conn(0x7fe4c80a2ee0 0x7fe4c8136940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:32.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cda01640 10 monclient: _finish_hunting 0 2024-09-16T04:40:32.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cda01640 1 monclient: found mon.noname-a 2024-09-16T04:40:32.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cda01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:32.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cda01640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:40:32.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cda01640 1 -- 172.21.15.46:0/3346102484 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c80a0bd0 con 0x7fe4c806e160 2024-09-16T04:40:32.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4cd200640 1 --2- 172.21.15.46:0/3346102484 >> v2:172.21.15.114:3300/0 conn(0x7fe4c80a2ee0 0x7fe4c8136940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:40:32.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.468+0000 7fe4ce202640 1 --2- 172.21.15.46:0/3346102484 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8136e80 0x7fe4c81372f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:40:32.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.469+0000 7fe4cda01640 1 --2- 172.21.15.46:0/3346102484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4c806e160 0x7fe4c80a29a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe4c809fec0 tx=0x7fe4c402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21d55a4574de0cf2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:32.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.469+0000 7fe4cc9ff640 1 -- 172.21.15.46:0/3346102484 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4037070 con 0x7fe4c806e160 2024-09-16T04:40:32.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.469+0000 7fe4cc9ff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:32.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.469+0000 7fe4cc9ff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:40:32.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.469+0000 7fe4cc9ff640 10 monclient: dump: 2024-09-16T04:40:32.476 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:32.476 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:32.476 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:32.477 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:32.477 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:32.477 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:32.477 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:32.477 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:32.477 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:32.477 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:32.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.469+0000 7fe4cc9ff640 10 monclient: _finish_auth 0 2024-09-16T04:40:32.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.469+0000 7fe4cc9ff640 10 monclient: _check_auth_tickets 2024-09-16T04:40:32.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.469+0000 7fe4cc9ff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:32.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.469+0000 7fe4cfc65640 5 monclient: authenticate success, global_id 4959 2024-09-16T04:40:32.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.469+0000 7fe4cc9ff640 1 -- 172.21.15.46:0/3346102484 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4c402ae10 con 0x7fe4c806e160 2024-09-16T04:40:32.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.469+0000 7fe4cfc65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:40:32.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.469+0000 7fe4cc9ff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:32.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.469+0000 7fe4cfc65640 10 monclient: get_monmap_and_config success 2024-09-16T04:40:32.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.469+0000 7fe4cfc65640 4 set_mon_vals no callback set 2024-09-16T04:40:32.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.469+0000 7fe4cc9ff640 1 -- 172.21.15.46:0/3346102484 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4032660 con 0x7fe4c806e160 2024-09-16T04:40:32.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.469+0000 7fe4cc9ff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:32.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.469+0000 7fe4cc9ff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:32.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.469+0000 7fe4cc9ff640 10 monclient: dump: 2024-09-16T04:40:32.478 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:32.478 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:32.478 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:32.478 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:32.478 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:32.478 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:32.478 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:32.478 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:32.479 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:32.479 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:32.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.470+0000 7fe4cfc65640 10 monclient: shutdown 2024-09-16T04:40:32.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.470+0000 7fe4cfc65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:32.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.470+0000 7fe4cfc65640 1 -- 172.21.15.46:0/3346102484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4c806e160 msgr2=0x7fe4c80a29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:32.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.470+0000 7fe4cfc65640 1 --2- 172.21.15.46:0/3346102484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4c806e160 0x7fe4c80a29a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe4c809fec0 tx=0x7fe4c402a100 comp rx=0 tx=0).stop 2024-09-16T04:40:32.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.470+0000 7fe4cfc65640 1 -- 172.21.15.46:0/3346102484 shutdown_connections 2024-09-16T04:40:32.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.470+0000 7fe4cfc65640 1 --2- 172.21.15.46:0/3346102484 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8136e80 0x7fe4c81372f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:32.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.470+0000 7fe4cfc65640 1 --2- 172.21.15.46:0/3346102484 >> v2:172.21.15.114:3300/0 conn(0x7fe4c80a2ee0 0x7fe4c8136940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:32.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.470+0000 7fe4cfc65640 1 --2- 172.21.15.46:0/3346102484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4c806e160 0x7fe4c80a29a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:32.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.470+0000 7fe4cfc65640 1 -- 172.21.15.46:0/3346102484 >> 172.21.15.46:0/3346102484 conn(0x7fe4c80a5c10 msgr2=0x7fe4c80a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:32.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.470+0000 7fe4cfc65640 1 -- 172.21.15.46:0/3346102484 shutdown_connections 2024-09-16T04:40:32.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.470+0000 7fe4cfc65640 1 -- 172.21.15.46:0/3346102484 wait complete. 2024-09-16T04:40:32.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.470+0000 7fe4cfc65640 10 monclient: build_initial_monmap 2024-09-16T04:40:32.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.470+0000 7fe4cfc65640 10 monclient: monmap: 2024-09-16T04:40:32.480 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:32.480 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:40:32.480 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:40:32.480 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:40:32.480 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:32.480 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:32.480 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:40:32.480 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:40:32.480 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:40:32.480 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:32.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cfc65640 1 Processor -- start 2024-09-16T04:40:32.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cfc65640 1 -- start start 2024-09-16T04:40:32.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cfc65640 10 monclient: init 2024-09-16T04:40:32.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cfc65640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:32.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cfc65640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:40:32.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cfc65640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c806e160 0x7fe4c8135de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:32.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cfc65640 10 monclient(hunting): picked mon.noname-c con 0x7fe4c806e160 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:32.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cfc65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4c80a2ee0 0x7fe4c8136320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:32.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cfc65640 10 monclient(hunting): picked mon.noname-a con 0x7fe4c80a2ee0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:40:32.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cfc65640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4c8136e80 0x7fe4c81238d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:32.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cfc65640 10 monclient(hunting): picked mon.noname-b con 0x7fe4c8136e80 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:40:32.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cfc65640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:32.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cfc65640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c80a5130 con 0x7fe4c80a2ee0 2024-09-16T04:40:32.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cfc65640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:32.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cfc65640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c80a1190 con 0x7fe4c8136e80 2024-09-16T04:40:32.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cfc65640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:32.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cfc65640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe4c80a34a0 con 0x7fe4c806e160 2024-09-16T04:40:32.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cfc65640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:32.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cfc65640 10 monclient(hunting): authenticate will time out at 830.886899s 2024-09-16T04:40:32.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cda01640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c806e160 0x7fe4c8135de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:32.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cd200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4c80a2ee0 0x7fe4c8136320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:32.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.471+0000 7fe4cda01640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c806e160 0x7fe4c8135de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:44124/0 (socket says 172.21.15.46:44124) 2024-09-16T04:40:32.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cda01640 1 -- 172.21.15.46:0/2547648300 learned_addr learned my addr 172.21.15.46:0/2547648300 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:32.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cda01640 10 monclient(hunting): get_auth_request con 0x7fe4c806e160 auth_method 0 2024-09-16T04:40:32.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cda01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:32.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cda01640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:32.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cda01640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:32.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cd200640 10 monclient(hunting): get_auth_request con 0x7fe4c80a2ee0 auth_method 0 2024-09-16T04:40:32.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cd200640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:32.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cd200640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:32.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cd200640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:32.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cda01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:32.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cda01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:32.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cda01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:32.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cd200640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:32.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cd200640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:32.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cd200640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:32.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4ce202640 1 --2- 172.21.15.46:0/2547648300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4c8136e80 0x7fe4c81238d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:32.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cd200640 10 monclient(hunting): handle_auth_done global_id 4962 payload 824 2024-09-16T04:40:32.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cd200640 1 -- 172.21.15.46:0/2547648300 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c806e160 msgr2=0x7fe4c8135de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:32.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cd200640 1 --2- 172.21.15.46:0/2547648300 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c806e160 0x7fe4c8135de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:32.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cd200640 1 -- 172.21.15.46:0/2547648300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4c8136e80 msgr2=0x7fe4c81238d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:32.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cd200640 1 --2- 172.21.15.46:0/2547648300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4c8136e80 0x7fe4c81238d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:32.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cd200640 10 monclient: _finish_hunting 0 2024-09-16T04:40:32.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cd200640 1 monclient: found mon.noname-a 2024-09-16T04:40:32.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cd200640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:32.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cd200640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:40:32.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cd200640 1 -- 172.21.15.46:0/2547648300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c81277e0 con 0x7fe4c80a2ee0 2024-09-16T04:40:32.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4ce202640 10 monclient: get_auth_request con 0x7fe4c8136e80 auth_method 0 2024-09-16T04:40:32.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4ce202640 1 --2- 172.21.15.46:0/2547648300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4c8136e80 0x7fe4c81238d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:40:32.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cd200640 1 --2- 172.21.15.46:0/2547648300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4c80a2ee0 0x7fe4c8136320 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8006fd0 tx=0x7fe4b800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:32.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4be7fc640 1 -- 172.21.15.46:0/2547648300 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b800f040 con 0x7fe4c80a2ee0 2024-09-16T04:40:32.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:32.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4be7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:40:32.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4be7fc640 10 monclient: dump: 2024-09-16T04:40:32.488 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:32.489 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:32.489 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:32.489 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:32.489 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:32.489 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:32.489 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:32.489 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:32.489 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:32.489 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:32.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4be7fc640 10 monclient: _finish_auth 0 2024-09-16T04:40:32.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4be7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:40:32.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:32.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4be7fc640 1 -- 172.21.15.46:0/2547648300 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4b8004030 con 0x7fe4c80a2ee0 2024-09-16T04:40:32.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cfc65640 5 monclient: authenticate success, global_id 4962 2024-09-16T04:40:32.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cfc65640 10 monclient: _renew_subs 2024-09-16T04:40:32.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cfc65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:32.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cfc65640 1 -- 172.21.15.46:0/2547648300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4c80a7f20 con 0x7fe4c80a2ee0 2024-09-16T04:40:32.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4be7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:32.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4be7fc640 1 -- 172.21.15.46:0/2547648300 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b80043e0 con 0x7fe4c80a2ee0 2024-09-16T04:40:32.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:32.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:32.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4be7fc640 10 monclient: dump: 2024-09-16T04:40:32.490 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:32.490 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:32.490 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:32.490 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:32.490 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:32.490 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:32.491 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:32.491 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:32.491 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:32.491 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:32.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4bffff640 4 set_mon_vals no callback set 2024-09-16T04:40:32.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cfc65640 10 monclient: _renew_subs 2024-09-16T04:40:32.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cfc65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:32.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.472+0000 7fe4cfc65640 1 -- 172.21.15.46:0/2547648300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4c80a8470 con 0x7fe4c80a2ee0 2024-09-16T04:40:32.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.473+0000 7fe4be7fc640 1 -- 172.21.15.46:0/2547648300 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fe4b800c6d0 con 0x7fe4c80a2ee0 2024-09-16T04:40:32.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.473+0000 7fe4cfc65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:32.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.473+0000 7fe4cfc65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:32.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.473+0000 7fe4cfc65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:32.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.473+0000 7fe4cfc65640 1 -- 172.21.15.46:0/2547648300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4c80a4d90 con 0x7fe4c80a2ee0 2024-09-16T04:40:32.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.473+0000 7fe4be7fc640 1 --2- 172.21.15.46:0/2547648300 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe4ac0383b0 0x7fe4ac03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:32.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.473+0000 7fe4be7fc640 1 -- 172.21.15.46:0/2547648300 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 4738+0+0 (secure 0 0 0) 0x7fe4b804b410 con 0x7fe4c80a2ee0 2024-09-16T04:40:32.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.474+0000 7fe4cda01640 1 --2- 172.21.15.46:0/2547648300 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe4ac0383b0 0x7fe4ac03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:32.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.474+0000 7fe4cda01640 10 monclient: get_auth_request con 0x7fe4ac0383b0 auth_method 0 2024-09-16T04:40:32.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.474+0000 7fe4cda01640 1 --2- 172.21.15.46:0/2547648300 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe4ac0383b0 0x7fe4ac03a870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe4c40090b0 tx=0x7fe4c402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:32.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.479+0000 7fe4be7fc640 1 -- 172.21.15.46:0/2547648300 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe4b8017020 con 0x7fe4c80a2ee0 2024-09-16T04:40:32.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.479+0000 7fe4be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:32.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.479+0000 7fe4be7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:40:32.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.724+0000 7fe4cfc65640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:40:32.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.724+0000 7fe4cfc65640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:40:32.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.724+0000 7fe4cfc65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:32.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.724+0000 7fe4cfc65640 1 -- 172.21.15.46:0/2547648300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe4c8127040 con 0x7fe4c80a2ee0 2024-09-16T04:40:32.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.725+0000 7fe4be7fc640 1 -- 172.21.15.46:0/2547648300 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fe4c8127040 con 0x7fe4c80a2ee0 2024-09-16T04:40:32.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.725+0000 7fe4be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:40:32.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.725+0000 7fe4be7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:40:32.727 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:40:32.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.727+0000 7fe4cfc65640 1 -- 172.21.15.46:0/2547648300 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe4ac0383b0 msgr2=0x7fe4ac03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:32.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.727+0000 7fe4cfc65640 1 --2- 172.21.15.46:0/2547648300 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe4ac0383b0 0x7fe4ac03a870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe4c40090b0 tx=0x7fe4c402e000 comp rx=0 tx=0).stop 2024-09-16T04:40:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.727+0000 7fe4cfc65640 10 monclient: shutdown 2024-09-16T04:40:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.727+0000 7fe4cfc65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.727+0000 7fe4cfc65640 1 -- 172.21.15.46:0/2547648300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4c80a2ee0 msgr2=0x7fe4c8136320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.727+0000 7fe4cfc65640 1 --2- 172.21.15.46:0/2547648300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4c80a2ee0 0x7fe4c8136320 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8006fd0 tx=0x7fe4b800ec70 comp rx=0 tx=0).stop 2024-09-16T04:40:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.727+0000 7fe4cfc65640 1 -- 172.21.15.46:0/2547648300 shutdown_connections 2024-09-16T04:40:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.727+0000 7fe4cfc65640 1 --2- 172.21.15.46:0/2547648300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4c8136e80 0x7fe4c81238d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.727+0000 7fe4cfc65640 1 --2- 172.21.15.46:0/2547648300 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe4ac0383b0 0x7fe4ac03a870 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.727+0000 7fe4cfc65640 1 --2- 172.21.15.46:0/2547648300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4c80a2ee0 0x7fe4c8136320 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.727+0000 7fe4cfc65640 1 --2- 172.21.15.46:0/2547648300 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c806e160 0x7fe4c8135de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.727+0000 7fe4cfc65640 1 -- 172.21.15.46:0/2547648300 >> 172.21.15.46:0/2547648300 conn(0x7fe4c80a5c10 msgr2=0x7fe4c80a6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.727+0000 7fe4cfc65640 1 -- 172.21.15.46:0/2547648300 shutdown_connections 2024-09-16T04:40:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.727+0000 7fe4cfc65640 1 -- 172.21.15.46:0/2547648300 wait complete. 2024-09-16T04:40:32.739 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:40:32.739 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:40:32.739 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:40:32.739 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:40:32.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.840+0000 7fd71847b640 10 monclient: get_monmap_and_config 2024-09-16T04:40:32.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.840+0000 7fd71847b640 10 monclient: build_initial_monmap 2024-09-16T04:40:32.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.840+0000 7fd71847b640 10 monclient: monmap: 2024-09-16T04:40:32.842 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:32.842 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:32.842 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:40:32.841014+0000 2024-09-16T04:40:32.842 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:40:32.841014+0000 2024-09-16T04:40:32.842 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:32.842 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:32.842 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:40:32.842 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:40:32.842 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:40:32.842 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:40:32.843 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:40:32.843 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.841+0000 7fd71847b640 1 Processor -- start 2024-09-16T04:40:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.841+0000 7fd71847b640 1 -- start start 2024-09-16T04:40:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.841+0000 7fd71847b640 10 monclient: init 2024-09-16T04:40:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.841+0000 7fd71847b640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.841+0000 7fd71847b640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-16T04:40:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.841+0000 7fd71847b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd7100a3c10 0x7fd71009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.841+0000 7fd71847b640 10 monclient(hunting): picked mon.noname-a con 0x7fd7100a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:40:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.841+0000 7fd71847b640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd7100a3ff0 0x7fd7100a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.841+0000 7fd71847b640 10 monclient(hunting): picked mon.noname-c con 0x7fd7100a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:40:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.841+0000 7fd71847b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd7101369d0 addr v1:172.21.15.114:6789/0 2024-09-16T04:40:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.841+0000 7fd71847b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71009b940 con 0x7fd7101369d0 2024-09-16T04:40:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.841+0000 7fd71847b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.841+0000 7fd71847b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd71009ab00 con 0x7fd7100a3c10 2024-09-16T04:40:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.841+0000 7fd71847b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.841+0000 7fd71847b640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fd71009b030 con 0x7fd7100a3ff0 2024-09-16T04:40:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.841+0000 7fd71847b640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.841+0000 7fd71847b640 10 monclient(hunting): authenticate will time out at 831.256822s 2024-09-16T04:40:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.841+0000 7fd716217640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd7100a3c10 0x7fd71009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd716217640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd7100a3c10 0x7fd71009bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58520/0 (socket says 172.21.15.46:58520) 2024-09-16T04:40:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd716217640 1 -- 172.21.15.46:0/4098935101 learned_addr learned my addr 172.21.15.46:0/4098935101 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd716217640 10 monclient(hunting): get_auth_request con 0x7fd7100a3c10 auth_method 0 2024-09-16T04:40:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd716217640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd716217640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd716217640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd715a16640 1 --2- 172.21.15.46:0/4098935101 >> v2:172.21.15.114:3300/0 conn(0x7fd7100a3ff0 0x7fd7100a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd716217640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd716217640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd716217640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd715a16640 10 monclient(hunting): get_auth_request con 0x7fd7100a3ff0 auth_method 0 2024-09-16T04:40:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd715a16640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd715a16640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd715a16640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd715a16640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd715a16640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd715a16640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd716217640 10 monclient(hunting): handle_auth_done global_id 4965 payload 274 2024-09-16T04:40:32.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd716217640 1 -- 172.21.15.46:0/4098935101 >> v2:172.21.15.114:3300/0 conn(0x7fd7100a3ff0 msgr2=0x7fd7100a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:40:32.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd716217640 1 --2- 172.21.15.46:0/4098935101 >> v2:172.21.15.114:3300/0 conn(0x7fd7100a3ff0 0x7fd7100a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:32.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd716217640 1 -- 172.21.15.46:0/4098935101 >> v1:172.21.15.114:6789/0 conn(0x7fd7101369d0 legacy=0x7fd710136e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:32.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd716217640 10 monclient: _finish_hunting 0 2024-09-16T04:40:32.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd716217640 1 monclient: found mon.noname-a 2024-09-16T04:40:32.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd716217640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:32.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd716217640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:40:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.842+0000 7fd716217640 1 -- 172.21.15.46:0/4098935101 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71009b310 con 0x7fd7100a3c10 2024-09-16T04:40:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd716217640 1 --2- 172.21.15.46:0/4098935101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7100a3c10 0x7fd71009bcf0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd7100a38d0 tx=0x7fd70c02a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f411e2c1b454924 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd715215640 1 -- 172.21.15.46:0/4098935101 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c037070 con 0x7fd7100a3c10 2024-09-16T04:40:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd715215640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd715215640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:40:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd715215640 10 monclient: dump: 2024-09-16T04:40:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:32.862 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd715215640 10 monclient: _finish_auth 0 2024-09-16T04:40:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd715215640 10 monclient: _check_auth_tickets 2024-09-16T04:40:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd715215640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd715215640 1 -- 172.21.15.46:0/4098935101 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd70c032440 con 0x7fd7100a3c10 2024-09-16T04:40:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd71847b640 5 monclient: authenticate success, global_id 4965 2024-09-16T04:40:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd71847b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:40:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd715215640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd71847b640 10 monclient: get_monmap_and_config success 2024-09-16T04:40:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd715215640 1 -- 172.21.15.46:0/4098935101 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c032810 con 0x7fd7100a3c10 2024-09-16T04:40:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd71847b640 4 set_mon_vals no callback set 2024-09-16T04:40:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd715215640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd715215640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd715215640 10 monclient: dump: 2024-09-16T04:40:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:32.864 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:32.864 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:32.864 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:32.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd71847b640 10 monclient: shutdown 2024-09-16T04:40:32.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd71847b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:32.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd71847b640 1 -- 172.21.15.46:0/4098935101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7100a3c10 msgr2=0x7fd71009bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:32.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd71847b640 1 --2- 172.21.15.46:0/4098935101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7100a3c10 0x7fd71009bcf0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd7100a38d0 tx=0x7fd70c02a150 comp rx=0 tx=0).stop 2024-09-16T04:40:32.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd71847b640 1 -- 172.21.15.46:0/4098935101 shutdown_connections 2024-09-16T04:40:32.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd71847b640 1 --2- 172.21.15.46:0/4098935101 >> v2:172.21.15.114:3300/0 conn(0x7fd7100a3ff0 0x7fd7100a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:32.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd71847b640 1 --2- 172.21.15.46:0/4098935101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7100a3c10 0x7fd71009bcf0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:32.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd71847b640 1 -- 172.21.15.46:0/4098935101 >> 172.21.15.46:0/4098935101 conn(0x7fd7100a6340 msgr2=0x7fd71006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:32.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd71847b640 1 -- 172.21.15.46:0/4098935101 shutdown_connections 2024-09-16T04:40:32.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.843+0000 7fd71847b640 1 -- 172.21.15.46:0/4098935101 wait complete. 2024-09-16T04:40:32.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.844+0000 7fd71847b640 10 monclient: build_initial_monmap 2024-09-16T04:40:32.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.844+0000 7fd71847b640 10 monclient: monmap: 2024-09-16T04:40:32.865 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:32.865 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:40:32.865 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:40:32.865 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:40:32.865 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:40:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:40:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:40:32.866 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.844+0000 7fd71847b640 1 Processor -- start 2024-09-16T04:40:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.844+0000 7fd71847b640 1 -- start start 2024-09-16T04:40:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.844+0000 7fd71847b640 10 monclient: init 2024-09-16T04:40:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd71847b640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd71847b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:40:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd71847b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7100a3c10 0x7fd710135b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd71847b640 10 monclient(hunting): picked mon.noname-c con 0x7fd7100a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd71847b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7100a3ff0 0x7fd710136040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd71847b640 10 monclient(hunting): picked mon.noname-a con 0x7fd7100a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:40:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd71847b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7101369d0 0x7fd710123570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd71847b640 10 monclient(hunting): picked mon.noname-b con 0x7fd7101369d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:40:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd71847b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd71847b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd7100a4910 con 0x7fd7100a3ff0 2024-09-16T04:40:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd71847b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd71847b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd7100a69c0 con 0x7fd7101369d0 2024-09-16T04:40:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd71847b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd71847b640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd7101cccb0 con 0x7fd7100a3c10 2024-09-16T04:40:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd71847b640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd71847b640 10 monclient(hunting): authenticate will time out at 831.260233s 2024-09-16T04:40:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd716217640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7100a3c10 0x7fd710135b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd715a16640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7100a3ff0 0x7fd710136040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd716217640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7100a3c10 0x7fd710135b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:44128/0 (socket says 172.21.15.46:44128) 2024-09-16T04:40:32.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd716217640 1 -- 172.21.15.46:0/3490122153 learned_addr learned my addr 172.21.15.46:0/3490122153 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:32.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd715a16640 10 monclient(hunting): get_auth_request con 0x7fd7100a3ff0 auth_method 0 2024-09-16T04:40:32.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd715a16640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:32.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd715a16640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:32.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd715a16640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:32.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd716217640 10 monclient(hunting): get_auth_request con 0x7fd7100a3c10 auth_method 0 2024-09-16T04:40:32.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd716217640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:32.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd716217640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:32.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd716217640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:32.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd716a18640 1 --2- 172.21.15.46:0/3490122153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7101369d0 0x7fd710123570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:32.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd715a16640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:32.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd715a16640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:32.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd715a16640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:32.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd716a18640 10 monclient(hunting): get_auth_request con 0x7fd7101369d0 auth_method 0 2024-09-16T04:40:32.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd716a18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:32.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd716a18640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:32.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd716a18640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:32.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd716217640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:32.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd716217640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:32.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd716217640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:32.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd716a18640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:32.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd716a18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:32.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd716a18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:32.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd715a16640 10 monclient(hunting): handle_auth_done global_id 4968 payload 824 2024-09-16T04:40:32.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd715a16640 1 -- 172.21.15.46:0/3490122153 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7100a3c10 msgr2=0x7fd710135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:32.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd715a16640 1 --2- 172.21.15.46:0/3490122153 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7100a3c10 0x7fd710135b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:32.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd715a16640 1 -- 172.21.15.46:0/3490122153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7101369d0 msgr2=0x7fd710123570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:32.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd715a16640 1 --2- 172.21.15.46:0/3490122153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7101369d0 0x7fd710123570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:32.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd715a16640 10 monclient: _finish_hunting 0 2024-09-16T04:40:32.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd715a16640 1 monclient: found mon.noname-a 2024-09-16T04:40:32.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd715a16640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:32.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd715a16640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:40:32.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd715a16640 1 -- 172.21.15.46:0/3490122153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7101256b0 con 0x7fd7100a3ff0 2024-09-16T04:40:32.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd715a16640 1 --2- 172.21.15.46:0/3490122153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7100a3ff0 0x7fd710136040 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd70000e850 tx=0x7fd70000ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:32.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd706ffd640 1 -- 172.21.15.46:0/3490122153 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd700017070 con 0x7fd7100a3ff0 2024-09-16T04:40:32.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd706ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:32.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd706ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:40:32.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd706ffd640 10 monclient: dump: 2024-09-16T04:40:32.872 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:32.872 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:32.872 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:32.872 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:32.872 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:32.872 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:32.872 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:32.872 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:32.872 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:32.873 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:32.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.845+0000 7fd706ffd640 10 monclient: _finish_auth 0 2024-09-16T04:40:32.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd706ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:40:32.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd706ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:32.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd706ffd640 1 -- 172.21.15.46:0/3490122153 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd70000c8e0 con 0x7fd7100a3ff0 2024-09-16T04:40:32.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd71847b640 5 monclient: authenticate success, global_id 4968 2024-09-16T04:40:32.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd71847b640 10 monclient: _renew_subs 2024-09-16T04:40:32.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd71847b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:32.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd71847b640 1 -- 172.21.15.46:0/3490122153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7100a2b00 con 0x7fd7100a3ff0 2024-09-16T04:40:32.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd706ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:32.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd706ffd640 1 -- 172.21.15.46:0/3490122153 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70000ccb0 con 0x7fd7100a3ff0 2024-09-16T04:40:32.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd706ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:32.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd706ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd706ffd640 10 monclient: dump: 2024-09-16T04:40:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:32.874 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd71847b640 10 monclient: _renew_subs 2024-09-16T04:40:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd71847b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:40:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd71847b640 1 -- 172.21.15.46:0/3490122153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd710126f40 con 0x7fd7100a3ff0 2024-09-16T04:40:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd707fff640 4 set_mon_vals no callback set 2024-09-16T04:40:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd71847b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd71847b640 1 --2- 172.21.15.46:0/3490122153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7101cbfc0 0x7fd710123ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd71847b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd71847b640 1 -- 172.21.15.46:0/3490122153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd7100a4a90 con 0x7fd7101cbfc0 2024-09-16T04:40:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd706ffd640 1 -- 172.21.15.46:0/3490122153 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fd70001a070 con 0x7fd7100a3ff0 2024-09-16T04:40:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd716217640 1 --2- 172.21.15.46:0/3490122153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7101cbfc0 0x7fd710123ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd716217640 10 monclient: get_auth_request con 0x7fd7101cbfc0 auth_method 0 2024-09-16T04:40:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd716217640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd716217640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.846+0000 7fd716217640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd716217640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd716217640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd716217640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd716217640 10 monclient(hunting): handle_auth_done global_id 4971 payload 824 2024-09-16T04:40:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd716217640 1 -- 172.21.15.46:0/3490122153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd71009fff0 con 0x7fd7101cbfc0 2024-09-16T04:40:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd706ffd640 1 --2- 172.21.15.46:0/3490122153 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd6f40383e0 0x7fd6f403a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd706ffd640 1 -- 172.21.15.46:0/3490122153 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 4738+0+0 (secure 0 0 0) 0x7fd70004b6d0 con 0x7fd7100a3ff0 2024-09-16T04:40:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd716217640 1 --2- 172.21.15.46:0/3490122153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7101cbfc0 0x7fd710123ab0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0090b0 tx=0x7fd70c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd716a18640 1 --2- 172.21.15.46:0/3490122153 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd6f40383e0 0x7fd6f403a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd716a18640 10 monclient: get_auth_request con 0x7fd6f40383e0 auth_method 0 2024-09-16T04:40:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd706ffd640 1 -- 172.21.15.46:0/3490122153 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c043070 con 0x7fd7101cbfc0 2024-09-16T04:40:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd706ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd706ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd706ffd640 10 monclient: dump: 2024-09-16T04:40:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:32.877 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd716a18640 1 --2- 172.21.15.46:0/3490122153 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd6f40383e0 0x7fd6f403a8a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd7100a8630 tx=0x7fd708009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd706ffd640 1 -- 172.21.15.46:0/3490122153 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd70c03c640 con 0x7fd7101cbfc0 2024-09-16T04:40:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd706ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd706ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:40:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd706ffd640 1 -- 172.21.15.46:0/3490122153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7101cbfc0 msgr2=0x7fd710123ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd706ffd640 1 --2- 172.21.15.46:0/3490122153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7101cbfc0 0x7fd710123ab0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0090b0 tx=0x7fd70c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:40:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd706ffd640 1 -- 172.21.15.46:0/3490122153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7101cbfc0 msgr2=0x7fd710123ab0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:40:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.847+0000 7fd706ffd640 1 --2- 172.21.15.46:0/3490122153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7101cbfc0 0x7fd710123ab0 secure :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0090b0 tx=0x7fd70c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:40:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd71847b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:40:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd71847b640 1 --2- 172.21.15.46:0/3490122153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd710123ff0 0x7fd71011f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd71847b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd71847b640 1 -- 172.21.15.46:0/3490122153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd710124400 con 0x7fd710123ff0 2024-09-16T04:40:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd716217640 1 --2- 172.21.15.46:0/3490122153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd710123ff0 0x7fd71011f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd716217640 10 monclient: get_auth_request con 0x7fd710123ff0 auth_method 0 2024-09-16T04:40:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd716217640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd716217640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd716217640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd716217640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd716217640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:32.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd716217640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:32.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd716217640 10 monclient(hunting): handle_auth_done global_id 4974 payload 824 2024-09-16T04:40:32.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd716217640 1 -- 172.21.15.46:0/3490122153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd71009bcf0 con 0x7fd710123ff0 2024-09-16T04:40:32.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd716217640 1 --2- 172.21.15.46:0/3490122153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd710123ff0 0x7fd71011f860 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd710124c40 tx=0x7fd70c009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:32.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd706ffd640 1 -- 172.21.15.46:0/3490122153 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c037040 con 0x7fd710123ff0 2024-09-16T04:40:32.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd706ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:32.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd706ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:40:32.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd706ffd640 10 monclient: dump: 2024-09-16T04:40:32.879 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:32.880 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:32.880 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:32.880 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:32.880 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:32.880 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:32.880 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:32.880 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:32.880 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:32.880 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:32.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd706ffd640 1 -- 172.21.15.46:0/3490122153 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7fd70c003820 con 0x7fd710123ff0 2024-09-16T04:40:32.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd706ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:40:32.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd706ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:40:32.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd706ffd640 1 -- 172.21.15.46:0/3490122153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd710123ff0 msgr2=0x7fd71011f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:32.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd706ffd640 1 --2- 172.21.15.46:0/3490122153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd710123ff0 0x7fd71011f860 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd710124c40 tx=0x7fd70c009240 comp rx=0 tx=0).stop 2024-09-16T04:40:32.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd706ffd640 1 -- 172.21.15.46:0/3490122153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd710123ff0 msgr2=0x7fd71011f860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:40:32.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.854+0000 7fd706ffd640 1 --2- 172.21.15.46:0/3490122153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd710123ff0 0x7fd71011f860 secure :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd710124c40 tx=0x7fd70c009240 comp rx=0 tx=0).stop 2024-09-16T04:40:32.882 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:40:32.882 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:40:32.882 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:40:32.882 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:40:32.882 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 20, 2024-09-16T04:40:32.882 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 118792, 2024-09-16T04:40:32.882 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:40:32.882 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:40:32.882 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:40:32.883 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:40:32.883 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:32.883 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 7, 2024-09-16T04:40:32.883 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:40:32.883 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:40:32.883 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:40:32.883 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:32.883 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:32.883 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:32.883 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:32.883 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:32.884 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:32.884 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:32.884 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:32.884 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:32.884 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:32.884 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:32.884 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:32.884 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:40:32.884 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:40:32.884 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:32.884 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:32.884 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:32.884 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:32.885 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:32.885 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:32.885 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:32.885 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:32.885 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:32.885 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:32.885 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:32.885 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:32.885 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:32.885 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:40:32.885 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:40:32.885 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:40:32.885 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:40:32.886 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:40:32.886 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:40:32.886 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:40:32.886 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:40:32.886 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:40:32.886 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:40:32.886 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:40:32.886 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:40:32.886 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:40:32.886 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:40:32.886 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:40:32.886 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:40:32.886 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:40:32.886 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:32.887 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:32.887 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:32.887 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:32.887 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:32.887 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:32.887 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:32.887 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:32.887 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:32.888 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:32.888 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:32.888 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:32.888 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:40:32.888 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:32.888 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:40:32.888 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:32.888 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:40:32.888 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:40:32.888 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:32.888 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:32.888 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:32.888 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:32.889 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:40:32.889 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:32.889 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:32.889 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:32.889 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:32.889 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:40:32.889 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:32.889 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:32.889 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:32.889 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:32.889 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:40:32.889 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:40:32.889 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:32.889 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:32.890 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:32.890 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:32.890 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:32.890 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:40:32.890 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:40:32.890 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:32.890 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:32.890 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:32.890 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:32.890 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:40:32.890 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:32.890 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:32.890 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:32.890 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:32.891 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:40:32.891 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:32.891 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:32.891 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:32.891 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:32.891 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:40:32.891 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:40:32.891 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:32.891 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:32.891 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:32.891 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:32.891 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:32.891 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:40:32.891 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:40:32.892 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:32.892 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:32.892 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:32.892 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:32.892 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:40:32.892 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:32.892 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:32.892 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:32.892 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:32.892 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:40:32.892 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:32.892 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:32.892 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:32.892 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:32.893 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:40:32.893 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:40:32.893 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:32.893 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:32.893 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:32.893 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:32.893 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:32.893 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:32.893 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:40:32.893 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:40:32.893 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:32.893 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:32.893 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:32.893 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:40:32.894 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:32.894 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:32.894 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:40:32.894 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:32.894 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:32.894 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:32.894 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:40:32.894 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:32.894 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:32.894 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:40:32.894 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:32.895 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:32.895 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:32.895 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:40:32.895 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:32.895 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:32.895 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:32.895 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:40:32.895 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:40:32.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.855+0000 7fd71847b640 1 -- 172.21.15.46:0/3490122153 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd6f40383e0 msgr2=0x7fd6f403a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:32.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.855+0000 7fd71847b640 1 --2- 172.21.15.46:0/3490122153 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd6f40383e0 0x7fd6f403a8a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd7100a8630 tx=0x7fd708009040 comp rx=0 tx=0).stop 2024-09-16T04:40:32.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.855+0000 7fd71847b640 10 monclient: shutdown 2024-09-16T04:40:32.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.855+0000 7fd71847b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:32.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.855+0000 7fd71847b640 1 -- 172.21.15.46:0/3490122153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7100a3ff0 msgr2=0x7fd710136040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:32.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.855+0000 7fd71847b640 1 --2- 172.21.15.46:0/3490122153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7100a3ff0 0x7fd710136040 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd70000e850 tx=0x7fd70000ed10 comp rx=0 tx=0).stop 2024-09-16T04:40:32.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.855+0000 7fd716a18640 1 -- 172.21.15.46:0/3490122153 reap_dead start 2024-09-16T04:40:32.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.855+0000 7fd716a18640 1 -- 172.21.15.46:0/3490122153 reap_dead start 2024-09-16T04:40:32.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.855+0000 7fd71847b640 1 -- 172.21.15.46:0/3490122153 shutdown_connections 2024-09-16T04:40:32.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.855+0000 7fd71847b640 1 -- 172.21.15.46:0/3490122153 >> 172.21.15.46:0/3490122153 conn(0x7fd7100a6340 msgr2=0x7fd71009f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:32.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.855+0000 7fd71847b640 1 -- 172.21.15.46:0/3490122153 shutdown_connections 2024-09-16T04:40:32.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.855+0000 7fd71847b640 1 -- 172.21.15.46:0/3490122153 wait complete. 2024-09-16T04:40:32.898 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:40:32.898 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:40:32.898 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:40:33.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.999+0000 7f1f09968640 10 monclient: get_monmap_and_config 2024-09-16T04:40:33.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.999+0000 7f1f09968640 10 monclient: build_initial_monmap 2024-09-16T04:40:33.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:32.999+0000 7f1f09968640 10 monclient: monmap: 2024-09-16T04:40:33.001 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:33.002 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:33.002 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:40:33.000116+0000 2024-09-16T04:40:33.002 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:40:33.000116+0000 2024-09-16T04:40:33.002 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:33.002 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.002 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:40:33.002 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:40:33.002 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:40:33.002 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:40:33.003 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:40:33.003 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f09968640 1 Processor -- start 2024-09-16T04:40:33.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f09968640 1 -- start start 2024-09-16T04:40:33.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f09968640 10 monclient: init 2024-09-16T04:40:33.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f09968640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:33.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f09968640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-16T04:40:33.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f09968640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f040a3ba0 addr v1:172.21.15.46:6789/0 2024-09-16T04:40:33.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f09968640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f040a3f80 0x7f1f040a4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:33.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f09968640 10 monclient(hunting): picked mon.noname-b con 0x7f1f040a3f80 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:33.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f09968640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1f0413bf00 0x7f1f0413e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:33.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f09968640 10 monclient(hunting): picked mon.noname-c con 0x7f1f0413bf00 addr v2:172.21.15.114:3300/0 2024-09-16T04:40:33.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f09968640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f0409c390 con 0x7f1f040a3ba0 2024-09-16T04:40:33.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f09968640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:33.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f09968640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f1f0409b860 con 0x7f1f0413bf00 2024-09-16T04:40:33.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f09968640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:33.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f09968640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f1f0409b2a0 con 0x7f1f040a3f80 2024-09-16T04:40:33.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f09968640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:33.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f09968640 10 monclient(hunting): authenticate will time out at 831.415786s 2024-09-16T04:40:33.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f027fc640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f040a3f80 0x7f1f040a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:33.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f02ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1f040a3ba0 0x7f1f0409bc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60966/0 (socket says 172.21.15.46:60966) 2024-09-16T04:40:33.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f02ffd640 1 -- 172.21.15.46:0/1374309492 learned_addr learned my addr 172.21.15.46:0/1374309492 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:40:33.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f027fc640 10 monclient(hunting): get_auth_request con 0x7f1f040a3f80 auth_method 0 2024-09-16T04:40:33.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f027fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:33.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:33.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.000+0000 7f1f027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:33.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.001+0000 7f1f027fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:33.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.001+0000 7f1f027fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:33.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.001+0000 7f1f027fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:33.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.001+0000 7f1f037fe640 1 --2- 172.21.15.46:0/1374309492 >> v2:172.21.15.114:3300/0 conn(0x7f1f0413bf00 0x7f1f0413e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:33.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.001+0000 7f1f027fc640 10 monclient(hunting): handle_auth_done global_id 24143 payload 274 2024-09-16T04:40:33.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.001+0000 7f1f027fc640 1 -- 172.21.15.46:0/1374309492 >> v2:172.21.15.114:3300/0 conn(0x7f1f0413bf00 msgr2=0x7f1f0413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:40:33.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.001+0000 7f1f027fc640 1 --2- 172.21.15.46:0/1374309492 >> v2:172.21.15.114:3300/0 conn(0x7f1f0413bf00 0x7f1f0413e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.001+0000 7f1f027fc640 1 -- 172.21.15.46:0/1374309492 >> v1:172.21.15.46:6789/0 conn(0x7f1f040a3ba0 legacy=0x7f1f0409bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.001+0000 7f1f01ffb640 1 -- 172.21.15.46:0/1374309492 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3511972998 0 0) 0x7f1f0409c390 con 0x7f1f040a3ba0 2024-09-16T04:40:33.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.001+0000 7f1f027fc640 10 monclient: _finish_hunting 0 2024-09-16T04:40:33.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.001+0000 7f1f027fc640 1 monclient: found mon.noname-b 2024-09-16T04:40:33.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.001+0000 7f1f027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:33.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.001+0000 7f1f027fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:40:33.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.001+0000 7f1f027fc640 1 -- 172.21.15.46:0/1374309492 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f0409afc0 con 0x7f1f040a3f80 2024-09-16T04:40:33.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.001+0000 7f1f037fe640 10 monclient: get_auth_request con 0x7f1f0413bf00 auth_method 0 2024-09-16T04:40:33.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.001+0000 7f1f037fe640 1 --2- 172.21.15.46:0/1374309492 >> v2:172.21.15.114:3300/0 conn(0x7f1f0413bf00 0x7f1f0413e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:40:33.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.001+0000 7f1f01ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:40:33.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.001+0000 7f1f027fc640 1 --2- 172.21.15.46:0/1374309492 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f040a3f80 0x7f1f040a4360 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1ef80097a0 tx=0x7f1ef802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=617ac160fe723546 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:33.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f01ffb640 1 -- 172.21.15.46:0/1374309492 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8037070 con 0x7f1f040a3f80 2024-09-16T04:40:33.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f01ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:33.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f01ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:40:33.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f01ffb640 10 monclient: dump: 2024-09-16T04:40:33.016 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:33.016 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:33.016 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.017 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.017 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:33.017 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.017 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:33.017 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:33.017 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:33.017 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f01ffb640 10 monclient: _finish_auth 0 2024-09-16T04:40:33.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f01ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:40:33.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:33.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f09968640 5 monclient: authenticate success, global_id 24143 2024-09-16T04:40:33.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f01ffb640 1 -- 172.21.15.46:0/1374309492 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ef802ae50 con 0x7f1f040a3f80 2024-09-16T04:40:33.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f09968640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:40:33.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f01ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:33.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f09968640 10 monclient: get_monmap_and_config success 2024-09-16T04:40:33.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f01ffb640 1 -- 172.21.15.46:0/1374309492 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8032660 con 0x7f1f040a3f80 2024-09-16T04:40:33.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f01ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:33.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f09968640 4 set_mon_vals no callback set 2024-09-16T04:40:33.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f01ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:33.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f01ffb640 10 monclient: dump: 2024-09-16T04:40:33.018 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:33.018 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:33.019 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.019 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.019 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:33.019 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.019 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:33.019 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:33.019 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:33.019 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f09968640 10 monclient: shutdown 2024-09-16T04:40:33.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f09968640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:33.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f09968640 1 -- 172.21.15.46:0/1374309492 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f040a3f80 msgr2=0x7f1f040a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f09968640 1 --2- 172.21.15.46:0/1374309492 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f040a3f80 0x7f1f040a4360 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1ef80097a0 tx=0x7f1ef802a100 comp rx=0 tx=0).stop 2024-09-16T04:40:33.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f09968640 1 -- 172.21.15.46:0/1374309492 shutdown_connections 2024-09-16T04:40:33.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f09968640 1 --2- 172.21.15.46:0/1374309492 >> v2:172.21.15.114:3300/0 conn(0x7f1f0413bf00 0x7f1f0413e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f09968640 1 --2- 172.21.15.46:0/1374309492 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f040a3f80 0x7f1f040a4360 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f09968640 1 -- 172.21.15.46:0/1374309492 >> 172.21.15.46:0/1374309492 conn(0x7f1f040a60c0 msgr2=0x7f1f040a64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:33.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f09968640 1 -- 172.21.15.46:0/1374309492 shutdown_connections 2024-09-16T04:40:33.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.002+0000 7f1f09968640 1 -- 172.21.15.46:0/1374309492 wait complete. 2024-09-16T04:40:33.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.003+0000 7f1f09968640 10 monclient: build_initial_monmap 2024-09-16T04:40:33.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.003+0000 7f1f09968640 10 monclient: monmap: 2024-09-16T04:40:33.021 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:33.021 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:40:33.021 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:40:33.021 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:40:33.021 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:33.021 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.022 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:40:33.022 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:40:33.022 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:40:33.022 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.003+0000 7f1f09968640 1 Processor -- start 2024-09-16T04:40:33.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.003+0000 7f1f09968640 1 -- start start 2024-09-16T04:40:33.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.003+0000 7f1f09968640 10 monclient: init 2024-09-16T04:40:33.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f09968640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:33.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f09968640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:40:33.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f09968640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f040a3ba0 0x7f1f04136320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:33.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f09968640 10 monclient(hunting): picked mon.noname-a con 0x7f1f040a3ba0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:40:33.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f09968640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f040a3f80 0x7f1f04136860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:33.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f09968640 10 monclient(hunting): picked mon.noname-b con 0x7f1f040a3f80 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:40:33.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f09968640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f0413bf00 0x7f1f04139a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:33.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f09968640 10 monclient(hunting): picked mon.noname-c con 0x7f1f0413bf00 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:33.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f09968640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:33.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f09968640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1f040a5540 con 0x7f1f040a3ba0 2024-09-16T04:40:33.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f09968640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:33.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f09968640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1f040a53c0 con 0x7f1f040a3f80 2024-09-16T04:40:33.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f09968640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:33.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f09968640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f1f0413e830 con 0x7f1f0413bf00 2024-09-16T04:40:33.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f09968640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:33.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f09968640 10 monclient(hunting): authenticate will time out at 831.419280s 2024-09-16T04:40:33.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f02ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f040a3ba0 0x7f1f04136320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:33.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f037fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f0413bf00 0x7f1f04139a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:33.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f02ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f040a3ba0 0x7f1f04136320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58544/0 (socket says 172.21.15.46:58544) 2024-09-16T04:40:33.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f02ffd640 1 -- 172.21.15.46:0/4171055823 learned_addr learned my addr 172.21.15.46:0/4171055823 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:33.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f02ffd640 10 monclient(hunting): get_auth_request con 0x7f1f040a3ba0 auth_method 0 2024-09-16T04:40:33.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f037fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f0413bf00 0x7f1f04139a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:44136/0 (socket says 172.21.15.46:44136) 2024-09-16T04:40:33.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:33.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f02ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:33.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:33.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f037fe640 10 monclient(hunting): get_auth_request con 0x7f1f0413bf00 auth_method 0 2024-09-16T04:40:33.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:33.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:33.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:33.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f02ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:33.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f02ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:33.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f027fc640 1 --2- 172.21.15.46:0/4171055823 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f040a3f80 0x7f1f04136860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:33.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f02ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:33.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:33.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:33.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:33.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f027fc640 10 monclient(hunting): get_auth_request con 0x7f1f040a3f80 auth_method 0 2024-09-16T04:40:33.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f027fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:33.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:33.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:33.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f027fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:33.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f027fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:33.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f027fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:33.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f037fe640 10 monclient(hunting): handle_auth_done global_id 24146 payload 824 2024-09-16T04:40:33.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f037fe640 1 -- 172.21.15.46:0/4171055823 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f040a3f80 msgr2=0x7f1f04136860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f037fe640 1 --2- 172.21.15.46:0/4171055823 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f040a3f80 0x7f1f04136860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f037fe640 1 -- 172.21.15.46:0/4171055823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f040a3ba0 msgr2=0x7f1f04136320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f037fe640 1 --2- 172.21.15.46:0/4171055823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f040a3ba0 0x7f1f04136320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f037fe640 10 monclient: _finish_hunting 0 2024-09-16T04:40:33.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f037fe640 1 monclient: found mon.noname-c 2024-09-16T04:40:33.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:33.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f037fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:40:33.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f037fe640 1 -- 172.21.15.46:0/4171055823 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f0411fae0 con 0x7f1f0413bf00 2024-09-16T04:40:33.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f02ffd640 1 --2- 172.21.15.46:0/4171055823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f040a3ba0 0x7f1f04136320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:40:33.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.004+0000 7f1f037fe640 1 --2- 172.21.15.46:0/4171055823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f0413bf00 0x7f1f04139a40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1efc00cdf0 tx=0x7f1efc00a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:33.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1eebfff640 1 -- 172.21.15.46:0/4171055823 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1efc01a070 con 0x7f1f0413bf00 2024-09-16T04:40:33.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1eebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:33.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1eebfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:40:33.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1eebfff640 10 monclient: dump: 2024-09-16T04:40:33.033 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:33.033 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:33.033 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.033 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.033 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:33.033 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.033 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:33.033 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:33.033 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:33.033 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1eebfff640 10 monclient: _finish_auth 0 2024-09-16T04:40:33.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1eebfff640 10 monclient: _check_auth_tickets 2024-09-16T04:40:33.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1eebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:33.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1eebfff640 1 -- 172.21.15.46:0/4171055823 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1efc00d840 con 0x7f1f0413bf00 2024-09-16T04:40:33.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1eebfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:33.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1eebfff640 1 -- 172.21.15.46:0/4171055823 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1efc00dc70 con 0x7f1f0413bf00 2024-09-16T04:40:33.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1f09968640 5 monclient: authenticate success, global_id 24146 2024-09-16T04:40:33.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1eebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:33.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1eebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:33.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1eebfff640 10 monclient: dump: 2024-09-16T04:40:33.034 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:33.034 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:33.034 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.034 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.034 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:33.035 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1f09968640 10 monclient: _renew_subs 2024-09-16T04:40:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1f09968640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:40:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1f09968640 1 -- 172.21.15.46:0/4171055823 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f0413c370 con 0x7f1f0413bf00 2024-09-16T04:40:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1f017fa640 4 set_mon_vals no callback set 2024-09-16T04:40:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1f09968640 10 monclient: _renew_subs 2024-09-16T04:40:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1f09968640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:40:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1f09968640 1 -- 172.21.15.46:0/4171055823 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f04120cb0 con 0x7f1f0413bf00 2024-09-16T04:40:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.005+0000 7f1eebfff640 1 -- 172.21.15.46:0/4171055823 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f1efc01d070 con 0x7f1f0413bf00 2024-09-16T04:40:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1f09968640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:33.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1f09968640 1 --2- 172.21.15.46:0/4171055823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f04062df0 0x7f1f040631d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:33.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1f09968640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:33.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1f09968640 1 -- 172.21.15.46:0/4171055823 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f1f040a4360 con 0x7f1f04062df0 2024-09-16T04:40:33.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1eebfff640 1 --2- 172.21.15.46:0/4171055823 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1ef00383b0 0x7f1ef003a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:33.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1f02ffd640 1 --2- 172.21.15.46:0/4171055823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f04062df0 0x7f1f040631d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:33.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1f02ffd640 10 monclient: get_auth_request con 0x7f1f04062df0 auth_method 0 2024-09-16T04:40:33.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1f02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:33.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1f02ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:33.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1f02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:33.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1eebfff640 1 -- 172.21.15.46:0/4171055823 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(78..78 src has 1..78) ==== 4738+0+0 (secure 0 0 0) 0x7f1efc04bfe0 con 0x7f1f0413bf00 2024-09-16T04:40:33.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1f02ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:33.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1f02ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:33.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1f02ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:33.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1f027fc640 1 --2- 172.21.15.46:0/4171055823 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1ef00383b0 0x7f1ef003a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:33.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1f027fc640 10 monclient: get_auth_request con 0x7f1ef00383b0 auth_method 0 2024-09-16T04:40:33.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1f02ffd640 10 monclient(hunting): handle_auth_done global_id 24149 payload 824 2024-09-16T04:40:33.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1f02ffd640 1 -- 172.21.15.46:0/4171055823 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1f0411f860 con 0x7f1f04062df0 2024-09-16T04:40:33.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1f02ffd640 1 --2- 172.21.15.46:0/4171055823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f04062df0 0x7f1f040631d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4007fd0 tx=0x7f1ef4001d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:33.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1eebfff640 1 -- 172.21.15.46:0/4171055823 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef400fa50 con 0x7f1f04062df0 2024-09-16T04:40:33.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1eebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:33.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1eebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:33.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1eebfff640 10 monclient: dump: 2024-09-16T04:40:33.038 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:33.038 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:33.038 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.038 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.038 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:33.038 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.038 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:33.038 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:33.038 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:33.038 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.006+0000 7f1f027fc640 1 --2- 172.21.15.46:0/4171055823 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1ef00383b0 0x7f1ef003a870 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8034000 tx=0x7f1ef802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:33.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.007+0000 7f1eebfff640 1 -- 172.21.15.46:0/4171055823 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1ef4018070 con 0x7f1f04062df0 2024-09-16T04:40:33.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.007+0000 7f1eebfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:33.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.007+0000 7f1eebfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:40:33.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.007+0000 7f1eebfff640 1 -- 172.21.15.46:0/4171055823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f04062df0 msgr2=0x7f1f040631d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.007+0000 7f1eebfff640 1 --2- 172.21.15.46:0/4171055823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f04062df0 0x7f1f040631d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4007fd0 tx=0x7f1ef4001d60 comp rx=0 tx=0).stop 2024-09-16T04:40:33.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.007+0000 7f1eebfff640 1 -- 172.21.15.46:0/4171055823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f04062df0 msgr2=0x7f1f040631d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:40:33.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.007+0000 7f1eebfff640 1 --2- 172.21.15.46:0/4171055823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f04062df0 0x7f1f040631d0 secure :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4007fd0 tx=0x7f1ef4001d60 comp rx=0 tx=0).stop 2024-09-16T04:40:33.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.013+0000 7f1f09968640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:40:33.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.013+0000 7f1f09968640 1 --2- 172.21.15.46:0/4171055823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f0402c640 0x7f1f0402ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:33.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.013+0000 7f1f09968640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:33.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.013+0000 7f1f09968640 1 -- 172.21.15.46:0/4171055823 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f1f040a1cb0 con 0x7f1f0402c640 2024-09-16T04:40:33.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.013+0000 7f1f02ffd640 1 --2- 172.21.15.46:0/4171055823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f0402c640 0x7f1f0402ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:33.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.013+0000 7f1f02ffd640 10 monclient: get_auth_request con 0x7f1f0402c640 auth_method 0 2024-09-16T04:40:33.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.013+0000 7f1f02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:33.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.013+0000 7f1f02ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:33.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.013+0000 7f1f02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:33.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.013+0000 7f1f02ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:33.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.013+0000 7f1f02ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:33.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.013+0000 7f1f02ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:33.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.013+0000 7f1f02ffd640 10 monclient(hunting): handle_auth_done global_id 24152 payload 824 2024-09-16T04:40:33.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.013+0000 7f1f02ffd640 1 -- 172.21.15.46:0/4171055823 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1f04067e20 con 0x7f1f0402c640 2024-09-16T04:40:33.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.013+0000 7f1f02ffd640 1 --2- 172.21.15.46:0/4171055823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f0402c640 0x7f1f0402ca20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4007fd0 tx=0x7f1ef4001080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:33.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1eebfff640 1 -- 172.21.15.46:0/4171055823 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef40136c0 con 0x7f1f0402c640 2024-09-16T04:40:33.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1eebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:33.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1eebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:33.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1eebfff640 10 monclient: dump: 2024-09-16T04:40:33.043 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:33.043 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:33.043 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.043 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.043 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:33.043 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.043 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:33.043 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:33.043 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:33.043 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1eebfff640 1 -- 172.21.15.46:0/4171055823 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f1ef40136c0 con 0x7f1f0402c640 2024-09-16T04:40:33.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1eebfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:40:33.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1eebfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:40:33.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1eebfff640 1 -- 172.21.15.46:0/4171055823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f0402c640 msgr2=0x7f1f0402ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1eebfff640 1 --2- 172.21.15.46:0/4171055823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f0402c640 0x7f1f0402ca20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4007fd0 tx=0x7f1ef4001080 comp rx=0 tx=0).stop 2024-09-16T04:40:33.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1eebfff640 1 -- 172.21.15.46:0/4171055823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f0402c640 msgr2=0x7f1f0402ca20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:40:33.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1eebfff640 1 --2- 172.21.15.46:0/4171055823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f0402c640 0x7f1f0402ca20 secure :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4007fd0 tx=0x7f1ef4001080 comp rx=0 tx=0).stop 2024-09-16T04:40:33.044 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:40:33.044 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:40:33.044 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:40:33.044 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:40:33.044 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 20, 2024-09-16T04:40:33.044 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 8213, 2024-09-16T04:40:33.044 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:40:33.044 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:40:33.045 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:40:33.045 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:40:33.045 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:33.045 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 7, 2024-09-16T04:40:33.045 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:40:33.045 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:40:33.045 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:40:33.045 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:33.045 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:33.045 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:33.045 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:33.045 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:33.045 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:33.046 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:33.046 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:33.046 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:33.046 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:33.046 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:33.046 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:33.046 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:40:33.046 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:40:33.046 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:33.046 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:33.046 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:33.046 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:33.046 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:33.046 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:33.046 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:33.047 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:33.047 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:33.047 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:33.047 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:33.047 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:33.047 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.047 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:40:33.047 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:40:33.047 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:40:33.047 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:40:33.048 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:40:33.048 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:40:33.048 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:40:33.048 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:40:33.048 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:40:33.048 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:40:33.048 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:40:33.048 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:40:33.048 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:40:33.048 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:40:33.048 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:40:33.048 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:40:33.048 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:40:33.049 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:33.049 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:33.049 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:33.049 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:33.049 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:33.049 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:33.049 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:33.049 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:33.049 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:33.049 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:33.049 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:33.049 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:33.049 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:40:33.049 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.050 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:40:33.050 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.050 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:40:33.050 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:40:33.050 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:33.050 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:33.050 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.050 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:33.050 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:40:33.050 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:33.050 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.050 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.050 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:33.050 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:40:33.051 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:33.051 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:33.051 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:33.051 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.051 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:40:33.051 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:40:33.051 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:33.051 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:33.051 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:33.051 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.051 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.051 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:40:33.051 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:40:33.051 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:33.051 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:33.052 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.052 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:33.052 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:40:33.052 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:33.052 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.052 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.052 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:33.052 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:40:33.052 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:33.052 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:33.052 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:33.052 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.052 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:40:33.052 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:40:33.053 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:33.053 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:33.053 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:33.053 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.053 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.053 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:40:33.053 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:40:33.053 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:33.053 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:33.053 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.053 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:33.053 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:40:33.053 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:33.054 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.054 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.054 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:33.054 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:40:33.054 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:33.054 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:33.054 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:33.054 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.054 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:40:33.054 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:40:33.054 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:33.054 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:33.054 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:33.057 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:33.057 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:33.058 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.058 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:40:33.058 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:40:33.058 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.059 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:33.059 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:33.059 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:40:33.059 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:33.059 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:33.059 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:40:33.059 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.059 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:33.059 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:33.060 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4 2024-09-16T04:40:33.060 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:33.060 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:33.060 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.060 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:40:33.061 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:40:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1f09968640 1 -- 172.21.15.46:0/4171055823 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1ef00383b0 msgr2=0x7f1ef003a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1f09968640 1 --2- 172.21.15.46:0/4171055823 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1ef00383b0 0x7f1ef003a870 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8034000 tx=0x7f1ef802e000 comp rx=0 tx=0).stop 2024-09-16T04:40:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1f09968640 10 monclient: shutdown 2024-09-16T04:40:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1f09968640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1f09968640 1 -- 172.21.15.46:0/4171055823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f0413bf00 msgr2=0x7f1f04139a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1f037fe640 1 -- 172.21.15.46:0/4171055823 reap_dead start 2024-09-16T04:40:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1f09968640 1 --2- 172.21.15.46:0/4171055823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f0413bf00 0x7f1f04139a40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1efc00cdf0 tx=0x7f1efc00a710 comp rx=0 tx=0).stop 2024-09-16T04:40:33.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1f09968640 1 -- 172.21.15.46:0/4171055823 shutdown_connections 2024-09-16T04:40:33.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1f09968640 1 --2- 172.21.15.46:0/4171055823 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1f0413bf00 0x7f1f04139a40 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1f09968640 1 -- 172.21.15.46:0/4171055823 >> 172.21.15.46:0/4171055823 conn(0x7f1f040a60c0 msgr2=0x7f1f0409f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:33.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1f09968640 1 -- 172.21.15.46:0/4171055823 shutdown_connections 2024-09-16T04:40:33.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.014+0000 7f1f09968640 1 -- 172.21.15.46:0/4171055823 wait complete. 2024-09-16T04:40:33.064 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:40:33.064 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:40:33.065 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:40:33.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.176+0000 7f7f43c5c640 10 monclient: get_monmap_and_config 2024-09-16T04:40:33.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.176+0000 7f7f43c5c640 10 monclient: build_initial_monmap 2024-09-16T04:40:33.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.176+0000 7f7f43c5c640 10 monclient: monmap: 2024-09-16T04:40:33.178 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:33.178 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:33.178 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:40:33.176568+0000 2024-09-16T04:40:33.178 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:40:33.176568+0000 2024-09-16T04:40:33.178 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:33.178 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.178 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:40:33.178 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:40:33.178 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:40:33.178 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:40:33.178 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:40:33.178 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.176+0000 7f7f43c5c640 1 Processor -- start 2024-09-16T04:40:33.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.176+0000 7f7f43c5c640 1 -- start start 2024-09-16T04:40:33.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.176+0000 7f7f43c5c640 10 monclient: init 2024-09-16T04:40:33.185 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:33.185 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36294]: starting. 2024-09-16T04:40:33.185 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36294]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-36259 2024-09-16T04:40:33.185 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36294]: created object 0... 2024-09-16T04:40:33.185 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36294]: created object 25... 2024-09-16T04:40:33.186 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36294]: created object 49... 2024-09-16T04:40:33.186 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36294]: finishing. 2024-09-16T04:40:33.186 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36294]: shutting down. 2024-09-16T04:40:33.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f43c5c640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:33.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f43c5c640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-16T04:40:33.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f43c5c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7f3c0a3c10 0x7f7f3c09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:33.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f43c5c640 10 monclient(hunting): picked mon.noname-b con 0x7f7f3c0a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:33.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f43c5c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7f3c0a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:40:33.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f43c5c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7f3c13be90 addr v1:172.21.15.46:6789/0 2024-09-16T04:40:33.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f43c5c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f3c09ab70 con 0x7f7f3c13be90 2024-09-16T04:40:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f43c5c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f3c0a4d60 con 0x7f7f3c0a3ff0 2024-09-16T04:40:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f43c5c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f43c5c640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f7f3c09b030 con 0x7f7f3c0a3c10 2024-09-16T04:40:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f43c5c640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f43c5c640 10 monclient(hunting): authenticate will time out at 831.592595s 2024-09-16T04:40:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f419f8640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7f3c0a3c10 0x7f7f3c09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f421f9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7f3c13be90 0x7f7f3c13e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60968/0 (socket says 172.21.15.46:60968) 2024-09-16T04:40:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f419f8640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7f3c0a3c10 0x7f7f3c09bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:44158/0 (socket says 172.21.15.46:44158) 2024-09-16T04:40:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f419f8640 1 -- 172.21.15.46:0/1795094203 learned_addr learned my addr 172.21.15.46:0/1795094203 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f419f8640 10 monclient(hunting): get_auth_request con 0x7f7f3c0a3c10 auth_method 0 2024-09-16T04:40:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f419f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f419f8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f419f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f419f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f419f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f419f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f419f8640 10 monclient(hunting): handle_auth_done global_id 24155 payload 274 2024-09-16T04:40:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f419f8640 1 -- 172.21.15.46:0/1795094203 >> v1:172.21.15.114:6789/0 conn(0x7f7f3c0a3ff0 legacy=0x7f7f3c0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f419f8640 1 -- 172.21.15.46:0/1795094203 >> v1:172.21.15.46:6789/0 conn(0x7f7f3c13be90 legacy=0x7f7f3c13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f419f8640 10 monclient: _finish_hunting 0 2024-09-16T04:40:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f419f8640 1 monclient: found mon.noname-b 2024-09-16T04:40:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f419f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f419f8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:40:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f419f8640 1 -- 172.21.15.46:0/1795094203 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f3c09b310 con 0x7f7f3c0a3c10 2024-09-16T04:40:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f409f6640 1 -- 172.21.15.46:0/1795094203 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 690120367 0 0) 0x7f7f3c09ab70 con 0x7f7f3c13be90 2024-09-16T04:40:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.177+0000 7f7f409f6640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:40:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.178+0000 7f7f419f8640 1 --2- 172.21.15.46:0/1795094203 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7f3c0a3c10 0x7f7f3c09bcf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7f380098a0 tx=0x7f7f3802a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c15ca747ce6547ff server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.178+0000 7f7f409f6640 1 -- 172.21.15.46:0/1795094203 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f38037070 con 0x7f7f3c0a3c10 2024-09-16T04:40:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.178+0000 7f7f409f6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.178+0000 7f7f409f6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:40:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.178+0000 7f7f409f6640 10 monclient: dump: 2024-09-16T04:40:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:33.193 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.178+0000 7f7f409f6640 10 monclient: _finish_auth 0 2024-09-16T04:40:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.178+0000 7f7f409f6640 10 monclient: _check_auth_tickets 2024-09-16T04:40:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.178+0000 7f7f409f6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.178+0000 7f7f43c5c640 5 monclient: authenticate success, global_id 24155 2024-09-16T04:40:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.178+0000 7f7f43c5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:40:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.178+0000 7f7f409f6640 1 -- 172.21.15.46:0/1795094203 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f38032440 con 0x7f7f3c0a3c10 2024-09-16T04:40:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.178+0000 7f7f409f6640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.178+0000 7f7f43c5c640 10 monclient: get_monmap_and_config success 2024-09-16T04:40:33.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.178+0000 7f7f43c5c640 4 set_mon_vals no callback set 2024-09-16T04:40:33.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.178+0000 7f7f409f6640 1 -- 172.21.15.46:0/1795094203 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f38032810 con 0x7f7f3c0a3c10 2024-09-16T04:40:33.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.178+0000 7f7f409f6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:33.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.178+0000 7f7f409f6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.178+0000 7f7f409f6640 10 monclient: dump: 2024-09-16T04:40:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:33.196 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.179+0000 7f7f43c5c640 10 monclient: shutdown 2024-09-16T04:40:33.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.179+0000 7f7f43c5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:33.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.179+0000 7f7f43c5c640 1 -- 172.21.15.46:0/1795094203 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7f3c0a3c10 msgr2=0x7f7f3c09bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.179+0000 7f7f43c5c640 1 --2- 172.21.15.46:0/1795094203 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7f3c0a3c10 0x7f7f3c09bcf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7f380098a0 tx=0x7f7f3802a1c0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.179+0000 7f7f43c5c640 1 -- 172.21.15.46:0/1795094203 shutdown_connections 2024-09-16T04:40:33.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.179+0000 7f7f43c5c640 1 --2- 172.21.15.46:0/1795094203 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7f3c0a3c10 0x7f7f3c09bcf0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.179+0000 7f7f43c5c640 1 -- 172.21.15.46:0/1795094203 >> 172.21.15.46:0/1795094203 conn(0x7f7f3c0a5d10 msgr2=0x7f7f3c0a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.179+0000 7f7f43c5c640 1 -- 172.21.15.46:0/1795094203 shutdown_connections 2024-09-16T04:40:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.179+0000 7f7f43c5c640 1 -- 172.21.15.46:0/1795094203 wait complete. 2024-09-16T04:40:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.180+0000 7f7f43c5c640 10 monclient: build_initial_monmap 2024-09-16T04:40:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.180+0000 7f7f43c5c640 10 monclient: monmap: 2024-09-16T04:40:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:40:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:40:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:40:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:40:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:40:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:40:33.198 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.180+0000 7f7f43c5c640 1 Processor -- start 2024-09-16T04:40:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.180+0000 7f7f43c5c640 1 -- start start 2024-09-16T04:40:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.180+0000 7f7f43c5c640 10 monclient: init 2024-09-16T04:40:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f43c5c640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f43c5c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:40:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f43c5c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c0a3c10 0x7f7f3c1366b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f43c5c640 10 monclient(hunting): picked mon.noname-b con 0x7f7f3c0a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:40:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f43c5c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3c0a3ff0 0x7f7f3c136bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f43c5c640 10 monclient(hunting): picked mon.noname-a con 0x7f7f3c0a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:40:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f43c5c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7f3c13be90 0x7f7f3c138b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f43c5c640 10 monclient(hunting): picked mon.noname-c con 0x7f7f3c13be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f43c5c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f43c5c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c06e550 con 0x7f7f3c0a3ff0 2024-09-16T04:40:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f43c5c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f43c5c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c06e3d0 con 0x7f7f3c0a3c10 2024-09-16T04:40:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f43c5c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f43c5c640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f7f3c13f9a0 con 0x7f7f3c13be90 2024-09-16T04:40:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f43c5c640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f43c5c640 10 monclient(hunting): authenticate will time out at 831.596160s 2024-09-16T04:40:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f411f7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3c0a3ff0 0x7f7f3c136bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f421f9640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7f3c13be90 0x7f7f3c138b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f411f7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3c0a3ff0 0x7f7f3c136bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58552/0 (socket says 172.21.15.46:58552) 2024-09-16T04:40:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f411f7640 1 -- 172.21.15.46:0/3223234670 learned_addr learned my addr 172.21.15.46:0/3223234670 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f411f7640 10 monclient(hunting): get_auth_request con 0x7f7f3c0a3ff0 auth_method 0 2024-09-16T04:40:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f411f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f411f7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f411f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f421f9640 10 monclient(hunting): get_auth_request con 0x7f7f3c13be90 auth_method 0 2024-09-16T04:40:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f421f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f421f9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f421f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f419f8640 1 --2- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c0a3c10 0x7f7f3c1366b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f419f8640 10 monclient(hunting): get_auth_request con 0x7f7f3c0a3c10 auth_method 0 2024-09-16T04:40:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f419f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f419f8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f419f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f411f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f411f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f411f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f421f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f421f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f421f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f419f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f419f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f419f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f421f9640 10 monclient(hunting): handle_auth_done global_id 24158 payload 824 2024-09-16T04:40:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f421f9640 1 -- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c0a3c10 msgr2=0x7f7f3c1366b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f421f9640 1 --2- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c0a3c10 0x7f7f3c1366b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f421f9640 1 -- 172.21.15.46:0/3223234670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3c0a3ff0 msgr2=0x7f7f3c136bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f421f9640 1 --2- 172.21.15.46:0/3223234670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3c0a3ff0 0x7f7f3c136bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f421f9640 10 monclient: _finish_hunting 0 2024-09-16T04:40:33.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f421f9640 1 monclient: found mon.noname-c 2024-09-16T04:40:33.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f421f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:33.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f421f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:40:33.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f421f9640 1 -- 172.21.15.46:0/3223234670 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f3c120270 con 0x7f7f3c13be90 2024-09-16T04:40:33.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f411f7640 1 --2- 172.21.15.46:0/3223234670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3c0a3ff0 0x7f7f3c136bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:40:33.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f421f9640 1 --2- 172.21.15.46:0/3223234670 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7f3c13be90 0x7f7f3c138b90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400c750 tx=0x7f7f3400cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:33.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f327fc640 1 -- 172.21.15.46:0/3223234670 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f34018070 con 0x7f7f3c13be90 2024-09-16T04:40:33.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:40:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f327fc640 10 monclient: dump: 2024-09-16T04:40:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:33.204 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f327fc640 10 monclient: _finish_auth 0 2024-09-16T04:40:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f327fc640 10 monclient: _check_auth_tickets 2024-09-16T04:40:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f43c5c640 5 monclient: authenticate success, global_id 24158 2024-09-16T04:40:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f43c5c640 10 monclient: _renew_subs 2024-09-16T04:40:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f43c5c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:40:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f43c5c640 1 -- 172.21.15.46:0/3223234670 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f3c120550 con 0x7f7f3c13be90 2024-09-16T04:40:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f327fc640 1 -- 172.21.15.46:0/3223234670 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f34004d70 con 0x7f7f3c13be90 2024-09-16T04:40:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f327fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f327fc640 1 -- 172.21.15.46:0/3223234670 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3400a820 con 0x7f7f3c13be90 2024-09-16T04:40:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f327fc640 10 monclient: dump: 2024-09-16T04:40:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:33.206 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.181+0000 7f7f33fff640 4 set_mon_vals no callback set 2024-09-16T04:40:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.182+0000 7f7f43c5c640 10 monclient: _renew_subs 2024-09-16T04:40:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.182+0000 7f7f43c5c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:40:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.182+0000 7f7f43c5c640 1 -- 172.21.15.46:0/3223234670 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f3c13f550 con 0x7f7f3c13be90 2024-09-16T04:40:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.182+0000 7f7f327fc640 1 -- 172.21.15.46:0/3223234670 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f7f3401b070 con 0x7f7f3c13be90 2024-09-16T04:40:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f43c5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f43c5c640 1 --2- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c1390d0 0x7f7f3c1394b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f43c5c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f43c5c640 1 -- 172.21.15.46:0/3223234670 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c09bcf0 con 0x7f7f3c1390d0 2024-09-16T04:40:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f327fc640 1 --2- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7f200383b0 0x7f7f2003a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f327fc640 1 -- 172.21.15.46:0/3223234670 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(79..79 src has 1..79) ==== 4738+0+0 (secure 0 0 0) 0x7f7f3404ab50 con 0x7f7f3c13be90 2024-09-16T04:40:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f419f8640 1 --2- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c1390d0 0x7f7f3c1394b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f419f8640 10 monclient: get_auth_request con 0x7f7f3c1390d0 auth_method 0 2024-09-16T04:40:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f419f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f419f8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f419f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f411f7640 1 --2- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7f200383b0 0x7f7f2003a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f419f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f419f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f419f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f411f7640 10 monclient: get_auth_request con 0x7f7f200383b0 auth_method 0 2024-09-16T04:40:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f419f8640 10 monclient(hunting): handle_auth_done global_id 4948 payload 824 2024-09-16T04:40:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f419f8640 1 -- 172.21.15.46:0/3223234670 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f3c13b110 con 0x7f7f3c1390d0 2024-09-16T04:40:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f419f8640 1 --2- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c1390d0 0x7f7f3c1394b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c09d0c0 tx=0x7f7f380046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f411f7640 1 --2- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7f200383b0 0x7f7f2003a870 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c13bb20 tx=0x7f7f2c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f327fc640 1 -- 172.21.15.46:0/3223234670 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f38043070 con 0x7f7f3c1390d0 2024-09-16T04:40:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:40:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.183+0000 7f7f327fc640 10 monclient: dump: 2024-09-16T04:40:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:33.209 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.184+0000 7f7f327fc640 1 -- 172.21.15.46:0/3223234670 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7f38032500 con 0x7f7f3c1390d0 2024-09-16T04:40:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.184+0000 7f7f327fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.184+0000 7f7f327fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:40:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.184+0000 7f7f327fc640 1 -- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c1390d0 msgr2=0x7f7f3c1394b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.184+0000 7f7f327fc640 1 --2- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c1390d0 0x7f7f3c1394b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c09d0c0 tx=0x7f7f380046c0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.184+0000 7f7f327fc640 1 -- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c1390d0 msgr2=0x7f7f3c1394b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:40:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.184+0000 7f7f327fc640 1 --2- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c1390d0 0x7f7f3c1394b0 secure :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c09d0c0 tx=0x7f7f380046c0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.210 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:33.210 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36296]: starting. 2024-09-16T04:40:33.211 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36296]: listing objects. 2024-09-16T04:40:33.211 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36296]: listed object 0... 2024-09-16T04:40:33.211 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36296]: listed object 25... 2024-09-16T04:40:33.211 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36296]: saw 50 objects 2024-09-16T04:40:33.211 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36296]: shutting down. 2024-09-16T04:40:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.190+0000 7f7f43c5c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:40:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.190+0000 7f7f43c5c640 1 --2- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c1399f0 0x7f7f3c09d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.190+0000 7f7f43c5c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.190+0000 7f7f43c5c640 1 -- 172.21.15.46:0/3223234670 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c067e40 con 0x7f7f3c1399f0 2024-09-16T04:40:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.190+0000 7f7f419f8640 1 --2- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c1399f0 0x7f7f3c09d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.190+0000 7f7f419f8640 10 monclient: get_auth_request con 0x7f7f3c1399f0 auth_method 0 2024-09-16T04:40:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.190+0000 7f7f419f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.190+0000 7f7f419f8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.190+0000 7f7f419f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.190+0000 7f7f419f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.190+0000 7f7f419f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.190+0000 7f7f419f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.190+0000 7f7f419f8640 10 monclient(hunting): handle_auth_done global_id 4951 payload 824 2024-09-16T04:40:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.191+0000 7f7f419f8640 1 -- 172.21.15.46:0/3223234670 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7f3c137130 con 0x7f7f3c1399f0 2024-09-16T04:40:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.191+0000 7f7f419f8640 1 --2- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c1399f0 0x7f7f3c09d0c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7f38000c00 tx=0x7f7f38008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.191+0000 7f7f327fc640 1 -- 172.21.15.46:0/3223234670 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f38037040 con 0x7f7f3c1399f0 2024-09-16T04:40:33.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.191+0000 7f7f327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:33.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.191+0000 7f7f327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:40:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.191+0000 7f7f327fc640 10 monclient: dump: 2024-09-16T04:40:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:33.217 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.191+0000 7f7f327fc640 1 -- 172.21.15.46:0/3223234670 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f7f38049a50 con 0x7f7f3c1399f0 2024-09-16T04:40:33.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.191+0000 7f7f327fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:40:33.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.191+0000 7f7f327fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:40:33.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.191+0000 7f7f327fc640 1 -- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c1399f0 msgr2=0x7f7f3c09d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.191+0000 7f7f327fc640 1 --2- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c1399f0 0x7f7f3c09d0c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7f38000c00 tx=0x7f7f38008030 comp rx=0 tx=0).stop 2024-09-16T04:40:33.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.191+0000 7f7f327fc640 1 -- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c1399f0 msgr2=0x7f7f3c09d0c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:40:33.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.191+0000 7f7f327fc640 1 --2- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c1399f0 0x7f7f3c09d0c0 secure :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7f38000c00 tx=0x7f7f38008030 comp rx=0 tx=0).stop 2024-09-16T04:40:33.218 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:40:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:40:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:40:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:40:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 20, 2024-09-16T04:40:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 119093, 2024-09-16T04:40:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:40:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:40:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:40:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:40:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 8, 2024-09-16T04:40:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:40:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:40:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:40:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:40:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:40:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:40:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:40:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:40:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:40:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:40:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:40:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:40:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:40:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:40:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:40:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:40:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:40:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:40:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:40:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:40:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:40:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:40:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:33.223 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:33.223 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:33.223 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:33.223 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:33.223 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:33.223 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:33.223 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:33.223 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:33.223 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:33.223 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:40:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:40:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:40:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:40:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:40:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:40:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:40:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:40:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:40:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:40:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:40:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:33.227 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.227 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.227 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:33.227 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:40:33.227 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:33.227 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:33.227 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:40:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:40:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:40:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:40:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:40:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:40:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:40:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:40:33.230 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:33.230 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:33.230 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:33.230 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:33.230 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:33.230 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.230 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:40:33.230 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:40:33.230 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.230 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:33.230 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:33.230 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:40:33.230 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:33.231 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:33.231 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:40:33.231 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.231 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:33.231 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:33.231 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:40:33.231 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:33.231 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:33.231 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:40:33.231 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.231 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:33.231 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:33.231 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6 2024-09-16T04:40:33.232 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:33.232 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:33.232 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:40:33.232 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:33.232 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:33.232 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:33.232 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:40:33.232 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:33.232 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:33.232 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:33.233 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:40:33.233 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:40:33.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.191+0000 7f7f43c5c640 1 -- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7f200383b0 msgr2=0x7f7f2003a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.191+0000 7f7f43c5c640 1 --2- 172.21.15.46:0/3223234670 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7f200383b0 0x7f7f2003a870 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c13bb20 tx=0x7f7f2c009040 comp rx=0 tx=0).stop 2024-09-16T04:40:33.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.191+0000 7f7f43c5c640 10 monclient: shutdown 2024-09-16T04:40:33.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.191+0000 7f7f43c5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:33.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.191+0000 7f7f43c5c640 1 -- 172.21.15.46:0/3223234670 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7f3c13be90 msgr2=0x7f7f3c138b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.191+0000 7f7f421f9640 1 -- 172.21.15.46:0/3223234670 reap_dead start 2024-09-16T04:40:33.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.191+0000 7f7f43c5c640 1 --2- 172.21.15.46:0/3223234670 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7f3c13be90 0x7f7f3c138b90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400c750 tx=0x7f7f3400cc10 comp rx=0 tx=0).stop 2024-09-16T04:40:33.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.192+0000 7f7f43c5c640 1 -- 172.21.15.46:0/3223234670 shutdown_connections 2024-09-16T04:40:33.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.192+0000 7f7f43c5c640 1 --2- 172.21.15.46:0/3223234670 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7f3c13be90 0x7f7f3c138b90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.192+0000 7f7f43c5c640 1 -- 172.21.15.46:0/3223234670 >> 172.21.15.46:0/3223234670 conn(0x7f7f3c0a5d10 msgr2=0x7f7f3c0a27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:33.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.192+0000 7f7f43c5c640 1 -- 172.21.15.46:0/3223234670 shutdown_connections 2024-09-16T04:40:33.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.192+0000 7f7f43c5c640 1 -- 172.21.15.46:0/3223234670 wait complete. 2024-09-16T04:40:33.236 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:40:33.236 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T04:40:33.236 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:40:33.236 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T04:40:33.236 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:40:33.236 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T04:40:33.236 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:40:33.248 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T04:40:33.248 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:40:33.248 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:40:33.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.387+0000 7f0c4b149640 10 monclient: get_monmap_and_config 2024-09-16T04:40:33.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.387+0000 7f0c4b149640 10 monclient: build_initial_monmap 2024-09-16T04:40:33.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.387+0000 7f0c4b149640 10 monclient: monmap: 2024-09-16T04:40:33.389 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:33.389 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:33.389 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:40:33.388047+0000 2024-09-16T04:40:33.389 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:40:33.388047+0000 2024-09-16T04:40:33.389 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:33.389 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.389 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:40:33.389 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:40:33.389 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:40:33.389 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:40:33.389 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:40:33.390 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.388+0000 7f0c4b149640 1 Processor -- start 2024-09-16T04:40:33.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.388+0000 7f0c4b149640 1 -- start start 2024-09-16T04:40:33.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.388+0000 7f0c4b149640 10 monclient: init 2024-09-16T04:40:33.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.388+0000 7f0c4b149640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:33.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.388+0000 7f0c4b149640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-16T04:40:33.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.388+0000 7f0c4b149640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0c440a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:40:33.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.388+0000 7f0c4b149640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0c440a3ff0 0x7f0c440a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:33.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.388+0000 7f0c4b149640 10 monclient(hunting): picked mon.noname-c con 0x7f0c440a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:40:33.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.388+0000 7f0c4b149640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c4413be90 0x7f0c4413e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:33.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.388+0000 7f0c4b149640 10 monclient(hunting): picked mon.noname-b con 0x7f0c4413be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:33.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.388+0000 7f0c4b149640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c4409b940 con 0x7f0c440a3c10 2024-09-16T04:40:33.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.388+0000 7f0c4b149640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:33.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.388+0000 7f0c4b149640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f0c4409ab00 con 0x7f0c440a3ff0 2024-09-16T04:40:33.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.388+0000 7f0c4b149640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:33.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.388+0000 7f0c4b149640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f0c4409b030 con 0x7f0c4413be90 2024-09-16T04:40:33.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.388+0000 7f0c4b149640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:33.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.388+0000 7f0c4b149640 10 monclient(hunting): authenticate will time out at 831.803868s 2024-09-16T04:40:33.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.388+0000 7f0c496e6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c4413be90 0x7f0c4413e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:33.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c496e6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c4413be90 0x7f0c4413e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:44198/0 (socket says 172.21.15.46:44198) 2024-09-16T04:40:33.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c496e6640 1 -- 172.21.15.46:0/2457778429 learned_addr learned my addr 172.21.15.46:0/2457778429 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:33.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c496e6640 10 monclient(hunting): get_auth_request con 0x7f0c4413be90 auth_method 0 2024-09-16T04:40:33.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c496e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:33.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c496e6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:33.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c496e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:33.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c496e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:33.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c496e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:33.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c43fff640 1 --2- 172.21.15.46:0/2457778429 >> v2:172.21.15.114:3300/0 conn(0x7f0c440a3ff0 0x7f0c440a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:33.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c496e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:33.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c43fff640 10 monclient(hunting): get_auth_request con 0x7f0c440a3ff0 auth_method 0 2024-09-16T04:40:33.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:33.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c43fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:33.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:33.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c496e6640 10 monclient(hunting): handle_auth_done global_id 24161 payload 274 2024-09-16T04:40:33.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c496e6640 1 -- 172.21.15.46:0/2457778429 >> v2:172.21.15.114:3300/0 conn(0x7f0c440a3ff0 msgr2=0x7f0c440a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:40:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c496e6640 1 --2- 172.21.15.46:0/2457778429 >> v2:172.21.15.114:3300/0 conn(0x7f0c440a3ff0 0x7f0c440a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c496e6640 1 -- 172.21.15.46:0/2457778429 >> v1:172.21.15.114:6789/0 conn(0x7f0c440a3c10 legacy=0x7f0c4409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c496e6640 10 monclient: _finish_hunting 0 2024-09-16T04:40:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c496e6640 1 monclient: found mon.noname-b 2024-09-16T04:40:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c496e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c496e6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:40:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c496e6640 1 -- 172.21.15.46:0/2457778429 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c4409b310 con 0x7f0c4413be90 2024-09-16T04:40:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.389+0000 7f0c43fff640 1 --2- 172.21.15.46:0/2457778429 >> v2:172.21.15.114:3300/0 conn(0x7f0c440a3ff0 0x7f0c440a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:40:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c496e6640 1 --2- 172.21.15.46:0/2457778429 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c4413be90 0x7f0c4413e280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c009e50 tx=0x7f0c3c02a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17e00ad50c4e7ab6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c437fe640 1 -- 172.21.15.46:0/2457778429 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c036070 con 0x7f0c4413be90 2024-09-16T04:40:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:33.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c437fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:40:33.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c437fe640 10 monclient: dump: 2024-09-16T04:40:33.402 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:33.402 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:33.402 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.402 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.402 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:33.402 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.402 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:33.402 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:33.402 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:33.402 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c437fe640 10 monclient: _finish_auth 0 2024-09-16T04:40:33.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c437fe640 10 monclient: _check_auth_tickets 2024-09-16T04:40:33.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:33.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c437fe640 1 -- 172.21.15.46:0/2457778429 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c3c031600 con 0x7f0c4413be90 2024-09-16T04:40:33.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c4b149640 5 monclient: authenticate success, global_id 24161 2024-09-16T04:40:33.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c4b149640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:40:33.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c437fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:33.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c4b149640 10 monclient: get_monmap_and_config success 2024-09-16T04:40:33.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c437fe640 1 -- 172.21.15.46:0/2457778429 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c0319d0 con 0x7f0c4413be90 2024-09-16T04:40:33.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c4b149640 4 set_mon_vals no callback set 2024-09-16T04:40:33.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:33.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:33.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c437fe640 10 monclient: dump: 2024-09-16T04:40:33.403 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:33.403 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:33.403 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:33.404 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c4b149640 10 monclient: shutdown 2024-09-16T04:40:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c4b149640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c4b149640 1 -- 172.21.15.46:0/2457778429 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c4413be90 msgr2=0x7f0c4413e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c4b149640 1 --2- 172.21.15.46:0/2457778429 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c4413be90 0x7f0c4413e280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c009e50 tx=0x7f0c3c02a680 comp rx=0 tx=0).stop 2024-09-16T04:40:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c4b149640 1 -- 172.21.15.46:0/2457778429 shutdown_connections 2024-09-16T04:40:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c4b149640 1 --2- 172.21.15.46:0/2457778429 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c4413be90 0x7f0c4413e280 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c4b149640 1 --2- 172.21.15.46:0/2457778429 >> v2:172.21.15.114:3300/0 conn(0x7f0c440a3ff0 0x7f0c440a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c4b149640 1 -- 172.21.15.46:0/2457778429 >> 172.21.15.46:0/2457778429 conn(0x7f0c440a6340 msgr2=0x7f0c4406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:33.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c4b149640 1 -- 172.21.15.46:0/2457778429 shutdown_connections 2024-09-16T04:40:33.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.390+0000 7f0c4b149640 1 -- 172.21.15.46:0/2457778429 wait complete. 2024-09-16T04:40:33.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.391+0000 7f0c4b149640 10 monclient: build_initial_monmap 2024-09-16T04:40:33.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.391+0000 7f0c4b149640 10 monclient: monmap: 2024-09-16T04:40:33.405 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:33.405 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:40:33.405 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:40:33.405 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:40:33.405 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:33.405 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.405 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:40:33.406 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:40:33.406 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:40:33.406 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.391+0000 7f0c4b149640 1 Processor -- start 2024-09-16T04:40:33.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.391+0000 7f0c4b149640 1 -- start start 2024-09-16T04:40:33.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.391+0000 7f0c4b149640 10 monclient: init 2024-09-16T04:40:33.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c4b149640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:33.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c4b149640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:40:33.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c4b149640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c440a3c10 0x7f0c44133cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:33.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c4b149640 10 monclient(hunting): picked mon.noname-b con 0x7f0c440a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:40:33.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c4b149640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c440a3ff0 0x7f0c44134230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:33.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c4b149640 10 monclient(hunting): picked mon.noname-c con 0x7f0c440a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:33.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c4b149640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c4413be90 0x7f0c441cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:33.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c4b149640 10 monclient(hunting): picked mon.noname-a con 0x7f0c4413be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:40:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c4b149640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c4b149640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0c440a6b40 con 0x7f0c4413be90 2024-09-16T04:40:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c4b149640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c4b149640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0c440a69c0 con 0x7f0c440a3c10 2024-09-16T04:40:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c4b149640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c4b149640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f0c4413e7c0 con 0x7f0c440a3ff0 2024-09-16T04:40:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c4b149640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c4b149640 10 monclient(hunting): authenticate will time out at 831.807368s 2024-09-16T04:40:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c496e6640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c4413be90 msgr2=0x7f0c441cb9e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:40:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c496e6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c4413be90 0x7f0c441cb9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c43fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c440a3ff0 0x7f0c44134230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c43fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c440a3ff0 0x7f0c44134230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:44212/0 (socket says 172.21.15.46:44212) 2024-09-16T04:40:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c43fff640 1 -- 172.21.15.46:0/3567579659 learned_addr learned my addr 172.21.15.46:0/3567579659 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c43fff640 10 monclient(hunting): get_auth_request con 0x7f0c440a3ff0 auth_method 0 2024-09-16T04:40:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c43fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c48ee5640 1 --2- 172.21.15.46:0/3567579659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c440a3c10 0x7f0c44133cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c48ee5640 10 monclient(hunting): get_auth_request con 0x7f0c440a3c10 auth_method 0 2024-09-16T04:40:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c48ee5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c48ee5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c48ee5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c48ee5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c48ee5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c48ee5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c43fff640 10 monclient(hunting): handle_auth_done global_id 24164 payload 824 2024-09-16T04:40:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c43fff640 1 -- 172.21.15.46:0/3567579659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c440a3c10 msgr2=0x7f0c44133cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c43fff640 1 --2- 172.21.15.46:0/3567579659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c440a3c10 0x7f0c44133cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c43fff640 1 -- 172.21.15.46:0/3567579659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c4413be90 msgr2=0x7f0c441cb9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:40:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c43fff640 1 --2- 172.21.15.46:0/3567579659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c4413be90 0x7f0c441cb9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c43fff640 10 monclient: _finish_hunting 0 2024-09-16T04:40:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c43fff640 1 monclient: found mon.noname-c 2024-09-16T04:40:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c43fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:40:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c43fff640 1 -- 172.21.15.46:0/3567579659 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c4413a4c0 con 0x7f0c440a3ff0 2024-09-16T04:40:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.392+0000 7f0c43fff640 1 --2- 172.21.15.46:0/3567579659 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c440a3ff0 0x7f0c44134230 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0c3000e760 tx=0x7f0c3000ec20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c417fa640 1 -- 172.21.15.46:0/3567579659 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c3000f040 con 0x7f0c440a3ff0 2024-09-16T04:40:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c417fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:40:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c417fa640 10 monclient: dump: 2024-09-16T04:40:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:33.410 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c417fa640 10 monclient: _finish_auth 0 2024-09-16T04:40:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c417fa640 10 monclient: _check_auth_tickets 2024-09-16T04:40:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c417fa640 1 -- 172.21.15.46:0/3567579659 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c30004230 con 0x7f0c440a3ff0 2024-09-16T04:40:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c4b149640 5 monclient: authenticate success, global_id 24164 2024-09-16T04:40:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c4b149640 10 monclient: _renew_subs 2024-09-16T04:40:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c4b149640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:40:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c4b149640 1 -- 172.21.15.46:0/3567579659 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c440a0400 con 0x7f0c440a3ff0 2024-09-16T04:40:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c417fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c417fa640 1 -- 172.21.15.46:0/3567579659 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c3000ee70 con 0x7f0c440a3ff0 2024-09-16T04:40:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c417fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c417fa640 10 monclient: dump: 2024-09-16T04:40:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:33.412 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:33.412 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:33.412 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:33.412 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:33.412 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:33.412 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:33.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c42ffd640 4 set_mon_vals no callback set 2024-09-16T04:40:33.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c4b149640 10 monclient: _renew_subs 2024-09-16T04:40:33.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c4b149640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:40:33.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c4b149640 1 -- 172.21.15.46:0/3567579659 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c4413a1e0 con 0x7f0c440a3ff0 2024-09-16T04:40:33.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.393+0000 7f0c417fa640 1 -- 172.21.15.46:0/3567579659 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f0c3000c6d0 con 0x7f0c440a3ff0 2024-09-16T04:40:33.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.394+0000 7f0c4b149640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:33.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.394+0000 7f0c4b149640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:33.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.394+0000 7f0c4b149640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:40:33.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.394+0000 7f0c4b149640 1 -- 172.21.15.46:0/3567579659 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c440a43d0 con 0x7f0c440a3ff0 2024-09-16T04:40:33.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.394+0000 7f0c417fa640 1 --2- 172.21.15.46:0/3567579659 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0c28038360 0x7f0c2803a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:33.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.394+0000 7f0c417fa640 1 -- 172.21.15.46:0/3567579659 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(79..79 src has 1..79) ==== 4738+0+0 (secure 0 0 0) 0x7f0c3004b440 con 0x7f0c440a3ff0 2024-09-16T04:40:33.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.394+0000 7f0c48ee5640 1 --2- 172.21.15.46:0/3567579659 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0c28038360 0x7f0c2803a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:33.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.394+0000 7f0c48ee5640 10 monclient: get_auth_request con 0x7f0c28038360 auth_method 0 2024-09-16T04:40:33.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.394+0000 7f0c48ee5640 1 --2- 172.21.15.46:0/3567579659 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0c28038360 0x7f0c2803a820 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0c38000f60 tx=0x7f0c38009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:33.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.399+0000 7f0c417fa640 1 -- 172.21.15.46:0/3567579659 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0c30017020 con 0x7f0c440a3ff0 2024-09-16T04:40:33.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.399+0000 7f0c417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:33.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.399+0000 7f0c417fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:40:33.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.646+0000 7f0c4b149640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:40:33.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.646+0000 7f0c4b149640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:40:33.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.646+0000 7f0c4b149640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:40:33.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.646+0000 7f0c4b149640 1 -- 172.21.15.46:0/3567579659 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0c44139010 con 0x7f0c440a3ff0 2024-09-16T04:40:33.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.646+0000 7f0c417fa640 1 -- 172.21.15.46:0/3567579659 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f0c30013dd0 con 0x7f0c440a3ff0 2024-09-16T04:40:33.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.646+0000 7f0c417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:40:33.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.646+0000 7f0c417fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:40:33.648 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:40:33.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.648+0000 7f0c4b149640 1 -- 172.21.15.46:0/3567579659 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0c28038360 msgr2=0x7f0c2803a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.648+0000 7f0c4b149640 1 --2- 172.21.15.46:0/3567579659 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0c28038360 0x7f0c2803a820 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0c38000f60 tx=0x7f0c38009040 comp rx=0 tx=0).stop 2024-09-16T04:40:33.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.648+0000 7f0c4b149640 10 monclient: shutdown 2024-09-16T04:40:33.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.648+0000 7f0c4b149640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:33.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.648+0000 7f0c4b149640 1 -- 172.21.15.46:0/3567579659 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c440a3ff0 msgr2=0x7f0c44134230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:33.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.648+0000 7f0c4b149640 1 --2- 172.21.15.46:0/3567579659 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c440a3ff0 0x7f0c44134230 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0c3000e760 tx=0x7f0c3000ec20 comp rx=0 tx=0).stop 2024-09-16T04:40:33.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.649+0000 7f0c4b149640 1 -- 172.21.15.46:0/3567579659 shutdown_connections 2024-09-16T04:40:33.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.649+0000 7f0c4b149640 1 --2- 172.21.15.46:0/3567579659 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0c28038360 0x7f0c2803a820 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.649+0000 7f0c4b149640 1 --2- 172.21.15.46:0/3567579659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c4413be90 0x7f0c441cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.649+0000 7f0c4b149640 1 --2- 172.21.15.46:0/3567579659 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c440a3ff0 0x7f0c44134230 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.649+0000 7f0c4b149640 1 --2- 172.21.15.46:0/3567579659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c440a3c10 0x7f0c44133cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.649+0000 7f0c4b149640 1 -- 172.21.15.46:0/3567579659 >> 172.21.15.46:0/3567579659 conn(0x7f0c440a6340 msgr2=0x7f0c44136c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.649+0000 7f0c4b149640 1 -- 172.21.15.46:0/3567579659 shutdown_connections 2024-09-16T04:40:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:33.649+0000 7f0c4b149640 1 -- 172.21.15.46:0/3567579659 wait complete. 2024-09-16T04:40:36.661 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:40:36.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.774+0000 7f05e8a77640 10 monclient: get_monmap_and_config 2024-09-16T04:40:36.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.774+0000 7f05e8a77640 10 monclient: build_initial_monmap 2024-09-16T04:40:36.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.774+0000 7f05e8a77640 10 monclient: monmap: 2024-09-16T04:40:36.776 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:36.776 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:36.776 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:40:36.774751+0000 2024-09-16T04:40:36.776 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:40:36.774751+0000 2024-09-16T04:40:36.777 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:36.777 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:36.777 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:40:36.777 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:40:36.777 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:40:36.777 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:40:36.777 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:40:36.777 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:36.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.774+0000 7f05e8a77640 1 Processor -- start 2024-09-16T04:40:36.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.774+0000 7f05e8a77640 1 -- start start 2024-09-16T04:40:36.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.774+0000 7f05e8a77640 10 monclient: init 2024-09-16T04:40:36.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e8a77640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:36.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e8a77640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-16T04:40:36.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e8a77640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f05e40a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:40:36.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e8a77640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f05e40a3ff0 0x7f05e40a4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:36.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e8a77640 10 monclient(hunting): picked mon.noname-c con 0x7f05e40a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:40:36.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e8a77640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f05e413be90 0x7f05e413e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:36.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e8a77640 10 monclient(hunting): picked mon.noname-b con 0x7f05e413be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:36.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e8a77640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05e409ab70 con 0x7f05e40a3c10 2024-09-16T04:40:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e8a77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e8a77640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f05e40a4d60 con 0x7f05e40a3ff0 2024-09-16T04:40:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e8a77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e8a77640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f05e409b030 con 0x7f05e413be90 2024-09-16T04:40:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e8a77640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e8a77640 10 monclient(hunting): authenticate will time out at 835.190687s 2024-09-16T04:40:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e259c640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f05e40a3c10 legacy=0x7f05e409bcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:40:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e2d9d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f05e413be90 0x7f05e413e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e2d9d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f05e413be90 0x7f05e413e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:37780/0 (socket says 172.21.15.46:37780) 2024-09-16T04:40:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e2d9d640 1 -- 172.21.15.46:0/3417002242 learned_addr learned my addr 172.21.15.46:0/3417002242 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e2d9d640 10 monclient(hunting): get_auth_request con 0x7f05e413be90 auth_method 0 2024-09-16T04:40:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:36.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:36.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:36.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:36.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.775+0000 7f05e2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:36.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.776+0000 7f05e1d9b640 1 --2- 172.21.15.46:0/3417002242 >> v2:172.21.15.114:3300/0 conn(0x7f05e40a3ff0 0x7f05e40a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:36.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.776+0000 7f05e1d9b640 10 monclient(hunting): get_auth_request con 0x7f05e40a3ff0 auth_method 0 2024-09-16T04:40:36.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.776+0000 7f05e1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:36.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.776+0000 7f05e1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:36.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.776+0000 7f05e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:36.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.776+0000 7f05e2d9d640 10 monclient(hunting): handle_auth_done global_id 24167 payload 274 2024-09-16T04:40:36.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.776+0000 7f05e2d9d640 1 -- 172.21.15.46:0/3417002242 >> v2:172.21.15.114:3300/0 conn(0x7f05e40a3ff0 msgr2=0x7f05e40a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:40:36.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.776+0000 7f05e2d9d640 1 --2- 172.21.15.46:0/3417002242 >> v2:172.21.15.114:3300/0 conn(0x7f05e40a3ff0 0x7f05e40a4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:36.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.776+0000 7f05e2d9d640 1 -- 172.21.15.46:0/3417002242 >> v1:172.21.15.46:6789/0 conn(0x7f05e40a3c10 legacy=0x7f05e409bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:40:36.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.776+0000 7f05e2d9d640 10 monclient: _finish_hunting 0 2024-09-16T04:40:36.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.776+0000 7f05e2d9d640 1 monclient: found mon.noname-b 2024-09-16T04:40:36.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.776+0000 7f05e2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:36.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.776+0000 7f05e2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:40:36.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.776+0000 7f05e2d9d640 1 -- 172.21.15.46:0/3417002242 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05e409b310 con 0x7f05e413be90 2024-09-16T04:40:36.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.776+0000 7f05e2d9d640 1 --2- 172.21.15.46:0/3417002242 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f05e413be90 0x7f05e413e280 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f05dc009e50 tx=0x7f05dc02a5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14b1a8fae4d3a06f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:36.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e159a640 1 -- 172.21.15.46:0/3417002242 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc036070 con 0x7f05e413be90 2024-09-16T04:40:36.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:36.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:40:36.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e159a640 10 monclient: dump: 2024-09-16T04:40:36.789 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:36.789 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:36.789 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:36.789 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:36.790 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:36.790 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:36.790 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:36.790 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:36.790 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:36.790 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:36.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e159a640 10 monclient: _finish_auth 0 2024-09-16T04:40:36.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e159a640 10 monclient: _check_auth_tickets 2024-09-16T04:40:36.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:36.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e8a77640 5 monclient: authenticate success, global_id 24167 2024-09-16T04:40:36.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e159a640 1 -- 172.21.15.46:0/3417002242 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05dc031600 con 0x7f05e413be90 2024-09-16T04:40:36.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e8a77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:40:36.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e8a77640 10 monclient: get_monmap_and_config success 2024-09-16T04:40:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e8a77640 4 set_mon_vals no callback set 2024-09-16T04:40:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e159a640 1 -- 172.21.15.46:0/3417002242 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc0319d0 con 0x7f05e413be90 2024-09-16T04:40:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e159a640 10 monclient: dump: 2024-09-16T04:40:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:36.792 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e8a77640 10 monclient: shutdown 2024-09-16T04:40:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e8a77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e8a77640 1 -- 172.21.15.46:0/3417002242 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f05e413be90 msgr2=0x7f05e413e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e8a77640 1 --2- 172.21.15.46:0/3417002242 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f05e413be90 0x7f05e413e280 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f05dc009e50 tx=0x7f05dc02a5b0 comp rx=0 tx=0).stop 2024-09-16T04:40:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e8a77640 1 -- 172.21.15.46:0/3417002242 shutdown_connections 2024-09-16T04:40:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e8a77640 1 --2- 172.21.15.46:0/3417002242 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f05e413be90 0x7f05e413e280 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e8a77640 1 --2- 172.21.15.46:0/3417002242 >> v2:172.21.15.114:3300/0 conn(0x7f05e40a3ff0 0x7f05e40a4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e8a77640 1 -- 172.21.15.46:0/3417002242 >> 172.21.15.46:0/3417002242 conn(0x7f05e40a5d10 msgr2=0x7f05e40a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e8a77640 1 -- 172.21.15.46:0/3417002242 shutdown_connections 2024-09-16T04:40:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.777+0000 7f05e8a77640 1 -- 172.21.15.46:0/3417002242 wait complete. 2024-09-16T04:40:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.778+0000 7f05e8a77640 10 monclient: build_initial_monmap 2024-09-16T04:40:36.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.778+0000 7f05e8a77640 10 monclient: monmap: 2024-09-16T04:40:36.793 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:36.793 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:40:36.793 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:40:36.793 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:40:36.793 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:36.793 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:36.793 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:40:36.793 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:40:36.793 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:40:36.793 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:36.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.778+0000 7f05e8a77640 1 Processor -- start 2024-09-16T04:40:36.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.778+0000 7f05e8a77640 1 -- start start 2024-09-16T04:40:36.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.778+0000 7f05e8a77640 10 monclient: init 2024-09-16T04:40:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e8a77640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e8a77640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:40:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e8a77640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f05e40a3c10 0x7f05e4133df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e8a77640 10 monclient(hunting): picked mon.noname-c con 0x7f05e40a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e8a77640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05e40a3ff0 0x7f05e4134330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e8a77640 10 monclient(hunting): picked mon.noname-a con 0x7f05e40a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:40:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e8a77640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05e413be90 0x7f05e41cba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e8a77640 10 monclient(hunting): picked mon.noname-b con 0x7f05e413be90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:40:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e8a77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e8a77640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f05e406e550 con 0x7f05e40a3ff0 2024-09-16T04:40:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e8a77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e8a77640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f05e406e3d0 con 0x7f05e413be90 2024-09-16T04:40:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e8a77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e8a77640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f05e413e7c0 con 0x7f05e40a3c10 2024-09-16T04:40:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e8a77640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e8a77640 10 monclient(hunting): authenticate will time out at 835.194504s 2024-09-16T04:40:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e1d9b640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05e40a3ff0 msgr2=0x7f05e4134330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:40:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e1d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05e40a3ff0 0x7f05e4134330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e259c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f05e40a3c10 0x7f05e4133df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e259c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f05e40a3c10 0x7f05e4133df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:37788/0 (socket says 172.21.15.46:37788) 2024-09-16T04:40:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e259c640 1 -- 172.21.15.46:0/1494438941 learned_addr learned my addr 172.21.15.46:0/1494438941 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e259c640 10 monclient(hunting): get_auth_request con 0x7f05e40a3c10 auth_method 0 2024-09-16T04:40:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e2d9d640 1 --2- 172.21.15.46:0/1494438941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05e413be90 0x7f05e41cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:36.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:36.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e2d9d640 10 monclient(hunting): get_auth_request con 0x7f05e413be90 auth_method 0 2024-09-16T04:40:36.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:36.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:36.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:36.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:36.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:36.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:36.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e259c640 10 monclient(hunting): handle_auth_done global_id 24170 payload 824 2024-09-16T04:40:36.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e259c640 1 -- 172.21.15.46:0/1494438941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05e413be90 msgr2=0x7f05e41cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:36.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e259c640 1 --2- 172.21.15.46:0/1494438941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05e413be90 0x7f05e41cba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:36.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e259c640 1 -- 172.21.15.46:0/1494438941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05e40a3ff0 msgr2=0x7f05e4134330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:40:36.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e259c640 1 --2- 172.21.15.46:0/1494438941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05e40a3ff0 0x7f05e4134330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:36.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e259c640 10 monclient: _finish_hunting 0 2024-09-16T04:40:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e259c640 1 monclient: found mon.noname-c 2024-09-16T04:40:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:40:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.779+0000 7f05e259c640 1 -- 172.21.15.46:0/1494438941 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05e4139e80 con 0x7f05e40a3c10 2024-09-16T04:40:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05e259c640 1 --2- 172.21.15.46:0/1494438941 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f05e40a3c10 0x7f05e4133df0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f05d400b550 tx=0x7f05d400ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05d2ffd640 1 -- 172.21.15.46:0/1494438941 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d4003dc0 con 0x7f05e40a3c10 2024-09-16T04:40:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05d2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:40:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05d2ffd640 10 monclient: dump: 2024-09-16T04:40:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:36.798 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:36.798 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:36.798 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:36.798 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:36.798 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:36.798 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:36.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05d2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:40:36.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05d2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:40:36.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:36.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05d2ffd640 1 -- 172.21.15.46:0/1494438941 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05d4015070 con 0x7f05e40a3c10 2024-09-16T04:40:36.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05d2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:36.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05d2ffd640 1 -- 172.21.15.46:0/1494438941 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d40052b0 con 0x7f05e40a3c10 2024-09-16T04:40:36.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05e8a77640 5 monclient: authenticate success, global_id 24170 2024-09-16T04:40:36.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:36.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:36.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05d2ffd640 10 monclient: dump: 2024-09-16T04:40:36.799 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:36.799 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:36.799 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:36.799 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:36.799 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:36.799 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:36.799 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:36.799 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:36.799 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:36.799 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:36.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05e8a77640 10 monclient: _renew_subs 2024-09-16T04:40:36.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05e0d99640 4 set_mon_vals no callback set 2024-09-16T04:40:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05e8a77640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:40:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05e8a77640 1 -- 172.21.15.46:0/1494438941 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05e413c9c0 con 0x7f05e40a3c10 2024-09-16T04:40:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05e8a77640 10 monclient: _renew_subs 2024-09-16T04:40:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05e8a77640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:40:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.780+0000 7f05e8a77640 1 -- 172.21.15.46:0/1494438941 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f05e40a0400 con 0x7f05e40a3c10 2024-09-16T04:40:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.781+0000 7f05d2ffd640 1 -- 172.21.15.46:0/1494438941 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f05d4005450 con 0x7f05e40a3c10 2024-09-16T04:40:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.781+0000 7f05e8a77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.781+0000 7f05e8a77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.781+0000 7f05e8a77640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:40:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.781+0000 7f05e8a77640 1 -- 172.21.15.46:0/1494438941 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05e40a4400 con 0x7f05e40a3c10 2024-09-16T04:40:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.781+0000 7f05d2ffd640 1 --2- 172.21.15.46:0/1494438941 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f05c4038360 0x7f05c403a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.781+0000 7f05d2ffd640 1 -- 172.21.15.46:0/1494438941 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(79..79 src has 1..79) ==== 4738+0+0 (secure 0 0 0) 0x7f05d404b4a0 con 0x7f05e40a3c10 2024-09-16T04:40:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.781+0000 7f05e1d9b640 1 --2- 172.21.15.46:0/1494438941 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f05c4038360 0x7f05c403a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:36.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.781+0000 7f05e1d9b640 10 monclient: get_auth_request con 0x7f05c4038360 auth_method 0 2024-09-16T04:40:36.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.782+0000 7f05e1d9b640 1 --2- 172.21.15.46:0/1494438941 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f05c4038360 0x7f05c403a820 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f05e4138410 tx=0x7f05d80092e0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:36.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.786+0000 7f05d2ffd640 1 -- 172.21.15.46:0/1494438941 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f05d40193c0 con 0x7f05e40a3c10 2024-09-16T04:40:36.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.786+0000 7f05d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:36.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:36.786+0000 7f05d2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:40:37.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:37.033+0000 7f05e8a77640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:40:37.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:37.033+0000 7f05e8a77640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:40:37.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:37.033+0000 7f05e8a77640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:40:37.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:37.033+0000 7f05e8a77640 1 -- 172.21.15.46:0/1494438941 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f05e4138470 con 0x7f05e40a3c10 2024-09-16T04:40:37.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:37.033+0000 7f05d2ffd640 1 -- 172.21.15.46:0/1494438941 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f05d4018360 con 0x7f05e40a3c10 2024-09-16T04:40:37.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:37.033+0000 7f05d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:40:37.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:37.033+0000 7f05d2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:40:37.035 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:40:37.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:37.035+0000 7f05e8a77640 1 -- 172.21.15.46:0/1494438941 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f05c4038360 msgr2=0x7f05c403a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:37.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:37.035+0000 7f05e8a77640 1 --2- 172.21.15.46:0/1494438941 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f05c4038360 0x7f05c403a820 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f05e4138410 tx=0x7f05d80092e0 comp rx=0 tx=0).stop 2024-09-16T04:40:37.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:37.035+0000 7f05e8a77640 10 monclient: shutdown 2024-09-16T04:40:37.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:37.035+0000 7f05e8a77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:37.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:37.035+0000 7f05e8a77640 1 -- 172.21.15.46:0/1494438941 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f05e40a3c10 msgr2=0x7f05e4133df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:37.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:37.035+0000 7f05e8a77640 1 --2- 172.21.15.46:0/1494438941 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f05e40a3c10 0x7f05e4133df0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f05d400b550 tx=0x7f05d400ba10 comp rx=0 tx=0).stop 2024-09-16T04:40:37.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:37.035+0000 7f05e8a77640 1 -- 172.21.15.46:0/1494438941 shutdown_connections 2024-09-16T04:40:37.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:37.035+0000 7f05e8a77640 1 --2- 172.21.15.46:0/1494438941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05e413be90 0x7f05e41cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:37.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:37.035+0000 7f05e8a77640 1 --2- 172.21.15.46:0/1494438941 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f05c4038360 0x7f05c403a820 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:37.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:37.035+0000 7f05e8a77640 1 --2- 172.21.15.46:0/1494438941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05e40a3ff0 0x7f05e4134330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:37.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:37.035+0000 7f05e8a77640 1 --2- 172.21.15.46:0/1494438941 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f05e40a3c10 0x7f05e4133df0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:37.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:37.035+0000 7f05e8a77640 1 -- 172.21.15.46:0/1494438941 >> 172.21.15.46:0/1494438941 conn(0x7f05e40a5d10 msgr2=0x7f05e413b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:37.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:37.035+0000 7f05e8a77640 1 -- 172.21.15.46:0/1494438941 shutdown_connections 2024-09-16T04:40:37.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:37.035+0000 7f05e8a77640 1 -- 172.21.15.46:0/1494438941 wait complete. 2024-09-16T04:40:40.049 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:40:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.161+0000 7f3de218d640 10 monclient: get_monmap_and_config 2024-09-16T04:40:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.161+0000 7f3de218d640 10 monclient: build_initial_monmap 2024-09-16T04:40:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.161+0000 7f3de218d640 10 monclient: monmap: 2024-09-16T04:40:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:40:40.162154+0000 2024-09-16T04:40:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:40:40.162154+0000 2024-09-16T04:40:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:40:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:40:40.164 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:40:40.164 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:40:40.164 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:40:40.164 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:40.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.162+0000 7f3de218d640 1 Processor -- start 2024-09-16T04:40:40.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.162+0000 7f3de218d640 1 -- start start 2024-09-16T04:40:40.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.162+0000 7f3de218d640 10 monclient: init 2024-09-16T04:40:40.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.162+0000 7f3de218d640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:40.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3de218d640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-16T04:40:40.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3de218d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3ddc0a3c10 0x7f3ddc09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:40.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3de218d640 10 monclient(hunting): picked mon.noname-a con 0x7f3ddc0a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:40:40.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3de218d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3ddc0a3ff0 0x7f3ddc0a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:40.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3de218d640 10 monclient(hunting): picked mon.noname-b con 0x7f3ddc0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:40.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3de218d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3ddc1369d0 addr v1:172.21.15.114:6789/0 2024-09-16T04:40:40.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3de218d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ddc09b940 con 0x7f3ddc1369d0 2024-09-16T04:40:40.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3de218d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:40.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3de218d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f3ddc09ab00 con 0x7f3ddc0a3c10 2024-09-16T04:40:40.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3de218d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:40.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3de218d640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f3ddc09b030 con 0x7f3ddc0a3ff0 2024-09-16T04:40:40.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3de218d640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:40.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3de218d640 10 monclient(hunting): authenticate will time out at 838.578112s 2024-09-16T04:40:40.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3ddb7fe640 1 -- >> v2:172.21.15.46:3300/0 conn(0x7f3ddc0a3c10 msgr2=0x7f3ddc09bcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:40:40.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3ddb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3ddc0a3c10 0x7f3ddc09bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:40.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3ddaffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3ddc0a3ff0 0x7f3ddc0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:40.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3ddaffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3ddc0a3ff0 0x7f3ddc0a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:37792/0 (socket says 172.21.15.46:37792) 2024-09-16T04:40:40.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3ddaffd640 1 -- 172.21.15.46:0/2431901849 learned_addr learned my addr 172.21.15.46:0/2431901849 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:40.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3ddaffd640 10 monclient(hunting): get_auth_request con 0x7f3ddc0a3ff0 auth_method 0 2024-09-16T04:40:40.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3ddaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:40.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3ddaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:40.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3ddaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:40.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3ddaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:40.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3ddaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:40.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3ddaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:40.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3ddaffd640 10 monclient(hunting): handle_auth_done global_id 24173 payload 274 2024-09-16T04:40:40.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3ddaffd640 1 -- 172.21.15.46:0/2431901849 >> v2:172.21.15.46:3300/0 conn(0x7f3ddc0a3c10 msgr2=0x7f3ddc09bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:40:40.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3ddaffd640 1 --2- 172.21.15.46:0/2431901849 >> v2:172.21.15.46:3300/0 conn(0x7f3ddc0a3c10 0x7f3ddc09bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:40.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3ddaffd640 1 -- 172.21.15.46:0/2431901849 >> v1:172.21.15.114:6789/0 conn(0x7f3ddc1369d0 legacy=0x7f3ddc136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:40.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3ddaffd640 10 monclient: _finish_hunting 0 2024-09-16T04:40:40.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3ddaffd640 1 monclient: found mon.noname-b 2024-09-16T04:40:40.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3ddaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:40.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3ddaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:40:40.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.163+0000 7f3ddaffd640 1 -- 172.21.15.46:0/2431901849 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ddc09b310 con 0x7f3ddc0a3ff0 2024-09-16T04:40:40.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3ddaffd640 1 --2- 172.21.15.46:0/2431901849 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3ddc0a3ff0 0x7f3ddc0a43d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0009870 tx=0x7f3dd002a150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=653038bf6cfc15e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:40.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3dda7fc640 1 -- 172.21.15.46:0/2431901849 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0037070 con 0x7f3ddc0a3ff0 2024-09-16T04:40:40.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3dda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:40.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3dda7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:40:40.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3dda7fc640 10 monclient: dump: 2024-09-16T04:40:40.176 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:40.177 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:40.177 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:40.177 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:40.177 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:40.177 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:40.177 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:40.177 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:40.177 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:40.177 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:40.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3dda7fc640 10 monclient: _finish_auth 0 2024-09-16T04:40:40.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3dda7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:40:40.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3dda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:40.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3dda7fc640 1 -- 172.21.15.46:0/2431901849 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dd0032440 con 0x7f3ddc0a3ff0 2024-09-16T04:40:40.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3de218d640 5 monclient: authenticate success, global_id 24173 2024-09-16T04:40:40.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3de218d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:40:40.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3dda7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:40.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3dda7fc640 1 -- 172.21.15.46:0/2431901849 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0032810 con 0x7f3ddc0a3ff0 2024-09-16T04:40:40.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3dda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:40.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3dda7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:40.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3dda7fc640 10 monclient: dump: 2024-09-16T04:40:40.179 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:40.179 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:40.179 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:40.179 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:40.179 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:40.179 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:40.179 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:40.179 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:40.179 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:40.180 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:40.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3de218d640 10 monclient: get_monmap_and_config success 2024-09-16T04:40:40.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3de218d640 4 set_mon_vals no callback set 2024-09-16T04:40:40.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3de218d640 10 monclient: shutdown 2024-09-16T04:40:40.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3de218d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:40.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3de218d640 1 -- 172.21.15.46:0/2431901849 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3ddc0a3ff0 msgr2=0x7f3ddc0a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:40.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.164+0000 7f3de218d640 1 --2- 172.21.15.46:0/2431901849 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3ddc0a3ff0 0x7f3ddc0a43d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0009870 tx=0x7f3dd002a150 comp rx=0 tx=0).stop 2024-09-16T04:40:40.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.165+0000 7f3de218d640 1 -- 172.21.15.46:0/2431901849 shutdown_connections 2024-09-16T04:40:40.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.165+0000 7f3de218d640 1 --2- 172.21.15.46:0/2431901849 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3ddc0a3ff0 0x7f3ddc0a43d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:40.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.165+0000 7f3de218d640 1 --2- 172.21.15.46:0/2431901849 >> v2:172.21.15.46:3300/0 conn(0x7f3ddc0a3c10 0x7f3ddc09bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:40.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.165+0000 7f3de218d640 1 -- 172.21.15.46:0/2431901849 >> 172.21.15.46:0/2431901849 conn(0x7f3ddc0a6340 msgr2=0x7f3ddc06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:40.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.165+0000 7f3de218d640 1 -- 172.21.15.46:0/2431901849 shutdown_connections 2024-09-16T04:40:40.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.165+0000 7f3de218d640 1 -- 172.21.15.46:0/2431901849 wait complete. 2024-09-16T04:40:40.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.165+0000 7f3de218d640 10 monclient: build_initial_monmap 2024-09-16T04:40:40.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.165+0000 7f3de218d640 10 monclient: monmap: 2024-09-16T04:40:40.181 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:40.181 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:40:40.181 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:40:40.181 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:40:40.181 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:40.181 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:40.181 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:40:40.182 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:40:40.182 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:40:40.182 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:40.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3de218d640 1 Processor -- start 2024-09-16T04:40:40.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3de218d640 1 -- start start 2024-09-16T04:40:40.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3de218d640 10 monclient: init 2024-09-16T04:40:40.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3de218d640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:40.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3de218d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:40:40.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3de218d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ddc0a3c10 0x7f3ddc1cc030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:40.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3de218d640 10 monclient(hunting): picked mon.noname-a con 0x7f3ddc0a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:40:40.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3de218d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3ddc0a3ff0 0x7f3ddc1cc570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:40.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3de218d640 10 monclient(hunting): picked mon.noname-c con 0x7f3ddc0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:40.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3de218d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ddc1369d0 0x7f3ddc123160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:40.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3de218d640 10 monclient(hunting): picked mon.noname-b con 0x7f3ddc1369d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:40:40.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3de218d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:40.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3de218d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3ddc0a4910 con 0x7f3ddc0a3c10 2024-09-16T04:40:40.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3de218d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:40.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3de218d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3ddc0a69c0 con 0x7f3ddc1369d0 2024-09-16T04:40:40.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3de218d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:40.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3de218d640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f3ddc127b20 con 0x7f3ddc0a3ff0 2024-09-16T04:40:40.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3de218d640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:40.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3de218d640 10 monclient(hunting): authenticate will time out at 838.581773s 2024-09-16T04:40:40.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3ddb7fe640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ddc0a3c10 msgr2=0x7f3ddc1cc030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:40:40.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3ddb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ddc0a3c10 0x7f3ddc1cc030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:40.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3ddaffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3ddc0a3ff0 0x7f3ddc1cc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:40.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3ddaffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3ddc0a3ff0 0x7f3ddc1cc570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:37796/0 (socket says 172.21.15.46:37796) 2024-09-16T04:40:40.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3ddaffd640 1 -- 172.21.15.46:0/3921868241 learned_addr learned my addr 172.21.15.46:0/3921868241 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:40.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3ddaffd640 10 monclient(hunting): get_auth_request con 0x7f3ddc0a3ff0 auth_method 0 2024-09-16T04:40:40.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3ddaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:40.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3ddaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:40.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3ddaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:40.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3ddaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:40.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3ddaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:40.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3ddaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:40.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.166+0000 7f3ddbfff640 1 --2- 172.21.15.46:0/3921868241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ddc1369d0 0x7f3ddc123160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:40.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3ddbfff640 10 monclient(hunting): get_auth_request con 0x7f3ddc1369d0 auth_method 0 2024-09-16T04:40:40.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3ddbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:40.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3ddbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:40.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3ddbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:40.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3ddaffd640 10 monclient(hunting): handle_auth_done global_id 24176 payload 824 2024-09-16T04:40:40.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3ddaffd640 1 -- 172.21.15.46:0/3921868241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ddc1369d0 msgr2=0x7f3ddc123160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:40.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3ddaffd640 1 --2- 172.21.15.46:0/3921868241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ddc1369d0 0x7f3ddc123160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:40.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3ddaffd640 1 -- 172.21.15.46:0/3921868241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ddc0a3c10 msgr2=0x7f3ddc1cc030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:40:40.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3ddaffd640 1 --2- 172.21.15.46:0/3921868241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ddc0a3c10 0x7f3ddc1cc030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:40.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3ddaffd640 10 monclient: _finish_hunting 0 2024-09-16T04:40:40.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3ddaffd640 1 monclient: found mon.noname-c 2024-09-16T04:40:40.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3ddaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:40.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3ddaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:40:40.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3ddaffd640 1 -- 172.21.15.46:0/3921868241 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ddc125960 con 0x7f3ddc0a3ff0 2024-09-16T04:40:40.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3ddaffd640 1 --2- 172.21.15.46:0/3921868241 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3ddc0a3ff0 0x7f3ddc1cc570 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0000c00 tx=0x7f3dd00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:40.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3dc3fff640 1 -- 172.21.15.46:0/3921868241 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0041070 con 0x7f3ddc0a3ff0 2024-09-16T04:40:40.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3dc3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:40.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3dc3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:40:40.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3dc3fff640 10 monclient: dump: 2024-09-16T04:40:40.186 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:40.186 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:40.186 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:40.186 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:40.186 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:40.186 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:40.186 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:40.187 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:40.187 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:40.187 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:40.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3dc3fff640 10 monclient: _finish_auth 0 2024-09-16T04:40:40.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3dc3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:40:40.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3dc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:40.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3dc3fff640 1 -- 172.21.15.46:0/3921868241 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dd00308d0 con 0x7f3ddc0a3ff0 2024-09-16T04:40:40.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3dc3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:40.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3dc3fff640 1 -- 172.21.15.46:0/3921868241 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0037040 con 0x7f3ddc0a3ff0 2024-09-16T04:40:40.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3dc3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:40.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3dc3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:40.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3dc3fff640 10 monclient: dump: 2024-09-16T04:40:40.187 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:40.187 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:40.188 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:40.188 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:40.188 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:40.188 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:40.188 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:40.188 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:40.188 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:40.188 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:40.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3de218d640 5 monclient: authenticate success, global_id 24176 2024-09-16T04:40:40.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3de218d640 10 monclient: _renew_subs 2024-09-16T04:40:40.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3de218d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:40:40.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3de218d640 1 -- 172.21.15.46:0/3921868241 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ddc1270f0 con 0x7f3ddc0a3ff0 2024-09-16T04:40:40.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3dd9ffb640 4 set_mon_vals no callback set 2024-09-16T04:40:40.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3de218d640 10 monclient: _renew_subs 2024-09-16T04:40:40.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3de218d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:40:40.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.167+0000 7f3de218d640 1 -- 172.21.15.46:0/3921868241 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ddc0a0400 con 0x7f3ddc0a3ff0 2024-09-16T04:40:40.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.168+0000 7f3dc3fff640 1 -- 172.21.15.46:0/3921868241 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f3dd0044070 con 0x7f3ddc0a3ff0 2024-09-16T04:40:40.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.168+0000 7f3de218d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:40.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.168+0000 7f3de218d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:40.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.168+0000 7f3de218d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:40:40.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.168+0000 7f3de218d640 1 -- 172.21.15.46:0/3921868241 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ddc09bd70 con 0x7f3ddc0a3ff0 2024-09-16T04:40:40.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.168+0000 7f3dc3fff640 1 --2- 172.21.15.46:0/3921868241 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3dc8038360 0x7f3dc803a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:40.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.168+0000 7f3dc3fff640 1 -- 172.21.15.46:0/3921868241 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(79..79 src has 1..79) ==== 4738+0+0 (secure 0 0 0) 0x7f3dd006c050 con 0x7f3ddc0a3ff0 2024-09-16T04:40:40.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.169+0000 7f3ddb7fe640 1 --2- 172.21.15.46:0/3921868241 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3dc8038360 0x7f3dc803a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:40.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.169+0000 7f3ddb7fe640 10 monclient: get_auth_request con 0x7f3dc8038360 auth_method 0 2024-09-16T04:40:40.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.169+0000 7f3ddb7fe640 1 --2- 172.21.15.46:0/3921868241 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3dc8038360 0x7f3dc803a820 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc006fd0 tx=0x7f3dcc006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:40.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.173+0000 7f3dc3fff640 1 -- 172.21.15.46:0/3921868241 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3dd004b070 con 0x7f3ddc0a3ff0 2024-09-16T04:40:40.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.173+0000 7f3dc3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:40.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.173+0000 7f3dc3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:40:40.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.417+0000 7f3de218d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:40:40.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.417+0000 7f3de218d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:40:40.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.417+0000 7f3de218d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:40:40.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.417+0000 7f3de218d640 1 -- 172.21.15.46:0/3921868241 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3ddc067e20 con 0x7f3ddc0a3ff0 2024-09-16T04:40:40.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.418+0000 7f3dc3fff640 1 -- 172.21.15.46:0/3921868241 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f3dd0045e90 con 0x7f3ddc0a3ff0 2024-09-16T04:40:40.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.418+0000 7f3dc3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:40:40.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.418+0000 7f3dc3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:40:40.420 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:40:40.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.420+0000 7f3de218d640 1 -- 172.21.15.46:0/3921868241 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3dc8038360 msgr2=0x7f3dc803a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:40.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.420+0000 7f3de218d640 1 --2- 172.21.15.46:0/3921868241 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3dc8038360 0x7f3dc803a820 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc006fd0 tx=0x7f3dcc006d80 comp rx=0 tx=0).stop 2024-09-16T04:40:40.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.420+0000 7f3de218d640 10 monclient: shutdown 2024-09-16T04:40:40.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.420+0000 7f3de218d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:40.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.420+0000 7f3de218d640 1 -- 172.21.15.46:0/3921868241 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3ddc0a3ff0 msgr2=0x7f3ddc1cc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:40.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.420+0000 7f3de218d640 1 --2- 172.21.15.46:0/3921868241 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3ddc0a3ff0 0x7f3ddc1cc570 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0000c00 tx=0x7f3dd00046c0 comp rx=0 tx=0).stop 2024-09-16T04:40:40.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.420+0000 7f3de218d640 1 -- 172.21.15.46:0/3921868241 shutdown_connections 2024-09-16T04:40:40.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.420+0000 7f3de218d640 1 --2- 172.21.15.46:0/3921868241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ddc1369d0 0x7f3ddc123160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:40.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.420+0000 7f3de218d640 1 --2- 172.21.15.46:0/3921868241 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3ddc0a3ff0 0x7f3ddc1cc570 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:40.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.420+0000 7f3de218d640 1 --2- 172.21.15.46:0/3921868241 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3dc8038360 0x7f3dc803a820 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:40.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.420+0000 7f3de218d640 1 --2- 172.21.15.46:0/3921868241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ddc0a3c10 0x7f3ddc1cc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:40.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.420+0000 7f3de218d640 1 -- 172.21.15.46:0/3921868241 >> 172.21.15.46:0/3921868241 conn(0x7f3ddc0a6340 msgr2=0x7f3ddc0a0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:40.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.420+0000 7f3de218d640 1 -- 172.21.15.46:0/3921868241 shutdown_connections 2024-09-16T04:40:40.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:40.420+0000 7f3de218d640 1 -- 172.21.15.46:0/3921868241 wait complete. 2024-09-16T04:40:43.433 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:40:43.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.545+0000 7f5427ed0640 10 monclient: get_monmap_and_config 2024-09-16T04:40:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.545+0000 7f5427ed0640 10 monclient: build_initial_monmap 2024-09-16T04:40:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.545+0000 7f5427ed0640 10 monclient: monmap: 2024-09-16T04:40:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:40:43.546076+0000 2024-09-16T04:40:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:40:43.546076+0000 2024-09-16T04:40:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:40:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:40:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:40:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:40:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:40:43.548 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:43.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.546+0000 7f5427ed0640 1 Processor -- start 2024-09-16T04:40:43.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.546+0000 7f5427ed0640 1 -- start start 2024-09-16T04:40:43.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.546+0000 7f5427ed0640 10 monclient: init 2024-09-16T04:40:43.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.546+0000 7f5427ed0640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:43.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.546+0000 7f5427ed0640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-16T04:40:43.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.546+0000 7f5427ed0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54200a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:40:43.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.546+0000 7f5427ed0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54200a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:40:43.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.546+0000 7f5427ed0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f542013cea0 0x7f542013f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:43.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.547+0000 7f5427ed0640 10 monclient(hunting): picked mon.noname-a con 0x7f542013cea0 addr v2:172.21.15.46:3300/0 2024-09-16T04:40:43.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.547+0000 7f5427ed0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f542009b940 con 0x7f54200a3c10 2024-09-16T04:40:43.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.547+0000 7f5427ed0640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f542009ab00 con 0x7f54200a3ff0 2024-09-16T04:40:43.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.547+0000 7f5427ed0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:43.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.547+0000 7f5427ed0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f542009b030 con 0x7f542013cea0 2024-09-16T04:40:43.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.547+0000 7f5427ed0640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:43.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.547+0000 7f5427ed0640 10 monclient(hunting): authenticate will time out at 841.962066s 2024-09-16T04:40:43.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.547+0000 7f5425c6c640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f54200a3c10 legacy=0x7f542009bcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:40:43.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.547+0000 7f542646d640 1 -- >> v2:172.21.15.46:3300/0 conn(0x7f542013cea0 msgr2=0x7f542013f290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:40:43.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.547+0000 7f542646d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f542013cea0 0x7f542013f290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:43.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.547+0000 7f542546b640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f54200a3ff0 0x7f54200a43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.46:48262/0 (socket says 172.21.15.46:48262) 2024-09-16T04:40:43.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.547+0000 7f542546b640 1 -- 172.21.15.46:0/467268803 learned_addr learned my addr 172.21.15.46:0/467268803 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:40:43.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.748+0000 7f5425c6c640 1 -- 172.21.15.46:0/467268803 >> v1:172.21.15.46:6789/0 conn(0x7f54200a3c10 legacy=0x7f542009bcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:40:43.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.748+0000 7f542646d640 1 -- 172.21.15.46:0/467268803 >> v2:172.21.15.46:3300/0 conn(0x7f542013cea0 msgr2=0x7f542013f290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:40:43.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:43.748+0000 7f542646d640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.46:3300/0 conn(0x7f542013cea0 0x7f542013f290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:40:44.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:44.149+0000 7f5425c6c640 1 -- 172.21.15.46:0/467268803 >> v1:172.21.15.46:6789/0 conn(0x7f54200a3c10 legacy=0x7f542009bcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:40:44.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:44.149+0000 7f542646d640 1 -- 172.21.15.46:0/467268803 >> v2:172.21.15.46:3300/0 conn(0x7f542013cea0 msgr2=0x7f542013f290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:40:44.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:44.149+0000 7f542646d640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.46:3300/0 conn(0x7f542013cea0 0x7f542013f290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:40:44.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:44.950+0000 7f5425c6c640 1 -- 172.21.15.46:0/467268803 >> v1:172.21.15.46:6789/0 conn(0x7f54200a3c10 legacy=0x7f542009bcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:40:44.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:44.950+0000 7f542646d640 1 -- 172.21.15.46:0/467268803 >> v2:172.21.15.46:3300/0 conn(0x7f542013cea0 msgr2=0x7f542013f290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:40:44.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:44.950+0000 7f542646d640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.46:3300/0 conn(0x7f542013cea0 0x7f542013f290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:40:46.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f54177fe640 10 monclient(hunting): tick 2024-09-16T04:40:46.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f54177fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:40:46.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f54177fe640 1 monclient(hunting): continuing hunt 2024-09-16T04:40:46.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f54177fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:40:46.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f54177fe640 1 -- 172.21.15.46:0/467268803 >> v2:172.21.15.46:3300/0 conn(0x7f542013cea0 msgr2=0x7f542013f290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:40:46.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f54177fe640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.46:3300/0 conn(0x7f542013cea0 0x7f542013f290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:46.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f54177fe640 1 -- 172.21.15.46:0/467268803 >> v1:172.21.15.114:6789/0 conn(0x7f54200a3ff0 legacy=0x7f54200a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:46.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f54177fe640 1 -- 172.21.15.46:0/467268803 >> v1:172.21.15.46:6789/0 conn(0x7f54200a3c10 legacy=0x7f542009bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:40:46.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f54177fe640 10 monclient: _add_conns ranks=[4,2] 2024-09-16T04:40:46.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f54177fe640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 0x7f5404005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:46.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f54177fe640 10 monclient(hunting): picked mon.noname-c con 0x7f54040036c0 addr v2:172.21.15.114:3300/0 2024-09-16T04:40:46.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f54177fe640 1 --2- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 0x7f54040085d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:46.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f54177fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5404006110 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:46.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f54177fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:46.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f54177fe640 1 -- 172.21.15.46:0/467268803 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f542009b030 con 0x7f54040036c0 2024-09-16T04:40:46.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f54177fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:46.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f54177fe640 1 -- 172.21.15.46:0/467268803 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f5404008d00 con 0x7f5404006110 2024-09-16T04:40:46.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f54177fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:46.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f542546b640 1 --2- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 0x7f54040085d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:46.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f542546b640 10 monclient(hunting): get_auth_request con 0x7f5404006110 auth_method 0 2024-09-16T04:40:46.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f542546b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:46.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f542546b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:46.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f542546b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:46.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f542546b640 1 -- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 msgr2=0x7f54040085d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:40:46.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f542546b640 1 -- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 msgr2=0x7f54040085d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:40:46.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f542546b640 1 --2- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 0x7f54040085d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:46.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f5425c6c640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 0x7f5404005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:46.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f542546b640 1 --2- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 0x7f54040085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:46.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f5425c6c640 10 monclient(hunting): get_auth_request con 0x7f54040036c0 auth_method 0 2024-09-16T04:40:46.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f5425c6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:46.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f5425c6c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:46.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f5425c6c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:46.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f5425c6c640 1 -- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 msgr2=0x7f5404005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:40:46.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f5425c6c640 1 -- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 msgr2=0x7f5404005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:40:46.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f5425c6c640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 0x7f5404005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:46.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.547+0000 7f5425c6c640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 0x7f5404005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:46.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.748+0000 7f542546b640 1 --2- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 0x7f54040085d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:46.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.748+0000 7f542546b640 10 monclient(hunting): get_auth_request con 0x7f5404006110 auth_method 0 2024-09-16T04:40:46.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.748+0000 7f542546b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.748+0000 7f542546b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.748+0000 7f542546b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.748+0000 7f542546b640 1 -- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 msgr2=0x7f54040085d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:40:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.748+0000 7f542546b640 1 -- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 msgr2=0x7f54040085d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:40:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.748+0000 7f542546b640 1 --2- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 0x7f54040085d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.748+0000 7f542546b640 1 --2- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 0x7f54040085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:40:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.748+0000 7f5425c6c640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 0x7f5404005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.748+0000 7f5425c6c640 10 monclient(hunting): get_auth_request con 0x7f54040036c0 auth_method 0 2024-09-16T04:40:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.748+0000 7f5425c6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.748+0000 7f5425c6c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.748+0000 7f5425c6c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.748+0000 7f5425c6c640 1 -- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 msgr2=0x7f5404005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:40:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.748+0000 7f5425c6c640 1 -- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 msgr2=0x7f5404005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:40:46.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.748+0000 7f5425c6c640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 0x7f5404005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:46.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:46.748+0000 7f5425c6c640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 0x7f5404005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:40:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.149+0000 7f542546b640 1 --2- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 0x7f54040085d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.149+0000 7f542546b640 10 monclient(hunting): get_auth_request con 0x7f5404006110 auth_method 0 2024-09-16T04:40:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.149+0000 7f542546b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.149+0000 7f542546b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.149+0000 7f542546b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.149+0000 7f542546b640 1 -- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 msgr2=0x7f54040085d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:40:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.149+0000 7f542546b640 1 -- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 msgr2=0x7f54040085d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:40:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.149+0000 7f542546b640 1 --2- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 0x7f54040085d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.149+0000 7f542546b640 1 --2- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 0x7f54040085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:40:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.149+0000 7f5425c6c640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 0x7f5404005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:47.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.149+0000 7f5425c6c640 10 monclient(hunting): get_auth_request con 0x7f54040036c0 auth_method 0 2024-09-16T04:40:47.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.149+0000 7f5425c6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:47.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.149+0000 7f5425c6c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:47.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.149+0000 7f5425c6c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:47.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.149+0000 7f5425c6c640 1 -- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 msgr2=0x7f5404005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:40:47.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.149+0000 7f5425c6c640 1 -- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 msgr2=0x7f5404005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:40:47.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.149+0000 7f5425c6c640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 0x7f5404005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:47.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.149+0000 7f5425c6c640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 0x7f5404005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:40:47.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.950+0000 7f542546b640 1 --2- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 0x7f54040085d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:47.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.950+0000 7f542546b640 10 monclient(hunting): get_auth_request con 0x7f5404006110 auth_method 0 2024-09-16T04:40:47.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.950+0000 7f542546b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:47.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.950+0000 7f542546b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:47.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.950+0000 7f542546b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:47.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.950+0000 7f542546b640 1 -- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 msgr2=0x7f54040085d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:40:47.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.950+0000 7f542546b640 1 -- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 msgr2=0x7f54040085d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:40:47.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.950+0000 7f542546b640 1 --2- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 0x7f54040085d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:47.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.950+0000 7f542546b640 1 --2- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 0x7f54040085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:40:47.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.950+0000 7f5425c6c640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 0x7f5404005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:47.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.950+0000 7f5425c6c640 10 monclient(hunting): get_auth_request con 0x7f54040036c0 auth_method 0 2024-09-16T04:40:47.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.950+0000 7f5425c6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:47.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.950+0000 7f5425c6c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:47.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.950+0000 7f5425c6c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:40:47.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.950+0000 7f5425c6c640 1 -- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 msgr2=0x7f5404005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:40:47.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.950+0000 7f5425c6c640 1 -- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 msgr2=0x7f5404005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:40:47.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.950+0000 7f5425c6c640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 0x7f5404005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:40:47.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:47.950+0000 7f5425c6c640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 0x7f5404005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:40:48.167 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:48.167 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36295]: starting. 2024-09-16T04:40:48.168 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36295]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-36259 2024-09-16T04:40:48.168 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36295]: shutting down. 2024-09-16T04:40:48.169 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:48.169 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:48.169 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:40:48.172 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:40:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f54177fe640 10 monclient(hunting): tick 2024-09-16T04:40:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f54177fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:40:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f54177fe640 1 monclient(hunting): continuing hunt 2024-09-16T04:40:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f54177fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:40:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f54177fe640 1 -- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 msgr2=0x7f54040085d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:40:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f54177fe640 1 --2- 172.21.15.46:0/467268803 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404006110 0x7f54040085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f54177fe640 1 -- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 msgr2=0x7f5404005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:40:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f54177fe640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f54040036c0 0x7f5404005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f542646d640 1 -- 172.21.15.46:0/467268803 reap_dead start 2024-09-16T04:40:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f54177fe640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-16T04:40:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f54177fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5404003b10 addr v1:172.21.15.46:6789/0 2024-09-16T04:40:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f54177fe640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.46:3300/0 conn(0x7f5404007ce0 0x7f54040080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f54177fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5404007ce0 addr v2:172.21.15.46:3300/0 2024-09-16T04:40:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f54177fe640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f5404008660 0x7f540400d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f54177fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5404008660 addr v2:172.21.15.114:3300/0 2024-09-16T04:40:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f54177fe640 1 -- 172.21.15.46:0/467268803 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f542009b310 con 0x7f5404003b10 2024-09-16T04:40:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f54177fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f54177fe640 1 -- 172.21.15.46:0/467268803 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f542009b030 con 0x7f5404007ce0 2024-09-16T04:40:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f54177fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f54177fe640 1 -- 172.21.15.46:0/467268803 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f5404008d00 con 0x7f5404008660 2024-09-16T04:40:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f54177fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f5425c6c640 1 -- 172.21.15.46:0/467268803 >> v1:172.21.15.46:6789/0 conn(0x7f5404003b10 legacy=0x7f54040065c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:40:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f542546b640 1 -- 172.21.15.46:0/467268803 >> v2:172.21.15.46:3300/0 conn(0x7f5404007ce0 msgr2=0x7f54040080c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:40:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.546+0000 7f542546b640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.46:3300/0 conn(0x7f5404007ce0 0x7f54040080c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.547+0000 7f542646d640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.114:3300/0 conn(0x7f5404008660 0x7f540400d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.547+0000 7f542646d640 10 monclient(hunting): get_auth_request con 0x7f5404008660 auth_method 0 2024-09-16T04:40:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.547+0000 7f542646d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.547+0000 7f542646d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.547+0000 7f542646d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.547+0000 7f542646d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.547+0000 7f542646d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.547+0000 7f542646d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.547+0000 7f542646d640 10 monclient(hunting): handle_auth_done global_id 4990 payload 274 2024-09-16T04:40:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.547+0000 7f542646d640 1 -- 172.21.15.46:0/467268803 >> v2:172.21.15.46:3300/0 conn(0x7f5404007ce0 msgr2=0x7f54040080c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:40:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.547+0000 7f542646d640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.46:3300/0 conn(0x7f5404007ce0 0x7f54040080c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.547+0000 7f542646d640 1 -- 172.21.15.46:0/467268803 >> v1:172.21.15.46:6789/0 conn(0x7f5404003b10 legacy=0x7f54040065c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:40:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.547+0000 7f542646d640 10 monclient: _finish_hunting 0 2024-09-16T04:40:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.547+0000 7f542646d640 1 monclient: found mon.noname-c 2024-09-16T04:40:49.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.547+0000 7f542646d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:49.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.547+0000 7f542646d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:40:49.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.547+0000 7f542646d640 1 -- 172.21.15.46:0/467268803 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542009b940 con 0x7f5404008660 2024-09-16T04:40:49.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.548+0000 7f542646d640 1 --2- 172.21.15.46:0/467268803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5404008660 0x7f540400d140 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f542009bc40 tx=0x7f541802a870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0d32fc3ae80a22c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:49.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.548+0000 7f5424c6a640 1 -- 172.21.15.46:0/467268803 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418005480 con 0x7f5404008660 2024-09-16T04:40:49.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.548+0000 7f5424c6a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:49.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.548+0000 7f5424c6a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:40:49.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.548+0000 7f5424c6a640 10 monclient: dump: 2024-09-16T04:40:49.554 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:49.554 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:49.554 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:49.554 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:49.554 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:49.555 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.548+0000 7f5424c6a640 10 monclient: _finish_auth 0 2024-09-16T04:40:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.548+0000 7f5424c6a640 10 monclient: _check_auth_tickets 2024-09-16T04:40:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.548+0000 7f5424c6a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.548+0000 7f5424c6a640 1 -- 172.21.15.46:0/467268803 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5418005620 con 0x7f5404008660 2024-09-16T04:40:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.548+0000 7f5424c6a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.548+0000 7f5424c6a640 1 -- 172.21.15.46:0/467268803 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54180059f0 con 0x7f5404008660 2024-09-16T04:40:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.548+0000 7f5427ed0640 5 monclient: authenticate success, global_id 4990 2024-09-16T04:40:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.548+0000 7f5427ed0640 10 monclient: get_monmap_and_config success 2024-09-16T04:40:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.548+0000 7f5424c6a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.548+0000 7f5427ed0640 4 set_mon_vals no callback set 2024-09-16T04:40:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.548+0000 7f5424c6a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:40:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.548+0000 7f5424c6a640 10 monclient: dump: 2024-09-16T04:40:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:49.556 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:49.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.549+0000 7f5427ed0640 10 monclient: shutdown 2024-09-16T04:40:49.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.549+0000 7f5427ed0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:49.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.549+0000 7f5427ed0640 1 -- 172.21.15.46:0/467268803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5404008660 msgr2=0x7f540400d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:49.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.549+0000 7f5427ed0640 1 --2- 172.21.15.46:0/467268803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5404008660 0x7f540400d140 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f542009bc40 tx=0x7f541802a870 comp rx=0 tx=0).stop 2024-09-16T04:40:49.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.549+0000 7f5427ed0640 1 -- 172.21.15.46:0/467268803 shutdown_connections 2024-09-16T04:40:49.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.549+0000 7f5427ed0640 1 --2- 172.21.15.46:0/467268803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5404008660 0x7f540400d140 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:49.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.549+0000 7f5427ed0640 1 --2- 172.21.15.46:0/467268803 >> v2:172.21.15.46:3300/0 conn(0x7f5404007ce0 0x7f54040080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:49.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.549+0000 7f5427ed0640 1 -- 172.21.15.46:0/467268803 >> 172.21.15.46:0/467268803 conn(0x7f54200a6340 msgr2=0x7f542006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:49.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.549+0000 7f5427ed0640 1 -- 172.21.15.46:0/467268803 shutdown_connections 2024-09-16T04:40:49.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.549+0000 7f5427ed0640 1 -- 172.21.15.46:0/467268803 wait complete. 2024-09-16T04:40:49.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.550+0000 7f5427ed0640 10 monclient: build_initial_monmap 2024-09-16T04:40:49.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.550+0000 7f5427ed0640 10 monclient: monmap: 2024-09-16T04:40:49.562 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:49.562 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:40:49.562 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:40:49.562 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:40:49.562 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:49.562 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:49.563 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:40:49.563 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:40:49.563 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:40:49.563 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:49.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.550+0000 7f5427ed0640 1 Processor -- start 2024-09-16T04:40:49.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5427ed0640 1 -- start start 2024-09-16T04:40:49.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5427ed0640 10 monclient: init 2024-09-16T04:40:49.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5427ed0640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:49.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5427ed0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:40:49.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5427ed0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5404003b10 0x7f542011f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:49.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5427ed0640 10 monclient(hunting): picked mon.noname-b con 0x7f5404003b10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:40:49.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5427ed0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5404007ce0 0x7f542011fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:49.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5427ed0640 10 monclient(hunting): picked mon.noname-a con 0x7f5404007ce0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:40:49.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5427ed0640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404008660 0x7f54201202e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:49.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5427ed0640 10 monclient(hunting): picked mon.noname-c con 0x7f5404008660 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:49.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5427ed0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:49.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5427ed0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f542009be70 con 0x7f5404007ce0 2024-09-16T04:40:49.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5427ed0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:49.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5427ed0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f542009bcf0 con 0x7f5404003b10 2024-09-16T04:40:49.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5427ed0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:49.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5427ed0640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f542009bff0 con 0x7f5404008660 2024-09-16T04:40:49.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5427ed0640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:49.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5427ed0640 10 monclient(hunting): authenticate will time out at 847.967608s 2024-09-16T04:40:49.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f542546b640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5404007ce0 msgr2=0x7f542011fda0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:40:49.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f542546b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5404007ce0 0x7f542011fda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:49.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f542646d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404008660 0x7f54201202e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:49.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f542646d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404008660 0x7f54201202e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39538/0 (socket says 172.21.15.46:39538) 2024-09-16T04:40:49.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f542646d640 1 -- 172.21.15.46:0/480756371 learned_addr learned my addr 172.21.15.46:0/480756371 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:49.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f542646d640 10 monclient(hunting): get_auth_request con 0x7f5404008660 auth_method 0 2024-09-16T04:40:49.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f542646d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:49.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f542646d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:49.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f542646d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:49.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5425c6c640 1 --2- 172.21.15.46:0/480756371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5404003b10 0x7f542011f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:49.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5425c6c640 10 monclient(hunting): get_auth_request con 0x7f5404003b10 auth_method 0 2024-09-16T04:40:49.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5425c6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:49.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5425c6c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:49.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.551+0000 7f5425c6c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:49.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f542646d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:49.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f542646d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:49.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f542646d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:49.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5425c6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:49.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5425c6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:49.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5425c6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:49.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5425c6c640 10 monclient(hunting): handle_auth_done global_id 4993 payload 824 2024-09-16T04:40:49.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5425c6c640 1 -- 172.21.15.46:0/480756371 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404008660 msgr2=0x7f54201202e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:49.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5425c6c640 1 --2- 172.21.15.46:0/480756371 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404008660 0x7f54201202e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:49.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5425c6c640 1 -- 172.21.15.46:0/480756371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5404007ce0 msgr2=0x7f542011fda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:40:49.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5425c6c640 1 --2- 172.21.15.46:0/480756371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5404007ce0 0x7f542011fda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:49.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5425c6c640 10 monclient: _finish_hunting 0 2024-09-16T04:40:49.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5425c6c640 1 monclient: found mon.noname-b 2024-09-16T04:40:49.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5425c6c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:49.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5425c6c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3300/0 2024-09-16T04:40:49.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5425c6c640 1 -- 172.21.15.46:0/480756371 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54200a0d30 con 0x7f5404003b10 2024-09-16T04:40:49.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f542646d640 1 --2- 172.21.15.46:0/480756371 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404008660 0x7f54201202e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:40:49.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5425c6c640 1 --2- 172.21.15.46:0/480756371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5404003b10 0x7f542011f860 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f541c003980 tx=0x7f541c00bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:49.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f54167fc640 1 -- 172.21.15.46:0/480756371 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c01b030 con 0x7f5404003b10 2024-09-16T04:40:49.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f54167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:49.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f54167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:40:49.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f54167fc640 10 monclient: dump: 2024-09-16T04:40:49.567 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:49.567 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:49.567 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:49.567 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:49.567 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:49.567 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:49.567 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:49.567 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:49.567 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:49.567 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:49.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f54167fc640 10 monclient: _finish_auth 0 2024-09-16T04:40:49.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f54167fc640 10 monclient: _check_auth_tickets 2024-09-16T04:40:49.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f54167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:49.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f54167fc640 1 -- 172.21.15.46:0/480756371 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f541c006e50 con 0x7f5404003b10 2024-09-16T04:40:49.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f54167fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:49.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f54167fc640 1 -- 172.21.15.46:0/480756371 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c005a80 con 0x7f5404003b10 2024-09-16T04:40:49.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f54167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:49.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f54167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:40:49.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f54167fc640 10 monclient: dump: 2024-09-16T04:40:49.568 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:49.568 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:49.568 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:49.568 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:49.569 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:49.569 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:49.569 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:49.569 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:49.569 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:49.569 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:49.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5427ed0640 5 monclient: authenticate success, global_id 4993 2024-09-16T04:40:49.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5427ed0640 10 monclient: _renew_subs 2024-09-16T04:40:49.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5427ed0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:40:49.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5427ed0640 1 -- 172.21.15.46:0/480756371 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54200a29d0 con 0x7f5404003b10 2024-09-16T04:40:49.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5417fff640 4 set_mon_vals no callback set 2024-09-16T04:40:49.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5427ed0640 10 monclient: _renew_subs 2024-09-16T04:40:49.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5427ed0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:40:49.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.552+0000 7f5427ed0640 1 -- 172.21.15.46:0/480756371 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5420122a10 con 0x7f5404003b10 2024-09-16T04:40:49.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.553+0000 7f54167fc640 1 -- 172.21.15.46:0/480756371 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f541c01e070 con 0x7f5404003b10 2024-09-16T04:40:49.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.554+0000 7f5427ed0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:49.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.554+0000 7f5427ed0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:49.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.554+0000 7f5427ed0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:40:49.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.554+0000 7f5427ed0640 1 -- 172.21.15.46:0/480756371 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54200a43d0 con 0x7f5404003b10 2024-09-16T04:40:49.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.554+0000 7f54167fc640 1 --2- 172.21.15.46:0/480756371 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5404039e80 0x7f540403c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:49.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.554+0000 7f54167fc640 1 -- 172.21.15.46:0/480756371 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 4738+0+0 (secure 0 0 0) 0x7f541c04bc30 con 0x7f5404003b10 2024-09-16T04:40:49.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.554+0000 7f542546b640 1 --2- 172.21.15.46:0/480756371 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5404039e80 0x7f540403c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:49.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.554+0000 7f542546b640 10 monclient: get_auth_request con 0x7f5404039e80 auth_method 0 2024-09-16T04:40:49.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.554+0000 7f542546b640 1 --2- 172.21.15.46:0/480756371 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5404039e80 0x7f540403c340 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5410007fd0 tx=0x7f5410009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:49.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.559+0000 7f54167fc640 1 -- 172.21.15.46:0/480756371 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f541c025070 con 0x7f5404003b10 2024-09-16T04:40:49.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.559+0000 7f54167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:49.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.559+0000 7f54167fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:40:49.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.811+0000 7f5427ed0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:40:49.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.811+0000 7f5427ed0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:40:49.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.811+0000 7f5427ed0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:40:49.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.811+0000 7f5427ed0640 1 -- 172.21.15.46:0/480756371 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5420121510 con 0x7f5404003b10 2024-09-16T04:40:49.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.812+0000 7f54167fc640 1 -- 172.21.15.46:0/480756371 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f54200a0d30 con 0x7f5404003b10 2024-09-16T04:40:49.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.812+0000 7f54167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:40:49.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.812+0000 7f54167fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:40:49.814 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":22,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:40:49.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.814+0000 7f5427ed0640 1 -- 172.21.15.46:0/480756371 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5404039e80 msgr2=0x7f540403c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:49.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.814+0000 7f5427ed0640 1 --2- 172.21.15.46:0/480756371 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5404039e80 0x7f540403c340 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5410007fd0 tx=0x7f5410009040 comp rx=0 tx=0).stop 2024-09-16T04:40:49.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.814+0000 7f5427ed0640 10 monclient: shutdown 2024-09-16T04:40:49.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.814+0000 7f5427ed0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:49.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.814+0000 7f5427ed0640 1 -- 172.21.15.46:0/480756371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5404003b10 msgr2=0x7f542011f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:49.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.814+0000 7f5427ed0640 1 --2- 172.21.15.46:0/480756371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5404003b10 0x7f542011f860 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f541c003980 tx=0x7f541c00bf40 comp rx=0 tx=0).stop 2024-09-16T04:40:49.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.814+0000 7f5427ed0640 1 -- 172.21.15.46:0/480756371 shutdown_connections 2024-09-16T04:40:49.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.814+0000 7f5427ed0640 1 --2- 172.21.15.46:0/480756371 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5404008660 0x7f54201202e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:49.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.814+0000 7f5427ed0640 1 --2- 172.21.15.46:0/480756371 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5404039e80 0x7f540403c340 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:49.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.814+0000 7f5427ed0640 1 --2- 172.21.15.46:0/480756371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5404007ce0 0x7f542011fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:49.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.814+0000 7f5427ed0640 1 --2- 172.21.15.46:0/480756371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5404003b10 0x7f542011f860 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:49.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.814+0000 7f5427ed0640 1 -- 172.21.15.46:0/480756371 >> 172.21.15.46:0/480756371 conn(0x7f54200a6340 msgr2=0x7f54200a13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:49.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.814+0000 7f5427ed0640 1 -- 172.21.15.46:0/480756371 shutdown_connections 2024-09-16T04:40:49.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.814+0000 7f5427ed0640 1 -- 172.21.15.46:0/480756371 wait complete. 2024-09-16T04:40:49.827 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:40:49.828 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T04:40:49.828 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:40:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.928+0000 7f9f78331640 10 monclient: get_monmap_and_config 2024-09-16T04:40:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.928+0000 7f9f78331640 10 monclient: build_initial_monmap 2024-09-16T04:40:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.928+0000 7f9f78331640 10 monclient: monmap: 2024-09-16T04:40:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:40:49.929995+0000 2024-09-16T04:40:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:40:49.929995+0000 2024-09-16T04:40:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:40:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:40:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:40:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:40:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:40:49.931 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:49.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.929+0000 7f9f78331640 1 Processor -- start 2024-09-16T04:40:49.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.929+0000 7f9f78331640 1 -- start start 2024-09-16T04:40:49.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.929+0000 7f9f78331640 10 monclient: init 2024-09-16T04:40:49.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.929+0000 7f9f78331640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:49.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.929+0000 7f9f78331640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-16T04:40:49.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.929+0000 7f9f78331640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9f700a3c10 0x7f9f7009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:49.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.929+0000 7f9f78331640 10 monclient(hunting): picked mon.noname-c con 0x7f9f700a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:40:49.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.929+0000 7f9f78331640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9f700a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:40:49.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.929+0000 7f9f78331640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f7013be90 0x7f9f7013e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:49.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.929+0000 7f9f78331640 10 monclient(hunting): picked mon.noname-b con 0x7f9f7013be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:49.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.929+0000 7f9f78331640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f7009b940 con 0x7f9f700a3ff0 2024-09-16T04:40:49.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.929+0000 7f9f78331640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:49.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.929+0000 7f9f78331640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f9f7009ab00 con 0x7f9f700a3c10 2024-09-16T04:40:49.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.929+0000 7f9f78331640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:49.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.929+0000 7f9f78331640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f9f7009b030 con 0x7f9f7013be90 2024-09-16T04:40:49.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.929+0000 7f9f78331640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:49.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.929+0000 7f9f78331640 10 monclient(hunting): authenticate will time out at 848.345846s 2024-09-16T04:40:49.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.929+0000 7f9f768ce640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f7013be90 0x7f9f7013e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:49.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.929+0000 7f9f768ce640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f7013be90 0x7f9f7013e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39554/0 (socket says 172.21.15.46:39554) 2024-09-16T04:40:49.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.929+0000 7f9f768ce640 1 -- 172.21.15.46:0/4281898550 learned_addr learned my addr 172.21.15.46:0/4281898550 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:49.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f768ce640 10 monclient(hunting): get_auth_request con 0x7f9f7013be90 auth_method 0 2024-09-16T04:40:49.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f768ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:49.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f768ce640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f768ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f768ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f768ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f760cd640 1 --2- 172.21.15.46:0/4281898550 >> v2:172.21.15.114:3300/0 conn(0x7f9f700a3c10 0x7f9f7009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f768ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f760cd640 10 monclient(hunting): get_auth_request con 0x7f9f700a3c10 auth_method 0 2024-09-16T04:40:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f760cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f760cd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f760cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f760cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f760cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f760cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f768ce640 10 monclient(hunting): handle_auth_done global_id 24191 payload 274 2024-09-16T04:40:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f768ce640 1 -- 172.21.15.46:0/4281898550 >> v2:172.21.15.114:3300/0 conn(0x7f9f700a3c10 msgr2=0x7f9f7009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:40:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f768ce640 1 --2- 172.21.15.46:0/4281898550 >> v2:172.21.15.114:3300/0 conn(0x7f9f700a3c10 0x7f9f7009bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f768ce640 1 -- 172.21.15.46:0/4281898550 >> v1:172.21.15.114:6789/0 conn(0x7f9f700a3ff0 legacy=0x7f9f700a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f768ce640 10 monclient: _finish_hunting 0 2024-09-16T04:40:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f768ce640 1 monclient: found mon.noname-b 2024-09-16T04:40:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f768ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f768ce640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:40:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.930+0000 7f9f768ce640 1 -- 172.21.15.46:0/4281898550 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f7009b310 con 0x7f9f7013be90 2024-09-16T04:40:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f768ce640 1 --2- 172.21.15.46:0/4281898550 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f7013be90 0x7f9f7013e280 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9f68009e50 tx=0x7f9f6802a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94d5563330c9a999 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f750cb640 1 -- 172.21.15.46:0/4281898550 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f68036070 con 0x7f9f7013be90 2024-09-16T04:40:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f750cb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f750cb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:40:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f750cb640 10 monclient: dump: 2024-09-16T04:40:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:49.946 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:49.946 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:49.946 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:49.946 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:49.946 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:49.946 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:49.946 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:49.946 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:49.946 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:49.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f750cb640 10 monclient: _finish_auth 0 2024-09-16T04:40:49.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f750cb640 10 monclient: _check_auth_tickets 2024-09-16T04:40:49.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f750cb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:49.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f750cb640 1 -- 172.21.15.46:0/4281898550 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f68031600 con 0x7f9f7013be90 2024-09-16T04:40:49.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f78331640 5 monclient: authenticate success, global_id 24191 2024-09-16T04:40:49.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f78331640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:40:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f750cb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f750cb640 1 -- 172.21.15.46:0/4281898550 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f680319d0 con 0x7f9f7013be90 2024-09-16T04:40:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f78331640 10 monclient: get_monmap_and_config success 2024-09-16T04:40:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f78331640 4 set_mon_vals no callback set 2024-09-16T04:40:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f750cb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f750cb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f750cb640 10 monclient: dump: 2024-09-16T04:40:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:49.948 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f78331640 10 monclient: shutdown 2024-09-16T04:40:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f78331640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f78331640 1 -- 172.21.15.46:0/4281898550 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f7013be90 msgr2=0x7f9f7013e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f78331640 1 --2- 172.21.15.46:0/4281898550 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f7013be90 0x7f9f7013e280 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9f68009e50 tx=0x7f9f6802a5f0 comp rx=0 tx=0).stop 2024-09-16T04:40:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f78331640 1 -- 172.21.15.46:0/4281898550 shutdown_connections 2024-09-16T04:40:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f78331640 1 --2- 172.21.15.46:0/4281898550 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f7013be90 0x7f9f7013e280 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f78331640 1 --2- 172.21.15.46:0/4281898550 >> v2:172.21.15.114:3300/0 conn(0x7f9f700a3c10 0x7f9f7009bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f78331640 1 -- 172.21.15.46:0/4281898550 >> 172.21.15.46:0/4281898550 conn(0x7f9f700a6340 msgr2=0x7f9f7006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f78331640 1 -- 172.21.15.46:0/4281898550 shutdown_connections 2024-09-16T04:40:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.931+0000 7f9f78331640 1 -- 172.21.15.46:0/4281898550 wait complete. 2024-09-16T04:40:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.932+0000 7f9f78331640 10 monclient: build_initial_monmap 2024-09-16T04:40:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.932+0000 7f9f78331640 10 monclient: monmap: 2024-09-16T04:40:49.949 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:49.949 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:40:49.949 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:40:49.949 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:40:49.949 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:49.949 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:49.949 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:40:49.949 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:40:49.949 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:40:49.949 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:49.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.932+0000 7f9f78331640 1 Processor -- start 2024-09-16T04:40:49.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.932+0000 7f9f78331640 1 -- start start 2024-09-16T04:40:49.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.932+0000 7f9f78331640 10 monclient: init 2024-09-16T04:40:49.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f78331640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:49.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f78331640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:40:49.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f78331640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f700a3c10 0x7f9f701cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:49.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f78331640 10 monclient(hunting): picked mon.noname-a con 0x7f9f700a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:40:49.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f78331640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f700a3ff0 0x7f9f701cc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:49.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f78331640 10 monclient(hunting): picked mon.noname-b con 0x7f9f700a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:40:49.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f78331640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f7013be90 0x7f9f701cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:49.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f760cd640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f700a3c10 msgr2=0x7f9f701cbc50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:40:49.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f78331640 10 monclient(hunting): picked mon.noname-c con 0x7f9f7013be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:49.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f78331640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:49.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f78331640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9f700a6b40 con 0x7f9f700a3c10 2024-09-16T04:40:49.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f78331640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:49.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f760cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f700a3c10 0x7f9f701cbc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:49.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f78331640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9f700a69c0 con 0x7f9f700a3ff0 2024-09-16T04:40:49.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f78331640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:49.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f78331640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f9f7013e7c0 con 0x7f9f7013be90 2024-09-16T04:40:49.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f78331640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:49.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f78331640 10 monclient(hunting): authenticate will time out at 848.349280s 2024-09-16T04:40:49.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f768ce640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f7013be90 0x7f9f701cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:49.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f758cc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f700a3ff0 0x7f9f701cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:49.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f768ce640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f7013be90 0x7f9f701cc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39568/0 (socket says 172.21.15.46:39568) 2024-09-16T04:40:49.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f768ce640 1 -- 172.21.15.46:0/2603565432 learned_addr learned my addr 172.21.15.46:0/2603565432 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:49.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f768ce640 10 monclient(hunting): get_auth_request con 0x7f9f7013be90 auth_method 0 2024-09-16T04:40:49.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f768ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:49.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f768ce640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:49.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f768ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:49.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f758cc640 10 monclient(hunting): get_auth_request con 0x7f9f700a3ff0 auth_method 0 2024-09-16T04:40:49.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f758cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:49.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f758cc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:49.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f758cc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:49.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f768ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:49.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f768ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:49.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f768ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:49.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f758cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:49.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f758cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:49.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f758cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:49.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f768ce640 10 monclient(hunting): handle_auth_done global_id 24194 payload 824 2024-09-16T04:40:49.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f768ce640 1 -- 172.21.15.46:0/2603565432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f700a3ff0 msgr2=0x7f9f701cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:49.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f768ce640 1 --2- 172.21.15.46:0/2603565432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f700a3ff0 0x7f9f701cc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:49.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f768ce640 1 -- 172.21.15.46:0/2603565432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f700a3c10 msgr2=0x7f9f701cbc50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:40:49.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f768ce640 1 --2- 172.21.15.46:0/2603565432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f700a3c10 0x7f9f701cbc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:49.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f768ce640 10 monclient: _finish_hunting 0 2024-09-16T04:40:49.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f768ce640 1 monclient: found mon.noname-c 2024-09-16T04:40:49.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f768ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:49.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f768ce640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:40:49.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f768ce640 1 -- 172.21.15.46:0/2603565432 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f700a3520 con 0x7f9f7013be90 2024-09-16T04:40:49.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f758cc640 1 --2- 172.21.15.46:0/2603565432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f700a3ff0 0x7f9f701cc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:40:49.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.933+0000 7f9f768ce640 1 --2- 172.21.15.46:0/2603565432 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f7013be90 0x7f9f701cc6d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9f68009e50 tx=0x7f9f68004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:49.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f66ffd640 1 -- 172.21.15.46:0/2603565432 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f68036050 con 0x7f9f7013be90 2024-09-16T04:40:49.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:49.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f66ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:40:49.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f66ffd640 10 monclient: dump: 2024-09-16T04:40:49.953 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:49.953 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:49.953 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:49.953 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:49.953 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:49.953 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:49.954 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:49.954 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:49.954 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:49.954 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:49.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f66ffd640 10 monclient: _finish_auth 0 2024-09-16T04:40:49.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f66ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:40:49.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:49.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f66ffd640 1 -- 172.21.15.46:0/2603565432 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f68042070 con 0x7f9f7013be90 2024-09-16T04:40:49.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f78331640 5 monclient: authenticate success, global_id 24194 2024-09-16T04:40:49.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f78331640 10 monclient: _renew_subs 2024-09-16T04:40:49.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f78331640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:40:49.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f78331640 1 -- 172.21.15.46:0/2603565432 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f7013c6a0 con 0x7f9f7013be90 2024-09-16T04:40:49.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f66ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:49.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f66ffd640 1 -- 172.21.15.46:0/2603565432 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f68007bf0 con 0x7f9f7013be90 2024-09-16T04:40:49.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:49.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:49.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f66ffd640 10 monclient: dump: 2024-09-16T04:40:49.955 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:49.955 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:49.955 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:49.955 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:49.955 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:49.955 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:49.955 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:49.955 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:49.955 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:49.955 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:49.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f67fff640 4 set_mon_vals no callback set 2024-09-16T04:40:49.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f78331640 10 monclient: _renew_subs 2024-09-16T04:40:49.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f78331640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:40:49.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f78331640 1 -- 172.21.15.46:0/2603565432 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f7013a550 con 0x7f9f7013be90 2024-09-16T04:40:49.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f66ffd640 1 -- 172.21.15.46:0/2603565432 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f9f68047460 con 0x7f9f7013be90 2024-09-16T04:40:49.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.934+0000 7f9f78331640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:49.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f78331640 1 --2- 172.21.15.46:0/2603565432 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f701339a0 0x7f9f70133d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:49.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f78331640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:49.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f78331640 1 -- 172.21.15.46:0/2603565432 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f9f7013b160 con 0x7f9f701339a0 2024-09-16T04:40:49.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f66ffd640 1 --2- 172.21.15.46:0/2603565432 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9f54038360 0x7f9f5403a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:49.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f760cd640 1 --2- 172.21.15.46:0/2603565432 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f701339a0 0x7f9f70133d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:49.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f760cd640 10 monclient: get_auth_request con 0x7f9f701339a0 auth_method 0 2024-09-16T04:40:49.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f760cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:49.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f760cd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:49.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f760cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:49.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f66ffd640 1 -- 172.21.15.46:0/2603565432 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(81..81 src has 1..81) ==== 4738+0+0 (secure 0 0 0) 0x7f9f6806c4c0 con 0x7f9f7013be90 2024-09-16T04:40:49.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f760cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:49.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f760cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:49.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f760cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:49.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f758cc640 1 --2- 172.21.15.46:0/2603565432 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9f54038360 0x7f9f5403a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:49.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f758cc640 10 monclient: get_auth_request con 0x7f9f54038360 auth_method 0 2024-09-16T04:40:49.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f760cd640 10 monclient(hunting): handle_auth_done global_id 24197 payload 824 2024-09-16T04:40:49.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f760cd640 1 -- 172.21.15.46:0/2603565432 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9f7009bcf0 con 0x7f9f701339a0 2024-09-16T04:40:49.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f760cd640 1 --2- 172.21.15.46:0/2603565432 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f701339a0 0x7f9f70133d80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c00e770 tx=0x7f9f6c00ec30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:49.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f66ffd640 1 -- 172.21.15.46:0/2603565432 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c017070 con 0x7f9f701339a0 2024-09-16T04:40:49.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f66ffd640 10 monclient: dump: 2024-09-16T04:40:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:49.958 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.935+0000 7f9f758cc640 1 --2- 172.21.15.46:0/2603565432 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9f54038360 0x7f9f5403a820 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9f7013b6e0 tx=0x7f9f60009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.936+0000 7f9f66ffd640 1 -- 172.21.15.46:0/2603565432 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9f6c004020 con 0x7f9f701339a0 2024-09-16T04:40:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.936+0000 7f9f66ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.936+0000 7f9f66ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:40:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.936+0000 7f9f66ffd640 1 -- 172.21.15.46:0/2603565432 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f701339a0 msgr2=0x7f9f70133d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.936+0000 7f9f66ffd640 1 --2- 172.21.15.46:0/2603565432 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f701339a0 0x7f9f70133d80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c00e770 tx=0x7f9f6c00ec30 comp rx=0 tx=0).stop 2024-09-16T04:40:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.936+0000 7f9f66ffd640 1 -- 172.21.15.46:0/2603565432 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f701339a0 msgr2=0x7f9f70133d80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:40:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.936+0000 7f9f66ffd640 1 --2- 172.21.15.46:0/2603565432 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f701339a0 0x7f9f70133d80 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c00e770 tx=0x7f9f6c00ec30 comp rx=0 tx=0).stop 2024-09-16T04:40:49.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.942+0000 7f9f78331640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:40:49.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.942+0000 7f9f78331640 1 --2- 172.21.15.46:0/2603565432 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f701342c0 0x7f9f7009d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:49.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.942+0000 7f9f78331640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:49.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.942+0000 7f9f78331640 1 -- 172.21.15.46:0/2603565432 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f9f700a8660 con 0x7f9f701342c0 2024-09-16T04:40:49.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.942+0000 7f9f760cd640 1 --2- 172.21.15.46:0/2603565432 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f701342c0 0x7f9f7009d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:49.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.942+0000 7f9f760cd640 10 monclient: get_auth_request con 0x7f9f701342c0 auth_method 0 2024-09-16T04:40:49.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.942+0000 7f9f760cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:49.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.942+0000 7f9f760cd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:49.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.942+0000 7f9f760cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:49.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.942+0000 7f9f760cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:49.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.942+0000 7f9f760cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:49.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.942+0000 7f9f760cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:49.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.942+0000 7f9f760cd640 10 monclient(hunting): handle_auth_done global_id 24200 payload 824 2024-09-16T04:40:49.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.942+0000 7f9f760cd640 1 -- 172.21.15.46:0/2603565432 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9f70067e20 con 0x7f9f701342c0 2024-09-16T04:40:49.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.942+0000 7f9f760cd640 1 --2- 172.21.15.46:0/2603565432 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f701342c0 0x7f9f7009d110 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c00e770 tx=0x7f9f6c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:49.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.942+0000 7f9f66ffd640 1 -- 172.21.15.46:0/2603565432 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c017040 con 0x7f9f701342c0 2024-09-16T04:40:49.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.942+0000 7f9f66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:49.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.942+0000 7f9f66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:49.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.942+0000 7f9f66ffd640 10 monclient: dump: 2024-09-16T04:40:49.962 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:49.962 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:49.962 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:49.962 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:49.962 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:49.962 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:49.962 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:49.962 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:49.963 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:49.963 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:49.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.943+0000 7f9f66ffd640 1 -- 172.21.15.46:0/2603565432 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4700 (secure 0 0 0) 0x7f9f6c0039b0 con 0x7f9f701342c0 2024-09-16T04:40:49.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.943+0000 7f9f66ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:40:49.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.943+0000 7f9f66ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:40:49.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.943+0000 7f9f66ffd640 1 -- 172.21.15.46:0/2603565432 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f701342c0 msgr2=0x7f9f7009d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:49.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.943+0000 7f9f66ffd640 1 --2- 172.21.15.46:0/2603565432 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f701342c0 0x7f9f7009d110 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c00e770 tx=0x7f9f6c004730 comp rx=0 tx=0).stop 2024-09-16T04:40:49.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.943+0000 7f9f66ffd640 1 -- 172.21.15.46:0/2603565432 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f701342c0 msgr2=0x7f9f7009d110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:40:49.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.943+0000 7f9f66ffd640 1 --2- 172.21.15.46:0/2603565432 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f701342c0 0x7f9f7009d110 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c00e770 tx=0x7f9f6c004730 comp rx=0 tx=0).stop 2024-09-16T04:40:49.964 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:40:49.964 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:40:49.964 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:40:49.964 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:40:49.964 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 22, 2024-09-16T04:40:49.964 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 25142, 2024-09-16T04:40:49.964 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:40:49.964 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:40:49.964 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:40:49.964 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:49.964 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 1, 2024-09-16T04:40:49.965 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:40:49.965 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:40:49.965 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:40:49.965 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:49.965 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:49.965 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:49.965 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:49.965 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:49.965 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:49.965 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:49.965 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:49.965 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:49.966 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:49.966 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:49.966 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:49.966 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:40:49.966 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:40:49.966 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:49.966 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:49.966 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:49.966 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:49.966 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:49.966 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:49.966 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:49.966 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:49.966 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:49.967 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:49.967 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:49.967 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:49.967 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:49.967 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:40:49.967 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:40:49.967 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:40:49.967 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:40:49.967 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:40:49.967 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:40:49.967 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:40:49.967 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:40:49.967 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:40:49.967 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:40:49.967 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:40:49.968 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:40:49.968 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:40:49.968 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:40:49.968 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:40:49.968 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:40:49.968 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:40:49.968 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:49.968 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:49.968 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:49.968 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:49.968 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:49.968 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:49.968 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:49.968 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:49.969 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:49.969 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:49.969 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:49.969 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:49.969 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:40:49.969 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:49.969 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:40:49.969 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:49.969 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:40:49.969 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:40:49.969 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:49.969 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:49.969 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:49.970 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:49.970 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:40:49.970 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:49.970 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:49.970 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:49.970 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:49.970 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:40:49.970 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:49.970 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:49.970 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:49.970 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:49.970 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:40:49.970 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:40:49.970 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:49.971 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:49.971 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:49.971 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:49.971 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:49.971 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:40:49.971 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:40:49.971 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:49.971 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:49.971 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:49.971 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:49.971 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:40:49.971 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:49.971 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:49.971 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:49.972 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:49.972 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:40:49.972 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:49.972 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:49.972 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:49.972 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:49.972 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:40:49.972 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:40:49.972 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:49.972 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:49.972 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:49.972 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:49.972 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:49.972 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:40:49.973 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:40:49.973 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:49.973 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:49.973 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:49.973 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:49.973 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:40:49.973 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:49.973 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:49.973 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:49.973 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:49.973 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:40:49.973 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:49.973 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:49.974 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:49.974 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:49.974 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:40:49.974 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:40:49.974 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:49.974 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:49.974 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:49.974 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:49.974 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:49.974 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:49.974 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:40:49.974 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:40:49.974 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:49.974 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:49.975 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:49.975 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:40:49.975 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:49.975 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:49.975 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:40:49.975 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:49.975 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:49.975 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:49.975 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:40:49.975 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:49.975 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:49.975 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:40:49.975 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:49.975 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:49.976 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:49.976 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:40:49.976 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:49.976 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:49.976 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:49.976 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:40:49.976 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:40:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.943+0000 7f9f78331640 1 -- 172.21.15.46:0/2603565432 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9f54038360 msgr2=0x7f9f5403a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.943+0000 7f9f78331640 1 --2- 172.21.15.46:0/2603565432 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9f54038360 0x7f9f5403a820 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9f7013b6e0 tx=0x7f9f60009040 comp rx=0 tx=0).stop 2024-09-16T04:40:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.943+0000 7f9f78331640 10 monclient: shutdown 2024-09-16T04:40:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.943+0000 7f9f78331640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.943+0000 7f9f78331640 1 -- 172.21.15.46:0/2603565432 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f7013be90 msgr2=0x7f9f701cc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.943+0000 7f9f78331640 1 --2- 172.21.15.46:0/2603565432 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9f7013be90 0x7f9f701cc6d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9f68009e50 tx=0x7f9f68004a40 comp rx=0 tx=0).stop 2024-09-16T04:40:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.943+0000 7f9f768ce640 1 -- 172.21.15.46:0/2603565432 reap_dead start 2024-09-16T04:40:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.943+0000 7f9f768ce640 1 -- 172.21.15.46:0/2603565432 reap_dead start 2024-09-16T04:40:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.943+0000 7f9f78331640 1 -- 172.21.15.46:0/2603565432 shutdown_connections 2024-09-16T04:40:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.943+0000 7f9f78331640 1 -- 172.21.15.46:0/2603565432 >> 172.21.15.46:0/2603565432 conn(0x7f9f700a6340 msgr2=0x7f9f7013da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.943+0000 7f9f78331640 1 -- 172.21.15.46:0/2603565432 shutdown_connections 2024-09-16T04:40:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:49.943+0000 7f9f78331640 1 -- 172.21.15.46:0/2603565432 wait complete. 2024-09-16T04:40:49.979 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:40:49.979 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:40:49.979 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:40:50.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.079+0000 7ff98e7e4640 10 monclient: get_monmap_and_config 2024-09-16T04:40:50.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.079+0000 7ff98e7e4640 10 monclient: build_initial_monmap 2024-09-16T04:40:50.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.079+0000 7ff98e7e4640 10 monclient: monmap: 2024-09-16T04:40:50.081 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:50.081 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:50.081 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:40:50.080582+0000 2024-09-16T04:40:50.081 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:40:50.080582+0000 2024-09-16T04:40:50.082 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:50.082 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:50.082 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:40:50.082 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:40:50.082 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:40:50.082 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:40:50.082 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:40:50.082 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:50.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.079+0000 7ff98e7e4640 1 Processor -- start 2024-09-16T04:40:50.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.079+0000 7ff98e7e4640 1 -- start start 2024-09-16T04:40:50.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.079+0000 7ff98e7e4640 10 monclient: init 2024-09-16T04:40:50.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98e7e4640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:50.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98e7e4640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-16T04:40:50.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98e7e4640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff9880a3c10 0x7ff98809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:50.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98e7e4640 10 monclient(hunting): picked mon.noname-b con 0x7ff9880a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:50.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98e7e4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff9880a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:40:50.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98e7e4640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff98813be90 0x7ff98813e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:50.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98e7e4640 10 monclient(hunting): picked mon.noname-c con 0x7ff98813be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:40:50.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98e7e4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff98809b940 con 0x7ff9880a3ff0 2024-09-16T04:40:50.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98e7e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:50.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98e7e4640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ff98809ab00 con 0x7ff98813be90 2024-09-16T04:40:50.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98e7e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:50.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98e7e4640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7ff98809b030 con 0x7ff9880a3c10 2024-09-16T04:40:50.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98e7e4640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:50.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98e7e4640 10 monclient(hunting): authenticate will time out at 848.496487s 2024-09-16T04:40:50.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff987fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff9880a3c10 0x7ff98809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:50.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff987fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff9880a3c10 0x7ff98809bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39588/0 (socket says 172.21.15.46:39588) 2024-09-16T04:40:50.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff987fff640 1 -- 172.21.15.46:0/1014554401 learned_addr learned my addr 172.21.15.46:0/1014554401 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:50.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff987fff640 10 monclient(hunting): get_auth_request con 0x7ff9880a3c10 auth_method 0 2024-09-16T04:40:50.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff987fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:50.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff987fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:50.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff987fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:50.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff987fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:50.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff987fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:50.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff987fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:50.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98cd81640 1 --2- 172.21.15.46:0/1014554401 >> v2:172.21.15.114:3300/0 conn(0x7ff98813be90 0x7ff98813e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:50.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98cd81640 10 monclient(hunting): get_auth_request con 0x7ff98813be90 auth_method 0 2024-09-16T04:40:50.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98cd81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98cd81640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98cd81640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98cd81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98cd81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff98cd81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff987fff640 10 monclient(hunting): handle_auth_done global_id 24203 payload 274 2024-09-16T04:40:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff987fff640 1 -- 172.21.15.46:0/1014554401 >> v2:172.21.15.114:3300/0 conn(0x7ff98813be90 msgr2=0x7ff98813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:40:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff987fff640 1 --2- 172.21.15.46:0/1014554401 >> v2:172.21.15.114:3300/0 conn(0x7ff98813be90 0x7ff98813e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff987fff640 1 -- 172.21.15.46:0/1014554401 >> v1:172.21.15.114:6789/0 conn(0x7ff9880a3ff0 legacy=0x7ff9880a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff987fff640 10 monclient: _finish_hunting 0 2024-09-16T04:40:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff987fff640 1 monclient: found mon.noname-b 2024-09-16T04:40:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff987fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:50.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff987fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:40:50.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.080+0000 7ff987fff640 1 -- 172.21.15.46:0/1014554401 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff98809b310 con 0x7ff9880a3c10 2024-09-16T04:40:50.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.081+0000 7ff987fff640 1 --2- 172.21.15.46:0/1014554401 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff9880a3c10 0x7ff98809bcf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff978009870 tx=0x7ff97802a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb726cd49cc7f692 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:50.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.081+0000 7ff986ffd640 1 -- 172.21.15.46:0/1014554401 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978037070 con 0x7ff9880a3c10 2024-09-16T04:40:50.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.081+0000 7ff986ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:50.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.081+0000 7ff986ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:40:50.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.081+0000 7ff986ffd640 10 monclient: dump: 2024-09-16T04:40:50.107 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:50.107 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:50.107 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:50.109 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:50.109 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:50.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.081+0000 7ff986ffd640 10 monclient: _finish_auth 0 2024-09-16T04:40:50.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.081+0000 7ff986ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:40:50.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.081+0000 7ff986ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:50.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.081+0000 7ff986ffd640 1 -- 172.21.15.46:0/1014554401 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff978032440 con 0x7ff9880a3c10 2024-09-16T04:40:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.081+0000 7ff98e7e4640 5 monclient: authenticate success, global_id 24203 2024-09-16T04:40:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.081+0000 7ff98e7e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:40:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.081+0000 7ff986ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.081+0000 7ff986ffd640 1 -- 172.21.15.46:0/1014554401 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978032810 con 0x7ff9880a3c10 2024-09-16T04:40:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.081+0000 7ff986ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.081+0000 7ff986ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:50.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.081+0000 7ff986ffd640 10 monclient: dump: 2024-09-16T04:40:50.111 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:50.111 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:50.111 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:50.111 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:50.112 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:50.112 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:50.112 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:50.113 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:50.113 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:50.113 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:50.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.081+0000 7ff98e7e4640 10 monclient: get_monmap_and_config success 2024-09-16T04:40:50.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.081+0000 7ff98e7e4640 4 set_mon_vals no callback set 2024-09-16T04:40:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.082+0000 7ff98e7e4640 10 monclient: shutdown 2024-09-16T04:40:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.082+0000 7ff98e7e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.082+0000 7ff98e7e4640 1 -- 172.21.15.46:0/1014554401 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff9880a3c10 msgr2=0x7ff98809bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.082+0000 7ff98e7e4640 1 --2- 172.21.15.46:0/1014554401 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff9880a3c10 0x7ff98809bcf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff978009870 tx=0x7ff97802a130 comp rx=0 tx=0).stop 2024-09-16T04:40:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.082+0000 7ff98e7e4640 1 -- 172.21.15.46:0/1014554401 shutdown_connections 2024-09-16T04:40:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.082+0000 7ff98e7e4640 1 --2- 172.21.15.46:0/1014554401 >> v2:172.21.15.114:3300/0 conn(0x7ff98813be90 0x7ff98813e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.082+0000 7ff98e7e4640 1 --2- 172.21.15.46:0/1014554401 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff9880a3c10 0x7ff98809bcf0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.082+0000 7ff98e7e4640 1 -- 172.21.15.46:0/1014554401 >> 172.21.15.46:0/1014554401 conn(0x7ff9880a6340 msgr2=0x7ff98806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.082+0000 7ff98e7e4640 1 -- 172.21.15.46:0/1014554401 shutdown_connections 2024-09-16T04:40:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.082+0000 7ff98e7e4640 1 -- 172.21.15.46:0/1014554401 wait complete. 2024-09-16T04:40:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.082+0000 7ff98e7e4640 10 monclient: build_initial_monmap 2024-09-16T04:40:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.082+0000 7ff98e7e4640 10 monclient: monmap: 2024-09-16T04:40:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:40:50.116 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:40:50.116 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:40:50.116 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:40:50.116 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:40:50.116 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:50.116 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:40:50.117 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:40:50.117 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:40:50.117 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:50.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.083+0000 7ff98e7e4640 1 Processor -- start 2024-09-16T04:40:50.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.083+0000 7ff98e7e4640 1 -- start start 2024-09-16T04:40:50.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.083+0000 7ff98e7e4640 10 monclient: init 2024-09-16T04:40:50.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.083+0000 7ff98e7e4640 10 monclient: _reopen_session rank -1 2024-09-16T04:40:50.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.083+0000 7ff98e7e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:40:50.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.083+0000 7ff98e7e4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9880a3c10 0x7ff98813a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:50.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.083+0000 7ff98e7e4640 10 monclient(hunting): picked mon.noname-b con 0x7ff9880a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:40:50.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.083+0000 7ff98e7e4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9880a3ff0 0x7ff9881cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:50.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.083+0000 7ff98e7e4640 10 monclient(hunting): picked mon.noname-a con 0x7ff9880a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:40:50.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.083+0000 7ff98e7e4640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff98813be90 0x7ff9881cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:50.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.083+0000 7ff98e7e4640 10 monclient(hunting): picked mon.noname-c con 0x7ff98813be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:40:50.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.083+0000 7ff98e7e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:50.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.083+0000 7ff98e7e4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff9880a6b40 con 0x7ff9880a3ff0 2024-09-16T04:40:50.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.083+0000 7ff98e7e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:50.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.083+0000 7ff98e7e4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff9880a69c0 con 0x7ff9880a3c10 2024-09-16T04:40:50.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.083+0000 7ff98e7e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:50.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.083+0000 7ff98e7e4640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7ff98813e7c0 con 0x7ff98813be90 2024-09-16T04:40:50.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.083+0000 7ff98e7e4640 10 monclient(hunting): _renew_subs 2024-09-16T04:40:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.083+0000 7ff98e7e4640 10 monclient(hunting): authenticate will time out at 848.499969s 2024-09-16T04:40:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.083+0000 7ff9877fe640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9880a3ff0 msgr2=0x7ff9881cbba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:40:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff9877fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9880a3ff0 0x7ff9881cbba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:40:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98cd81640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff98813be90 0x7ff9881cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98cd81640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff98813be90 0x7ff9881cc0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39590/0 (socket says 172.21.15.46:39590) 2024-09-16T04:40:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98cd81640 1 -- 172.21.15.46:0/1638007523 learned_addr learned my addr 172.21.15.46:0/1638007523 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:40:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98cd81640 10 monclient(hunting): get_auth_request con 0x7ff98813be90 auth_method 0 2024-09-16T04:40:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98cd81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98cd81640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98cd81640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff987fff640 1 --2- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9880a3c10 0x7ff98813a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff987fff640 10 monclient(hunting): get_auth_request con 0x7ff9880a3c10 auth_method 0 2024-09-16T04:40:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff987fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:50.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff987fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:50.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff987fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:50.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98cd81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:50.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98cd81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:50.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98cd81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:50.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff987fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:50.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff987fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:50.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff987fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:50.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98cd81640 10 monclient(hunting): handle_auth_done global_id 24206 payload 824 2024-09-16T04:40:50.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98cd81640 1 -- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9880a3c10 msgr2=0x7ff98813a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:50.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98cd81640 1 --2- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9880a3c10 0x7ff98813a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:50.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98cd81640 1 -- 172.21.15.46:0/1638007523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9880a3ff0 msgr2=0x7ff9881cbba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:40:50.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98cd81640 1 --2- 172.21.15.46:0/1638007523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9880a3ff0 0x7ff9881cbba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:50.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98cd81640 10 monclient: _finish_hunting 0 2024-09-16T04:40:50.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98cd81640 1 monclient: found mon.noname-c 2024-09-16T04:40:50.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98cd81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:40:50.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98cd81640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:40:50.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98cd81640 1 -- 172.21.15.46:0/1638007523 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff98813c690 con 0x7ff98813be90 2024-09-16T04:40:50.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98cd81640 1 --2- 172.21.15.46:0/1638007523 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff98813be90 0x7ff9881cc0e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff980009d80 tx=0x7ff98000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:50.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff984ff9640 1 -- 172.21.15.46:0/1638007523 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff980018020 con 0x7ff98813be90 2024-09-16T04:40:50.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff984ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:50.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff984ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:40:50.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff984ff9640 10 monclient: dump: 2024-09-16T04:40:50.121 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:50.121 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:50.122 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:50.122 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:50.122 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:50.122 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:50.122 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:50.122 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:50.122 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:50.122 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:50.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff984ff9640 10 monclient: _finish_auth 0 2024-09-16T04:40:50.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff984ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:40:50.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff984ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:40:50.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff984ff9640 1 -- 172.21.15.46:0/1638007523 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff980014480 con 0x7ff98813be90 2024-09-16T04:40:50.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff984ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:40:50.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff984ff9640 1 -- 172.21.15.46:0/1638007523 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff980014870 con 0x7ff98813be90 2024-09-16T04:40:50.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff984ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:50.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff984ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:40:50.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff984ff9640 10 monclient: dump: 2024-09-16T04:40:50.123 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:50.123 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:50.123 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:50.123 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:50.123 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:50.123 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:50.123 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:50.123 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:50.123 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:50.123 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:50.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98e7e4640 5 monclient: authenticate success, global_id 24206 2024-09-16T04:40:50.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98e7e4640 10 monclient: _renew_subs 2024-09-16T04:40:50.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98e7e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:40:50.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98e7e4640 1 -- 172.21.15.46:0/1638007523 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9881389d0 con 0x7ff98813be90 2024-09-16T04:40:50.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff9867fc640 4 set_mon_vals no callback set 2024-09-16T04:40:50.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98e7e4640 10 monclient: _renew_subs 2024-09-16T04:40:50.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98e7e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:40:50.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.084+0000 7ff98e7e4640 1 -- 172.21.15.46:0/1638007523 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff988139e80 con 0x7ff98813be90 2024-09-16T04:40:50.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.085+0000 7ff984ff9640 1 -- 172.21.15.46:0/1638007523 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7ff980005d10 con 0x7ff98813be90 2024-09-16T04:40:50.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.085+0000 7ff98e7e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:50.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.085+0000 7ff98e7e4640 1 --2- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9881cc620 0x7ff9881339a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:50.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.085+0000 7ff98e7e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:50.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.085+0000 7ff98e7e4640 1 -- 172.21.15.46:0/1638007523 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff9881cca00 con 0x7ff9881cc620 2024-09-16T04:40:50.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.085+0000 7ff984ff9640 1 --2- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff974038360 0x7ff97403a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:50.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.085+0000 7ff984ff9640 1 -- 172.21.15.46:0/1638007523 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(81..81 src has 1..81) ==== 4738+0+0 (secure 0 0 0) 0x7ff98004c560 con 0x7ff98813be90 2024-09-16T04:40:50.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.085+0000 7ff987fff640 1 --2- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9881cc620 0x7ff9881339a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:50.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.085+0000 7ff987fff640 10 monclient: get_auth_request con 0x7ff9881cc620 auth_method 0 2024-09-16T04:40:50.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.085+0000 7ff987fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:50.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.085+0000 7ff987fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:50.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.085+0000 7ff987fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:50.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.086+0000 7ff9877fe640 1 --2- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff974038360 0x7ff97403a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:50.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.086+0000 7ff9877fe640 10 monclient: get_auth_request con 0x7ff974038360 auth_method 0 2024-09-16T04:40:50.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.086+0000 7ff987fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:50.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.086+0000 7ff987fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:50.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.086+0000 7ff987fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:50.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.086+0000 7ff987fff640 10 monclient(hunting): handle_auth_done global_id 5011 payload 824 2024-09-16T04:40:50.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.086+0000 7ff987fff640 1 -- 172.21.15.46:0/1638007523 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff9880a43d0 con 0x7ff9881cc620 2024-09-16T04:40:50.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.086+0000 7ff9877fe640 1 --2- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff974038360 0x7ff97403a820 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff9881397f0 tx=0x7ff97c0079f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:50.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.086+0000 7ff987fff640 1 --2- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9881cc620 0x7ff9881339a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff9880689b0 tx=0x7ff9780046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:50.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.086+0000 7ff984ff9640 1 -- 172.21.15.46:0/1638007523 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978037040 con 0x7ff9881cc620 2024-09-16T04:40:50.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.086+0000 7ff984ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:50.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.086+0000 7ff984ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:40:50.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.086+0000 7ff984ff9640 10 monclient: dump: 2024-09-16T04:40:50.126 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:50.126 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:50.126 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:50.126 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:50.126 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:50.126 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:50.126 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:50.126 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:50.127 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:50.127 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:50.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.086+0000 7ff984ff9640 1 -- 172.21.15.46:0/1638007523 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff97803c640 con 0x7ff9881cc620 2024-09-16T04:40:50.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.087+0000 7ff984ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:40:50.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.087+0000 7ff984ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:40:50.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.087+0000 7ff984ff9640 1 -- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9881cc620 msgr2=0x7ff9881339a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:50.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.087+0000 7ff984ff9640 1 --2- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9881cc620 0x7ff9881339a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff9880689b0 tx=0x7ff9780046c0 comp rx=0 tx=0).stop 2024-09-16T04:40:50.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.087+0000 7ff984ff9640 1 -- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9881cc620 msgr2=0x7ff9881339a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:40:50.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.087+0000 7ff984ff9640 1 --2- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9881cc620 0x7ff9881339a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:50.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.093+0000 7ff98e7e4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:40:50.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.093+0000 7ff98e7e4640 1 --2- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff988133ee0 0x7ff9881342f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:40:50.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.093+0000 7ff98e7e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:40:50.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.093+0000 7ff98e7e4640 1 -- 172.21.15.46:0/1638007523 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff988134830 con 0x7ff988133ee0 2024-09-16T04:40:50.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.093+0000 7ff987fff640 1 --2- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff988133ee0 0x7ff9881342f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:40:50.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.093+0000 7ff987fff640 10 monclient: get_auth_request con 0x7ff988133ee0 auth_method 0 2024-09-16T04:40:50.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.093+0000 7ff987fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:40:50.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.093+0000 7ff987fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:40:50.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.093+0000 7ff987fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:40:50.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.093+0000 7ff987fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:40:50.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.093+0000 7ff987fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:40:50.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.093+0000 7ff987fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:40:50.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.093+0000 7ff987fff640 10 monclient(hunting): handle_auth_done global_id 5014 payload 824 2024-09-16T04:40:50.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.093+0000 7ff987fff640 1 -- 172.21.15.46:0/1638007523 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff988067e20 con 0x7ff988133ee0 2024-09-16T04:40:50.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff987fff640 1 --2- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff988133ee0 0x7ff9881342f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff97802a170 tx=0x7ff978008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:40:50.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff984ff9640 1 -- 172.21.15.46:0/1638007523 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978048a10 con 0x7ff988133ee0 2024-09-16T04:40:50.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff984ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:40:50.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff984ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:40:50.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff984ff9640 10 monclient: dump: 2024-09-16T04:40:50.129 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:40:50.129 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:40:50.129 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:50.129 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:40:50.129 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:40:50.129 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:40:50.129 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:40:50.129 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:40:50.130 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:40:50.130 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:40:50.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff984ff9640 1 -- 172.21.15.46:0/1638007523 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4870 (secure 0 0 0) 0x7ff9780489f0 con 0x7ff988133ee0 2024-09-16T04:40:50.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff984ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:40:50.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff984ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:40:50.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff984ff9640 1 -- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff988133ee0 msgr2=0x7ff9881342f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:50.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff984ff9640 1 --2- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff988133ee0 0x7ff9881342f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff97802a170 tx=0x7ff978008030 comp rx=0 tx=0).stop 2024-09-16T04:40:50.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff984ff9640 1 -- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff988133ee0 msgr2=0x7ff9881342f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:40:50.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff984ff9640 1 --2- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff988133ee0 0x7ff9881342f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:40:50.130 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:40:50.131 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:40:50.131 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:40:50.131 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:40:50.131 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 22, 2024-09-16T04:40:50.131 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 135996, 2024-09-16T04:40:50.131 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:40:50.131 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:40:50.131 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:40:50.131 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:50.131 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 1, 2024-09-16T04:40:50.131 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:40:50.131 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:40:50.131 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:40:50.132 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:50.132 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:50.132 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:50.132 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:50.132 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:50.132 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:50.132 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:50.132 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:50.132 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:50.132 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:50.132 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:50.132 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:50.132 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:40:50.132 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:40:50.133 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:50.133 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:50.133 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:50.133 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:50.133 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:50.133 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:50.133 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:50.133 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:50.133 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:50.133 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:50.133 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:50.133 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:50.133 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:50.134 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:40:50.134 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:40:50.134 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:40:50.134 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:40:50.134 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:40:50.134 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:40:50.134 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:40:50.134 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:40:50.134 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:40:50.134 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:40:50.134 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:40:50.134 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:40:50.134 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:40:50.134 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:40:50.135 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:40:50.135 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:40:50.135 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:40:50.135 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:40:50.135 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:40:50.135 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:40:50.135 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:40:50.135 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:40:50.135 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:40:50.136 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:40:50.136 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:40:50.136 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:40:50.136 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:40:50.136 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:40:50.136 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:50.136 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:40:50.136 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:50.136 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:40:50.136 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:50.136 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:40:50.136 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:40:50.137 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:50.137 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:50.137 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:50.137 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:50.137 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:40:50.137 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:50.137 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:50.137 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:50.137 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:50.137 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:40:50.137 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:50.137 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:50.137 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:50.138 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:50.138 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:40:50.138 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:40:50.138 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:50.138 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:50.138 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:50.138 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:50.138 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:50.138 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:40:50.138 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:40:50.138 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:50.138 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:50.139 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:50.139 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:50.139 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:40:50.139 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:50.139 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:50.139 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:50.139 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:50.139 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:40:50.139 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:50.139 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:50.139 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:50.139 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:50.140 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:40:50.140 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:40:50.140 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:50.140 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:50.140 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:50.140 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:50.140 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:50.140 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:40:50.140 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:40:50.140 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:40:50.140 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:40:50.140 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:50.140 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:40:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:40:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:40:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:40:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:40:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:40:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:40:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:40:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:40:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:40:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:40:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:40:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:40:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:40:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4 2024-09-16T04:40:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:40:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 7 2024-09-16T04:40:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:40:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:40:50.144 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:40:50.144 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:40:50.144 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:40:50.144 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:40:50.144 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:40:50.144 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:40:50.144 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:40:50.144 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:40:50.144 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:40:50.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff98e7e4640 1 -- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff974038360 msgr2=0x7ff97403a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:50.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff98e7e4640 1 --2- 172.21.15.46:0/1638007523 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff974038360 0x7ff97403a820 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff9881397f0 tx=0x7ff97c0079f0 comp rx=0 tx=0).stop 2024-09-16T04:40:50.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff98e7e4640 10 monclient: shutdown 2024-09-16T04:40:50.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff98e7e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:40:50.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff98e7e4640 1 -- 172.21.15.46:0/1638007523 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff98813be90 msgr2=0x7ff9881cc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:40:50.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff98e7e4640 1 --2- 172.21.15.46:0/1638007523 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff98813be90 0x7ff9881cc0e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff980009d80 tx=0x7ff98000c490 comp rx=0 tx=0).stop 2024-09-16T04:40:50.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff98cd81640 1 -- 172.21.15.46:0/1638007523 reap_dead start 2024-09-16T04:40:50.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff98cd81640 1 -- 172.21.15.46:0/1638007523 reap_dead start 2024-09-16T04:40:50.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff98e7e4640 1 -- 172.21.15.46:0/1638007523 shutdown_connections 2024-09-16T04:40:50.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff98e7e4640 1 -- 172.21.15.46:0/1638007523 >> 172.21.15.46:0/1638007523 conn(0x7ff9880a6340 msgr2=0x7ff98813ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:40:50.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff98e7e4640 1 -- 172.21.15.46:0/1638007523 shutdown_connections 2024-09-16T04:40:50.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:40:50.094+0000 7ff98e7e4640 1 -- 172.21.15.46:0/1638007523 wait complete. 2024-09-16T04:40:50.147 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:40:50.147 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:40:50.199 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36363]: starting. 2024-09-16T04:40:50.199 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36363]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-36362 2024-09-16T04:40:50.200 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36363]: created object 0... 2024-09-16T04:40:50.200 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36363]: created object 25... 2024-09-16T04:40:50.200 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36363]: created object 49... 2024-09-16T04:40:50.200 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36363]: finishing. 2024-09-16T04:40:50.200 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36363]: shutting down. 2024-09-16T04:40:51.126 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36364]: starting. 2024-09-16T04:40:51.127 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36364]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-36362 2024-09-16T04:40:51.127 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36364]: shutting down. 2024-09-16T04:40:53.215 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:53.215 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36399]: starting. 2024-09-16T04:40:53.215 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36399]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-36362 2024-09-16T04:40:53.215 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36399]: created object 0... 2024-09-16T04:40:53.215 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36399]: created object 25... 2024-09-16T04:40:53.215 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36399]: created object 49... 2024-09-16T04:40:53.216 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36399]: finishing. 2024-09-16T04:40:53.216 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36399]: shutting down. 2024-09-16T04:40:53.228 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:53.228 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36401]: starting. 2024-09-16T04:40:53.228 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36401]: listing objects. 2024-09-16T04:40:53.228 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36401]: listed object 0... 2024-09-16T04:40:53.228 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36401]: listed object 25... 2024-09-16T04:40:53.228 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36401]: saw 50 objects 2024-09-16T04:40:53.228 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36401]: shutting down. 2024-09-16T04:40:54.142 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:54.142 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36400]: starting. 2024-09-16T04:40:54.142 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36400]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-36362 2024-09-16T04:40:54.142 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36400]: shutting down. 2024-09-16T04:40:54.143 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:54.143 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:54.143 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:40:54.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:40:56.241 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36470]: starting. 2024-09-16T04:40:56.241 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36470]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-36469 2024-09-16T04:40:56.241 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36470]: created object 0... 2024-09-16T04:40:56.241 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36470]: created object 25... 2024-09-16T04:40:56.241 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36470]: created object 49... 2024-09-16T04:40:56.242 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36470]: finishing. 2024-09-16T04:40:56.242 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36470]: shutting down. 2024-09-16T04:40:57.160 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36471]: starting. 2024-09-16T04:40:57.160 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36471]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-36469 2024-09-16T04:40:57.160 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36471]: shutting down. 2024-09-16T04:40:59.215 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:59.215 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36506]: starting. 2024-09-16T04:40:59.215 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36506]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-36469 2024-09-16T04:40:59.215 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36506]: created object 0... 2024-09-16T04:40:59.215 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36506]: created object 25... 2024-09-16T04:40:59.215 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36506]: created object 49... 2024-09-16T04:40:59.215 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36506]: finishing. 2024-09-16T04:40:59.215 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36506]: shutting down. 2024-09-16T04:40:59.228 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:40:59.228 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36508]: starting. 2024-09-16T04:40:59.228 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36508]: listing objects. 2024-09-16T04:40:59.228 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36508]: listed object 0... 2024-09-16T04:40:59.228 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36508]: listed object 25... 2024-09-16T04:40:59.229 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36508]: saw 50 objects 2024-09-16T04:40:59.229 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36508]: shutting down. 2024-09-16T04:41:00.126 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:00.127 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36507]: starting. 2024-09-16T04:41:00.127 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36507]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-36469 2024-09-16T04:41:00.127 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36507]: shutting down. 2024-09-16T04:41:00.128 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:00.128 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:00.128 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:41:00.133 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:41:02.267 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36577]: starting. 2024-09-16T04:41:02.267 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36577]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-36576 2024-09-16T04:41:02.267 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36577]: created object 0... 2024-09-16T04:41:02.267 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36577]: created object 25... 2024-09-16T04:41:02.267 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36577]: created object 49... 2024-09-16T04:41:02.267 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36577]: finishing. 2024-09-16T04:41:02.267 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36577]: shutting down. 2024-09-16T04:41:03.190 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36578]: starting. 2024-09-16T04:41:03.190 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36578]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-36576 2024-09-16T04:41:03.190 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36578]: shutting down. 2024-09-16T04:41:05.285 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:05.285 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36613]: starting. 2024-09-16T04:41:05.285 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36613]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-36576 2024-09-16T04:41:05.285 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36613]: created object 0... 2024-09-16T04:41:05.285 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36613]: created object 25... 2024-09-16T04:41:05.286 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36613]: created object 49... 2024-09-16T04:41:05.286 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36613]: finishing. 2024-09-16T04:41:05.286 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36613]: shutting down. 2024-09-16T04:41:05.297 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:05.297 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36615]: starting. 2024-09-16T04:41:05.297 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36615]: listing objects. 2024-09-16T04:41:05.297 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36615]: listed object 0... 2024-09-16T04:41:05.297 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36615]: listed object 25... 2024-09-16T04:41:05.297 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36615]: saw 50 objects 2024-09-16T04:41:05.297 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36615]: shutting down. 2024-09-16T04:41:06.199 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:06.199 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36614]: starting. 2024-09-16T04:41:06.200 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36614]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-36576 2024-09-16T04:41:06.200 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36614]: shutting down. 2024-09-16T04:41:06.201 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:06.201 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:06.201 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:41:06.204 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:41:08.300 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36684]: starting. 2024-09-16T04:41:08.300 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36684]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-36683 2024-09-16T04:41:08.300 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36684]: created object 0... 2024-09-16T04:41:08.300 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36684]: created object 25... 2024-09-16T04:41:08.300 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36684]: created object 49... 2024-09-16T04:41:08.300 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36684]: finishing. 2024-09-16T04:41:08.300 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36684]: shutting down. 2024-09-16T04:41:09.129 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36685]: starting. 2024-09-16T04:41:09.129 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36685]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-36683 2024-09-16T04:41:09.129 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36685]: shutting down. 2024-09-16T04:41:10.148 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:41:10.148 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T04:41:10.149 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T04:41:10.149 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T04:41:10.152 INFO:tasks.ceph.mon.a:Started 2024-09-16T04:41:10.152 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:41:10.152 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:41:10.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.262+0000 7fe863bdf640 10 monclient: get_monmap_and_config 2024-09-16T04:41:10.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.262+0000 7fe863bdf640 10 monclient: build_initial_monmap 2024-09-16T04:41:10.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.262+0000 7fe863bdf640 10 monclient: monmap: 2024-09-16T04:41:10.273 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:10.273 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:10.273 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:41:10.263559+0000 2024-09-16T04:41:10.273 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:41:10.263559+0000 2024-09-16T04:41:10.273 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:10.273 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:10.273 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:41:10.274 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:41:10.274 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:41:10.274 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:41:10.274 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:41:10.274 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:10.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.262+0000 7fe863bdf640 1 Processor -- start 2024-09-16T04:41:10.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.262+0000 7fe863bdf640 1 -- start start 2024-09-16T04:41:10.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.262+0000 7fe863bdf640 10 monclient: init 2024-09-16T04:41:10.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe863bdf640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:10.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe863bdf640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-16T04:41:10.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe863bdf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe85c0a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:41:10.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe863bdf640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe85c0a3ff0 0x7fe85c0a4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:10.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe863bdf640 10 monclient(hunting): picked mon.noname-a con 0x7fe85c0a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:41:10.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe863bdf640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe85c13be90 0x7fe85c13e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:10.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe863bdf640 10 monclient(hunting): picked mon.noname-b con 0x7fe85c13be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:10.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe863bdf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe85c09ab70 con 0x7fe85c0a3c10 2024-09-16T04:41:10.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe863bdf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:10.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe863bdf640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe85c0a4d60 con 0x7fe85c0a3ff0 2024-09-16T04:41:10.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe863bdf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:10.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe863bdf640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe85c09b030 con 0x7fe85c13be90 2024-09-16T04:41:10.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe863bdf640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:10.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe863bdf640 10 monclient(hunting): authenticate will time out at 868.679437s 2024-09-16T04:41:10.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86197b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe85c0a3c10 0x7fe85c09bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42300/0 (socket says 172.21.15.46:42300) 2024-09-16T04:41:10.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86197b640 1 -- 172.21.15.46:0/69401756 learned_addr learned my addr 172.21.15.46:0/69401756 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:41:10.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86117a640 1 --2- 172.21.15.46:0/69401756 >> v2:172.21.15.46:3300/0 conn(0x7fe85c0a3ff0 0x7fe85c0a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:10.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86217c640 1 --2- 172.21.15.46:0/69401756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe85c13be90 0x7fe85c13e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:10.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86117a640 10 monclient(hunting): get_auth_request con 0x7fe85c0a3ff0 auth_method 0 2024-09-16T04:41:10.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86117a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86117a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86117a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86217c640 10 monclient(hunting): get_auth_request con 0x7fe85c13be90 auth_method 0 2024-09-16T04:41:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86217c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86217c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86217c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86217c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86217c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86117a640 1 -- 172.21.15.46:0/69401756 >> v2:172.21.15.46:3300/0 conn(0x7fe85c0a3ff0 msgr2=0x7fe85c0a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:41:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86117a640 1 -- 172.21.15.46:0/69401756 >> v2:172.21.15.46:3300/0 conn(0x7fe85c0a3ff0 msgr2=0x7fe85c0a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:41:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86217c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86117a640 1 --2- 172.21.15.46:0/69401756 >> v2:172.21.15.46:3300/0 conn(0x7fe85c0a3ff0 0x7fe85c0a4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:41:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86117a640 1 --2- 172.21.15.46:0/69401756 >> v2:172.21.15.46:3300/0 conn(0x7fe85c0a3ff0 0x7fe85c0a4400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:41:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86217c640 10 monclient(hunting): handle_auth_done global_id 24302 payload 274 2024-09-16T04:41:10.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86217c640 1 -- 172.21.15.46:0/69401756 >> v2:172.21.15.46:3300/0 conn(0x7fe85c0a3ff0 msgr2=0x7fe85c0a4400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:41:10.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86217c640 1 --2- 172.21.15.46:0/69401756 >> v2:172.21.15.46:3300/0 conn(0x7fe85c0a3ff0 0x7fe85c0a4400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:10.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86217c640 1 -- 172.21.15.46:0/69401756 >> v1:172.21.15.46:6789/0 conn(0x7fe85c0a3c10 legacy=0x7fe85c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:10.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86217c640 10 monclient: _finish_hunting 0 2024-09-16T04:41:10.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86217c640 1 monclient: found mon.noname-b 2024-09-16T04:41:10.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86217c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:10.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86217c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:41:10.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.263+0000 7fe86217c640 1 -- 172.21.15.46:0/69401756 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe85c09b310 con 0x7fe85c13be90 2024-09-16T04:41:10.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.264+0000 7fe86217c640 1 --2- 172.21.15.46:0/69401756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe85c13be90 0x7fe85c13e280 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe854009d90 tx=0x7fe85402a530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=403f279f63891975 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:10.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.264+0000 7fe860979640 1 -- 172.21.15.46:0/69401756 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe854007740 con 0x7fe85c13be90 2024-09-16T04:41:10.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.264+0000 7fe860979640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:10.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.264+0000 7fe860979640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:41:10.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.264+0000 7fe860979640 10 monclient: dump: 2024-09-16T04:41:10.277 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:10.278 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:10.278 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:10.278 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:10.278 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:10.278 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:10.278 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:10.278 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:10.278 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:10.278 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:10.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.264+0000 7fe860979640 10 monclient: _finish_auth 0 2024-09-16T04:41:10.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.264+0000 7fe860979640 10 monclient: _check_auth_tickets 2024-09-16T04:41:10.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.264+0000 7fe860979640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:10.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.264+0000 7fe860979640 1 -- 172.21.15.46:0/69401756 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8540078e0 con 0x7fe85c13be90 2024-09-16T04:41:10.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.264+0000 7fe863bdf640 5 monclient: authenticate success, global_id 24302 2024-09-16T04:41:10.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.264+0000 7fe863bdf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:41:10.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.264+0000 7fe860979640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:10.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.264+0000 7fe863bdf640 10 monclient: get_monmap_and_config success 2024-09-16T04:41:10.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.264+0000 7fe860979640 1 -- 172.21.15.46:0/69401756 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe854007cb0 con 0x7fe85c13be90 2024-09-16T04:41:10.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.264+0000 7fe863bdf640 4 set_mon_vals no callback set 2024-09-16T04:41:10.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.264+0000 7fe860979640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:10.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.264+0000 7fe860979640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:10.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.264+0000 7fe860979640 10 monclient: dump: 2024-09-16T04:41:10.279 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:10.279 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:10.279 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:10.279 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:10.279 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:10.279 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:10.279 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:10.280 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.265+0000 7fe863bdf640 10 monclient: shutdown 2024-09-16T04:41:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.265+0000 7fe863bdf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.265+0000 7fe863bdf640 1 -- 172.21.15.46:0/69401756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe85c13be90 msgr2=0x7fe85c13e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.265+0000 7fe863bdf640 1 --2- 172.21.15.46:0/69401756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe85c13be90 0x7fe85c13e280 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe854009d90 tx=0x7fe85402a530 comp rx=0 tx=0).stop 2024-09-16T04:41:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.265+0000 7fe863bdf640 1 -- 172.21.15.46:0/69401756 shutdown_connections 2024-09-16T04:41:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.265+0000 7fe863bdf640 1 --2- 172.21.15.46:0/69401756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe85c13be90 0x7fe85c13e280 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.265+0000 7fe863bdf640 1 --2- 172.21.15.46:0/69401756 >> v2:172.21.15.46:3300/0 conn(0x7fe85c0a3ff0 0x7fe85c0a4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.265+0000 7fe863bdf640 1 -- 172.21.15.46:0/69401756 >> 172.21.15.46:0/69401756 conn(0x7fe85c0a5d10 msgr2=0x7fe85c0a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.265+0000 7fe863bdf640 1 -- 172.21.15.46:0/69401756 shutdown_connections 2024-09-16T04:41:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.265+0000 7fe863bdf640 1 -- 172.21.15.46:0/69401756 wait complete. 2024-09-16T04:41:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.266+0000 7fe863bdf640 10 monclient: build_initial_monmap 2024-09-16T04:41:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.266+0000 7fe863bdf640 10 monclient: monmap: 2024-09-16T04:41:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:41:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:41:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:41:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:41:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:41:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:41:10.281 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.266+0000 7fe863bdf640 1 Processor -- start 2024-09-16T04:41:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.266+0000 7fe863bdf640 1 -- start start 2024-09-16T04:41:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.266+0000 7fe863bdf640 10 monclient: init 2024-09-16T04:41:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.266+0000 7fe863bdf640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.266+0000 7fe863bdf640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:41:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.266+0000 7fe863bdf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe85c0a3c10 0x7fe85c133d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.266+0000 7fe863bdf640 10 monclient(hunting): picked mon.noname-a con 0x7fe85c0a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:41:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.266+0000 7fe863bdf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe85c0a3ff0 0x7fe85c1342a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.266+0000 7fe863bdf640 10 monclient(hunting): picked mon.noname-b con 0x7fe85c0a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:41:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.266+0000 7fe863bdf640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe85c13be90 0x7fe85c1cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.266+0000 7fe863bdf640 10 monclient(hunting): picked mon.noname-c con 0x7fe85c13be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.266+0000 7fe863bdf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.266+0000 7fe863bdf640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe85c06e550 con 0x7fe85c0a3c10 2024-09-16T04:41:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.266+0000 7fe863bdf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.266+0000 7fe863bdf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe85c06e3d0 con 0x7fe85c0a3ff0 2024-09-16T04:41:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.266+0000 7fe863bdf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.266+0000 7fe863bdf640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe85c13e7c0 con 0x7fe85c13be90 2024-09-16T04:41:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.266+0000 7fe863bdf640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.266+0000 7fe863bdf640 10 monclient(hunting): authenticate will time out at 868.682957s 2024-09-16T04:41:10.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86197b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe85c0a3c10 0x7fe85c133d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:10.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86217c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe85c13be90 0x7fe85c1cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:10.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86197b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe85c0a3c10 0x7fe85c133d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39554/0 (socket says 172.21.15.46:39554) 2024-09-16T04:41:10.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86197b640 1 -- 172.21.15.46:0/2964411971 learned_addr learned my addr 172.21.15.46:0/2964411971 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:10.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86217c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe85c13be90 0x7fe85c1cb940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:48498/0 (socket says 172.21.15.46:48498) 2024-09-16T04:41:10.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86197b640 10 monclient(hunting): get_auth_request con 0x7fe85c0a3c10 auth_method 0 2024-09-16T04:41:10.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86197b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:10.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86197b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:10.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86197b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:10.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86217c640 10 monclient(hunting): get_auth_request con 0x7fe85c13be90 auth_method 0 2024-09-16T04:41:10.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86217c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:10.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86217c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:10.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86217c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:10.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86197b640 1 -- 172.21.15.46:0/2964411971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe85c0a3c10 msgr2=0x7fe85c133d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:41:10.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86197b640 1 -- 172.21.15.46:0/2964411971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe85c0a3c10 msgr2=0x7fe85c133d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:41:10.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86197b640 1 --2- 172.21.15.46:0/2964411971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe85c0a3c10 0x7fe85c133d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:41:10.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86217c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:10.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86217c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:10.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86217c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:10.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86197b640 1 --2- 172.21.15.46:0/2964411971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe85c0a3c10 0x7fe85c133d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:41:10.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86117a640 1 --2- 172.21.15.46:0/2964411971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe85c0a3ff0 0x7fe85c1342a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:10.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86117a640 10 monclient(hunting): get_auth_request con 0x7fe85c0a3ff0 auth_method 0 2024-09-16T04:41:10.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86117a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:10.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86117a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:10.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86117a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:10.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86217c640 10 monclient(hunting): handle_auth_done global_id 24305 payload 824 2024-09-16T04:41:10.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86217c640 1 -- 172.21.15.46:0/2964411971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe85c0a3ff0 msgr2=0x7fe85c1342a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:10.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86217c640 1 --2- 172.21.15.46:0/2964411971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe85c0a3ff0 0x7fe85c1342a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:10.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86217c640 1 -- 172.21.15.46:0/2964411971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe85c0a3c10 msgr2=0x7fe85c133d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:41:10.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86217c640 1 --2- 172.21.15.46:0/2964411971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe85c0a3c10 0x7fe85c133d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:10.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86217c640 10 monclient: _finish_hunting 0 2024-09-16T04:41:10.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86217c640 1 monclient: found mon.noname-c 2024-09-16T04:41:10.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86217c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:10.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86217c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:41:10.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86217c640 1 -- 172.21.15.46:0/2964411971 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe85c1347e0 con 0x7fe85c13be90 2024-09-16T04:41:10.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe86217c640 1 --2- 172.21.15.46:0/2964411971 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe85c13be90 0x7fe85c1cb940 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe854031000 tx=0x7fe854004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:10.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe8527fc640 1 -- 172.21.15.46:0/2964411971 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe854040070 con 0x7fe85c13be90 2024-09-16T04:41:10.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe8527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:10.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe8527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:41:10.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe8527fc640 10 monclient: dump: 2024-09-16T04:41:10.285 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:10.285 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:10.286 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:10.286 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:10.286 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:10.286 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:10.286 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:10.286 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:10.286 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:10.286 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:10.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe8527fc640 10 monclient: _finish_auth 0 2024-09-16T04:41:10.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe8527fc640 10 monclient: _check_auth_tickets 2024-09-16T04:41:10.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe8527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:10.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe8527fc640 1 -- 172.21.15.46:0/2964411971 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe854031c00 con 0x7fe85c13be90 2024-09-16T04:41:10.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe8527fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:10.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe8527fc640 1 -- 172.21.15.46:0/2964411971 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe854009450 con 0x7fe85c13be90 2024-09-16T04:41:10.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe863bdf640 5 monclient: authenticate success, global_id 24305 2024-09-16T04:41:10.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe863bdf640 10 monclient: _renew_subs 2024-09-16T04:41:10.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe863bdf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:10.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe863bdf640 1 -- 172.21.15.46:0/2964411971 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe85c139de0 con 0x7fe85c13be90 2024-09-16T04:41:10.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe8527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:10.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe853fff640 4 set_mon_vals no callback set 2024-09-16T04:41:10.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe8527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:10.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe8527fc640 10 monclient: dump: 2024-09-16T04:41:10.287 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:10.287 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:10.287 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:10.287 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:10.287 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:10.287 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:10.287 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:10.287 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:10.288 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:10.288 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:10.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe863bdf640 10 monclient: _renew_subs 2024-09-16T04:41:10.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe863bdf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:10.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.267+0000 7fe863bdf640 1 -- 172.21.15.46:0/2964411971 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe85c13c8c0 con 0x7fe85c13be90 2024-09-16T04:41:10.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.268+0000 7fe8527fc640 1 -- 172.21.15.46:0/2964411971 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fe854039040 con 0x7fe85c13be90 2024-09-16T04:41:10.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.268+0000 7fe863bdf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:10.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.268+0000 7fe863bdf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:10.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.268+0000 7fe863bdf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:10.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.268+0000 7fe8527fc640 1 --2- 172.21.15.46:0/2964411971 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe8400383b0 0x7fe84003a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:10.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.268+0000 7fe863bdf640 1 -- 172.21.15.46:0/2964411971 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe85c0a4400 con 0x7fe85c13be90 2024-09-16T04:41:10.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.269+0000 7fe8527fc640 1 -- 172.21.15.46:0/2964411971 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(102..102 src has 1..102) ==== 4738+0+0 (secure 0 0 0) 0x7fe85406d0b0 con 0x7fe85c13be90 2024-09-16T04:41:10.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.269+0000 7fe86197b640 1 --2- 172.21.15.46:0/2964411971 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe8400383b0 0x7fe84003a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:10.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.269+0000 7fe86197b640 10 monclient: get_auth_request con 0x7fe8400383b0 auth_method 0 2024-09-16T04:41:10.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.269+0000 7fe86197b640 1 --2- 172.21.15.46:0/2964411971 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe8400383b0 0x7fe84003a870 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe858007d30 tx=0x7fe858009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:10.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.274+0000 7fe8527fc640 1 -- 172.21.15.46:0/2964411971 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe85403b570 con 0x7fe85c13be90 2024-09-16T04:41:10.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.274+0000 7fe8527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:10.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.274+0000 7fe8527fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:41:10.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.519+0000 7fe863bdf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:41:10.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.519+0000 7fe863bdf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:41:10.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.519+0000 7fe863bdf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:10.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.519+0000 7fe863bdf640 1 -- 172.21.15.46:0/2964411971 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe85c069150 con 0x7fe85c13be90 2024-09-16T04:41:10.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.520+0000 7fe8527fc640 1 -- 172.21.15.46:0/2964411971 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fe854045c10 con 0x7fe85c13be90 2024-09-16T04:41:10.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.520+0000 7fe8527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:41:10.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.520+0000 7fe8527fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:41:10.522 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":22,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:41:10.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.522+0000 7fe863bdf640 1 -- 172.21.15.46:0/2964411971 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe8400383b0 msgr2=0x7fe84003a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:10.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.522+0000 7fe863bdf640 1 --2- 172.21.15.46:0/2964411971 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe8400383b0 0x7fe84003a870 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe858007d30 tx=0x7fe858009040 comp rx=0 tx=0).stop 2024-09-16T04:41:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.522+0000 7fe863bdf640 10 monclient: shutdown 2024-09-16T04:41:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.522+0000 7fe863bdf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.522+0000 7fe863bdf640 1 -- 172.21.15.46:0/2964411971 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe85c13be90 msgr2=0x7fe85c1cb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.522+0000 7fe863bdf640 1 --2- 172.21.15.46:0/2964411971 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe85c13be90 0x7fe85c1cb940 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe854031000 tx=0x7fe854004a40 comp rx=0 tx=0).stop 2024-09-16T04:41:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.522+0000 7fe863bdf640 1 -- 172.21.15.46:0/2964411971 shutdown_connections 2024-09-16T04:41:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.522+0000 7fe863bdf640 1 --2- 172.21.15.46:0/2964411971 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe85c13be90 0x7fe85c1cb940 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.522+0000 7fe863bdf640 1 --2- 172.21.15.46:0/2964411971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe85c0a3ff0 0x7fe85c1342a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.522+0000 7fe863bdf640 1 --2- 172.21.15.46:0/2964411971 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe8400383b0 0x7fe84003a870 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.522+0000 7fe863bdf640 1 --2- 172.21.15.46:0/2964411971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe85c0a3c10 0x7fe85c133d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.522+0000 7fe863bdf640 1 -- 172.21.15.46:0/2964411971 >> 172.21.15.46:0/2964411971 conn(0x7fe85c0a5d10 msgr2=0x7fe85c13b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.522+0000 7fe863bdf640 1 -- 172.21.15.46:0/2964411971 shutdown_connections 2024-09-16T04:41:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:10.522+0000 7fe863bdf640 1 -- 172.21.15.46:0/2964411971 wait complete. 2024-09-16T04:41:11.212 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:11.212 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36720]: starting. 2024-09-16T04:41:11.212 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36720]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-36683 2024-09-16T04:41:11.213 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36720]: created object 0... 2024-09-16T04:41:11.213 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36720]: created object 25... 2024-09-16T04:41:11.213 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36720]: created object 49... 2024-09-16T04:41:11.213 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36720]: finishing. 2024-09-16T04:41:11.213 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36720]: shutting down. 2024-09-16T04:41:11.225 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:11.225 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36722]: starting. 2024-09-16T04:41:11.225 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36722]: listing objects. 2024-09-16T04:41:11.225 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36722]: listed object 0... 2024-09-16T04:41:11.225 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36722]: listed object 25... 2024-09-16T04:41:11.225 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36722]: saw 50 objects 2024-09-16T04:41:11.225 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36722]: shutting down. 2024-09-16T04:41:12.137 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:12.137 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36721]: starting. 2024-09-16T04:41:12.137 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36721]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-36683 2024-09-16T04:41:12.137 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36721]: shutting down. 2024-09-16T04:41:12.138 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:12.138 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:12.138 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:41:12.142 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:41:13.535 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:41:13.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.646+0000 7fd6f03e4640 10 monclient: get_monmap_and_config 2024-09-16T04:41:13.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.646+0000 7fd6f03e4640 10 monclient: build_initial_monmap 2024-09-16T04:41:13.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.646+0000 7fd6f03e4640 10 monclient: monmap: 2024-09-16T04:41:13.649 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:13.649 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:13.649 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:41:13.648207+0000 2024-09-16T04:41:13.649 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:41:13.648207+0000 2024-09-16T04:41:13.649 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:13.649 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:13.649 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:41:13.649 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:41:13.650 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:41:13.650 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:41:13.650 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:41:13.650 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:13.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.647+0000 7fd6f03e4640 1 Processor -- start 2024-09-16T04:41:13.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.647+0000 7fd6f03e4640 1 -- start start 2024-09-16T04:41:13.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.647+0000 7fd6f03e4640 10 monclient: init 2024-09-16T04:41:13.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6f03e4640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:13.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6f03e4640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-16T04:41:13.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6f03e4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd6e80a3c10 0x7fd6e809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:13.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6f03e4640 10 monclient(hunting): picked mon.noname-a con 0x7fd6e80a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:41:13.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6f03e4640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6e80a3ff0 0x7fd6e80a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:13.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6f03e4640 10 monclient(hunting): picked mon.noname-b con 0x7fd6e80a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:13.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6f03e4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6e81369d0 addr v1:172.21.15.114:6789/0 2024-09-16T04:41:13.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6f03e4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6e809b940 con 0x7fd6e81369d0 2024-09-16T04:41:13.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6f03e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:13.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6f03e4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd6e809ab00 con 0x7fd6e80a3c10 2024-09-16T04:41:13.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6f03e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:13.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6f03e4640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd6e809b030 con 0x7fd6e80a3ff0 2024-09-16T04:41:13.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6f03e4640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:13.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6f03e4640 10 monclient(hunting): authenticate will time out at 872.064257s 2024-09-16T04:41:13.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ee180640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd6e80a3c10 0x7fd6e809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:13.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ee180640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd6e80a3c10 0x7fd6e809bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39564/0 (socket says 172.21.15.46:39564) 2024-09-16T04:41:13.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ee180640 1 -- 172.21.15.46:0/2215661330 learned_addr learned my addr 172.21.15.46:0/2215661330 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:13.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ee180640 10 monclient(hunting): get_auth_request con 0x7fd6e80a3c10 auth_method 0 2024-09-16T04:41:13.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ee180640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:13.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ee180640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:13.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ee180640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:13.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ed97f640 1 --2- 172.21.15.46:0/2215661330 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6e80a3ff0 0x7fd6e80a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:13.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ed97f640 10 monclient(hunting): get_auth_request con 0x7fd6e80a3ff0 auth_method 0 2024-09-16T04:41:13.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ed97f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:13.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ed97f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:13.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ed97f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:13.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ee180640 1 -- 172.21.15.46:0/2215661330 >> v2:172.21.15.46:3300/0 conn(0x7fd6e80a3c10 msgr2=0x7fd6e809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:41:13.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ee180640 1 -- 172.21.15.46:0/2215661330 >> v2:172.21.15.46:3300/0 conn(0x7fd6e80a3c10 msgr2=0x7fd6e809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:41:13.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ee180640 1 --2- 172.21.15.46:0/2215661330 >> v2:172.21.15.46:3300/0 conn(0x7fd6e80a3c10 0x7fd6e809bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:41:13.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ed97f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:13.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ee180640 1 --2- 172.21.15.46:0/2215661330 >> v2:172.21.15.46:3300/0 conn(0x7fd6e80a3c10 0x7fd6e809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:41:13.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ed97f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:13.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ed97f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:13.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ed97f640 10 monclient(hunting): handle_auth_done global_id 24317 payload 274 2024-09-16T04:41:13.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ed97f640 1 -- 172.21.15.46:0/2215661330 >> v2:172.21.15.46:3300/0 conn(0x7fd6e80a3c10 msgr2=0x7fd6e809bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:41:13.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ed97f640 1 --2- 172.21.15.46:0/2215661330 >> v2:172.21.15.46:3300/0 conn(0x7fd6e80a3c10 0x7fd6e809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:13.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ed97f640 1 -- 172.21.15.46:0/2215661330 >> v1:172.21.15.114:6789/0 conn(0x7fd6e81369d0 legacy=0x7fd6e8136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:13.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ed97f640 10 monclient: _finish_hunting 0 2024-09-16T04:41:13.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ed97f640 1 monclient: found mon.noname-b 2024-09-16T04:41:13.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ed97f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ed97f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:41:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.648+0000 7fd6ed97f640 1 -- 172.21.15.46:0/2215661330 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e809b310 con 0x7fd6e80a3ff0 2024-09-16T04:41:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.649+0000 7fd6ed97f640 1 --2- 172.21.15.46:0/2215661330 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6e80a3ff0 0x7fd6e80a43d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc0097a0 tx=0x7fd6dc02a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32dc95ff1485902e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.649+0000 7fd6ed17e640 1 -- 172.21.15.46:0/2215661330 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc037070 con 0x7fd6e80a3ff0 2024-09-16T04:41:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.649+0000 7fd6ed17e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.649+0000 7fd6ed17e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:41:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.649+0000 7fd6ed17e640 10 monclient: dump: 2024-09-16T04:41:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:13.663 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:13.663 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:13.663 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:13.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.649+0000 7fd6ed17e640 10 monclient: _finish_auth 0 2024-09-16T04:41:13.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.649+0000 7fd6ed17e640 10 monclient: _check_auth_tickets 2024-09-16T04:41:13.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.649+0000 7fd6ed17e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:13.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.649+0000 7fd6ed17e640 1 -- 172.21.15.46:0/2215661330 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6dc02adb0 con 0x7fd6e80a3ff0 2024-09-16T04:41:13.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.649+0000 7fd6ed17e640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:13.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.649+0000 7fd6ed17e640 1 -- 172.21.15.46:0/2215661330 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc032660 con 0x7fd6e80a3ff0 2024-09-16T04:41:13.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.649+0000 7fd6f03e4640 5 monclient: authenticate success, global_id 24317 2024-09-16T04:41:13.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.649+0000 7fd6f03e4640 10 monclient: get_monmap_and_config success 2024-09-16T04:41:13.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.649+0000 7fd6f03e4640 4 set_mon_vals no callback set 2024-09-16T04:41:13.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.649+0000 7fd6ed17e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:13.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.649+0000 7fd6ed17e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:13.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.649+0000 7fd6ed17e640 10 monclient: dump: 2024-09-16T04:41:13.666 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:13.666 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:13.666 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:13.666 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:13.666 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:13.666 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:13.666 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:13.666 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:13.666 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:13.666 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:13.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.650+0000 7fd6f03e4640 10 monclient: shutdown 2024-09-16T04:41:13.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.650+0000 7fd6f03e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:13.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.650+0000 7fd6f03e4640 1 -- 172.21.15.46:0/2215661330 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6e80a3ff0 msgr2=0x7fd6e80a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:13.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.650+0000 7fd6f03e4640 1 --2- 172.21.15.46:0/2215661330 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6e80a3ff0 0x7fd6e80a43d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc0097a0 tx=0x7fd6dc02a060 comp rx=0 tx=0).stop 2024-09-16T04:41:13.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.650+0000 7fd6f03e4640 1 -- 172.21.15.46:0/2215661330 shutdown_connections 2024-09-16T04:41:13.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.650+0000 7fd6f03e4640 1 --2- 172.21.15.46:0/2215661330 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6e80a3ff0 0x7fd6e80a43d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:13.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.650+0000 7fd6f03e4640 1 --2- 172.21.15.46:0/2215661330 >> v2:172.21.15.46:3300/0 conn(0x7fd6e80a3c10 0x7fd6e809bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:13.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.650+0000 7fd6f03e4640 1 -- 172.21.15.46:0/2215661330 >> 172.21.15.46:0/2215661330 conn(0x7fd6e80a6340 msgr2=0x7fd6e806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:13.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.650+0000 7fd6f03e4640 1 -- 172.21.15.46:0/2215661330 shutdown_connections 2024-09-16T04:41:13.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.650+0000 7fd6f03e4640 1 -- 172.21.15.46:0/2215661330 wait complete. 2024-09-16T04:41:13.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.650+0000 7fd6f03e4640 10 monclient: build_initial_monmap 2024-09-16T04:41:13.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.651+0000 7fd6f03e4640 10 monclient: monmap: 2024-09-16T04:41:13.667 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:13.667 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:41:13.667 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:41:13.668 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:41:13.668 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:13.668 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:13.668 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:41:13.668 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:41:13.668 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:41:13.669 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:13.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.651+0000 7fd6f03e4640 1 Processor -- start 2024-09-16T04:41:13.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.651+0000 7fd6f03e4640 1 -- start start 2024-09-16T04:41:13.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.651+0000 7fd6f03e4640 10 monclient: init 2024-09-16T04:41:13.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.651+0000 7fd6f03e4640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:13.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.651+0000 7fd6f03e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:41:13.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.651+0000 7fd6f03e4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6e80a3c10 0x7fd6e81cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:13.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.651+0000 7fd6f03e4640 10 monclient(hunting): picked mon.noname-b con 0x7fd6e80a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:41:13.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.651+0000 7fd6f03e4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e80a3ff0 0x7fd6e81cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:13.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.651+0000 7fd6f03e4640 10 monclient(hunting): picked mon.noname-a con 0x7fd6e80a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:41:13.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.651+0000 7fd6f03e4640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6e81369d0 0x7fd6e81232d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:13.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.651+0000 7fd6f03e4640 10 monclient(hunting): picked mon.noname-c con 0x7fd6e81369d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:13.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.651+0000 7fd6f03e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:13.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.651+0000 7fd6f03e4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e80a4910 con 0x7fd6e80a3ff0 2024-09-16T04:41:13.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.651+0000 7fd6f03e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:13.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.651+0000 7fd6f03e4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e80a69c0 con 0x7fd6e80a3c10 2024-09-16T04:41:13.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.651+0000 7fd6f03e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:13.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.651+0000 7fd6f03e4640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd6e8127260 con 0x7fd6e81369d0 2024-09-16T04:41:13.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.651+0000 7fd6f03e4640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:13.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.651+0000 7fd6f03e4640 10 monclient(hunting): authenticate will time out at 872.067949s 2024-09-16T04:41:13.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ed97f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e80a3ff0 0x7fd6e81cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:13.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee981640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6e81369d0 0x7fd6e81232d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:13.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ed97f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e80a3ff0 0x7fd6e81cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39580/0 (socket says 172.21.15.46:39580) 2024-09-16T04:41:13.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ed97f640 1 -- 172.21.15.46:0/3952419151 learned_addr learned my addr 172.21.15.46:0/3952419151 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:13.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee981640 10 monclient(hunting): get_auth_request con 0x7fd6e81369d0 auth_method 0 2024-09-16T04:41:13.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee981640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:13.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee981640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:13.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee981640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:13.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ed97f640 10 monclient(hunting): get_auth_request con 0x7fd6e80a3ff0 auth_method 0 2024-09-16T04:41:13.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ed97f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:13.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ed97f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:13.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ed97f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:13.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee180640 1 --2- 172.21.15.46:0/3952419151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6e80a3c10 0x7fd6e81cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:13.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee180640 10 monclient(hunting): get_auth_request con 0x7fd6e80a3c10 auth_method 0 2024-09-16T04:41:13.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee180640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:13.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee180640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:13.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee180640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:13.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee981640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:13.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee981640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:13.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ed97f640 1 -- 172.21.15.46:0/3952419151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e80a3ff0 msgr2=0x7fd6e81cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:41:13.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ed97f640 1 -- 172.21.15.46:0/3952419151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e80a3ff0 msgr2=0x7fd6e81cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:41:13.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ed97f640 1 --2- 172.21.15.46:0/3952419151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e80a3ff0 0x7fd6e81cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:41:13.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee981640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:13.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ed97f640 1 --2- 172.21.15.46:0/3952419151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e80a3ff0 0x7fd6e81cc420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:41:13.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee180640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:13.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee180640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:13.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee180640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:13.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee981640 10 monclient(hunting): handle_auth_done global_id 24320 payload 824 2024-09-16T04:41:13.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee981640 1 -- 172.21.15.46:0/3952419151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6e80a3c10 msgr2=0x7fd6e81cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:13.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee981640 1 --2- 172.21.15.46:0/3952419151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6e80a3c10 0x7fd6e81cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:13.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee981640 1 -- 172.21.15.46:0/3952419151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e80a3ff0 msgr2=0x7fd6e81cc420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:41:13.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee981640 1 --2- 172.21.15.46:0/3952419151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e80a3ff0 0x7fd6e81cc420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:13.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee981640 10 monclient: _finish_hunting 0 2024-09-16T04:41:13.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee981640 1 monclient: found mon.noname-c 2024-09-16T04:41:13.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee981640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee981640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:41:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee981640 1 -- 172.21.15.46:0/3952419151 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e8126ca0 con 0x7fd6e81369d0 2024-09-16T04:41:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ee981640 1 --2- 172.21.15.46:0/3952419151 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6e81369d0 0x7fd6e81232d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd6e000c730 tx=0x7fd6e000cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6daffd640 1 -- 172.21.15.46:0/3952419151 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0003b70 con 0x7fd6e81369d0 2024-09-16T04:41:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:41:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6daffd640 10 monclient: dump: 2024-09-16T04:41:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:13.675 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:13.675 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:13.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6daffd640 10 monclient: _finish_auth 0 2024-09-16T04:41:13.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6daffd640 10 monclient: _check_auth_tickets 2024-09-16T04:41:13.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:13.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6daffd640 1 -- 172.21.15.46:0/3952419151 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6e0003d10 con 0x7fd6e81369d0 2024-09-16T04:41:13.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6daffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:13.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6daffd640 1 -- 172.21.15.46:0/3952419151 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0005310 con 0x7fd6e81369d0 2024-09-16T04:41:13.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:13.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:13.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6daffd640 10 monclient: dump: 2024-09-16T04:41:13.675 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:13.675 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:13.675 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:13.676 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6f03e4640 5 monclient: authenticate success, global_id 24320 2024-09-16T04:41:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6f03e4640 10 monclient: _renew_subs 2024-09-16T04:41:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6f03e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6f03e4640 1 -- 172.21.15.46:0/3952419151 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6e8125ad0 con 0x7fd6e81369d0 2024-09-16T04:41:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6ec97d640 4 set_mon_vals no callback set 2024-09-16T04:41:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6f03e4640 10 monclient: _renew_subs 2024-09-16T04:41:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6f03e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.652+0000 7fd6f03e4640 1 -- 172.21.15.46:0/3952419151 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6e81253f0 con 0x7fd6e81369d0 2024-09-16T04:41:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.653+0000 7fd6f03e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.653+0000 7fd6f03e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.653+0000 7fd6f03e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.653+0000 7fd6daffd640 1 -- 172.21.15.46:0/3952419151 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fd6e00054b0 con 0x7fd6e81369d0 2024-09-16T04:41:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.653+0000 7fd6f03e4640 1 -- 172.21.15.46:0/3952419151 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6e809bd70 con 0x7fd6e81369d0 2024-09-16T04:41:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.653+0000 7fd6daffd640 1 --2- 172.21.15.46:0/3952419151 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd6cc0383b0 0x7fd6cc03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.654+0000 7fd6daffd640 1 -- 172.21.15.46:0/3952419151 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(105..105 src has 1..105) ==== 4738+0+0 (secure 0 0 0) 0x7fd6e004b480 con 0x7fd6e81369d0 2024-09-16T04:41:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.654+0000 7fd6ee180640 1 --2- 172.21.15.46:0/3952419151 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd6cc0383b0 0x7fd6cc03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.654+0000 7fd6ee180640 10 monclient: get_auth_request con 0x7fd6cc0383b0 auth_method 0 2024-09-16T04:41:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.654+0000 7fd6ee180640 1 --2- 172.21.15.46:0/3952419151 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd6cc0383b0 0x7fd6cc03a870 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4009100 tx=0x7fd6e4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.658+0000 7fd6daffd640 1 -- 172.21.15.46:0/3952419151 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd6e000adf0 con 0x7fd6e81369d0 2024-09-16T04:41:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.658+0000 7fd6daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.658+0000 7fd6daffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:41:13.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.901+0000 7fd6f03e4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:41:13.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.901+0000 7fd6f03e4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:41:13.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.901+0000 7fd6f03e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:13.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.901+0000 7fd6f03e4640 1 -- 172.21.15.46:0/3952419151 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd6e8067e20 con 0x7fd6e81369d0 2024-09-16T04:41:13.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.902+0000 7fd6daffd640 1 -- 172.21.15.46:0/3952419151 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fd6e0011540 con 0x7fd6e81369d0 2024-09-16T04:41:13.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.902+0000 7fd6daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:41:13.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.902+0000 7fd6daffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:41:13.905 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":22,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":25,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:41:13.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.904+0000 7fd6f03e4640 1 -- 172.21.15.46:0/3952419151 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd6cc0383b0 msgr2=0x7fd6cc03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:13.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.904+0000 7fd6f03e4640 1 --2- 172.21.15.46:0/3952419151 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd6cc0383b0 0x7fd6cc03a870 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4009100 tx=0x7fd6e4008040 comp rx=0 tx=0).stop 2024-09-16T04:41:13.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.904+0000 7fd6f03e4640 10 monclient: shutdown 2024-09-16T04:41:13.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.904+0000 7fd6f03e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:13.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.904+0000 7fd6f03e4640 1 -- 172.21.15.46:0/3952419151 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6e81369d0 msgr2=0x7fd6e81232d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:13.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.904+0000 7fd6f03e4640 1 --2- 172.21.15.46:0/3952419151 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6e81369d0 0x7fd6e81232d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd6e000c730 tx=0x7fd6e000cbf0 comp rx=0 tx=0).stop 2024-09-16T04:41:13.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.904+0000 7fd6f03e4640 1 -- 172.21.15.46:0/3952419151 shutdown_connections 2024-09-16T04:41:13.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.904+0000 7fd6f03e4640 1 --2- 172.21.15.46:0/3952419151 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6e81369d0 0x7fd6e81232d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:13.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.904+0000 7fd6f03e4640 1 --2- 172.21.15.46:0/3952419151 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd6cc0383b0 0x7fd6cc03a870 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:13.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.904+0000 7fd6f03e4640 1 --2- 172.21.15.46:0/3952419151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e80a3ff0 0x7fd6e81cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:13.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.904+0000 7fd6f03e4640 1 --2- 172.21.15.46:0/3952419151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6e80a3c10 0x7fd6e81cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:13.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.904+0000 7fd6f03e4640 1 -- 172.21.15.46:0/3952419151 >> 172.21.15.46:0/3952419151 conn(0x7fd6e80a6340 msgr2=0x7fd6e809f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:13.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.904+0000 7fd6f03e4640 1 -- 172.21.15.46:0/3952419151 shutdown_connections 2024-09-16T04:41:13.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:13.904+0000 7fd6f03e4640 1 -- 172.21.15.46:0/3952419151 wait complete. 2024-09-16T04:41:14.216 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36791]: starting. 2024-09-16T04:41:14.216 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36791]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-36790 2024-09-16T04:41:14.216 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36791]: created object 0... 2024-09-16T04:41:14.216 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36791]: created object 25... 2024-09-16T04:41:14.216 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36791]: created object 49... 2024-09-16T04:41:14.216 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36791]: finishing. 2024-09-16T04:41:14.216 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36791]: shutting down. 2024-09-16T04:41:15.145 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36792]: starting. 2024-09-16T04:41:15.145 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36792]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-36790 2024-09-16T04:41:15.145 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36792]: shutting down. 2024-09-16T04:41:16.622 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:16.622 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36827]: starting. 2024-09-16T04:41:16.622 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36827]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-36790 2024-09-16T04:41:16.622 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36827]: created object 0... 2024-09-16T04:41:16.622 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36827]: created object 25... 2024-09-16T04:41:16.622 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36827]: created object 49... 2024-09-16T04:41:16.622 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36827]: finishing. 2024-09-16T04:41:16.622 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36827]: shutting down. 2024-09-16T04:41:16.634 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:16.634 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36829]: starting. 2024-09-16T04:41:16.635 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36829]: listing objects. 2024-09-16T04:41:16.635 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36829]: listed object 0... 2024-09-16T04:41:16.635 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36829]: listed object 25... 2024-09-16T04:41:16.635 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36829]: saw 50 objects 2024-09-16T04:41:16.635 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36829]: shutting down. 2024-09-16T04:41:16.917 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:41:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.026+0000 7f318fe77640 10 monclient: get_monmap_and_config 2024-09-16T04:41:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.026+0000 7f318fe77640 10 monclient: build_initial_monmap 2024-09-16T04:41:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.027+0000 7f318fe77640 10 monclient: monmap: 2024-09-16T04:41:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:41:17.028484+0000 2024-09-16T04:41:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:41:17.028484+0000 2024-09-16T04:41:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:41:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:41:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:41:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:41:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:41:17.030 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.027+0000 7f318fe77640 1 Processor -- start 2024-09-16T04:41:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.027+0000 7f318fe77640 1 -- start start 2024-09-16T04:41:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.027+0000 7f318fe77640 10 monclient: init 2024-09-16T04:41:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318fe77640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318fe77640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-16T04:41:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318fe77640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f31880a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:41:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318fe77640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31880a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:41:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318fe77640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f318813cea0 0x7f318813f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318fe77640 10 monclient(hunting): picked mon.noname-b con 0x7f318813cea0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318fe77640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f318809b940 con 0x7f31880a3ff0 2024-09-16T04:41:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318fe77640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f318809ab00 con 0x7f31880a3c10 2024-09-16T04:41:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318fe77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318fe77640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f318809b030 con 0x7f318813cea0 2024-09-16T04:41:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318fe77640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318fe77640 10 monclient(hunting): authenticate will time out at 875.444396s 2024-09-16T04:41:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318e414640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f318813cea0 0x7f318813f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318d412640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f31880a3ff0 0x7f31880a43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36010/0 (socket says 172.21.15.46:36010) 2024-09-16T04:41:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318d412640 1 -- 172.21.15.46:0/641415993 learned_addr learned my addr 172.21.15.46:0/641415993 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:41:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318e414640 10 monclient(hunting): get_auth_request con 0x7f318813cea0 auth_method 0 2024-09-16T04:41:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318e414640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318e414640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318e414640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318e414640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318e414640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318e414640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318e414640 10 monclient(hunting): handle_auth_done global_id 24338 payload 274 2024-09-16T04:41:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318cc11640 1 -- 172.21.15.46:0/641415993 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 937340675 0 0) 0x7f318809b940 con 0x7f31880a3ff0 2024-09-16T04:41:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318e414640 1 -- 172.21.15.46:0/641415993 >> v1:172.21.15.114:6789/0 conn(0x7f31880a3c10 legacy=0x7f318809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318e414640 1 -- 172.21.15.46:0/641415993 >> v1:172.21.15.46:6789/0 conn(0x7f31880a3ff0 legacy=0x7f31880a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318e414640 10 monclient: _finish_hunting 0 2024-09-16T04:41:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318e414640 1 monclient: found mon.noname-b 2024-09-16T04:41:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318e414640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318e414640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:41:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318e414640 1 -- 172.21.15.46:0/641415993 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f318809b310 con 0x7f318813cea0 2024-09-16T04:41:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.028+0000 7f318cc11640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:41:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.029+0000 7f318e414640 1 --2- 172.21.15.46:0/641415993 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f318813cea0 0x7f318813f290 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3180009750 tx=0x7f318002a010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b7f7cc5efb4e89 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:17.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.029+0000 7f318cc11640 1 -- 172.21.15.46:0/641415993 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3180037070 con 0x7f318813cea0 2024-09-16T04:41:17.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.029+0000 7f318cc11640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.029+0000 7f318cc11640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:41:17.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.029+0000 7f318cc11640 10 monclient: dump: 2024-09-16T04:41:17.036 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.036 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.036 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.036 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.036 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.036 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.036 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.036 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.036 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.037 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.029+0000 7f318cc11640 10 monclient: _finish_auth 0 2024-09-16T04:41:17.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.029+0000 7f318cc11640 10 monclient: _check_auth_tickets 2024-09-16T04:41:17.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.029+0000 7f318cc11640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:17.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.029+0000 7f318fe77640 5 monclient: authenticate success, global_id 24338 2024-09-16T04:41:17.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.029+0000 7f318fe77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:41:17.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.029+0000 7f318cc11640 1 -- 172.21.15.46:0/641415993 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f318002ad20 con 0x7f318813cea0 2024-09-16T04:41:17.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.029+0000 7f318cc11640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:17.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.029+0000 7f318fe77640 10 monclient: get_monmap_and_config success 2024-09-16T04:41:17.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.029+0000 7f318cc11640 1 -- 172.21.15.46:0/641415993 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3180032660 con 0x7f318813cea0 2024-09-16T04:41:17.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.029+0000 7f318cc11640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.029+0000 7f318fe77640 4 set_mon_vals no callback set 2024-09-16T04:41:17.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.029+0000 7f318cc11640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:17.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.029+0000 7f318cc11640 10 monclient: dump: 2024-09-16T04:41:17.038 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.038 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.038 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.038 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.038 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.038 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.038 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.038 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.038 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.038 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.030+0000 7f318fe77640 10 monclient: shutdown 2024-09-16T04:41:17.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.030+0000 7f318fe77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:17.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.030+0000 7f318fe77640 1 -- 172.21.15.46:0/641415993 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f318813cea0 msgr2=0x7f318813f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.030+0000 7f318fe77640 1 --2- 172.21.15.46:0/641415993 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f318813cea0 0x7f318813f290 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3180009750 tx=0x7f318002a010 comp rx=0 tx=0).stop 2024-09-16T04:41:17.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.030+0000 7f318fe77640 1 -- 172.21.15.46:0/641415993 shutdown_connections 2024-09-16T04:41:17.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.030+0000 7f318fe77640 1 --2- 172.21.15.46:0/641415993 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f318813cea0 0x7f318813f290 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.030+0000 7f318fe77640 1 -- 172.21.15.46:0/641415993 >> 172.21.15.46:0/641415993 conn(0x7f31880a6340 msgr2=0x7f318806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:17.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.030+0000 7f318fe77640 1 -- 172.21.15.46:0/641415993 shutdown_connections 2024-09-16T04:41:17.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.030+0000 7f318fe77640 1 -- 172.21.15.46:0/641415993 wait complete. 2024-09-16T04:41:17.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.031+0000 7f318fe77640 10 monclient: build_initial_monmap 2024-09-16T04:41:17.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.031+0000 7f318fe77640 10 monclient: monmap: 2024-09-16T04:41:17.039 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:17.039 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:41:17.039 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:41:17.039 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:41:17.039 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:17.039 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.040 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:41:17.040 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:41:17.040 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:41:17.040 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.031+0000 7f318fe77640 1 Processor -- start 2024-09-16T04:41:17.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.031+0000 7f318fe77640 1 -- start start 2024-09-16T04:41:17.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.031+0000 7f318fe77640 10 monclient: init 2024-09-16T04:41:17.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.031+0000 7f318fe77640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:17.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.031+0000 7f318fe77640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:41:17.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.031+0000 7f318fe77640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31880a3c10 0x7f31881cbb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.031+0000 7f318fe77640 10 monclient(hunting): picked mon.noname-b con 0x7f31880a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:41:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.031+0000 7f318fe77640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31880a3ff0 0x7f3188136de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318fe77640 10 monclient(hunting): picked mon.noname-a con 0x7f31880a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:41:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318fe77640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f318813cea0 0x7f3188139e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318fe77640 10 monclient(hunting): picked mon.noname-c con 0x7f318813cea0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318fe77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318fe77640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f318813f950 con 0x7f31880a3ff0 2024-09-16T04:41:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318fe77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318fe77640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f318813f7d0 con 0x7f31880a3c10 2024-09-16T04:41:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318fe77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318fe77640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f318813fad0 con 0x7f318813cea0 2024-09-16T04:41:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318fe77640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318fe77640 10 monclient(hunting): authenticate will time out at 875.448054s 2024-09-16T04:41:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318d412640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31880a3ff0 0x7f3188136de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318e414640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f318813cea0 0x7f3188139e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318d412640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31880a3ff0 0x7f3188136de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60600/0 (socket says 172.21.15.46:60600) 2024-09-16T04:41:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318d412640 1 -- 172.21.15.46:0/491651338 learned_addr learned my addr 172.21.15.46:0/491651338 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318d412640 10 monclient(hunting): get_auth_request con 0x7f31880a3ff0 auth_method 0 2024-09-16T04:41:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318d412640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318d412640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318d412640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318e414640 10 monclient(hunting): get_auth_request con 0x7f318813cea0 auth_method 0 2024-09-16T04:41:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318e414640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318e414640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318e414640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318d412640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318d412640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318d412640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318e414640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318e414640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318e414640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318dc13640 1 --2- 172.21.15.46:0/491651338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31880a3c10 0x7f31881cbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318dc13640 10 monclient(hunting): get_auth_request con 0x7f31880a3c10 auth_method 0 2024-09-16T04:41:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318dc13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318dc13640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318dc13640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318d412640 10 monclient(hunting): handle_auth_done global_id 34101 payload 824 2024-09-16T04:41:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318d412640 1 -- 172.21.15.46:0/491651338 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f318813cea0 msgr2=0x7f3188139e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318d412640 1 --2- 172.21.15.46:0/491651338 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f318813cea0 0x7f3188139e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318d412640 1 -- 172.21.15.46:0/491651338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31880a3c10 msgr2=0x7f31881cbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318d412640 1 --2- 172.21.15.46:0/491651338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31880a3c10 0x7f31881cbb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318d412640 10 monclient: _finish_hunting 0 2024-09-16T04:41:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318d412640 1 monclient: found mon.noname-a 2024-09-16T04:41:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318d412640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318d412640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:41:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318d412640 1 -- 172.21.15.46:0/491651338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f318813c390 con 0x7f31880a3ff0 2024-09-16T04:41:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318dc13640 1 --2- 172.21.15.46:0/491651338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31880a3c10 0x7f31881cbb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:41:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318d412640 1 --2- 172.21.15.46:0/491651338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31880a3ff0 0x7f3188136de0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f317800c4a0 tx=0x7f317800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f317e7fc640 1 -- 172.21.15.46:0/491651338 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3178018070 con 0x7f31880a3ff0 2024-09-16T04:41:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f317e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f317e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:41:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f317e7fc640 10 monclient: dump: 2024-09-16T04:41:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.049 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.049 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.049 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.049 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.049 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.049 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f317e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:41:17.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f317e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:41:17.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f317e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:17.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f317e7fc640 1 -- 172.21.15.46:0/491651338 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31780074e0 con 0x7f31880a3ff0 2024-09-16T04:41:17.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318fe77640 5 monclient: authenticate success, global_id 34101 2024-09-16T04:41:17.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318fe77640 10 monclient: _renew_subs 2024-09-16T04:41:17.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318fe77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:41:17.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318fe77640 1 -- 172.21.15.46:0/491651338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3188120760 con 0x7f31880a3ff0 2024-09-16T04:41:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f317e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f317e7fc640 1 -- 172.21.15.46:0/491651338 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3178004a20 con 0x7f31880a3ff0 2024-09-16T04:41:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f317e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f317e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:41:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f317e7fc640 10 monclient: dump: 2024-09-16T04:41:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.051 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f317f7fe640 4 set_mon_vals no callback set 2024-09-16T04:41:17.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318fe77640 10 monclient: _renew_subs 2024-09-16T04:41:17.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318fe77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:41:17.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.032+0000 7f318fe77640 1 -- 172.21.15.46:0/491651338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f318813d280 con 0x7f31880a3ff0 2024-09-16T04:41:17.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.033+0000 7f317e7fc640 1 -- 172.21.15.46:0/491651338 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f317801b070 con 0x7f31880a3ff0 2024-09-16T04:41:17.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.033+0000 7f318fe77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:17.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.033+0000 7f318fe77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:17.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.033+0000 7f318fe77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:41:17.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.033+0000 7f317e7fc640 1 --2- 172.21.15.46:0/491651338 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f316c0383b0 0x7f316c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.033+0000 7f318fe77640 1 -- 172.21.15.46:0/491651338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f318806ecc0 con 0x7f31880a3ff0 2024-09-16T04:41:17.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.034+0000 7f317e7fc640 1 -- 172.21.15.46:0/491651338 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 4738+0+0 (secure 0 0 0) 0x7f317804aa50 con 0x7f31880a3ff0 2024-09-16T04:41:17.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.034+0000 7f318dc13640 1 --2- 172.21.15.46:0/491651338 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f316c0383b0 0x7f316c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.034+0000 7f318dc13640 10 monclient: get_auth_request con 0x7f316c0383b0 auth_method 0 2024-09-16T04:41:17.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.034+0000 7f318dc13640 1 --2- 172.21.15.46:0/491651338 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f316c0383b0 0x7f316c03a870 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3184007fd0 tx=0x7f3184009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:17.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.038+0000 7f317e7fc640 1 -- 172.21.15.46:0/491651338 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f318806ecc0 con 0x7f31880a3ff0 2024-09-16T04:41:17.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.038+0000 7f317e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:17.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.038+0000 7f317e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:41:17.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.285+0000 7f318fe77640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:41:17.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.285+0000 7f318fe77640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:41:17.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.285+0000 7f318fe77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:41:17.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.285+0000 7f318fe77640 1 -- 172.21.15.46:0/491651338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f318809cbe0 con 0x7f31880a3ff0 2024-09-16T04:41:17.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.286+0000 7f317e7fc640 1 -- 172.21.15.46:0/491651338 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f318809cbe0 con 0x7f31880a3ff0 2024-09-16T04:41:17.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.286+0000 7f317e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:41:17.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.286+0000 7f317e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:41:17.289 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:41:17.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.288+0000 7f318fe77640 1 -- 172.21.15.46:0/491651338 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f316c0383b0 msgr2=0x7f316c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.288+0000 7f318fe77640 1 --2- 172.21.15.46:0/491651338 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f316c0383b0 0x7f316c03a870 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3184007fd0 tx=0x7f3184009040 comp rx=0 tx=0).stop 2024-09-16T04:41:17.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.288+0000 7f318fe77640 10 monclient: shutdown 2024-09-16T04:41:17.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.288+0000 7f318fe77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:17.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.288+0000 7f318fe77640 1 -- 172.21.15.46:0/491651338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31880a3ff0 msgr2=0x7f3188136de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.288+0000 7f318fe77640 1 --2- 172.21.15.46:0/491651338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31880a3ff0 0x7f3188136de0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f317800c4a0 tx=0x7f317800c960 comp rx=0 tx=0).stop 2024-09-16T04:41:17.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.288+0000 7f318fe77640 1 -- 172.21.15.46:0/491651338 shutdown_connections 2024-09-16T04:41:17.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.288+0000 7f318fe77640 1 --2- 172.21.15.46:0/491651338 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f318813cea0 0x7f3188139e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.288+0000 7f318fe77640 1 --2- 172.21.15.46:0/491651338 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f316c0383b0 0x7f316c03a870 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.288+0000 7f318fe77640 1 --2- 172.21.15.46:0/491651338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31880a3ff0 0x7f3188136de0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.288+0000 7f318fe77640 1 --2- 172.21.15.46:0/491651338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31880a3c10 0x7f31881cbb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.288+0000 7f318fe77640 1 -- 172.21.15.46:0/491651338 >> 172.21.15.46:0/491651338 conn(0x7f31880a6340 msgr2=0x7f31880a2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:17.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.288+0000 7f318fe77640 1 -- 172.21.15.46:0/491651338 shutdown_connections 2024-09-16T04:41:17.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.288+0000 7f318fe77640 1 -- 172.21.15.46:0/491651338 wait complete. 2024-09-16T04:41:17.300 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:41:17.301 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:41:17.301 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:41:17.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.401+0000 7f4864082640 10 monclient: get_monmap_and_config 2024-09-16T04:41:17.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.401+0000 7f4864082640 10 monclient: build_initial_monmap 2024-09-16T04:41:17.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.401+0000 7f4864082640 10 monclient: monmap: 2024-09-16T04:41:17.404 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:17.404 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.404 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:41:17.402970+0000 2024-09-16T04:41:17.404 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:41:17.402970+0000 2024-09-16T04:41:17.404 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:17.404 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.404 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:41:17.404 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:41:17.404 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:41:17.405 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:41:17.405 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:41:17.405 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4864082640 1 Processor -- start 2024-09-16T04:41:17.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4864082640 1 -- start start 2024-09-16T04:41:17.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4864082640 10 monclient: init 2024-09-16T04:41:17.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4864082640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:17.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4864082640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-16T04:41:17.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4864082640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f485c0a3c10 0x7f485c09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4864082640 10 monclient(hunting): picked mon.noname-a con 0x7f485c0a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:41:17.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4864082640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f485c0a3ff0 0x7f485c0a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4864082640 10 monclient(hunting): picked mon.noname-b con 0x7f485c0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:17.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4864082640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f485c1369d0 addr v1:172.21.15.46:6789/0 2024-09-16T04:41:17.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4864082640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f485c09b940 con 0x7f485c1369d0 2024-09-16T04:41:17.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4864082640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4864082640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f485c09ab00 con 0x7f485c0a3c10 2024-09-16T04:41:17.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4864082640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4864082640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f485c09b030 con 0x7f485c0a3ff0 2024-09-16T04:41:17.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4864082640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:17.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4864082640 10 monclient(hunting): authenticate will time out at 875.818807s 2024-09-16T04:41:17.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f486161d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f485c0a3ff0 0x7f485c0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4861e1e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f485c0a3c10 0x7f485c09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f486161d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f485c0a3ff0 0x7f485c0a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38044/0 (socket says 172.21.15.46:38044) 2024-09-16T04:41:17.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f486261f640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f485c1369d0 0x7f485c136e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36024/0 (socket says 172.21.15.46:36024) 2024-09-16T04:41:17.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f486161d640 1 -- 172.21.15.46:0/4210663978 learned_addr learned my addr 172.21.15.46:0/4210663978 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:17.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4861e1e640 10 monclient(hunting): get_auth_request con 0x7f485c0a3c10 auth_method 0 2024-09-16T04:41:17.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4861e1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4861e1e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f4861e1e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f486161d640 10 monclient(hunting): get_auth_request con 0x7f485c0a3ff0 auth_method 0 2024-09-16T04:41:17.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f486161d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f486161d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.402+0000 7f486161d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f4861e1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f4861e1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f4861e1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f486161d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f486161d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f486161d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f4860e1c640 1 -- 172.21.15.46:0/4210663978 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 505763800 0 0) 0x7f485c09b940 con 0x7f485c1369d0 2024-09-16T04:41:17.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f4860e1c640 10 monclient(hunting): _negotiate 2024-09-16T04:41:17.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f4860e1c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f4860e1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f4860e1c640 10 monclient(hunting): my global_id is 34107 2024-09-16T04:41:17.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f4860e1c640 1 -- 172.21.15.46:0/4210663978 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4848003520 con 0x7f485c1369d0 2024-09-16T04:41:17.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f486161d640 10 monclient(hunting): handle_auth_done global_id 24344 payload 274 2024-09-16T04:41:17.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f486161d640 1 -- 172.21.15.46:0/4210663978 >> v2:172.21.15.46:3300/0 conn(0x7f485c0a3c10 msgr2=0x7f485c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:41:17.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f486161d640 1 --2- 172.21.15.46:0/4210663978 >> v2:172.21.15.46:3300/0 conn(0x7f485c0a3c10 0x7f485c09bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f486161d640 1 -- 172.21.15.46:0/4210663978 >> v1:172.21.15.46:6789/0 conn(0x7f485c1369d0 legacy=0x7f485c136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f486161d640 10 monclient: _finish_hunting 0 2024-09-16T04:41:17.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f486161d640 1 monclient: found mon.noname-b 2024-09-16T04:41:17.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f486161d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:17.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f486161d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:41:17.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f486161d640 1 -- 172.21.15.46:0/4210663978 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f485c09b310 con 0x7f485c0a3ff0 2024-09-16T04:41:17.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.403+0000 7f4861e1e640 1 --2- 172.21.15.46:0/4210663978 >> v2:172.21.15.46:3300/0 conn(0x7f485c0a3c10 0x7f485c09bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:41:17.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f486161d640 1 --2- 172.21.15.46:0/4210663978 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f485c0a3ff0 0x7f485c0a43d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f484c009870 tx=0x7f484c02a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e09eb9bca6f0de87 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:17.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4860e1c640 1 -- 172.21.15.46:0/4210663978 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484c037070 con 0x7f485c0a3ff0 2024-09-16T04:41:17.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4860e1c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4860e1c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:41:17.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4860e1c640 10 monclient: dump: 2024-09-16T04:41:17.420 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.420 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.420 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.420 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.420 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.420 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.420 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.420 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.420 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.420 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4860e1c640 10 monclient: _finish_auth 0 2024-09-16T04:41:17.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4860e1c640 10 monclient: _check_auth_tickets 2024-09-16T04:41:17.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4860e1c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:17.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4860e1c640 1 -- 172.21.15.46:0/4210663978 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f484c032440 con 0x7f485c0a3ff0 2024-09-16T04:41:17.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4864082640 5 monclient: authenticate success, global_id 24344 2024-09-16T04:41:17.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4864082640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:41:17.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4860e1c640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:17.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4864082640 10 monclient: get_monmap_and_config success 2024-09-16T04:41:17.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4864082640 4 set_mon_vals no callback set 2024-09-16T04:41:17.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4860e1c640 1 -- 172.21.15.46:0/4210663978 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484c032810 con 0x7f485c0a3ff0 2024-09-16T04:41:17.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4860e1c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4860e1c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:17.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4860e1c640 10 monclient: dump: 2024-09-16T04:41:17.422 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.422 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.422 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.422 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.422 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.422 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.422 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.422 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.422 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.422 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4864082640 10 monclient: shutdown 2024-09-16T04:41:17.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4864082640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:17.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4864082640 1 -- 172.21.15.46:0/4210663978 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f485c0a3ff0 msgr2=0x7f485c0a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4864082640 1 --2- 172.21.15.46:0/4210663978 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f485c0a3ff0 0x7f485c0a43d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f484c009870 tx=0x7f484c02a130 comp rx=0 tx=0).stop 2024-09-16T04:41:17.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4864082640 1 -- 172.21.15.46:0/4210663978 shutdown_connections 2024-09-16T04:41:17.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4864082640 1 --2- 172.21.15.46:0/4210663978 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f485c0a3ff0 0x7f485c0a43d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4864082640 1 --2- 172.21.15.46:0/4210663978 >> v2:172.21.15.46:3300/0 conn(0x7f485c0a3c10 0x7f485c09bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4864082640 1 -- 172.21.15.46:0/4210663978 >> 172.21.15.46:0/4210663978 conn(0x7f485c0a6340 msgr2=0x7f485c06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:17.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4864082640 1 -- 172.21.15.46:0/4210663978 shutdown_connections 2024-09-16T04:41:17.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.404+0000 7f4864082640 1 -- 172.21.15.46:0/4210663978 wait complete. 2024-09-16T04:41:17.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.405+0000 7f4864082640 10 monclient: build_initial_monmap 2024-09-16T04:41:17.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.405+0000 7f4864082640 10 monclient: monmap: 2024-09-16T04:41:17.423 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:17.423 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:41:17.424 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:41:17.424 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:41:17.424 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:17.424 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.424 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:41:17.424 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:41:17.424 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:41:17.424 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.405+0000 7f4864082640 1 Processor -- start 2024-09-16T04:41:17.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.405+0000 7f4864082640 1 -- start start 2024-09-16T04:41:17.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.405+0000 7f4864082640 10 monclient: init 2024-09-16T04:41:17.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f4864082640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:17.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f4864082640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:41:17.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f4864082640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f485c0a3c10 0x7f485c1cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f4864082640 10 monclient(hunting): picked mon.noname-b con 0x7f485c0a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:41:17.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f4864082640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f485c0a3ff0 0x7f485c1cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f4864082640 10 monclient(hunting): picked mon.noname-a con 0x7f485c0a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:41:17.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f4864082640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f485c1369d0 0x7f485c123210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f4864082640 10 monclient(hunting): picked mon.noname-c con 0x7f485c1369d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:17.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f4864082640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f4864082640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f485c0a4910 con 0x7f485c0a3ff0 2024-09-16T04:41:17.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f4864082640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f4864082640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f485c0a69c0 con 0x7f485c0a3c10 2024-09-16T04:41:17.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f4864082640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f4864082640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f485c1265d0 con 0x7f485c1369d0 2024-09-16T04:41:17.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f4864082640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:17.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f4864082640 10 monclient(hunting): authenticate will time out at 875.822328s 2024-09-16T04:41:17.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486161d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f485c0a3ff0 0x7f485c1cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486161d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f485c0a3ff0 0x7f485c1cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60614/0 (socket says 172.21.15.46:60614) 2024-09-16T04:41:17.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486161d640 1 -- 172.21.15.46:0/3188786306 learned_addr learned my addr 172.21.15.46:0/3188786306 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:17.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486161d640 10 monclient(hunting): get_auth_request con 0x7f485c0a3ff0 auth_method 0 2024-09-16T04:41:17.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486161d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486161d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486161d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486261f640 1 --2- 172.21.15.46:0/3188786306 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f485c1369d0 0x7f485c123210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486261f640 10 monclient(hunting): get_auth_request con 0x7f485c1369d0 auth_method 0 2024-09-16T04:41:17.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486261f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486261f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486261f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486161d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486161d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486161d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486261f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486261f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486261f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f4861e1e640 1 --2- 172.21.15.46:0/3188786306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f485c0a3c10 0x7f485c1cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f4861e1e640 10 monclient(hunting): get_auth_request con 0x7f485c0a3c10 auth_method 0 2024-09-16T04:41:17.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f4861e1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f4861e1e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f4861e1e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486161d640 10 monclient(hunting): handle_auth_done global_id 34110 payload 824 2024-09-16T04:41:17.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486161d640 1 -- 172.21.15.46:0/3188786306 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f485c1369d0 msgr2=0x7f485c123210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486161d640 1 --2- 172.21.15.46:0/3188786306 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f485c1369d0 0x7f485c123210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486161d640 1 -- 172.21.15.46:0/3188786306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f485c0a3c10 msgr2=0x7f485c1cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486161d640 1 --2- 172.21.15.46:0/3188786306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f485c0a3c10 0x7f485c1cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486161d640 10 monclient: _finish_hunting 0 2024-09-16T04:41:17.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486161d640 1 monclient: found mon.noname-a 2024-09-16T04:41:17.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486161d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:17.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486161d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:41:17.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486161d640 1 -- 172.21.15.46:0/3188786306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f485c125a10 con 0x7f485c0a3ff0 2024-09-16T04:41:17.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486261f640 1 --2- 172.21.15.46:0/3188786306 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f485c1369d0 0x7f485c123210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:41:17.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.406+0000 7f486161d640 1 --2- 172.21.15.46:0/3188786306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f485c0a3ff0 0x7f485c1cc420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f484c000c00 tx=0x7f484c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:17.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f48527fc640 1 -- 172.21.15.46:0/3188786306 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484c041020 con 0x7f485c0a3ff0 2024-09-16T04:41:17.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f48527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f48527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:41:17.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f48527fc640 10 monclient: dump: 2024-09-16T04:41:17.429 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.429 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.429 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.429 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.429 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.430 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.430 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.430 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.430 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.430 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f48527fc640 10 monclient: _finish_auth 0 2024-09-16T04:41:17.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f48527fc640 10 monclient: _check_auth_tickets 2024-09-16T04:41:17.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f48527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:17.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f48527fc640 1 -- 172.21.15.46:0/3188786306 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f484c03d620 con 0x7f485c0a3ff0 2024-09-16T04:41:17.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f4864082640 5 monclient: authenticate success, global_id 34110 2024-09-16T04:41:17.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f4864082640 10 monclient: _renew_subs 2024-09-16T04:41:17.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f4864082640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:41:17.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f4864082640 1 -- 172.21.15.46:0/3188786306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f485c1271a0 con 0x7f485c0a3ff0 2024-09-16T04:41:17.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f48527fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:17.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f48527fc640 1 -- 172.21.15.46:0/3188786306 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484c037040 con 0x7f485c0a3ff0 2024-09-16T04:41:17.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f48527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f48527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:41:17.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f48527fc640 10 monclient: dump: 2024-09-16T04:41:17.431 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.431 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.432 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f4864082640 10 monclient: _renew_subs 2024-09-16T04:41:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f4864082640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:41:17.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f4864082640 1 -- 172.21.15.46:0/3188786306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f485c126ec0 con 0x7f485c0a3ff0 2024-09-16T04:41:17.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f4853fff640 4 set_mon_vals no callback set 2024-09-16T04:41:17.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f4864082640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:17.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f4864082640 1 --2- 172.21.15.46:0/3188786306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f485c11f860 0x7f485c11fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f4864082640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f4864082640 1 -- 172.21.15.46:0/3188786306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f485c0a2900 con 0x7f485c11f860 2024-09-16T04:41:17.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.407+0000 7f48527fc640 1 -- 172.21.15.46:0/3188786306 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f484c044020 con 0x7f485c0a3ff0 2024-09-16T04:41:17.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f4861e1e640 1 --2- 172.21.15.46:0/3188786306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f485c11f860 0x7f485c11fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f4861e1e640 10 monclient: get_auth_request con 0x7f485c11f860 auth_method 0 2024-09-16T04:41:17.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f4861e1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f4861e1e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f4861e1e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f4861e1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f4861e1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f4861e1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f48527fc640 1 --2- 172.21.15.46:0/3188786306 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f48400383b0 0x7f484003a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f4861e1e640 10 monclient(hunting): handle_auth_done global_id 34113 payload 824 2024-09-16T04:41:17.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f4861e1e640 1 -- 172.21.15.46:0/3188786306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f485c0a84d0 con 0x7f485c11f860 2024-09-16T04:41:17.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f48527fc640 1 -- 172.21.15.46:0/3188786306 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 4738+0+0 (secure 0 0 0) 0x7f484c06be50 con 0x7f485c0a3ff0 2024-09-16T04:41:17.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f4861e1e640 1 --2- 172.21.15.46:0/3188786306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f485c11f860 0x7f485c11fc40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4858011590 tx=0x7f4858011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:17.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f486261f640 1 --2- 172.21.15.46:0/3188786306 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f48400383b0 0x7f484003a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f486261f640 10 monclient: get_auth_request con 0x7f48400383b0 auth_method 0 2024-09-16T04:41:17.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f48527fc640 1 -- 172.21.15.46:0/3188786306 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f485800f910 con 0x7f485c11f860 2024-09-16T04:41:17.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f48527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f48527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:41:17.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f48527fc640 10 monclient: dump: 2024-09-16T04:41:17.435 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.435 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.435 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.435 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.435 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.435 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.436 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f48527fc640 1 -- 172.21.15.46:0/3188786306 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f485800f910 con 0x7f485c11f860 2024-09-16T04:41:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f48527fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f48527fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:41:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f48527fc640 1 -- 172.21.15.46:0/3188786306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f485c11f860 msgr2=0x7f485c11fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f48527fc640 1 --2- 172.21.15.46:0/3188786306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f485c11f860 0x7f485c11fc40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4858011590 tx=0x7f4858011a50 comp rx=0 tx=0).stop 2024-09-16T04:41:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f48527fc640 1 -- 172.21.15.46:0/3188786306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f485c11f860 msgr2=0x7f485c11fc40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:41:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f48527fc640 1 --2- 172.21.15.46:0/3188786306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f485c11f860 0x7f485c11fc40 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4858011590 tx=0x7f4858011a50 comp rx=0 tx=0).stop 2024-09-16T04:41:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.408+0000 7f486261f640 1 --2- 172.21.15.46:0/3188786306 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f48400383b0 0x7f484003a870 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4854007e20 tx=0x7f4854007d70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:17.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.414+0000 7f4864082640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:41:17.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.414+0000 7f4864082640 1 --2- 172.21.15.46:0/3188786306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f485c120180 0x7f485c123750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f4864082640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f4864082640 1 -- 172.21.15.46:0/3188786306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f485c0a4a90 con 0x7f485c120180 2024-09-16T04:41:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f4861e1e640 1 --2- 172.21.15.46:0/3188786306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f485c120180 0x7f485c123750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f4861e1e640 10 monclient: get_auth_request con 0x7f485c120180 auth_method 0 2024-09-16T04:41:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f4861e1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f4861e1e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f4861e1e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f4861e1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f4861e1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f4861e1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f4861e1e640 10 monclient(hunting): handle_auth_done global_id 34116 payload 824 2024-09-16T04:41:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f4861e1e640 1 -- 172.21.15.46:0/3188786306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f485c09bcf0 con 0x7f485c120180 2024-09-16T04:41:17.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f4861e1e640 1 --2- 172.21.15.46:0/3188786306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f485c120180 0x7f485c123750 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4858011590 tx=0x7f48580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:17.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f48527fc640 1 -- 172.21.15.46:0/3188786306 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4858006300 con 0x7f485c120180 2024-09-16T04:41:17.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f48527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f48527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:41:17.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f48527fc640 10 monclient: dump: 2024-09-16T04:41:17.440 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.440 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.440 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.440 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.440 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.440 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.440 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.441 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.441 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.441 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f48527fc640 1 -- 172.21.15.46:0/3188786306 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f4858006300 con 0x7f485c120180 2024-09-16T04:41:17.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f48527fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:41:17.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f48527fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:41:17.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f48527fc640 1 -- 172.21.15.46:0/3188786306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f485c120180 msgr2=0x7f485c123750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f48527fc640 1 --2- 172.21.15.46:0/3188786306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f485c120180 0x7f485c123750 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4858011590 tx=0x7f48580046c0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f48527fc640 1 -- 172.21.15.46:0/3188786306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f485c120180 msgr2=0x7f485c123750 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:41:17.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.415+0000 7f48527fc640 1 --2- 172.21.15.46:0/3188786306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f485c120180 0x7f485c123750 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4858011590 tx=0x7f48580046c0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.442 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:41:17.442 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:41:17.442 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:41:17.442 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:41:17.442 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 26, 2024-09-16T04:41:17.442 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 7174, 2024-09-16T04:41:17.442 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:41:17.442 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:41:17.442 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:41:17.442 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:41:17.443 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:17.443 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:41:17.443 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:41:17.443 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:41:17.443 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:41:17.443 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:17.443 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:17.443 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:17.443 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:17.443 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:17.443 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:17.443 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:17.444 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:17.444 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:17.444 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:17.444 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:17.444 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:17.444 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:41:17.444 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:41:17.444 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:17.444 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:17.444 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:17.444 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:17.444 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:17.445 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:17.445 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:17.445 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:17.445 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:17.445 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:17.445 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:17.445 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:17.445 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.445 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:41:17.445 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:41:17.445 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:41:17.445 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:41:17.446 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:41:17.446 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:41:17.446 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:41:17.446 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:41:17.446 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:41:17.446 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:41:17.446 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:41:17.446 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:41:17.446 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:41:17.446 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:41:17.446 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:41:17.446 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:41:17.447 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:41:17.447 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:17.447 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:17.447 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:17.447 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:17.447 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:17.447 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:17.447 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:17.447 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:17.447 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:17.447 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:17.448 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:17.448 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:17.448 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:41:17.448 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.448 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:41:17.448 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.448 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:41:17.448 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:41:17.448 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:17.448 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:17.448 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.448 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:17.449 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:41:17.449 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:17.449 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.449 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.449 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:17.449 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:41:17.449 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:17.449 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:17.449 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:17.449 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.449 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:41:17.449 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:41:17.450 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:17.450 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:17.450 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:17.450 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.450 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.450 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:41:17.450 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:41:17.450 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:17.450 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:17.450 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.450 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:17.450 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:41:17.451 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:17.451 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.451 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.451 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:17.451 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:41:17.451 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:17.451 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:17.451 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:17.451 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.451 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:41:17.451 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:41:17.451 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:17.452 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:17.452 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:17.452 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.452 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.452 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:41:17.452 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:41:17.452 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:17.452 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:17.452 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.452 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:17.452 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:41:17.453 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:17.453 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.453 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.453 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:17.453 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:41:17.453 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:17.453 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:17.453 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:17.453 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.453 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:41:17.453 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:41:17.453 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:17.454 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:17.454 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:17.454 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:17.454 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:17.454 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.454 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:41:17.454 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:41:17.454 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.454 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:17.455 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:17.455 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:41:17.455 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:17.455 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:17.455 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:41:17.455 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.455 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:17.455 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:17.455 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:41:17.455 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:17.455 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:17.455 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.456 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:41:17.456 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:41:17.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.416+0000 7f4864082640 1 -- 172.21.15.46:0/3188786306 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f48400383b0 msgr2=0x7f484003a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.416+0000 7f4864082640 1 --2- 172.21.15.46:0/3188786306 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f48400383b0 0x7f484003a870 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4854007e20 tx=0x7f4854007d70 comp rx=0 tx=0).stop 2024-09-16T04:41:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.416+0000 7f4864082640 10 monclient: shutdown 2024-09-16T04:41:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.416+0000 7f4864082640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.416+0000 7f4864082640 1 -- 172.21.15.46:0/3188786306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f485c0a3ff0 msgr2=0x7f485c1cc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.416+0000 7f4864082640 1 --2- 172.21.15.46:0/3188786306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f485c0a3ff0 0x7f485c1cc420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f484c000c00 tx=0x7f484c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.416+0000 7f486261f640 1 -- 172.21.15.46:0/3188786306 reap_dead start 2024-09-16T04:41:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.416+0000 7f486261f640 1 -- 172.21.15.46:0/3188786306 reap_dead start 2024-09-16T04:41:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.416+0000 7f4864082640 1 -- 172.21.15.46:0/3188786306 shutdown_connections 2024-09-16T04:41:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.416+0000 7f4864082640 1 -- 172.21.15.46:0/3188786306 >> 172.21.15.46:0/3188786306 conn(0x7f485c0a6340 msgr2=0x7f485c0a1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.416+0000 7f4864082640 1 -- 172.21.15.46:0/3188786306 shutdown_connections 2024-09-16T04:41:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.416+0000 7f4864082640 1 -- 172.21.15.46:0/3188786306 wait complete. 2024-09-16T04:41:17.459 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:41:17.459 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:41:17.459 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:41:17.545 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:17.545 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36828]: starting. 2024-09-16T04:41:17.546 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36828]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-36790 2024-09-16T04:41:17.546 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36828]: shutting down. 2024-09-16T04:41:17.546 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:17.547 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:17.547 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:41:17.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:41:17.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.562+0000 7f4293ff8640 10 monclient: get_monmap_and_config 2024-09-16T04:41:17.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.562+0000 7f4293ff8640 10 monclient: build_initial_monmap 2024-09-16T04:41:17.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.562+0000 7f4293ff8640 10 monclient: monmap: 2024-09-16T04:41:17.564 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:17.564 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.564 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:41:17.563725+0000 2024-09-16T04:41:17.564 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:41:17.563725+0000 2024-09-16T04:41:17.565 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:17.565 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.565 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:41:17.565 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:41:17.565 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:41:17.565 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:41:17.565 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:41:17.565 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.562+0000 7f4293ff8640 1 Processor -- start 2024-09-16T04:41:17.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4293ff8640 1 -- start start 2024-09-16T04:41:17.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4293ff8640 10 monclient: init 2024-09-16T04:41:17.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4293ff8640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:17.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4293ff8640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-16T04:41:17.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4293ff8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f428c0a3c10 0x7f428c09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4293ff8640 10 monclient(hunting): picked mon.noname-a con 0x7f428c0a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:41:17.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4293ff8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f428c0a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:41:17.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4293ff8640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f428c13be90 0x7f428c13e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4293ff8640 10 monclient(hunting): picked mon.noname-c con 0x7f428c13be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:41:17.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4293ff8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f428c09ab70 con 0x7f428c0a3ff0 2024-09-16T04:41:17.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4293ff8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4293ff8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f428c0a4d60 con 0x7f428c0a3c10 2024-09-16T04:41:17.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4293ff8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4293ff8640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f428c09b030 con 0x7f428c13be90 2024-09-16T04:41:17.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4293ff8640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:17.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4293ff8640 10 monclient(hunting): authenticate will time out at 875.979635s 2024-09-16T04:41:17.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4291d94640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f428c0a3c10 0x7f428c09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4291d94640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f428c0a3c10 0x7f428c09bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60638/0 (socket says 172.21.15.46:60638) 2024-09-16T04:41:17.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4291d94640 1 -- 172.21.15.46:0/4130486617 learned_addr learned my addr 172.21.15.46:0/4130486617 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:17.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4291d94640 10 monclient(hunting): get_auth_request con 0x7f428c0a3c10 auth_method 0 2024-09-16T04:41:17.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4291d94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4291d94640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4291d94640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4291d94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4291d94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4291d94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4292595640 1 --2- 172.21.15.46:0/4130486617 >> v2:172.21.15.114:3300/0 conn(0x7f428c13be90 0x7f428c13e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4292595640 10 monclient(hunting): get_auth_request con 0x7f428c13be90 auth_method 0 2024-09-16T04:41:17.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4292595640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4292595640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.563+0000 7f4292595640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4291d94640 10 monclient(hunting): handle_auth_done global_id 34119 payload 274 2024-09-16T04:41:17.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4291d94640 1 -- 172.21.15.46:0/4130486617 >> v2:172.21.15.114:3300/0 conn(0x7f428c13be90 msgr2=0x7f428c13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:41:17.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4291d94640 1 --2- 172.21.15.46:0/4130486617 >> v2:172.21.15.114:3300/0 conn(0x7f428c13be90 0x7f428c13e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4291d94640 1 -- 172.21.15.46:0/4130486617 >> v1:172.21.15.114:6789/0 conn(0x7f428c0a3ff0 legacy=0x7f428c0a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4291d94640 10 monclient: _finish_hunting 0 2024-09-16T04:41:17.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4291d94640 1 monclient: found mon.noname-a 2024-09-16T04:41:17.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4291d94640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:17.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4291d94640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:41:17.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4291d94640 1 -- 172.21.15.46:0/4130486617 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f428c09b310 con 0x7f428c0a3c10 2024-09-16T04:41:17.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4292595640 1 --2- 172.21.15.46:0/4130486617 >> v2:172.21.15.114:3300/0 conn(0x7f428c13be90 0x7f428c13e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:41:17.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4291d94640 1 --2- 172.21.15.46:0/4130486617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f428c0a3c10 0x7f428c09bcf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f428c09b8d0 tx=0x7f428802a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ce8f0e52e9c8927 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:17.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4290d92640 1 -- 172.21.15.46:0/4130486617 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4288037070 con 0x7f428c0a3c10 2024-09-16T04:41:17.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4290d92640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4290d92640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:41:17.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4290d92640 10 monclient: dump: 2024-09-16T04:41:17.578 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.578 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.578 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.578 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.578 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.578 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.578 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.578 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.578 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.578 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4290d92640 10 monclient: _finish_auth 0 2024-09-16T04:41:17.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4290d92640 10 monclient: _check_auth_tickets 2024-09-16T04:41:17.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4290d92640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:17.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4290d92640 1 -- 172.21.15.46:0/4130486617 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4288032440 con 0x7f428c0a3c10 2024-09-16T04:41:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4293ff8640 5 monclient: authenticate success, global_id 34119 2024-09-16T04:41:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4290d92640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4293ff8640 10 monclient: get_monmap_and_config success 2024-09-16T04:41:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4290d92640 1 -- 172.21.15.46:0/4130486617 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4288032810 con 0x7f428c0a3c10 2024-09-16T04:41:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4290d92640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4293ff8640 4 set_mon_vals no callback set 2024-09-16T04:41:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4290d92640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:41:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.564+0000 7f4290d92640 10 monclient: dump: 2024-09-16T04:41:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.580 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.580 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.580 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.580 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.565+0000 7f4293ff8640 10 monclient: shutdown 2024-09-16T04:41:17.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.565+0000 7f4293ff8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:17.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.565+0000 7f4293ff8640 1 -- 172.21.15.46:0/4130486617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f428c0a3c10 msgr2=0x7f428c09bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.565+0000 7f4293ff8640 1 --2- 172.21.15.46:0/4130486617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f428c0a3c10 0x7f428c09bcf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f428c09b8d0 tx=0x7f428802a150 comp rx=0 tx=0).stop 2024-09-16T04:41:17.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.565+0000 7f4293ff8640 1 -- 172.21.15.46:0/4130486617 shutdown_connections 2024-09-16T04:41:17.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.565+0000 7f4293ff8640 1 --2- 172.21.15.46:0/4130486617 >> v2:172.21.15.114:3300/0 conn(0x7f428c13be90 0x7f428c13e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.565+0000 7f4293ff8640 1 --2- 172.21.15.46:0/4130486617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f428c0a3c10 0x7f428c09bcf0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.565+0000 7f4293ff8640 1 -- 172.21.15.46:0/4130486617 >> 172.21.15.46:0/4130486617 conn(0x7f428c0a5d10 msgr2=0x7f428c0a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:17.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.565+0000 7f4293ff8640 1 -- 172.21.15.46:0/4130486617 shutdown_connections 2024-09-16T04:41:17.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.565+0000 7f4293ff8640 1 -- 172.21.15.46:0/4130486617 wait complete. 2024-09-16T04:41:17.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.566+0000 7f4293ff8640 10 monclient: build_initial_monmap 2024-09-16T04:41:17.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.566+0000 7f4293ff8640 10 monclient: monmap: 2024-09-16T04:41:17.581 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:17.581 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:41:17.581 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:41:17.581 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:41:17.581 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:17.581 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.581 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:41:17.581 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:41:17.581 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:41:17.581 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.566+0000 7f4293ff8640 1 Processor -- start 2024-09-16T04:41:17.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.566+0000 7f4293ff8640 1 -- start start 2024-09-16T04:41:17.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.566+0000 7f4293ff8640 10 monclient: init 2024-09-16T04:41:17.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.566+0000 7f4293ff8640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:17.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.566+0000 7f4293ff8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:41:17.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.566+0000 7f4293ff8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f428c0a3c10 0x7f428c1cba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.566+0000 7f4293ff8640 10 monclient(hunting): picked mon.noname-b con 0x7f428c0a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:41:17.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.566+0000 7f4293ff8640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f428c0a3ff0 0x7f428c1cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.566+0000 7f4293ff8640 10 monclient(hunting): picked mon.noname-c con 0x7f428c0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:17.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.566+0000 7f4293ff8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f428c13be90 0x7f428c1cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.566+0000 7f4293ff8640 10 monclient(hunting): picked mon.noname-a con 0x7f428c13be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:41:17.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.566+0000 7f4293ff8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.566+0000 7f4293ff8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f428c06e550 con 0x7f428c13be90 2024-09-16T04:41:17.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.566+0000 7f4293ff8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.566+0000 7f4293ff8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f428c06e3d0 con 0x7f428c0a3c10 2024-09-16T04:41:17.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.566+0000 7f4293ff8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.566+0000 7f4293ff8640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f428c13e7c0 con 0x7f428c0a3ff0 2024-09-16T04:41:17.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.566+0000 7f4293ff8640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:17.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.566+0000 7f4293ff8640 10 monclient(hunting): authenticate will time out at 875.982982s 2024-09-16T04:41:17.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4292595640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f428c13be90 0x7f428c1cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291593640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f428c0a3ff0 0x7f428c1cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4292595640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f428c13be90 0x7f428c1cc4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60644/0 (socket says 172.21.15.46:60644) 2024-09-16T04:41:17.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4292595640 1 -- 172.21.15.46:0/3755732586 learned_addr learned my addr 172.21.15.46:0/3755732586 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:17.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4292595640 10 monclient(hunting): get_auth_request con 0x7f428c13be90 auth_method 0 2024-09-16T04:41:17.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4292595640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4292595640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4292595640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291593640 10 monclient(hunting): get_auth_request con 0x7f428c0a3ff0 auth_method 0 2024-09-16T04:41:17.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291593640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291593640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291593640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291d94640 1 --2- 172.21.15.46:0/3755732586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f428c0a3c10 0x7f428c1cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291d94640 10 monclient(hunting): get_auth_request con 0x7f428c0a3c10 auth_method 0 2024-09-16T04:41:17.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291d94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291d94640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291d94640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291593640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291593640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291593640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4292595640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4292595640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4292595640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291d94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291d94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291d94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291593640 10 monclient(hunting): handle_auth_done global_id 24350 payload 824 2024-09-16T04:41:17.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291593640 1 -- 172.21.15.46:0/3755732586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f428c0a3c10 msgr2=0x7f428c1cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291593640 1 --2- 172.21.15.46:0/3755732586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f428c0a3c10 0x7f428c1cba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291593640 1 -- 172.21.15.46:0/3755732586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f428c13be90 msgr2=0x7f428c1cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291593640 1 --2- 172.21.15.46:0/3755732586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f428c13be90 0x7f428c1cc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291593640 10 monclient: _finish_hunting 0 2024-09-16T04:41:17.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291593640 1 monclient: found mon.noname-c 2024-09-16T04:41:17.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291593640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:17.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291593640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:41:17.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291593640 1 -- 172.21.15.46:0/3755732586 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f428c13a450 con 0x7f428c0a3ff0 2024-09-16T04:41:17.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4292595640 1 --2- 172.21.15.46:0/3755732586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f428c13be90 0x7f428c1cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:41:17.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4291593640 1 --2- 172.21.15.46:0/3755732586 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f428c0a3ff0 0x7f428c1cbfb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f427c00c3c0 tx=0x7f427c00c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:17.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f42827fc640 1 -- 172.21.15.46:0/3755732586 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427c017070 con 0x7f428c0a3ff0 2024-09-16T04:41:17.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f42827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f42827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:41:17.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f42827fc640 10 monclient: dump: 2024-09-16T04:41:17.586 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.586 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.586 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.586 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.586 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.586 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.587 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.587 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.587 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.587 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f42827fc640 10 monclient: _finish_auth 0 2024-09-16T04:41:17.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f42827fc640 10 monclient: _check_auth_tickets 2024-09-16T04:41:17.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f42827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:17.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f42827fc640 1 -- 172.21.15.46:0/3755732586 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f427c0052d0 con 0x7f428c0a3ff0 2024-09-16T04:41:17.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f42827fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:17.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f42827fc640 1 -- 172.21.15.46:0/3755732586 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427c003830 con 0x7f428c0a3ff0 2024-09-16T04:41:17.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4293ff8640 5 monclient: authenticate success, global_id 24350 2024-09-16T04:41:17.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f42827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f42827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:17.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f42827fc640 10 monclient: dump: 2024-09-16T04:41:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.588 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4283fff640 4 set_mon_vals no callback set 2024-09-16T04:41:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4293ff8640 10 monclient: _renew_subs 2024-09-16T04:41:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4293ff8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4293ff8640 1 -- 172.21.15.46:0/3755732586 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f428c13a730 con 0x7f428c0a3ff0 2024-09-16T04:41:17.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4293ff8640 10 monclient: _renew_subs 2024-09-16T04:41:17.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4293ff8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:17.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.567+0000 7f4293ff8640 1 -- 172.21.15.46:0/3755732586 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f428c0a0400 con 0x7f428c0a3ff0 2024-09-16T04:41:17.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.568+0000 7f4293ff8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:17.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.568+0000 7f4293ff8640 1 --2- 172.21.15.46:0/3755732586 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f428c1339a0 0x7f428c133d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.568+0000 7f4293ff8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.568+0000 7f4293ff8640 1 -- 172.21.15.46:0/3755732586 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f428c09bcf0 con 0x7f428c1339a0 2024-09-16T04:41:17.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.568+0000 7f42827fc640 1 -- 172.21.15.46:0/3755732586 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f427c01a070 con 0x7f428c0a3ff0 2024-09-16T04:41:17.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.568+0000 7f4291d94640 1 --2- 172.21.15.46:0/3755732586 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f428c1339a0 0x7f428c133d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.568+0000 7f4291d94640 10 monclient: get_auth_request con 0x7f428c1339a0 auth_method 0 2024-09-16T04:41:17.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.568+0000 7f4291d94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.568+0000 7f4291d94640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.568+0000 7f4291d94640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.568+0000 7f4291d94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.568+0000 7f4291d94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.568+0000 7f4291d94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.569+0000 7f42827fc640 1 --2- 172.21.15.46:0/3755732586 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f42700383e0 0x7f427003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.569+0000 7f42827fc640 1 -- 172.21.15.46:0/3755732586 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(110..110 src has 1..110) ==== 4346+0+0 (secure 0 0 0) 0x7f427c04a990 con 0x7f428c0a3ff0 2024-09-16T04:41:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.569+0000 7f4291d94640 10 monclient(hunting): handle_auth_done global_id 24353 payload 824 2024-09-16T04:41:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.569+0000 7f4291d94640 1 -- 172.21.15.46:0/3755732586 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f428c138d20 con 0x7f428c1339a0 2024-09-16T04:41:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.569+0000 7f4292595640 1 --2- 172.21.15.46:0/3755732586 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f42700383e0 0x7f427003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.569+0000 7f4291d94640 1 --2- 172.21.15.46:0/3755732586 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f428c1339a0 0x7f428c133d80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f428c06edc0 tx=0x7f42880046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.569+0000 7f4292595640 10 monclient: get_auth_request con 0x7f42700383e0 auth_method 0 2024-09-16T04:41:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.569+0000 7f42827fc640 1 -- 172.21.15.46:0/3755732586 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4288043070 con 0x7f428c1339a0 2024-09-16T04:41:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.569+0000 7f42827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.569+0000 7f42827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.569+0000 7f42827fc640 10 monclient: dump: 2024-09-16T04:41:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.591 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.591 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.591 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.591 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.591 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.591 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.591 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.591 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.591 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.569+0000 7f42827fc640 1 -- 172.21.15.46:0/3755732586 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f428803c640 con 0x7f428c1339a0 2024-09-16T04:41:17.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.569+0000 7f42827fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:17.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.569+0000 7f42827fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:41:17.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.569+0000 7f42827fc640 1 -- 172.21.15.46:0/3755732586 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f428c1339a0 msgr2=0x7f428c133d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.569+0000 7f4292595640 1 --2- 172.21.15.46:0/3755732586 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f42700383e0 0x7f427003a8a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4284009fd0 tx=0x7f4284009860 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:17.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.569+0000 7f42827fc640 1 --2- 172.21.15.46:0/3755732586 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f428c1339a0 0x7f428c133d80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f428c06edc0 tx=0x7f42880046c0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.569+0000 7f42827fc640 1 -- 172.21.15.46:0/3755732586 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f428c1339a0 msgr2=0x7f428c133d80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:41:17.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.569+0000 7f42827fc640 1 --2- 172.21.15.46:0/3755732586 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f428c1339a0 0x7f428c133d80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.575+0000 7f4293ff8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:41:17.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.575+0000 7f4293ff8640 1 --2- 172.21.15.46:0/3755732586 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f428c1342c0 0x7f428c0a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.575+0000 7f4293ff8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.575+0000 7f4293ff8640 1 -- 172.21.15.46:0/3755732586 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f428c067e40 con 0x7f428c1342c0 2024-09-16T04:41:17.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.575+0000 7f4291d94640 1 --2- 172.21.15.46:0/3755732586 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f428c1342c0 0x7f428c0a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.575+0000 7f4291d94640 10 monclient: get_auth_request con 0x7f428c1342c0 auth_method 0 2024-09-16T04:41:17.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.575+0000 7f4291d94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.575+0000 7f4291d94640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.575+0000 7f4291d94640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.576+0000 7f4291d94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.576+0000 7f4291d94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.576+0000 7f4291d94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.576+0000 7f4291d94640 10 monclient(hunting): handle_auth_done global_id 24356 payload 824 2024-09-16T04:41:17.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.576+0000 7f4291d94640 1 -- 172.21.15.46:0/3755732586 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f428c1346d0 con 0x7f428c1342c0 2024-09-16T04:41:17.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.576+0000 7f4291d94640 1 --2- 172.21.15.46:0/3755732586 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f428c1342c0 0x7f428c0a05d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f428802a190 tx=0x7f4288008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:17.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.576+0000 7f42827fc640 1 -- 172.21.15.46:0/3755732586 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4288054060 con 0x7f428c1342c0 2024-09-16T04:41:17.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.576+0000 7f42827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.576+0000 7f42827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:17.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.576+0000 7f42827fc640 10 monclient: dump: 2024-09-16T04:41:17.595 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.595 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.595 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.595 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.595 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.595 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.595 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.596 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.596 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.596 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.576+0000 7f42827fc640 1 -- 172.21.15.46:0/3755732586 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f42880039b0 con 0x7f428c1342c0 2024-09-16T04:41:17.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.576+0000 7f42827fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:41:17.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.576+0000 7f42827fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:41:17.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.576+0000 7f42827fc640 1 -- 172.21.15.46:0/3755732586 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f428c1342c0 msgr2=0x7f428c0a05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.576+0000 7f42827fc640 1 --2- 172.21.15.46:0/3755732586 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f428c1342c0 0x7f428c0a05d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f428802a190 tx=0x7f4288008030 comp rx=0 tx=0).stop 2024-09-16T04:41:17.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.576+0000 7f42827fc640 1 -- 172.21.15.46:0/3755732586 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f428c1342c0 msgr2=0x7f428c0a05d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:41:17.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.576+0000 7f42827fc640 1 --2- 172.21.15.46:0/3755732586 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f428c1342c0 0x7f428c0a05d0 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f428802a190 tx=0x7f4288008030 comp rx=0 tx=0).stop 2024-09-16T04:41:17.596 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:41:17.597 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:41:17.597 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:41:17.597 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:41:17.597 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 26, 2024-09-16T04:41:17.597 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 52776, 2024-09-16T04:41:17.597 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:41:17.597 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:41:17.597 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:41:17.597 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:41:17.597 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:17.597 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 1, 2024-09-16T04:41:17.597 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:41:17.598 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:41:17.598 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:41:17.598 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:17.598 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:17.598 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:17.598 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:17.598 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:17.598 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:17.598 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:17.598 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:17.598 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:17.598 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:17.598 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:17.598 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:17.598 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:41:17.599 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:41:17.599 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:17.599 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:17.599 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:17.599 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:17.599 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:17.599 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:17.599 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:17.599 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:17.599 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:17.599 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:17.599 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:17.599 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:17.599 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.600 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:41:17.600 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:41:17.600 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:41:17.600 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:41:17.600 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:41:17.600 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:41:17.600 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:41:17.600 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:41:17.600 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:41:17.600 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:41:17.600 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:41:17.600 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:41:17.600 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:41:17.601 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:41:17.601 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:41:17.601 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:41:17.601 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:41:17.601 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:17.601 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:17.601 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:17.601 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:17.601 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:17.601 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:17.601 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:17.601 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:17.601 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:17.602 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:17.602 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:17.602 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:17.602 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:41:17.602 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.602 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:41:17.602 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.602 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:41:17.602 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:41:17.602 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:17.602 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:17.602 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.602 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:17.603 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:41:17.603 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:17.603 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.603 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.603 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:17.603 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:41:17.603 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:17.603 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:17.603 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:17.603 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.603 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:41:17.603 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:41:17.603 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:17.603 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:17.604 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:17.604 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.604 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.604 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:41:17.604 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:41:17.604 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:17.604 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:17.604 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.604 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:17.604 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:41:17.604 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:17.604 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.604 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.605 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:17.605 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:41:17.605 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:17.605 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:17.605 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:17.605 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.605 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:41:17.605 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:41:17.605 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:17.605 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:17.605 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:17.605 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.605 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.606 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:41:17.606 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:41:17.606 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:17.606 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:17.606 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.606 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:17.606 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:41:17.606 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:17.606 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.606 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.606 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:17.606 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:41:17.606 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:17.606 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:17.606 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:17.607 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.607 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:41:17.607 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:41:17.607 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:17.607 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:17.607 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:17.607 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:17.607 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:17.607 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.607 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:41:17.607 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:41:17.607 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.607 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:17.607 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:17.608 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:41:17.608 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:17.608 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:17.608 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:41:17.608 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.608 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:17.608 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:17.608 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:41:17.608 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:17.608 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:17.608 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:41:17.608 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.608 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:17.608 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:17.609 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:41:17.609 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:17.609 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:17.609 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.609 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:41:17.609 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:41:17.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.577+0000 7f4293ff8640 1 -- 172.21.15.46:0/3755732586 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f42700383e0 msgr2=0x7f427003a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.577+0000 7f4293ff8640 1 --2- 172.21.15.46:0/3755732586 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f42700383e0 0x7f427003a8a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4284009fd0 tx=0x7f4284009860 comp rx=0 tx=0).stop 2024-09-16T04:41:17.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.577+0000 7f4293ff8640 10 monclient: shutdown 2024-09-16T04:41:17.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.577+0000 7f4293ff8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:17.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.577+0000 7f4293ff8640 1 -- 172.21.15.46:0/3755732586 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f428c0a3ff0 msgr2=0x7f428c1cbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.577+0000 7f4293ff8640 1 --2- 172.21.15.46:0/3755732586 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f428c0a3ff0 0x7f428c1cbfb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f427c00c3c0 tx=0x7f427c00c880 comp rx=0 tx=0).stop 2024-09-16T04:41:17.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.577+0000 7f4292595640 1 -- 172.21.15.46:0/3755732586 reap_dead start 2024-09-16T04:41:17.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.577+0000 7f4293ff8640 1 -- 172.21.15.46:0/3755732586 shutdown_connections 2024-09-16T04:41:17.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.577+0000 7f4293ff8640 1 --2- 172.21.15.46:0/3755732586 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f428c0a3ff0 0x7f428c1cbfb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.577+0000 7f4293ff8640 1 -- 172.21.15.46:0/3755732586 >> 172.21.15.46:0/3755732586 conn(0x7f428c0a5d10 msgr2=0x7f428c0a70c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:17.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.577+0000 7f4293ff8640 1 -- 172.21.15.46:0/3755732586 shutdown_connections 2024-09-16T04:41:17.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.577+0000 7f4293ff8640 1 -- 172.21.15.46:0/3755732586 wait complete. 2024-09-16T04:41:17.611 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:41:17.611 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:41:17.612 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:41:17.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.710+0000 7f6081f31640 10 monclient: get_monmap_and_config 2024-09-16T04:41:17.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.710+0000 7f6081f31640 10 monclient: build_initial_monmap 2024-09-16T04:41:17.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.710+0000 7f6081f31640 10 monclient: monmap: 2024-09-16T04:41:17.713 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:17.713 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.713 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:41:17.712334+0000 2024-09-16T04:41:17.713 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:41:17.712334+0000 2024-09-16T04:41:17.714 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:17.714 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.714 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:41:17.714 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:41:17.714 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:41:17.714 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:41:17.714 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:41:17.714 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.711+0000 7f6081f31640 1 Processor -- start 2024-09-16T04:41:17.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.711+0000 7f6081f31640 1 -- start start 2024-09-16T04:41:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.711+0000 7f6081f31640 10 monclient: init 2024-09-16T04:41:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f6081f31640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f6081f31640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-16T04:41:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f6081f31640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f607c0a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:41:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f6081f31640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f607c0a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:41:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f6081f31640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f607c13cea0 0x7f607c13f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f6081f31640 10 monclient(hunting): picked mon.noname-c con 0x7f607c13cea0 addr v2:172.21.15.114:3300/0 2024-09-16T04:41:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f6081f31640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f607c09b940 con 0x7f607c0a3c10 2024-09-16T04:41:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f6081f31640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f607c09ab00 con 0x7f607c0a3ff0 2024-09-16T04:41:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f6081f31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f6081f31640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f607c09b030 con 0x7f607c13cea0 2024-09-16T04:41:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f6081f31640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:17.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f6081f31640 10 monclient(hunting): authenticate will time out at 876.128159s 2024-09-16T04:41:17.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f607c0a3c10 0x7f607c09bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36038/0 (socket says 172.21.15.46:36038) 2024-09-16T04:41:17.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607b7fe640 1 -- 172.21.15.46:0/3828036477 learned_addr learned my addr 172.21.15.46:0/3828036477 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:41:17.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607bfff640 1 --2- 172.21.15.46:0/3828036477 >> v2:172.21.15.114:3300/0 conn(0x7f607c13cea0 0x7f607c13f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607bfff640 10 monclient(hunting): get_auth_request con 0x7f607c13cea0 auth_method 0 2024-09-16T04:41:17.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607a7fc640 1 -- 172.21.15.46:0/3828036477 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3872147656 0 0) 0x7f607c09b940 con 0x7f607c0a3c10 2024-09-16T04:41:17.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:41:17.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607a7fc640 10 monclient(hunting): my global_id is 34137 2024-09-16T04:41:17.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607a7fc640 1 -- 172.21.15.46:0/3828036477 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6064003520 con 0x7f607c0a3c10 2024-09-16T04:41:17.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607a7fc640 1 -- 172.21.15.46:0/3828036477 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3332481133 0 0) 0x7f607c09ab00 con 0x7f607c0a3ff0 2024-09-16T04:41:17.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:41:17.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607a7fc640 10 monclient(hunting): my global_id is 5188 2024-09-16T04:41:17.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607a7fc640 1 -- 172.21.15.46:0/3828036477 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f607c09b940 con 0x7f607c0a3ff0 2024-09-16T04:41:17.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607bfff640 10 monclient(hunting): handle_auth_done global_id 5185 payload 274 2024-09-16T04:41:17.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607bfff640 1 -- 172.21.15.46:0/3828036477 >> v1:172.21.15.114:6789/0 conn(0x7f607c0a3ff0 legacy=0x7f607c0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607bfff640 1 -- 172.21.15.46:0/3828036477 >> v1:172.21.15.46:6789/0 conn(0x7f607c0a3c10 legacy=0x7f607c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:41:17.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607bfff640 1 monclient: found mon.noname-c 2024-09-16T04:41:17.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:17.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:41:17.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.712+0000 7f607bfff640 1 -- 172.21.15.46:0/3828036477 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f607c09b310 con 0x7f607c13cea0 2024-09-16T04:41:17.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.713+0000 7f607bfff640 1 --2- 172.21.15.46:0/3828036477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c13cea0 0x7f607c13f290 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6074009d90 tx=0x7f607402a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9379c201ef7cc5b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:17.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.713+0000 7f607a7fc640 1 -- 172.21.15.46:0/3828036477 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6074007740 con 0x7f607c13cea0 2024-09-16T04:41:17.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.713+0000 7f607a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.713+0000 7f607a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:41:17.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.713+0000 7f607a7fc640 10 monclient: dump: 2024-09-16T04:41:17.727 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.728 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.728 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.728 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.728 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.728 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.728 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.728 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.728 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.728 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.713+0000 7f607a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:41:17.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.713+0000 7f607a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:41:17.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.713+0000 7f607a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:17.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.713+0000 7f607a7fc640 1 -- 172.21.15.46:0/3828036477 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60740078e0 con 0x7f607c13cea0 2024-09-16T04:41:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.713+0000 7f6081f31640 5 monclient: authenticate success, global_id 5185 2024-09-16T04:41:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.713+0000 7f6081f31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:41:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.713+0000 7f607a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.713+0000 7f6081f31640 10 monclient: get_monmap_and_config success 2024-09-16T04:41:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.713+0000 7f607a7fc640 1 -- 172.21.15.46:0/3828036477 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6074007c90 con 0x7f607c13cea0 2024-09-16T04:41:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.713+0000 7f607a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.713+0000 7f6081f31640 4 set_mon_vals no callback set 2024-09-16T04:41:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.713+0000 7f607a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:41:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.713+0000 7f607a7fc640 10 monclient: dump: 2024-09-16T04:41:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.730 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.714+0000 7f6081f31640 10 monclient: shutdown 2024-09-16T04:41:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.714+0000 7f6081f31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.714+0000 7f6081f31640 1 -- 172.21.15.46:0/3828036477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c13cea0 msgr2=0x7f607c13f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.714+0000 7f6081f31640 1 --2- 172.21.15.46:0/3828036477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c13cea0 0x7f607c13f290 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6074009d90 tx=0x7f607402a660 comp rx=0 tx=0).stop 2024-09-16T04:41:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.714+0000 7f6081f31640 1 -- 172.21.15.46:0/3828036477 shutdown_connections 2024-09-16T04:41:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.714+0000 7f6081f31640 1 --2- 172.21.15.46:0/3828036477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c13cea0 0x7f607c13f290 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.714+0000 7f6081f31640 1 -- 172.21.15.46:0/3828036477 >> 172.21.15.46:0/3828036477 conn(0x7f607c0a6340 msgr2=0x7f607c06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.714+0000 7f6081f31640 1 -- 172.21.15.46:0/3828036477 shutdown_connections 2024-09-16T04:41:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.714+0000 7f6081f31640 1 -- 172.21.15.46:0/3828036477 wait complete. 2024-09-16T04:41:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.714+0000 7f6081f31640 10 monclient: build_initial_monmap 2024-09-16T04:41:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.714+0000 7f6081f31640 10 monclient: monmap: 2024-09-16T04:41:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:41:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:41:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:41:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:41:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:41:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:41:17.731 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.715+0000 7f6081f31640 1 Processor -- start 2024-09-16T04:41:17.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.715+0000 7f6081f31640 1 -- start start 2024-09-16T04:41:17.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.715+0000 7f6081f31640 10 monclient: init 2024-09-16T04:41:17.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.715+0000 7f6081f31640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:17.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.715+0000 7f6081f31640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:41:17.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.715+0000 7f6081f31640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c0a3c10 0x7f607c136920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.715+0000 7f6081f31640 10 monclient(hunting): picked mon.noname-b con 0x7f607c0a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:41:17.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.715+0000 7f6081f31640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f607c0a3ff0 0x7f607c139aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.715+0000 7f6081f31640 10 monclient(hunting): picked mon.noname-a con 0x7f607c0a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:41:17.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.715+0000 7f6081f31640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f607c13cea0 0x7f607c139fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.715+0000 7f6081f31640 10 monclient(hunting): picked mon.noname-c con 0x7f607c13cea0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:17.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.715+0000 7f6081f31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.715+0000 7f6081f31640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f607c13f950 con 0x7f607c0a3ff0 2024-09-16T04:41:17.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.715+0000 7f6081f31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.715+0000 7f6081f31640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f607c13f7d0 con 0x7f607c0a3c10 2024-09-16T04:41:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.715+0000 7f6081f31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.715+0000 7f6081f31640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f607c13fad0 con 0x7f607c13cea0 2024-09-16T04:41:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.715+0000 7f6081f31640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.715+0000 7f6081f31640 10 monclient(hunting): authenticate will time out at 876.131897s 2024-09-16T04:41:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.715+0000 7f607affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f607c0a3ff0 0x7f607c139aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607bfff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f607c13cea0 0x7f607c139fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f607c0a3ff0 0x7f607c139aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60658/0 (socket says 172.21.15.46:60658) 2024-09-16T04:41:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607affd640 1 -- 172.21.15.46:0/571802659 learned_addr learned my addr 172.21.15.46:0/571802659 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607affd640 10 monclient(hunting): get_auth_request con 0x7f607c0a3ff0 auth_method 0 2024-09-16T04:41:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607bfff640 10 monclient(hunting): get_auth_request con 0x7f607c13cea0 auth_method 0 2024-09-16T04:41:17.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607b7fe640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c0a3c10 0x7f607c136920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607b7fe640 10 monclient(hunting): get_auth_request con 0x7f607c0a3c10 auth_method 0 2024-09-16T04:41:17.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607affd640 10 monclient(hunting): handle_auth_done global_id 34140 payload 824 2024-09-16T04:41:17.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607affd640 1 -- 172.21.15.46:0/571802659 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f607c13cea0 msgr2=0x7f607c139fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607affd640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f607c13cea0 0x7f607c139fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607affd640 1 -- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c0a3c10 msgr2=0x7f607c136920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607affd640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c0a3c10 0x7f607c136920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607affd640 10 monclient: _finish_hunting 0 2024-09-16T04:41:17.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607affd640 1 monclient: found mon.noname-a 2024-09-16T04:41:17.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:17.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:41:17.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607affd640 1 -- 172.21.15.46:0/571802659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f607c120120 con 0x7f607c0a3ff0 2024-09-16T04:41:17.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607bfff640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f607c13cea0 0x7f607c139fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:41:17.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607b7fe640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c0a3c10 0x7f607c136920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:41:17.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f607affd640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f607c0a3ff0 0x7f607c139aa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f607000c500 tx=0x7f607000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:17.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f6063fff640 1 -- 172.21.15.46:0/571802659 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6070017070 con 0x7f607c0a3ff0 2024-09-16T04:41:17.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f6063fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f6063fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:41:17.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f6063fff640 10 monclient: dump: 2024-09-16T04:41:17.737 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.738 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.738 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.738 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.738 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.738 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.738 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.738 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.738 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.738 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f6063fff640 10 monclient: _finish_auth 0 2024-09-16T04:41:17.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f6063fff640 10 monclient: _check_auth_tickets 2024-09-16T04:41:17.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f6063fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:17.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f6081f31640 5 monclient: authenticate success, global_id 34140 2024-09-16T04:41:17.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f6063fff640 1 -- 172.21.15.46:0/571802659 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6070005310 con 0x7f607c0a3ff0 2024-09-16T04:41:17.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f6081f31640 10 monclient: _renew_subs 2024-09-16T04:41:17.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f6081f31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:41:17.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f6081f31640 1 -- 172.21.15.46:0/571802659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f607c11f860 con 0x7f607c0a3ff0 2024-09-16T04:41:17.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f6063fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:17.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f6063fff640 1 -- 172.21.15.46:0/571802659 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6070003830 con 0x7f607c0a3ff0 2024-09-16T04:41:17.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f6063fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f6063fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:41:17.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f6063fff640 10 monclient: dump: 2024-09-16T04:41:17.739 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.739 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.739 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.739 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.739 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.739 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.740 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.740 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.740 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.740 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f6081f31640 10 monclient: _renew_subs 2024-09-16T04:41:17.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f6079ffb640 4 set_mon_vals no callback set 2024-09-16T04:41:17.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f6081f31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:41:17.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.716+0000 7f6081f31640 1 -- 172.21.15.46:0/571802659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f607c0a0400 con 0x7f607c0a3ff0 2024-09-16T04:41:17.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.717+0000 7f6081f31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:17.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.717+0000 7f6081f31640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c13ace0 0x7f607c02c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.717+0000 7f6081f31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.717+0000 7f6081f31640 1 -- 172.21.15.46:0/571802659 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f607c09bcf0 con 0x7f607c13ace0 2024-09-16T04:41:17.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.717+0000 7f6063fff640 1 -- 172.21.15.46:0/571802659 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f607001a070 con 0x7f607c0a3ff0 2024-09-16T04:41:17.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.717+0000 7f607b7fe640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c13ace0 0x7f607c02c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.717+0000 7f6063fff640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f60680383b0 0x7f606803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.717+0000 7f607b7fe640 10 monclient: get_auth_request con 0x7f607c13ace0 auth_method 0 2024-09-16T04:41:17.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.717+0000 7f607b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.717+0000 7f607b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.717+0000 7f607b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.718+0000 7f6063fff640 1 -- 172.21.15.46:0/571802659 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 4346+0+0 (secure 0 0 0) 0x7f607004a910 con 0x7f607c0a3ff0 2024-09-16T04:41:17.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.718+0000 7f607b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.718+0000 7f607b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.718+0000 7f607b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.718+0000 7f607bfff640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f60680383b0 0x7f606803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.718+0000 7f607bfff640 10 monclient: get_auth_request con 0x7f60680383b0 auth_method 0 2024-09-16T04:41:17.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.718+0000 7f607b7fe640 10 monclient(hunting): handle_auth_done global_id 5194 payload 824 2024-09-16T04:41:17.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.718+0000 7f607b7fe640 1 -- 172.21.15.46:0/571802659 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f607c11ff00 con 0x7f607c13ace0 2024-09-16T04:41:17.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.718+0000 7f607b7fe640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c13ace0 0x7f607c02c640 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f606c0011e0 tx=0x7f606c00eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:17.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.718+0000 7f607bfff640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f60680383b0 0x7f606803a870 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6074034000 tx=0x7f607402f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:17.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.718+0000 7f6063fff640 1 -- 172.21.15.46:0/571802659 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f606c0058b0 con 0x7f607c13ace0 2024-09-16T04:41:17.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.718+0000 7f6063fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.718+0000 7f6063fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:41:17.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.718+0000 7f6063fff640 10 monclient: dump: 2024-09-16T04:41:17.742 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.742 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.742 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.742 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.742 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.742 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.742 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.742 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.743 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.743 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.719+0000 7f6063fff640 1 -- 172.21.15.46:0/571802659 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f606c018070 con 0x7f607c13ace0 2024-09-16T04:41:17.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.719+0000 7f6063fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:17.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.719+0000 7f6063fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:41:17.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.719+0000 7f6063fff640 1 -- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c13ace0 msgr2=0x7f607c02c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.719+0000 7f6063fff640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c13ace0 0x7f607c02c640 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f606c0011e0 tx=0x7f606c00eb90 comp rx=0 tx=0).stop 2024-09-16T04:41:17.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.719+0000 7f6063fff640 1 -- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c13ace0 msgr2=0x7f607c02c640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:41:17.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.719+0000 7f6063fff640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c13ace0 0x7f607c02c640 secure :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f606c0011e0 tx=0x7f606c00eb90 comp rx=0 tx=0).stop 2024-09-16T04:41:17.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.725+0000 7f6081f31640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:41:17.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.725+0000 7f6081f31640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c136e60 0x7f607c02cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.725+0000 7f6081f31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.725+0000 7f6081f31640 1 -- 172.21.15.46:0/571802659 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f607c09e820 con 0x7f607c136e60 2024-09-16T04:41:17.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.725+0000 7f607b7fe640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c136e60 0x7f607c02cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.725+0000 7f607b7fe640 10 monclient: get_auth_request con 0x7f607c136e60 auth_method 0 2024-09-16T04:41:17.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.725+0000 7f607b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.725+0000 7f607b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.725+0000 7f607b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.725+0000 7f607b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.725+0000 7f607b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.725+0000 7f607b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.725+0000 7f607b7fe640 10 monclient(hunting): handle_auth_done global_id 5197 payload 824 2024-09-16T04:41:17.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.725+0000 7f607b7fe640 1 -- 172.21.15.46:0/571802659 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f607c067e20 con 0x7f607c136e60 2024-09-16T04:41:17.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.725+0000 7f607b7fe640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c136e60 0x7f607c02cb80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f606c0011e0 tx=0x7f606c001150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:17.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6063fff640 1 -- 172.21.15.46:0/571802659 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f606c007680 con 0x7f607c136e60 2024-09-16T04:41:17.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6063fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6063fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:41:17.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6063fff640 10 monclient: dump: 2024-09-16T04:41:17.746 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.747 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.747 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.747 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.747 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.747 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.747 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.747 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.747 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.747 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6063fff640 1 -- 172.21.15.46:0/571802659 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f606c007680 con 0x7f607c136e60 2024-09-16T04:41:17.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6063fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:41:17.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6063fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:41:17.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6063fff640 1 -- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c136e60 msgr2=0x7f607c02cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6063fff640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c136e60 0x7f607c02cb80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f606c0011e0 tx=0x7f606c001150 comp rx=0 tx=0).stop 2024-09-16T04:41:17.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6063fff640 1 -- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c136e60 msgr2=0x7f607c02cb80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:41:17.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6063fff640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c136e60 0x7f607c02cb80 secure :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f606c0011e0 tx=0x7f606c001150 comp rx=0 tx=0).stop 2024-09-16T04:41:17.748 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:41:17.748 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:41:17.748 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:41:17.748 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:41:17.748 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 26, 2024-09-16T04:41:17.748 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 163628, 2024-09-16T04:41:17.748 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:41:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:41:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:41:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:41:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 1, 2024-09-16T04:41:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:41:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:41:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:41:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:41:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:41:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:41:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:41:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:41:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:41:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:41:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:41:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:41:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:41:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:41:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:41:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:41:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:41:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:41:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:41:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:41:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:41:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:41:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:41:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:41:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:41:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:41:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:41:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:41:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:41:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:41:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:41:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:41:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:41:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:41:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:41:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:41:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:41:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:41:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:41:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:41:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:41:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:41:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.759 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:41:17.759 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:41:17.759 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.759 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:17.759 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:17.759 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:41:17.759 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:17.759 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:17.759 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:41:17.759 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.759 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:17.759 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:17.759 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4 2024-09-16T04:41:17.759 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:17.759 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:17.760 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:41:17.760 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.760 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:17.760 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:17.760 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5 2024-09-16T04:41:17.760 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:17.760 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:17.760 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:41:17.760 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:17.760 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:17.760 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:17.760 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:41:17.760 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:17.760 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:17.761 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:17.761 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:41:17.761 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:41:17.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6081f31640 1 -- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f60680383b0 msgr2=0x7f606803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6081f31640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f60680383b0 0x7f606803a870 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6074034000 tx=0x7f607402f000 comp rx=0 tx=0).stop 2024-09-16T04:41:17.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6081f31640 10 monclient: shutdown 2024-09-16T04:41:17.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6081f31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:17.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6081f31640 1 -- 172.21.15.46:0/571802659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f607c0a3ff0 msgr2=0x7f607c139aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6081f31640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f607c0a3ff0 0x7f607c139aa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f607000c500 tx=0x7f607000c9c0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f607bfff640 1 -- 172.21.15.46:0/571802659 reap_dead start 2024-09-16T04:41:17.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6081f31640 1 -- 172.21.15.46:0/571802659 shutdown_connections 2024-09-16T04:41:17.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6081f31640 1 --2- 172.21.15.46:0/571802659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f607c0a3ff0 0x7f607c139aa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6081f31640 1 -- 172.21.15.46:0/571802659 >> 172.21.15.46:0/571802659 conn(0x7f607c0a6340 msgr2=0x7f607c0a2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:17.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6081f31640 1 -- 172.21.15.46:0/571802659 shutdown_connections 2024-09-16T04:41:17.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.726+0000 7f6081f31640 1 -- 172.21.15.46:0/571802659 wait complete. 2024-09-16T04:41:17.763 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:41:17.763 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:41:17.764 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:41:17.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.862+0000 7fda632e6640 10 monclient: get_monmap_and_config 2024-09-16T04:41:17.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.862+0000 7fda632e6640 10 monclient: build_initial_monmap 2024-09-16T04:41:17.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.862+0000 7fda632e6640 10 monclient: monmap: 2024-09-16T04:41:17.865 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:17.865 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.865 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:41:17.864348+0000 2024-09-16T04:41:17.865 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:41:17.864348+0000 2024-09-16T04:41:17.865 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:17.865 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.865 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:41:17.865 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:41:17.866 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:41:17.866 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:41:17.866 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:41:17.866 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.863+0000 7fda632e6640 1 Processor -- start 2024-09-16T04:41:17.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.863+0000 7fda632e6640 1 -- start start 2024-09-16T04:41:17.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.863+0000 7fda632e6640 10 monclient: init 2024-09-16T04:41:17.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda632e6640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:17.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda632e6640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-16T04:41:17.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda632e6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda5c133b30 0x7fda5c133f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda632e6640 10 monclient(hunting): picked mon.noname-a con 0x7fda5c133b30 addr v2:172.21.15.46:3300/0 2024-09-16T04:41:17.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda632e6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c09aa70 0x7fda5c0a1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda632e6640 10 monclient(hunting): picked mon.noname-b con 0x7fda5c09aa70 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:17.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda632e6640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fda5c0a1e70 0x7fda5c0a2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda632e6640 10 monclient(hunting): picked mon.noname-c con 0x7fda5c0a1e70 addr v2:172.21.15.114:3300/0 2024-09-16T04:41:17.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda632e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda632e6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fda5c06c710 con 0x7fda5c133b30 2024-09-16T04:41:17.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda632e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda632e6640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fda5c06c440 con 0x7fda5c0a1e70 2024-09-16T04:41:17.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda632e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda632e6640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fda5c09b1c0 con 0x7fda5c09aa70 2024-09-16T04:41:17.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda632e6640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:17.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda632e6640 10 monclient(hunting): authenticate will time out at 876.280169s 2024-09-16T04:41:17.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda60881640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c09aa70 0x7fda5c0a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda61082640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda5c133b30 0x7fda5c133f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda61082640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda5c133b30 0x7fda5c133f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60674/0 (socket says 172.21.15.46:60674) 2024-09-16T04:41:17.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda60881640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c09aa70 0x7fda5c0a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38108/0 (socket says 172.21.15.46:38108) 2024-09-16T04:41:17.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda61082640 1 -- 172.21.15.46:0/1167897642 learned_addr learned my addr 172.21.15.46:0/1167897642 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:17.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda61082640 10 monclient(hunting): get_auth_request con 0x7fda5c133b30 auth_method 0 2024-09-16T04:41:17.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda61082640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda61082640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda61082640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda60881640 10 monclient(hunting): get_auth_request con 0x7fda5c09aa70 auth_method 0 2024-09-16T04:41:17.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda60881640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda60881640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda60881640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda60881640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda60881640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda60881640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda61082640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda61082640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda61082640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda61883640 1 --2- 172.21.15.46:0/1167897642 >> v2:172.21.15.114:3300/0 conn(0x7fda5c0a1e70 0x7fda5c0a2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda60881640 10 monclient(hunting): handle_auth_done global_id 24371 payload 274 2024-09-16T04:41:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda60881640 1 -- 172.21.15.46:0/1167897642 >> v2:172.21.15.114:3300/0 conn(0x7fda5c0a1e70 msgr2=0x7fda5c0a2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:41:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda60881640 1 --2- 172.21.15.46:0/1167897642 >> v2:172.21.15.114:3300/0 conn(0x7fda5c0a1e70 0x7fda5c0a2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda60881640 1 -- 172.21.15.46:0/1167897642 >> v2:172.21.15.46:3300/0 conn(0x7fda5c133b30 msgr2=0x7fda5c133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:41:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda60881640 1 --2- 172.21.15.46:0/1167897642 >> v2:172.21.15.46:3300/0 conn(0x7fda5c133b30 0x7fda5c133f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda60881640 10 monclient: _finish_hunting 0 2024-09-16T04:41:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda60881640 1 monclient: found mon.noname-b 2024-09-16T04:41:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda60881640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:17.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda60881640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:41:17.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda60881640 1 -- 172.21.15.46:0/1167897642 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda5c09b4a0 con 0x7fda5c09aa70 2024-09-16T04:41:17.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda61883640 10 monclient: get_auth_request con 0x7fda5c0a1e70 auth_method 0 2024-09-16T04:41:17.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda61883640 1 --2- 172.21.15.46:0/1167897642 >> v2:172.21.15.114:3300/0 conn(0x7fda5c0a1e70 0x7fda5c0a2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:41:17.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.864+0000 7fda61082640 1 --2- 172.21.15.46:0/1167897642 >> v2:172.21.15.46:3300/0 conn(0x7fda5c133b30 0x7fda5c133f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:41:17.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda60881640 1 --2- 172.21.15.46:0/1167897642 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c09aa70 0x7fda5c0a1930 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fda5c06f2f0 tx=0x7fda5802a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c19f5359ffb87d1d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:17.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda4ffff640 1 -- 172.21.15.46:0/1167897642 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda58037070 con 0x7fda5c09aa70 2024-09-16T04:41:17.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda4ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:41:17.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda4ffff640 10 monclient: dump: 2024-09-16T04:41:17.877 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.877 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.877 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.878 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.878 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.878 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.878 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.878 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.878 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.878 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda4ffff640 10 monclient: _finish_auth 0 2024-09-16T04:41:17.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda4ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:41:17.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:17.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda632e6640 5 monclient: authenticate success, global_id 24371 2024-09-16T04:41:17.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda632e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:41:17.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda4ffff640 1 -- 172.21.15.46:0/1167897642 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda5802add0 con 0x7fda5c09aa70 2024-09-16T04:41:17.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda4ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:17.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda632e6640 10 monclient: get_monmap_and_config success 2024-09-16T04:41:17.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda632e6640 4 set_mon_vals no callback set 2024-09-16T04:41:17.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda4ffff640 1 -- 172.21.15.46:0/1167897642 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda58032660 con 0x7fda5c09aa70 2024-09-16T04:41:17.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:17.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda4ffff640 10 monclient: dump: 2024-09-16T04:41:17.879 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.879 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.879 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.879 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.879 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.879 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.879 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.879 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.880 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda632e6640 10 monclient: shutdown 2024-09-16T04:41:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda632e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda632e6640 1 -- 172.21.15.46:0/1167897642 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c09aa70 msgr2=0x7fda5c0a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.865+0000 7fda632e6640 1 --2- 172.21.15.46:0/1167897642 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c09aa70 0x7fda5c0a1930 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fda5c06f2f0 tx=0x7fda5802a080 comp rx=0 tx=0).stop 2024-09-16T04:41:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.866+0000 7fda632e6640 1 -- 172.21.15.46:0/1167897642 shutdown_connections 2024-09-16T04:41:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.866+0000 7fda632e6640 1 --2- 172.21.15.46:0/1167897642 >> v2:172.21.15.114:3300/0 conn(0x7fda5c0a1e70 0x7fda5c0a2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.866+0000 7fda632e6640 1 --2- 172.21.15.46:0/1167897642 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c09aa70 0x7fda5c0a1930 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.866+0000 7fda632e6640 1 --2- 172.21.15.46:0/1167897642 >> v2:172.21.15.46:3300/0 conn(0x7fda5c133b30 0x7fda5c133f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.866+0000 7fda632e6640 1 -- 172.21.15.46:0/1167897642 >> 172.21.15.46:0/1167897642 conn(0x7fda5c0a6630 msgr2=0x7fda5c0a2a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.866+0000 7fda632e6640 1 -- 172.21.15.46:0/1167897642 shutdown_connections 2024-09-16T04:41:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.866+0000 7fda632e6640 1 -- 172.21.15.46:0/1167897642 wait complete. 2024-09-16T04:41:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.866+0000 7fda632e6640 10 monclient: build_initial_monmap 2024-09-16T04:41:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.866+0000 7fda632e6640 10 monclient: monmap: 2024-09-16T04:41:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:41:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:41:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:41:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:41:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:41:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:41:17.881 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda632e6640 1 Processor -- start 2024-09-16T04:41:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda632e6640 1 -- start start 2024-09-16T04:41:17.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda632e6640 10 monclient: init 2024-09-16T04:41:17.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda632e6640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:17.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda632e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:41:17.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda632e6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c09aa70 0x7fda5c1cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda632e6640 10 monclient(hunting): picked mon.noname-b con 0x7fda5c09aa70 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:41:17.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda632e6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda5c0a1e70 0x7fda5c132920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda632e6640 10 monclient(hunting): picked mon.noname-a con 0x7fda5c0a1e70 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:41:17.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda632e6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c133b30 0x7fda5c1203e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda632e6640 10 monclient(hunting): picked mon.noname-c con 0x7fda5c133b30 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:17.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda632e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda632e6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fda5c134450 con 0x7fda5c0a1e70 2024-09-16T04:41:17.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda632e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda632e6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fda5c09c420 con 0x7fda5c09aa70 2024-09-16T04:41:17.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda632e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:17.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda632e6640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fda5c0a33a0 con 0x7fda5c133b30 2024-09-16T04:41:17.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda632e6640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:17.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda632e6640 10 monclient(hunting): authenticate will time out at 876.283578s 2024-09-16T04:41:17.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda60881640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda5c0a1e70 0x7fda5c132920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda61883640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c133b30 0x7fda5c1203e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda60881640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda5c0a1e70 0x7fda5c132920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60682/0 (socket says 172.21.15.46:60682) 2024-09-16T04:41:17.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda60881640 1 -- 172.21.15.46:0/1733907520 learned_addr learned my addr 172.21.15.46:0/1733907520 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:17.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda61883640 10 monclient(hunting): get_auth_request con 0x7fda5c133b30 auth_method 0 2024-09-16T04:41:17.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda61883640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda61883640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda61883640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda60881640 10 monclient(hunting): get_auth_request con 0x7fda5c0a1e70 auth_method 0 2024-09-16T04:41:17.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda60881640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda60881640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda60881640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda61883640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda61883640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda61883640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda60881640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:17.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda60881640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:17.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda60881640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:17.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda61082640 1 --2- 172.21.15.46:0/1733907520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c09aa70 0x7fda5c1cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda61082640 10 monclient(hunting): get_auth_request con 0x7fda5c09aa70 auth_method 0 2024-09-16T04:41:17.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda61082640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:17.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda61082640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:17.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda61082640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:17.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda61883640 10 monclient(hunting): handle_auth_done global_id 24374 payload 824 2024-09-16T04:41:17.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda61883640 1 -- 172.21.15.46:0/1733907520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c09aa70 msgr2=0x7fda5c1cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda61883640 1 --2- 172.21.15.46:0/1733907520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c09aa70 0x7fda5c1cba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.867+0000 7fda61883640 1 -- 172.21.15.46:0/1733907520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda5c0a1e70 msgr2=0x7fda5c132920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:17.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda61883640 1 --2- 172.21.15.46:0/1733907520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda5c0a1e70 0x7fda5c132920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:17.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda61883640 10 monclient: _finish_hunting 0 2024-09-16T04:41:17.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda61883640 1 monclient: found mon.noname-c 2024-09-16T04:41:17.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda61883640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:17.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda61883640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:41:17.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda61883640 1 -- 172.21.15.46:0/1733907520 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda5c123d60 con 0x7fda5c133b30 2024-09-16T04:41:17.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda60881640 1 --2- 172.21.15.46:0/1733907520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda5c0a1e70 0x7fda5c132920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:41:17.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda61883640 1 --2- 172.21.15.46:0/1733907520 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c133b30 0x7fda5c1203e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fda54007fd0 tx=0x7fda5400a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:17.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda4dffb640 1 -- 172.21.15.46:0/1733907520 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda5401a070 con 0x7fda5c133b30 2024-09-16T04:41:17.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda4dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:41:17.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda4dffb640 10 monclient: dump: 2024-09-16T04:41:17.886 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.886 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.886 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.886 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.886 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.886 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.886 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.886 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.886 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.886 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda4dffb640 10 monclient: _finish_auth 0 2024-09-16T04:41:17.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda4dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:41:17.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:17.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda4dffb640 1 -- 172.21.15.46:0/1733907520 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda5400d840 con 0x7fda5c133b30 2024-09-16T04:41:17.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda632e6640 5 monclient: authenticate success, global_id 24374 2024-09-16T04:41:17.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda632e6640 10 monclient: _renew_subs 2024-09-16T04:41:17.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda632e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:17.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda632e6640 1 -- 172.21.15.46:0/1733907520 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda5c124040 con 0x7fda5c133b30 2024-09-16T04:41:17.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda4dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:17.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda4dffb640 1 -- 172.21.15.46:0/1733907520 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda5400dc30 con 0x7fda5c133b30 2024-09-16T04:41:17.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:17.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda4dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:17.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda4dffb640 10 monclient: dump: 2024-09-16T04:41:17.887 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:17.887 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:17.887 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.888 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:17.888 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:17.888 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:17.888 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:17.888 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:17.888 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:17.888 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:17.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda4f7fe640 4 set_mon_vals no callback set 2024-09-16T04:41:17.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda632e6640 10 monclient: _renew_subs 2024-09-16T04:41:17.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda632e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:17.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.868+0000 7fda632e6640 1 -- 172.21.15.46:0/1733907520 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fda5c09d110 con 0x7fda5c133b30 2024-09-16T04:41:17.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.869+0000 7fda4dffb640 1 -- 172.21.15.46:0/1733907520 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fda5401d070 con 0x7fda5c133b30 2024-09-16T04:41:17.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.869+0000 7fda632e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:17.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.869+0000 7fda632e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:17.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.869+0000 7fda632e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:17.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.869+0000 7fda4dffb640 1 --2- 172.21.15.46:0/1733907520 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda400383e0 0x7fda4003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:17.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.869+0000 7fda632e6640 1 -- 172.21.15.46:0/1733907520 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda5c0a2300 con 0x7fda5c133b30 2024-09-16T04:41:17.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.869+0000 7fda4dffb640 1 -- 172.21.15.46:0/1733907520 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(110..110 src has 1..110) ==== 4346+0+0 (secure 0 0 0) 0x7fda5404bd80 con 0x7fda5c133b30 2024-09-16T04:41:17.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.869+0000 7fda61082640 1 --2- 172.21.15.46:0/1733907520 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda400383e0 0x7fda4003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:17.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.869+0000 7fda61082640 10 monclient: get_auth_request con 0x7fda400383e0 auth_method 0 2024-09-16T04:41:17.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.870+0000 7fda61082640 1 --2- 172.21.15.46:0/1733907520 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda400383e0 0x7fda4003a8a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fda50009b70 tx=0x7fda50008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:17.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.878+0000 7fda4dffb640 1 -- 172.21.15.46:0/1733907520 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fda5400ae20 con 0x7fda5c133b30 2024-09-16T04:41:17.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.878+0000 7fda4dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:17.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:17.878+0000 7fda4dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:41:18.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:18.123+0000 7fda632e6640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:41:18.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:18.123+0000 7fda632e6640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:41:18.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:18.123+0000 7fda632e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:18.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:18.123+0000 7fda632e6640 1 -- 172.21.15.46:0/1733907520 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fda5c122710 con 0x7fda5c133b30 2024-09-16T04:41:18.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:18.125+0000 7fda4dffb640 1 -- 172.21.15.46:0/1733907520 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fda5c122710 con 0x7fda5c133b30 2024-09-16T04:41:18.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:18.125+0000 7fda4dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:41:18.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:18.125+0000 7fda4dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:41:18.128 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.125+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:41:18.128 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.125+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=11,kv=11,logm=67} crc {auth=2915999823,config=3541143443,health=2119319422,kv=3989597926,logm=2291210792}) 2024-09-16T04:41:18.128 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.125+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=10,kv=12,logm=67} crc {auth=4187658030,config=3541143443,health=3431202026,kv=4017288519,logm=1898087277}) 2024-09-16T04:41:18.128 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.125+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:41:18.128 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.125+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=11,kv=11,logm=67} crc {auth=2915999823,config=3541143443,health=2119319422,kv=3989597926,logm=2291210792}) 2024-09-16T04:41:18.128 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.125+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=12,kv=11,logm=65} crc {auth=3073953044,config=3541143443,health=2337716963,kv=242601124,logm=2386583538}) 2024-09-16T04:41:18.128 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.126+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:41:18.128 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.126+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1166510972}) 2024-09-16T04:41:18.129 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.126+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=401091384}) 2024-09-16T04:41:18.129 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.126+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:41:18.129 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.126+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1166510972}) 2024-09-16T04:41:18.129 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.126+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3115367073}) 2024-09-16T04:41:18.130 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.127+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:41:18.130 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.127+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3717906900}) 2024-09-16T04:41:18.130 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.127+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4153356952}) 2024-09-16T04:41:18.130 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.127+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:41:18.130 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.127+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3717906900}) 2024-09-16T04:41:18.130 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.127+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2288255136}) 2024-09-16T04:41:18.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:18.127+0000 7fda632e6640 1 -- 172.21.15.46:0/1733907520 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda400383e0 msgr2=0x7fda4003a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:18.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:18.127+0000 7fda632e6640 1 --2- 172.21.15.46:0/1733907520 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda400383e0 0x7fda4003a8a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fda50009b70 tx=0x7fda50008040 comp rx=0 tx=0).stop 2024-09-16T04:41:18.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:18.127+0000 7fda632e6640 10 monclient: shutdown 2024-09-16T04:41:18.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:18.127+0000 7fda632e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:18.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:18.127+0000 7fda632e6640 1 -- 172.21.15.46:0/1733907520 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c133b30 msgr2=0x7fda5c1203e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:18.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:18.127+0000 7fda632e6640 1 --2- 172.21.15.46:0/1733907520 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c133b30 0x7fda5c1203e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fda54007fd0 tx=0x7fda5400a710 comp rx=0 tx=0).stop 2024-09-16T04:41:18.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:18.127+0000 7fda632e6640 1 -- 172.21.15.46:0/1733907520 shutdown_connections 2024-09-16T04:41:18.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:18.127+0000 7fda632e6640 1 --2- 172.21.15.46:0/1733907520 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c133b30 0x7fda5c1203e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:18.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:18.127+0000 7fda632e6640 1 --2- 172.21.15.46:0/1733907520 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda400383e0 0x7fda4003a8a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:18.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:18.127+0000 7fda632e6640 1 --2- 172.21.15.46:0/1733907520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda5c0a1e70 0x7fda5c132920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:18.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:18.127+0000 7fda632e6640 1 --2- 172.21.15.46:0/1733907520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c09aa70 0x7fda5c1cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:18.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:18.127+0000 7fda632e6640 1 -- 172.21.15.46:0/1733907520 >> 172.21.15.46:0/1733907520 conn(0x7fda5c0a6630 msgr2=0x7fda5c1368b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:18.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:18.127+0000 7fda632e6640 1 -- 172.21.15.46:0/1733907520 shutdown_connections 2024-09-16T04:41:18.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:18.127+0000 7fda632e6640 1 -- 172.21.15.46:0/1733907520 wait complete. 2024-09-16T04:41:18.134 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.127+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:41:18.135 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.127+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=397637357}) 2024-09-16T04:41:18.135 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.127+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4134456248}) 2024-09-16T04:41:18.135 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.128+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:41:18.136 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.128+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=397637357}) 2024-09-16T04:41:18.136 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.128+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1968155263}) 2024-09-16T04:41:18.136 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.128+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:41:18.136 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.128+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3890868713}) 2024-09-16T04:41:18.136 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.128+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=216200980}) 2024-09-16T04:41:18.136 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.128+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:41:18.136 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.128+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3890868713}) 2024-09-16T04:41:18.137 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.128+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2690814732}) 2024-09-16T04:41:18.137 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.129+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:41:18.137 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.129+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=996994573}) 2024-09-16T04:41:18.137 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.129+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=439598804}) 2024-09-16T04:41:18.137 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.129+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:41:18.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.129+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=996994573}) 2024-09-16T04:41:18.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.129+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1556123481}) 2024-09-16T04:41:18.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.130+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:41:18.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.130+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3852813461}) 2024-09-16T04:41:18.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.130+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1943247683}) 2024-09-16T04:41:18.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.130+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:41:18.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.130+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3852813461}) 2024-09-16T04:41:18.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.130+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1366862728}) 2024-09-16T04:41:18.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.131+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:41:18.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.131+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=80,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=8} crc {logm=1821319177,mdsmap=3956149183,mgr=1081907435,mgr_metadata=4119166318,mgrstat=2189279020}) 2024-09-16T04:41:18.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.131+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=75,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=3} crc {logm=2974222292,mdsmap=3956149183,mgr=677368822,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=2837416674,mon_config_key=3136301670}) 2024-09-16T04:41:18.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.131+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:41:18.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.131+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=80,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=8} crc {logm=1821319177,mdsmap=3956149183,mgr=1081907435,mgr_metadata=4119166318,mgrstat=2189279020}) 2024-09-16T04:41:18.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.131+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=1} crc {logm=2776505434,mdsmap=3956149183,mgr=2336270728,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=2913864721,mon_config_key=814484709}) 2024-09-16T04:41:18.139 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.132+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:41:18.139 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.132+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=2,mon_config_key=12,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=74} crc {mgrstat=4045869366,mon_config_key=2926520444,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=2058134480,osd_snap=723727393,osdmap=1420094148}) 2024-09-16T04:41:18.139 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.132+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=2,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=73} crc {mgrstat=4045869366,mon_config_key=3236504823,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=2058134480,osd_snap=723727393,osdmap=3423348028}) 2024-09-16T04:41:18.139 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.132+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:41:18.139 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.132+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=2,mon_config_key=12,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=74} crc {mgrstat=4045869366,mon_config_key=2926520444,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=2058134480,osd_snap=723727393,osdmap=1420094148}) 2024-09-16T04:41:18.139 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.132+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=1,mon_config_key=12,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=75} crc {mgrstat=2306239533,mon_config_key=54419307,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=2058134480,osd_snap=723727393,osdmap=3889051525}) 2024-09-16T04:41:18.140 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.133+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:41:18.140 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.133+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2150188175}) 2024-09-16T04:41:18.140 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.133+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4276390628}) 2024-09-16T04:41:18.140 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.133+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:41:18.140 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.133+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2150188175}) 2024-09-16T04:41:18.140 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:41:18.133+0000 7f552b332640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=674869468}) 2024-09-16T04:41:18.141 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:41:19.142 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:41:19.142 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:41:19.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.253+0000 7ff47e931640 10 monclient: get_monmap_and_config 2024-09-16T04:41:19.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.253+0000 7ff47e931640 10 monclient: build_initial_monmap 2024-09-16T04:41:19.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.253+0000 7ff47e931640 10 monclient: monmap: 2024-09-16T04:41:19.260 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:19.260 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.260 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:41:19.254576+0000 2024-09-16T04:41:19.260 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:41:19.254576+0000 2024-09-16T04:41:19.260 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:19.260 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.260 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:41:19.260 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:41:19.260 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:41:19.260 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:41:19.261 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:41:19.261 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.253+0000 7ff47e931640 1 Processor -- start 2024-09-16T04:41:19.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.253+0000 7ff47e931640 1 -- start start 2024-09-16T04:41:19.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.253+0000 7ff47e931640 10 monclient: init 2024-09-16T04:41:19.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff47e931640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:19.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff47e931640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-16T04:41:19.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff47e931640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4780a3c10 0x7ff47809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff47e931640 10 monclient(hunting): picked mon.noname-b con 0x7ff4780a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:19.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff47e931640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff4780a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:41:19.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff47e931640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff47813be90 0x7ff47813e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff47e931640 10 monclient(hunting): picked mon.noname-c con 0x7ff47813be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:41:19.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff47e931640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff47809ab70 con 0x7ff4780a3ff0 2024-09-16T04:41:19.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff47e931640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff47e931640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ff4780a4d60 con 0x7ff47813be90 2024-09-16T04:41:19.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff47e931640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff47e931640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7ff47809b030 con 0x7ff4780a3c10 2024-09-16T04:41:19.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff47e931640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:19.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff47e931640 10 monclient(hunting): authenticate will time out at 877.670525s 2024-09-16T04:41:19.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff477fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4780a3c10 0x7ff47809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff477fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4780a3c10 0x7ff47809bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38124/0 (socket says 172.21.15.46:38124) 2024-09-16T04:41:19.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff477fff640 1 -- 172.21.15.46:0/1511483792 learned_addr learned my addr 172.21.15.46:0/1511483792 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:19.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff477fff640 10 monclient(hunting): get_auth_request con 0x7ff4780a3c10 auth_method 0 2024-09-16T04:41:19.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff477fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff477fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff477fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff477fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff477fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff477fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff47cece640 1 --2- 172.21.15.46:0/1511483792 >> v2:172.21.15.114:3300/0 conn(0x7ff47813be90 0x7ff47813e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff47cece640 10 monclient(hunting): get_auth_request con 0x7ff47813be90 auth_method 0 2024-09-16T04:41:19.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff47cece640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff47cece640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff47cece640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff477fff640 10 monclient(hunting): handle_auth_done global_id 24377 payload 274 2024-09-16T04:41:19.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff477fff640 1 -- 172.21.15.46:0/1511483792 >> v2:172.21.15.114:3300/0 conn(0x7ff47813be90 msgr2=0x7ff47813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:41:19.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff477fff640 1 --2- 172.21.15.46:0/1511483792 >> v2:172.21.15.114:3300/0 conn(0x7ff47813be90 0x7ff47813e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.254+0000 7ff477fff640 1 -- 172.21.15.46:0/1511483792 >> v1:172.21.15.114:6789/0 conn(0x7ff4780a3ff0 legacy=0x7ff4780a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:41:19.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.255+0000 7ff477fff640 10 monclient: _finish_hunting 0 2024-09-16T04:41:19.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.255+0000 7ff477fff640 1 monclient: found mon.noname-b 2024-09-16T04:41:19.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.255+0000 7ff477fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:19.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.255+0000 7ff477fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:41:19.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.255+0000 7ff477fff640 1 -- 172.21.15.46:0/1511483792 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff47809b310 con 0x7ff4780a3c10 2024-09-16T04:41:19.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.255+0000 7ff477fff640 1 --2- 172.21.15.46:0/1511483792 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4780a3c10 0x7ff47809bcf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff468009870 tx=0x7ff46802a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68df9b113cfe39bb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:19.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.255+0000 7ff476ffd640 1 -- 172.21.15.46:0/1511483792 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff468037070 con 0x7ff4780a3c10 2024-09-16T04:41:19.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.255+0000 7ff476ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:19.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.255+0000 7ff476ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:41:19.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.255+0000 7ff476ffd640 10 monclient: dump: 2024-09-16T04:41:19.264 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:19.264 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.264 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.264 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.264 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:19.264 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.264 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:19.265 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:19.265 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:19.265 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.255+0000 7ff476ffd640 10 monclient: _finish_auth 0 2024-09-16T04:41:19.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.255+0000 7ff476ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:41:19.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.255+0000 7ff476ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:19.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.255+0000 7ff47e931640 5 monclient: authenticate success, global_id 24377 2024-09-16T04:41:19.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.255+0000 7ff476ffd640 1 -- 172.21.15.46:0/1511483792 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff468032440 con 0x7ff4780a3c10 2024-09-16T04:41:19.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.255+0000 7ff47e931640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:41:19.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.255+0000 7ff476ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:19.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.255+0000 7ff47e931640 10 monclient: get_monmap_and_config success 2024-09-16T04:41:19.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.255+0000 7ff47e931640 4 set_mon_vals no callback set 2024-09-16T04:41:19.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.255+0000 7ff476ffd640 1 -- 172.21.15.46:0/1511483792 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff468032810 con 0x7ff4780a3c10 2024-09-16T04:41:19.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.255+0000 7ff476ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:19.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.256+0000 7ff476ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:19.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.256+0000 7ff476ffd640 10 monclient: dump: 2024-09-16T04:41:19.266 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:19.266 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.266 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.266 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.266 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:19.266 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.266 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:19.266 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:19.266 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:19.266 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.256+0000 7ff47e931640 10 monclient: shutdown 2024-09-16T04:41:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.256+0000 7ff47e931640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.256+0000 7ff47e931640 1 -- 172.21.15.46:0/1511483792 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4780a3c10 msgr2=0x7ff47809bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.256+0000 7ff47e931640 1 --2- 172.21.15.46:0/1511483792 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4780a3c10 0x7ff47809bcf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff468009870 tx=0x7ff46802a130 comp rx=0 tx=0).stop 2024-09-16T04:41:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.256+0000 7ff47e931640 1 -- 172.21.15.46:0/1511483792 shutdown_connections 2024-09-16T04:41:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.256+0000 7ff47e931640 1 --2- 172.21.15.46:0/1511483792 >> v2:172.21.15.114:3300/0 conn(0x7ff47813be90 0x7ff47813e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.256+0000 7ff47e931640 1 --2- 172.21.15.46:0/1511483792 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4780a3c10 0x7ff47809bcf0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.256+0000 7ff47e931640 1 -- 172.21.15.46:0/1511483792 >> 172.21.15.46:0/1511483792 conn(0x7ff4780a5d10 msgr2=0x7ff4780a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.256+0000 7ff47e931640 1 -- 172.21.15.46:0/1511483792 shutdown_connections 2024-09-16T04:41:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.256+0000 7ff47e931640 1 -- 172.21.15.46:0/1511483792 wait complete. 2024-09-16T04:41:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.257+0000 7ff47e931640 10 monclient: build_initial_monmap 2024-09-16T04:41:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.257+0000 7ff47e931640 10 monclient: monmap: 2024-09-16T04:41:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:41:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:41:19.268 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:41:19.268 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:19.268 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.268 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:41:19.268 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:41:19.268 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:41:19.268 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.257+0000 7ff47e931640 1 Processor -- start 2024-09-16T04:41:19.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.257+0000 7ff47e931640 1 -- start start 2024-09-16T04:41:19.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.257+0000 7ff47e931640 10 monclient: init 2024-09-16T04:41:19.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47e931640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:19.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47e931640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:41:19.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47e931640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4780a3c10 0x7ff4781cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47e931640 10 monclient(hunting): picked mon.noname-b con 0x7ff4780a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:41:19.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47e931640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4780a3ff0 0x7ff478136e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47e931640 10 monclient(hunting): picked mon.noname-a con 0x7ff4780a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:41:19.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47e931640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff47813be90 0x7ff47809dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47e931640 10 monclient(hunting): picked mon.noname-c con 0x7ff47813be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:19.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47e931640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47e931640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff47806e550 con 0x7ff4780a3ff0 2024-09-16T04:41:19.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47e931640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47e931640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff47806e3d0 con 0x7ff4780a3c10 2024-09-16T04:41:19.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47e931640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47e931640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7ff47813e7c0 con 0x7ff47813be90 2024-09-16T04:41:19.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47e931640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:19.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47e931640 10 monclient(hunting): authenticate will time out at 877.674458s 2024-09-16T04:41:19.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47cece640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff47813be90 0x7ff47809dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff4777fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4780a3ff0 0x7ff478136e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47cece640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff47813be90 0x7ff47809dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38134/0 (socket says 172.21.15.46:38134) 2024-09-16T04:41:19.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47cece640 1 -- 172.21.15.46:0/194728169 learned_addr learned my addr 172.21.15.46:0/194728169 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:19.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47cece640 10 monclient(hunting): get_auth_request con 0x7ff47813be90 auth_method 0 2024-09-16T04:41:19.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47cece640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47cece640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47cece640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff4777fe640 10 monclient(hunting): get_auth_request con 0x7ff4780a3ff0 auth_method 0 2024-09-16T04:41:19.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff4777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff4777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff4777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff477fff640 1 --2- 172.21.15.46:0/194728169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4780a3c10 0x7ff4781cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff477fff640 10 monclient(hunting): get_auth_request con 0x7ff4780a3c10 auth_method 0 2024-09-16T04:41:19.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff477fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff477fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff477fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47cece640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47cece640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47cece640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff4777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff4777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff4777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff477fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff477fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff477fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47cece640 10 monclient(hunting): handle_auth_done global_id 24380 payload 824 2024-09-16T04:41:19.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47cece640 1 -- 172.21.15.46:0/194728169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4780a3c10 msgr2=0x7ff4781cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47cece640 1 --2- 172.21.15.46:0/194728169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4780a3c10 0x7ff4781cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47cece640 1 -- 172.21.15.46:0/194728169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4780a3ff0 msgr2=0x7ff478136e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47cece640 1 --2- 172.21.15.46:0/194728169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4780a3ff0 0x7ff478136e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47cece640 10 monclient: _finish_hunting 0 2024-09-16T04:41:19.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47cece640 1 monclient: found mon.noname-c 2024-09-16T04:41:19.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47cece640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:19.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47cece640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:41:19.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff47cece640 1 -- 172.21.15.46:0/194728169 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff47813ccb0 con 0x7ff47813be90 2024-09-16T04:41:19.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.258+0000 7ff4777fe640 1 --2- 172.21.15.46:0/194728169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4780a3ff0 0x7ff478136e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:41:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff47cece640 1 --2- 172.21.15.46:0/194728169 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff47813be90 0x7ff47809dc30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff470007dc0 tx=0x7ff47000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff474ff9640 1 -- 172.21.15.46:0/194728169 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff470014420 con 0x7ff47813be90 2024-09-16T04:41:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff474ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff474ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:41:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff474ff9640 10 monclient: dump: 2024-09-16T04:41:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:19.278 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:19.278 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff474ff9640 10 monclient: _finish_auth 0 2024-09-16T04:41:19.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff474ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:41:19.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff474ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:19.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff474ff9640 1 -- 172.21.15.46:0/194728169 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff470019070 con 0x7ff47813be90 2024-09-16T04:41:19.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff47e931640 5 monclient: authenticate success, global_id 24380 2024-09-16T04:41:19.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff47e931640 10 monclient: _renew_subs 2024-09-16T04:41:19.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff47e931640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:19.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff47e931640 1 -- 172.21.15.46:0/194728169 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff47813c760 con 0x7ff47813be90 2024-09-16T04:41:19.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff474ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:19.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff474ff9640 1 -- 172.21.15.46:0/194728169 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff470014820 con 0x7ff47813be90 2024-09-16T04:41:19.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff474ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:19.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff474ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:19.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff474ff9640 10 monclient: dump: 2024-09-16T04:41:19.279 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:19.279 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.279 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.279 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.279 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:19.279 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.279 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:19.279 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:19.279 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:19.279 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff47e931640 10 monclient: _renew_subs 2024-09-16T04:41:19.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff47e931640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:19.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff47e931640 1 -- 172.21.15.46:0/194728169 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff47809f880 con 0x7ff47813be90 2024-09-16T04:41:19.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.259+0000 7ff4767fc640 4 set_mon_vals no callback set 2024-09-16T04:41:19.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.260+0000 7ff474ff9640 1 -- 172.21.15.46:0/194728169 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7ff4700149c0 con 0x7ff47813be90 2024-09-16T04:41:19.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.260+0000 7ff47e931640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:19.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.260+0000 7ff47e931640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:19.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.260+0000 7ff47e931640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:19.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.260+0000 7ff47e931640 1 -- 172.21.15.46:0/194728169 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff478120380 con 0x7ff47813be90 2024-09-16T04:41:19.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.260+0000 7ff474ff9640 1 --2- 172.21.15.46:0/194728169 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff4640383b0 0x7ff46403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.260+0000 7ff474ff9640 1 -- 172.21.15.46:0/194728169 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(111..111 src has 1..111) ==== 4738+0+0 (secure 0 0 0) 0x7ff47004c0c0 con 0x7ff47813be90 2024-09-16T04:41:19.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.260+0000 7ff477fff640 1 --2- 172.21.15.46:0/194728169 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff4640383b0 0x7ff46403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.260+0000 7ff477fff640 10 monclient: get_auth_request con 0x7ff4640383b0 auth_method 0 2024-09-16T04:41:19.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.261+0000 7ff477fff640 1 --2- 172.21.15.46:0/194728169 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff4640383b0 0x7ff46403a870 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff468034000 tx=0x7ff46802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:19.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.265+0000 7ff474ff9640 1 -- 172.21.15.46:0/194728169 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff478120380 con 0x7ff47813be90 2024-09-16T04:41:19.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.265+0000 7ff474ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:19.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.265+0000 7ff474ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:41:19.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.511+0000 7ff47e931640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:41:19.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.511+0000 7ff47e931640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:41:19.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.511+0000 7ff47e931640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:19.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.511+0000 7ff47e931640 1 -- 172.21.15.46:0/194728169 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff47809bcf0 con 0x7ff47813be90 2024-09-16T04:41:19.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.511+0000 7ff474ff9640 1 -- 172.21.15.46:0/194728169 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7ff47001c360 con 0x7ff47813be90 2024-09-16T04:41:19.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.511+0000 7ff474ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:41:19.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.511+0000 7ff474ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:41:19.514 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:41:19.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.514+0000 7ff47e931640 1 -- 172.21.15.46:0/194728169 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff4640383b0 msgr2=0x7ff46403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.514+0000 7ff47e931640 1 --2- 172.21.15.46:0/194728169 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff4640383b0 0x7ff46403a870 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff468034000 tx=0x7ff46802e000 comp rx=0 tx=0).stop 2024-09-16T04:41:19.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.514+0000 7ff47e931640 10 monclient: shutdown 2024-09-16T04:41:19.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.514+0000 7ff47e931640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:19.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.514+0000 7ff47e931640 1 -- 172.21.15.46:0/194728169 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff47813be90 msgr2=0x7ff47809dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.514+0000 7ff47e931640 1 --2- 172.21.15.46:0/194728169 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff47813be90 0x7ff47809dc30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff470007dc0 tx=0x7ff47000e4b0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.514+0000 7ff47e931640 1 -- 172.21.15.46:0/194728169 shutdown_connections 2024-09-16T04:41:19.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.514+0000 7ff47e931640 1 --2- 172.21.15.46:0/194728169 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff47813be90 0x7ff47809dc30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.514+0000 7ff47e931640 1 --2- 172.21.15.46:0/194728169 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff4640383b0 0x7ff46403a870 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.514+0000 7ff47e931640 1 --2- 172.21.15.46:0/194728169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4780a3ff0 0x7ff478136e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.514+0000 7ff47e931640 1 --2- 172.21.15.46:0/194728169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4780a3c10 0x7ff4781cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.514+0000 7ff47e931640 1 -- 172.21.15.46:0/194728169 >> 172.21.15.46:0/194728169 conn(0x7ff4780a5d10 msgr2=0x7ff47813b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:19.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.514+0000 7ff47e931640 1 -- 172.21.15.46:0/194728169 shutdown_connections 2024-09-16T04:41:19.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.514+0000 7ff47e931640 1 -- 172.21.15.46:0/194728169 wait complete. 2024-09-16T04:41:19.527 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:41:19.527 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:41:19.527 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:41:19.527 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:41:19.641 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36898]: starting. 2024-09-16T04:41:19.641 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36898]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-36897 2024-09-16T04:41:19.641 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36898]: created object 0... 2024-09-16T04:41:19.641 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36898]: created object 25... 2024-09-16T04:41:19.641 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36898]: created object 49... 2024-09-16T04:41:19.641 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36898]: finishing. 2024-09-16T04:41:19.641 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[36898]: shutting down. 2024-09-16T04:41:19.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.643+0000 7f245aed0640 10 monclient: get_monmap_and_config 2024-09-16T04:41:19.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.643+0000 7f245aed0640 10 monclient: build_initial_monmap 2024-09-16T04:41:19.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.643+0000 7f245aed0640 10 monclient: monmap: 2024-09-16T04:41:19.645 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:19.645 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.645 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:41:19.644961+0000 2024-09-16T04:41:19.645 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:41:19.644961+0000 2024-09-16T04:41:19.646 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:19.646 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.646 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:41:19.646 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:41:19.646 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:41:19.646 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:41:19.646 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:41:19.646 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f245aed0640 1 Processor -- start 2024-09-16T04:41:19.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f245aed0640 1 -- start start 2024-09-16T04:41:19.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f245aed0640 10 monclient: init 2024-09-16T04:41:19.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f245aed0640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:19.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f245aed0640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-16T04:41:19.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f245aed0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f24540a3c10 0x7f245409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f245aed0640 10 monclient(hunting): picked mon.noname-a con 0x7f24540a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:41:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f245aed0640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f24540a3ff0 0x7f24540a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f245aed0640 10 monclient(hunting): picked mon.noname-c con 0x7f24540a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:41:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f245aed0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f24541369d0 addr v1:172.21.15.46:6789/0 2024-09-16T04:41:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f245aed0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f245409b940 con 0x7f24541369d0 2024-09-16T04:41:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f245aed0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f245aed0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f245409ab00 con 0x7f24540a3c10 2024-09-16T04:41:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f245aed0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f245aed0640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f245409b030 con 0x7f24540a3ff0 2024-09-16T04:41:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f245aed0640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f245aed0640 10 monclient(hunting): authenticate will time out at 878.060793s 2024-09-16T04:41:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f2458c6c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f24540a3c10 0x7f245409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f245946d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f24541369d0 0x7f2454136e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36044/0 (socket says 172.21.15.46:36044) 2024-09-16T04:41:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f245946d640 1 -- 172.21.15.46:0/1795773579 learned_addr learned my addr 172.21.15.46:0/1795773579 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:41:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f2458c6c640 10 monclient(hunting): get_auth_request con 0x7f24540a3c10 auth_method 0 2024-09-16T04:41:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f2458c6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f2458c6c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.644+0000 7f2458c6c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.645+0000 7f2458c6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.645+0000 7f2458c6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.645+0000 7f2458c6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.645+0000 7f2453fff640 1 --2- 172.21.15.46:0/1795773579 >> v2:172.21.15.114:3300/0 conn(0x7f24540a3ff0 0x7f24540a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.645+0000 7f2453fff640 10 monclient(hunting): get_auth_request con 0x7f24540a3ff0 auth_method 0 2024-09-16T04:41:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.645+0000 7f2453fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.645+0000 7f2453fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.645+0000 7f2453fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.645+0000 7f2458c6c640 10 monclient(hunting): handle_auth_done global_id 34152 payload 274 2024-09-16T04:41:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.645+0000 7f2458c6c640 1 -- 172.21.15.46:0/1795773579 >> v2:172.21.15.114:3300/0 conn(0x7f24540a3ff0 msgr2=0x7f24540a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:41:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.645+0000 7f2458c6c640 1 --2- 172.21.15.46:0/1795773579 >> v2:172.21.15.114:3300/0 conn(0x7f24540a3ff0 0x7f24540a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.645+0000 7f2458c6c640 1 -- 172.21.15.46:0/1795773579 >> v1:172.21.15.46:6789/0 conn(0x7f24541369d0 legacy=0x7f2454136e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.645+0000 7f2458c6c640 10 monclient: _finish_hunting 0 2024-09-16T04:41:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.645+0000 7f2458c6c640 1 monclient: found mon.noname-a 2024-09-16T04:41:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.645+0000 7f2458c6c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.645+0000 7f2458c6c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:41:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.645+0000 7f24537fe640 1 -- 172.21.15.46:0/1795773579 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3614641444 0 0) 0x7f245409b940 con 0x7f24541369d0 2024-09-16T04:41:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.645+0000 7f2458c6c640 1 -- 172.21.15.46:0/1795773579 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f245409b310 con 0x7f24540a3c10 2024-09-16T04:41:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.645+0000 7f24537fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:41:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.645+0000 7f2458c6c640 1 --2- 172.21.15.46:0/1795773579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24540a3c10 0x7f245409bcf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f24540a38d0 tx=0x7f244402a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a39a1d55279c73d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f24537fe640 1 -- 172.21.15.46:0/1795773579 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2444037070 con 0x7f24540a3c10 2024-09-16T04:41:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f24537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f24537fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:41:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f24537fe640 10 monclient: dump: 2024-09-16T04:41:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:19.661 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f24537fe640 10 monclient: _finish_auth 0 2024-09-16T04:41:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f24537fe640 10 monclient: _check_auth_tickets 2024-09-16T04:41:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f24537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f245aed0640 5 monclient: authenticate success, global_id 34152 2024-09-16T04:41:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f24537fe640 1 -- 172.21.15.46:0/1795773579 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f244402add0 con 0x7f24540a3c10 2024-09-16T04:41:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f245aed0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:41:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f24537fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f245aed0640 10 monclient: get_monmap_and_config success 2024-09-16T04:41:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f245aed0640 4 set_mon_vals no callback set 2024-09-16T04:41:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f24537fe640 1 -- 172.21.15.46:0/1795773579 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2444032660 con 0x7f24540a3c10 2024-09-16T04:41:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f24537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f24537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:41:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f24537fe640 10 monclient: dump: 2024-09-16T04:41:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:19.662 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.662 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.662 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.662 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:19.662 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.662 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:19.662 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:19.662 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:19.662 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f245aed0640 10 monclient: shutdown 2024-09-16T04:41:19.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f245aed0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:19.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f245aed0640 1 -- 172.21.15.46:0/1795773579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24540a3c10 msgr2=0x7f245409bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f245aed0640 1 --2- 172.21.15.46:0/1795773579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24540a3c10 0x7f245409bcf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f24540a38d0 tx=0x7f244402a080 comp rx=0 tx=0).stop 2024-09-16T04:41:19.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f245aed0640 1 -- 172.21.15.46:0/1795773579 shutdown_connections 2024-09-16T04:41:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f245aed0640 1 --2- 172.21.15.46:0/1795773579 >> v2:172.21.15.114:3300/0 conn(0x7f24540a3ff0 0x7f24540a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f245aed0640 1 --2- 172.21.15.46:0/1795773579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24540a3c10 0x7f245409bcf0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f245aed0640 1 -- 172.21.15.46:0/1795773579 >> 172.21.15.46:0/1795773579 conn(0x7f24540a6340 msgr2=0x7f245406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f245aed0640 1 -- 172.21.15.46:0/1795773579 shutdown_connections 2024-09-16T04:41:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.646+0000 7f245aed0640 1 -- 172.21.15.46:0/1795773579 wait complete. 2024-09-16T04:41:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.647+0000 7f245aed0640 10 monclient: build_initial_monmap 2024-09-16T04:41:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.647+0000 7f245aed0640 10 monclient: monmap: 2024-09-16T04:41:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:41:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:41:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:41:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:41:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:41:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:41:19.664 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.647+0000 7f245aed0640 1 Processor -- start 2024-09-16T04:41:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.647+0000 7f245aed0640 1 -- start start 2024-09-16T04:41:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.647+0000 7f245aed0640 10 monclient: init 2024-09-16T04:41:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245aed0640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245aed0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:41:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245aed0640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f24540a3c10 0x7f24541cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245aed0640 10 monclient(hunting): picked mon.noname-c con 0x7f24540a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245aed0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24540a3ff0 0x7f24541cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245aed0640 10 monclient(hunting): picked mon.noname-a con 0x7f24540a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:41:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245aed0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24541369d0 0x7f2454123020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245aed0640 10 monclient(hunting): picked mon.noname-b con 0x7f24541369d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:41:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245aed0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245aed0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f24540a4910 con 0x7f24540a3ff0 2024-09-16T04:41:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245aed0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245aed0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f24540a69c0 con 0x7f24541369d0 2024-09-16T04:41:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245aed0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245aed0640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f24540a29b0 con 0x7f24540a3c10 2024-09-16T04:41:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245aed0640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245aed0640 10 monclient(hunting): authenticate will time out at 878.064338s 2024-09-16T04:41:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2453fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24540a3ff0 0x7f24541cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2458c6c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f24540a3c10 0x7f24541cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2453fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24540a3ff0 0x7f24541cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60698/0 (socket says 172.21.15.46:60698) 2024-09-16T04:41:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2458c6c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f24540a3c10 0x7f24541cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38140/0 (socket says 172.21.15.46:38140) 2024-09-16T04:41:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2453fff640 1 -- 172.21.15.46:0/1084458624 learned_addr learned my addr 172.21.15.46:0/1084458624 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2453fff640 10 monclient(hunting): get_auth_request con 0x7f24540a3ff0 auth_method 0 2024-09-16T04:41:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2453fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2453fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2453fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2458c6c640 10 monclient(hunting): get_auth_request con 0x7f24540a3c10 auth_method 0 2024-09-16T04:41:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2458c6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2458c6c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2458c6c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245946d640 1 --2- 172.21.15.46:0/1084458624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24541369d0 0x7f2454123020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245946d640 10 monclient(hunting): get_auth_request con 0x7f24541369d0 auth_method 0 2024-09-16T04:41:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245946d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245946d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245946d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2453fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2453fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2453fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2458c6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2458c6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2458c6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245946d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245946d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f245946d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2453fff640 10 monclient(hunting): handle_auth_done global_id 34158 payload 824 2024-09-16T04:41:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2453fff640 1 -- 172.21.15.46:0/1084458624 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f24540a3c10 msgr2=0x7f24541cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2453fff640 1 --2- 172.21.15.46:0/1084458624 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f24540a3c10 0x7f24541cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2453fff640 1 -- 172.21.15.46:0/1084458624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24541369d0 msgr2=0x7f2454123020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2453fff640 1 --2- 172.21.15.46:0/1084458624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24541369d0 0x7f2454123020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2453fff640 10 monclient: _finish_hunting 0 2024-09-16T04:41:19.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2453fff640 1 monclient: found mon.noname-a 2024-09-16T04:41:19.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2453fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:19.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2453fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:41:19.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2453fff640 1 -- 172.21.15.46:0/1084458624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2454125160 con 0x7f24540a3ff0 2024-09-16T04:41:19.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2458c6c640 1 --2- 172.21.15.46:0/1084458624 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f24540a3c10 0x7f24541cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:41:19.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.648+0000 7f2453fff640 1 --2- 172.21.15.46:0/1084458624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24540a3ff0 0x7f24541cc420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f243c00d800 tx=0x7f243c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:19.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f24517fa640 1 -- 172.21.15.46:0/1084458624 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c00b6c0 con 0x7f24540a3ff0 2024-09-16T04:41:19.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f24517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:19.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f24517fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:41:19.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f24517fa640 10 monclient: dump: 2024-09-16T04:41:19.668 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:19.668 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.668 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.669 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.669 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:19.669 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.669 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:19.669 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:19.669 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:19.669 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f24517fa640 10 monclient: _finish_auth 0 2024-09-16T04:41:19.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f24517fa640 10 monclient: _check_auth_tickets 2024-09-16T04:41:19.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f24517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:19.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f24517fa640 1 -- 172.21.15.46:0/1084458624 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f243c019070 con 0x7f24540a3ff0 2024-09-16T04:41:19.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f245aed0640 5 monclient: authenticate success, global_id 34158 2024-09-16T04:41:19.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f245aed0640 10 monclient: _renew_subs 2024-09-16T04:41:19.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f245aed0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:41:19.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f245aed0640 1 -- 172.21.15.46:0/1084458624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2454125820 con 0x7f24540a3ff0 2024-09-16T04:41:19.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f24517fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:19.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f24517fa640 1 -- 172.21.15.46:0/1084458624 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c00ba80 con 0x7f24540a3ff0 2024-09-16T04:41:19.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f24517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:19.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f24517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:41:19.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f24517fa640 10 monclient: dump: 2024-09-16T04:41:19.670 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:19.670 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.670 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.671 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.671 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:19.671 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.671 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:19.671 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:19.671 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:19.671 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f2452ffd640 4 set_mon_vals no callback set 2024-09-16T04:41:19.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f245aed0640 10 monclient: _renew_subs 2024-09-16T04:41:19.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f245aed0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:41:19.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f245aed0640 1 -- 172.21.15.46:0/1084458624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24541269f0 con 0x7f24540a3ff0 2024-09-16T04:41:19.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f245aed0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:19.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f245aed0640 1 --2- 172.21.15.46:0/1084458624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2454126bc0 0x7f2454123560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f245aed0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.649+0000 7f245aed0640 1 -- 172.21.15.46:0/1084458624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f24540a4a90 con 0x7f2454126bc0 2024-09-16T04:41:19.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.650+0000 7f24517fa640 1 -- 172.21.15.46:0/1084458624 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f243c00bc20 con 0x7f24540a3ff0 2024-09-16T04:41:19.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.650+0000 7f2458c6c640 1 --2- 172.21.15.46:0/1084458624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2454126bc0 0x7f2454123560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.650+0000 7f2458c6c640 10 monclient: get_auth_request con 0x7f2454126bc0 auth_method 0 2024-09-16T04:41:19.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.650+0000 7f2458c6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.650+0000 7f2458c6c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.650+0000 7f2458c6c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.650+0000 7f2458c6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.650+0000 7f2458c6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.650+0000 7f2458c6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.650+0000 7f24517fa640 1 --2- 172.21.15.46:0/1084458624 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f24340383b0 0x7f243403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.650+0000 7f24517fa640 1 -- 172.21.15.46:0/1084458624 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 4738+0+0 (secure 0 0 0) 0x7f243c04bb50 con 0x7f24540a3ff0 2024-09-16T04:41:19.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.650+0000 7f2458c6c640 10 monclient(hunting): handle_auth_done global_id 34161 payload 824 2024-09-16T04:41:19.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.650+0000 7f2458c6c640 1 -- 172.21.15.46:0/1084458624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f245409bcf0 con 0x7f2454126bc0 2024-09-16T04:41:19.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.650+0000 7f245946d640 1 --2- 172.21.15.46:0/1084458624 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f24340383b0 0x7f243403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.650+0000 7f245946d640 10 monclient: get_auth_request con 0x7f24340383b0 auth_method 0 2024-09-16T04:41:19.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.650+0000 7f2458c6c640 1 --2- 172.21.15.46:0/1084458624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2454126bc0 0x7f2454123560 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f245406ee70 tx=0x7f24440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:19.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.650+0000 7f24517fa640 1 -- 172.21.15.46:0/1084458624 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2444043070 con 0x7f2454126bc0 2024-09-16T04:41:19.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.650+0000 7f24517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:19.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.650+0000 7f24517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:41:19.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.650+0000 7f24517fa640 10 monclient: dump: 2024-09-16T04:41:19.673 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:19.673 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.673 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.673 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.673 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:19.673 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.674 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:19.674 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:19.674 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:19.674 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.650+0000 7f245946d640 1 --2- 172.21.15.46:0/1084458624 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f24340383b0 0x7f243403a870 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f244c001820 tx=0x7f244c0017b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:19.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.651+0000 7f24517fa640 1 -- 172.21.15.46:0/1084458624 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f244403c640 con 0x7f2454126bc0 2024-09-16T04:41:19.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.651+0000 7f24517fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:19.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.651+0000 7f24517fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:41:19.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.651+0000 7f24517fa640 1 -- 172.21.15.46:0/1084458624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2454126bc0 msgr2=0x7f2454123560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.651+0000 7f24517fa640 1 --2- 172.21.15.46:0/1084458624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2454126bc0 0x7f2454123560 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f245406ee70 tx=0x7f24440046c0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.651+0000 7f24517fa640 1 -- 172.21.15.46:0/1084458624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2454126bc0 msgr2=0x7f2454123560 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:41:19.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.651+0000 7f24517fa640 1 --2- 172.21.15.46:0/1084458624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2454126bc0 0x7f2454123560 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.657+0000 7f245aed0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:41:19.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.657+0000 7f245aed0640 1 --2- 172.21.15.46:0/1084458624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2454123aa0 0x7f2454123eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.657+0000 7f245aed0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.657+0000 7f245aed0640 1 -- 172.21.15.46:0/1084458624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f24541243f0 con 0x7f2454123aa0 2024-09-16T04:41:19.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.657+0000 7f2458c6c640 1 --2- 172.21.15.46:0/1084458624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2454123aa0 0x7f2454123eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.657+0000 7f2458c6c640 10 monclient: get_auth_request con 0x7f2454123aa0 auth_method 0 2024-09-16T04:41:19.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.657+0000 7f2458c6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.657+0000 7f2458c6c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.657+0000 7f2458c6c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.657+0000 7f2458c6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.657+0000 7f2458c6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.657+0000 7f2458c6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.657+0000 7f2458c6c640 10 monclient(hunting): handle_auth_done global_id 34164 payload 824 2024-09-16T04:41:19.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.657+0000 7f2458c6c640 1 -- 172.21.15.46:0/1084458624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f24541cc960 con 0x7f2454123aa0 2024-09-16T04:41:19.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.657+0000 7f2458c6c640 1 --2- 172.21.15.46:0/1084458624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2454123aa0 0x7f2454123eb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2444000c00 tx=0x7f2444008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:19.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.657+0000 7f24517fa640 1 -- 172.21.15.46:0/1084458624 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2444037040 con 0x7f2454123aa0 2024-09-16T04:41:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.657+0000 7f24517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.657+0000 7f24517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:41:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.657+0000 7f24517fa640 10 monclient: dump: 2024-09-16T04:41:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:19.678 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.658+0000 7f24517fa640 1 -- 172.21.15.46:0/1084458624 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f2444003b70 con 0x7f2454123aa0 2024-09-16T04:41:19.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.658+0000 7f24517fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:41:19.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.658+0000 7f24517fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:41:19.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.658+0000 7f24517fa640 1 -- 172.21.15.46:0/1084458624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2454123aa0 msgr2=0x7f2454123eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.658+0000 7f24517fa640 1 --2- 172.21.15.46:0/1084458624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2454123aa0 0x7f2454123eb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2444000c00 tx=0x7f2444008030 comp rx=0 tx=0).stop 2024-09-16T04:41:19.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.658+0000 7f24517fa640 1 -- 172.21.15.46:0/1084458624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2454123aa0 msgr2=0x7f2454123eb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:41:19.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.658+0000 7f24517fa640 1 --2- 172.21.15.46:0/1084458624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2454123aa0 0x7f2454123eb0 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2444000c00 tx=0x7f2444008030 comp rx=0 tx=0).stop 2024-09-16T04:41:19.679 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:41:19.679 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:41:19.680 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:41:19.680 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:41:19.680 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 26, 2024-09-16T04:41:19.680 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 9416, 2024-09-16T04:41:19.680 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:41:19.680 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:41:19.680 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:41:19.680 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:41:19.680 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:19.680 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 3, 2024-09-16T04:41:19.680 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:41:19.681 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:41:19.681 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:41:19.681 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:19.681 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:19.681 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:19.681 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:19.681 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:19.681 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:19.681 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:19.681 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:19.681 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:19.681 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:19.681 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:19.681 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:19.682 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:41:19.682 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:41:19.682 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:19.682 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:19.682 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:19.682 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:19.682 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:19.682 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:19.682 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:19.682 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:19.682 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:19.682 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:19.682 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:19.682 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:19.683 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.683 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:41:19.683 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:41:19.683 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:41:19.683 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:41:19.683 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:41:19.683 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:41:19.683 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:41:19.683 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:41:19.683 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:41:19.683 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:41:19.683 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:41:19.683 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:41:19.683 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:41:19.684 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:41:19.684 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:41:19.684 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:41:19.684 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:41:19.684 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:19.684 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:19.684 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:19.684 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:19.684 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:19.684 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:19.684 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:19.684 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:19.684 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:19.684 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:19.685 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:19.685 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:19.685 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:41:19.685 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.685 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:41:19.685 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.685 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:41:19.685 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:41:19.685 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:19.685 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:19.685 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.685 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:19.685 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:41:19.685 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:19.686 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.686 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.686 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:19.686 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:41:19.686 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:19.686 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:19.686 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:19.686 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.686 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:41:19.686 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:41:19.686 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:19.686 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:19.686 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:19.686 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.687 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.687 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:41:19.687 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:41:19.687 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:19.687 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:19.687 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.687 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:19.687 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:41:19.687 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:19.687 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.687 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.687 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:19.687 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:41:19.687 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:19.688 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:19.688 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:19.688 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.688 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:41:19.688 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:41:19.688 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:19.688 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:19.688 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:19.688 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.688 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.688 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:41:19.688 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:41:19.688 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:19.688 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:19.689 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.689 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:19.689 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:41:19.689 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:19.689 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.689 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.689 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:19.689 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:41:19.689 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:19.689 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:19.689 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:19.689 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.689 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:41:19.689 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:41:19.690 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:19.690 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:19.690 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:19.690 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:19.690 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:19.690 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.690 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:41:19.690 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:41:19.690 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.690 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:19.690 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:19.690 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:41:19.690 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:19.690 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:19.690 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:41:19.691 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.691 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:19.691 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:19.691 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:41:19.691 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:19.691 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:19.691 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.691 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:41:19.691 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:41:19.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.658+0000 7f245aed0640 1 -- 172.21.15.46:0/1084458624 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f24340383b0 msgr2=0x7f243403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.658+0000 7f245aed0640 1 --2- 172.21.15.46:0/1084458624 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f24340383b0 0x7f243403a870 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f244c001820 tx=0x7f244c0017b0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.658+0000 7f245aed0640 10 monclient: shutdown 2024-09-16T04:41:19.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.658+0000 7f245aed0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:19.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.658+0000 7f245aed0640 1 -- 172.21.15.46:0/1084458624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24540a3ff0 msgr2=0x7f24541cc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.658+0000 7f245aed0640 1 --2- 172.21.15.46:0/1084458624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24540a3ff0 0x7f24541cc420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f243c00d800 tx=0x7f243c00dcc0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.658+0000 7f245946d640 1 -- 172.21.15.46:0/1084458624 reap_dead start 2024-09-16T04:41:19.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.658+0000 7f245946d640 1 -- 172.21.15.46:0/1084458624 reap_dead start 2024-09-16T04:41:19.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.658+0000 7f245aed0640 1 -- 172.21.15.46:0/1084458624 shutdown_connections 2024-09-16T04:41:19.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.658+0000 7f245aed0640 1 -- 172.21.15.46:0/1084458624 >> 172.21.15.46:0/1084458624 conn(0x7f24540a6340 msgr2=0x7f24540a14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:19.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.658+0000 7f245aed0640 1 -- 172.21.15.46:0/1084458624 shutdown_connections 2024-09-16T04:41:19.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.658+0000 7f245aed0640 1 -- 172.21.15.46:0/1084458624 wait complete. 2024-09-16T04:41:19.694 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:41:19.694 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:41:19.694 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:41:19.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.793+0000 7fce30d03640 10 monclient: get_monmap_and_config 2024-09-16T04:41:19.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.793+0000 7fce30d03640 10 monclient: build_initial_monmap 2024-09-16T04:41:19.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.794+0000 7fce30d03640 10 monclient: monmap: 2024-09-16T04:41:19.796 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:19.796 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.796 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:41:19.795478+0000 2024-09-16T04:41:19.796 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:41:19.795478+0000 2024-09-16T04:41:19.796 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:19.797 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.797 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:41:19.797 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:41:19.797 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:41:19.797 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:41:19.797 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:41:19.797 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.794+0000 7fce30d03640 1 Processor -- start 2024-09-16T04:41:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.794+0000 7fce30d03640 1 -- start start 2024-09-16T04:41:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.794+0000 7fce30d03640 10 monclient: init 2024-09-16T04:41:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce30d03640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce30d03640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-16T04:41:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce30d03640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c0a3c10 0x7fce2c09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce30d03640 10 monclient(hunting): picked mon.noname-b con 0x7fce2c0a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce30d03640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fce2c0a3ff0 0x7fce2c0a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce30d03640 10 monclient(hunting): picked mon.noname-c con 0x7fce2c0a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:41:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce30d03640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fce2c1369d0 0x7fce2c136e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce30d03640 10 monclient(hunting): picked mon.noname-a con 0x7fce2c1369d0 addr v2:172.21.15.46:3300/0 2024-09-16T04:41:19.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce30d03640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce30d03640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fce2c09b940 con 0x7fce2c1369d0 2024-09-16T04:41:19.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce30d03640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce30d03640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fce2c09ab00 con 0x7fce2c0a3ff0 2024-09-16T04:41:19.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce30d03640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce30d03640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fce2c09b030 con 0x7fce2c0a3c10 2024-09-16T04:41:19.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce30d03640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:19.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce30d03640 10 monclient(hunting): authenticate will time out at 878.211287s 2024-09-16T04:41:19.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2a59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c0a3c10 0x7fce2c09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fce2c1369d0 0x7fce2c136e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2a59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c0a3c10 0x7fce2c09bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38142/0 (socket says 172.21.15.46:38142) 2024-09-16T04:41:19.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fce2c1369d0 0x7fce2c136e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60724/0 (socket says 172.21.15.46:60724) 2024-09-16T04:41:19.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2a59c640 1 -- 172.21.15.46:0/1190282287 learned_addr learned my addr 172.21.15.46:0/1190282287 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:19.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2a59c640 10 monclient(hunting): get_auth_request con 0x7fce2c0a3c10 auth_method 0 2024-09-16T04:41:19.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2ad9d640 10 monclient(hunting): get_auth_request con 0x7fce2c1369d0 auth_method 0 2024-09-16T04:41:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce29d9b640 1 --2- 172.21.15.46:0/1190282287 >> v2:172.21.15.114:3300/0 conn(0x7fce2c0a3ff0 0x7fce2c0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce29d9b640 10 monclient(hunting): get_auth_request con 0x7fce2c0a3ff0 auth_method 0 2024-09-16T04:41:19.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce29d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce29d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce29d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce29d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce29d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2ad9d640 10 monclient(hunting): handle_auth_done global_id 34167 payload 274 2024-09-16T04:41:19.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2ad9d640 1 -- 172.21.15.46:0/1190282287 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c0a3c10 msgr2=0x7fce2c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:41:19.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2ad9d640 1 --2- 172.21.15.46:0/1190282287 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c0a3c10 0x7fce2c09bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2ad9d640 1 -- 172.21.15.46:0/1190282287 >> v2:172.21.15.114:3300/0 conn(0x7fce2c0a3ff0 msgr2=0x7fce2c0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:41:19.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2ad9d640 1 --2- 172.21.15.46:0/1190282287 >> v2:172.21.15.114:3300/0 conn(0x7fce2c0a3ff0 0x7fce2c0a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2ad9d640 10 monclient: _finish_hunting 0 2024-09-16T04:41:19.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2ad9d640 1 monclient: found mon.noname-a 2024-09-16T04:41:19.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:19.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:41:19.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2ad9d640 1 -- 172.21.15.46:0/1190282287 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce2c09b310 con 0x7fce2c1369d0 2024-09-16T04:41:19.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.795+0000 7fce2a59c640 1 --2- 172.21.15.46:0/1190282287 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c0a3c10 0x7fce2c09bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:41:19.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.796+0000 7fce2ad9d640 1 --2- 172.21.15.46:0/1190282287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce2c1369d0 0x7fce2c136e20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fce2c0689b0 tx=0x7fce1c02a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1422adb090e7dcd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:19.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.796+0000 7fce2959a640 1 -- 172.21.15.46:0/1190282287 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1c007740 con 0x7fce2c1369d0 2024-09-16T04:41:19.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.796+0000 7fce2959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:19.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.796+0000 7fce2959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:41:19.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.796+0000 7fce2959a640 10 monclient: dump: 2024-09-16T04:41:19.823 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:19.823 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.823 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.824 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.824 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:19.824 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.824 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:19.824 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:19.825 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:19.825 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.796+0000 7fce2959a640 10 monclient: _finish_auth 0 2024-09-16T04:41:19.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.796+0000 7fce2959a640 10 monclient: _check_auth_tickets 2024-09-16T04:41:19.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.796+0000 7fce2959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:19.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.796+0000 7fce30d03640 5 monclient: authenticate success, global_id 34167 2024-09-16T04:41:19.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.796+0000 7fce30d03640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:41:19.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.796+0000 7fce2959a640 1 -- 172.21.15.46:0/1190282287 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce1c0078e0 con 0x7fce2c1369d0 2024-09-16T04:41:19.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.796+0000 7fce2959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:19.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.796+0000 7fce30d03640 10 monclient: get_monmap_and_config success 2024-09-16T04:41:19.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.796+0000 7fce2959a640 1 -- 172.21.15.46:0/1190282287 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1c007cb0 con 0x7fce2c1369d0 2024-09-16T04:41:19.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.796+0000 7fce30d03640 4 set_mon_vals no callback set 2024-09-16T04:41:19.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.796+0000 7fce2959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:19.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.796+0000 7fce2959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:41:19.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.796+0000 7fce2959a640 10 monclient: dump: 2024-09-16T04:41:19.827 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:19.827 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.827 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:19.829 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.797+0000 7fce30d03640 10 monclient: shutdown 2024-09-16T04:41:19.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.797+0000 7fce30d03640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:19.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.797+0000 7fce30d03640 1 -- 172.21.15.46:0/1190282287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce2c1369d0 msgr2=0x7fce2c136e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.797+0000 7fce30d03640 1 --2- 172.21.15.46:0/1190282287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce2c1369d0 0x7fce2c136e20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fce2c0689b0 tx=0x7fce1c02a4b0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.797+0000 7fce30d03640 1 -- 172.21.15.46:0/1190282287 shutdown_connections 2024-09-16T04:41:19.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.797+0000 7fce30d03640 1 --2- 172.21.15.46:0/1190282287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce2c1369d0 0x7fce2c136e20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.797+0000 7fce30d03640 1 --2- 172.21.15.46:0/1190282287 >> v2:172.21.15.114:3300/0 conn(0x7fce2c0a3ff0 0x7fce2c0a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.797+0000 7fce30d03640 1 --2- 172.21.15.46:0/1190282287 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c0a3c10 0x7fce2c09bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.797+0000 7fce30d03640 1 -- 172.21.15.46:0/1190282287 >> 172.21.15.46:0/1190282287 conn(0x7fce2c0a6340 msgr2=0x7fce2c06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:19.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.797+0000 7fce30d03640 1 -- 172.21.15.46:0/1190282287 shutdown_connections 2024-09-16T04:41:19.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.797+0000 7fce30d03640 1 -- 172.21.15.46:0/1190282287 wait complete. 2024-09-16T04:41:19.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce30d03640 10 monclient: build_initial_monmap 2024-09-16T04:41:19.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce30d03640 10 monclient: monmap: 2024-09-16T04:41:19.831 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:19.831 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:41:19.831 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:41:19.832 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:41:19.832 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:19.832 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.832 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:41:19.832 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:41:19.832 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:41:19.833 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce30d03640 1 Processor -- start 2024-09-16T04:41:19.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce30d03640 1 -- start start 2024-09-16T04:41:19.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce30d03640 10 monclient: init 2024-09-16T04:41:19.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce30d03640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:19.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce30d03640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:41:19.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce30d03640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c0a3c10 0x7fce2c1cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce30d03640 10 monclient(hunting): picked mon.noname-c con 0x7fce2c0a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:19.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce30d03640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce2c0a3ff0 0x7fce2c1cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce30d03640 10 monclient(hunting): picked mon.noname-b con 0x7fce2c0a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:41:19.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce30d03640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce2c1369d0 0x7fce2c1233e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce30d03640 10 monclient(hunting): picked mon.noname-a con 0x7fce2c1369d0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:41:19.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce30d03640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce30d03640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fce2c0a4910 con 0x7fce2c1369d0 2024-09-16T04:41:19.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce30d03640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce30d03640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fce2c0a69c0 con 0x7fce2c0a3ff0 2024-09-16T04:41:19.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce30d03640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce30d03640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fce2c1259e0 con 0x7fce2c0a3c10 2024-09-16T04:41:19.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce30d03640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:19.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce30d03640 10 monclient(hunting): authenticate will time out at 878.214856s 2024-09-16T04:41:19.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce2a59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c0a3c10 0x7fce2c1cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce2ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce2c1369d0 0x7fce2c1233e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce2a59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c0a3c10 0x7fce2c1cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38150/0 (socket says 172.21.15.46:38150) 2024-09-16T04:41:19.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce2a59c640 1 -- 172.21.15.46:0/3693874316 learned_addr learned my addr 172.21.15.46:0/3693874316 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:19.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce2a59c640 10 monclient(hunting): get_auth_request con 0x7fce2c0a3c10 auth_method 0 2024-09-16T04:41:19.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce2a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.798+0000 7fce2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2ad9d640 10 monclient(hunting): get_auth_request con 0x7fce2c1369d0 auth_method 0 2024-09-16T04:41:19.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce29d9b640 1 --2- 172.21.15.46:0/3693874316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce2c0a3ff0 0x7fce2c1cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce29d9b640 10 monclient(hunting): get_auth_request con 0x7fce2c0a3ff0 auth_method 0 2024-09-16T04:41:19.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce29d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce29d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce29d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce29d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce29d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2a59c640 10 monclient(hunting): handle_auth_done global_id 24389 payload 824 2024-09-16T04:41:19.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2a59c640 1 -- 172.21.15.46:0/3693874316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce2c0a3ff0 msgr2=0x7fce2c1cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2a59c640 1 --2- 172.21.15.46:0/3693874316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce2c0a3ff0 0x7fce2c1cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2a59c640 1 -- 172.21.15.46:0/3693874316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce2c1369d0 msgr2=0x7fce2c1233e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2a59c640 1 --2- 172.21.15.46:0/3693874316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce2c1369d0 0x7fce2c1233e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2a59c640 10 monclient: _finish_hunting 0 2024-09-16T04:41:19.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2a59c640 1 monclient: found mon.noname-c 2024-09-16T04:41:19.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:19.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:41:19.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2a59c640 1 -- 172.21.15.46:0/3693874316 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce2c0a8db0 con 0x7fce2c0a3c10 2024-09-16T04:41:19.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2ad9d640 1 --2- 172.21.15.46:0/3693874316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce2c1369d0 0x7fce2c1233e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:41:19.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce2a59c640 1 --2- 172.21.15.46:0/3693874316 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c0a3c10 0x7fce2c1cbee0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fce2400ea00 tx=0x7fce2400eec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:19.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce22ffd640 1 -- 172.21.15.46:0/3693874316 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce24017070 con 0x7fce2c0a3c10 2024-09-16T04:41:19.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:19.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce22ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:41:19.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce22ffd640 10 monclient: dump: 2024-09-16T04:41:19.840 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:19.841 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce22ffd640 10 monclient: _finish_auth 0 2024-09-16T04:41:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce22ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:41:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce22ffd640 1 -- 172.21.15.46:0/3693874316 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce2400ca60 con 0x7fce2c0a3c10 2024-09-16T04:41:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce22ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce22ffd640 1 -- 172.21.15.46:0/3693874316 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce2400ce10 con 0x7fce2c0a3c10 2024-09-16T04:41:19.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce30d03640 5 monclient: authenticate success, global_id 24389 2024-09-16T04:41:19.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce30d03640 10 monclient: _renew_subs 2024-09-16T04:41:19.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce30d03640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:19.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce30d03640 1 -- 172.21.15.46:0/3693874316 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce2c0a8880 con 0x7fce2c0a3c10 2024-09-16T04:41:19.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:19.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:19.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce22ffd640 10 monclient: dump: 2024-09-16T04:41:19.842 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:19.842 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.842 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.842 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:19.843 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce28d99640 4 set_mon_vals no callback set 2024-09-16T04:41:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce30d03640 10 monclient: _renew_subs 2024-09-16T04:41:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce30d03640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.799+0000 7fce30d03640 1 -- 172.21.15.46:0/3693874316 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fce2c127170 con 0x7fce2c0a3c10 2024-09-16T04:41:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.800+0000 7fce30d03640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.800+0000 7fce30d03640 1 --2- 172.21.15.46:0/3693874316 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c123920 0x7fce2c123d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.800+0000 7fce30d03640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.800+0000 7fce30d03640 1 -- 172.21.15.46:0/3693874316 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fce2c126a60 con 0x7fce2c123920 2024-09-16T04:41:19.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.800+0000 7fce22ffd640 1 -- 172.21.15.46:0/3693874316 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fce2401a070 con 0x7fce2c0a3c10 2024-09-16T04:41:19.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.800+0000 7fce29d9b640 1 --2- 172.21.15.46:0/3693874316 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c123920 0x7fce2c123d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.800+0000 7fce29d9b640 10 monclient: get_auth_request con 0x7fce2c123920 auth_method 0 2024-09-16T04:41:19.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.800+0000 7fce29d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.800+0000 7fce29d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.800+0000 7fce29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.800+0000 7fce29d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.800+0000 7fce29d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.800+0000 7fce29d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.800+0000 7fce22ffd640 1 --2- 172.21.15.46:0/3693874316 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fce0c0383b0 0x7fce0c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.801+0000 7fce29d9b640 10 monclient(hunting): handle_auth_done global_id 24392 payload 824 2024-09-16T04:41:19.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.801+0000 7fce22ffd640 1 -- 172.21.15.46:0/3693874316 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(112..112 src has 1..112) ==== 4738+0+0 (secure 0 0 0) 0x7fce2404b690 con 0x7fce2c0a3c10 2024-09-16T04:41:19.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.801+0000 7fce29d9b640 1 -- 172.21.15.46:0/3693874316 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fce2c09bcf0 con 0x7fce2c123920 2024-09-16T04:41:19.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.801+0000 7fce2ad9d640 1 --2- 172.21.15.46:0/3693874316 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fce0c0383b0 0x7fce0c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.801+0000 7fce2ad9d640 10 monclient: get_auth_request con 0x7fce0c0383b0 auth_method 0 2024-09-16T04:41:19.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.801+0000 7fce29d9b640 1 --2- 172.21.15.46:0/3693874316 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c123920 0x7fce2c123d00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fce14006a90 tx=0x7fce14011cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:19.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.801+0000 7fce22ffd640 1 -- 172.21.15.46:0/3693874316 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1400faa0 con 0x7fce2c123920 2024-09-16T04:41:19.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.801+0000 7fce22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:19.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.801+0000 7fce22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:19.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.801+0000 7fce22ffd640 10 monclient: dump: 2024-09-16T04:41:19.845 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:19.846 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.846 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.846 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.846 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:19.846 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.846 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:19.846 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:19.846 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:19.846 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.801+0000 7fce2ad9d640 1 --2- 172.21.15.46:0/3693874316 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fce0c0383b0 0x7fce0c03a870 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fce1c034000 tx=0x7fce1c02f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:19.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.801+0000 7fce22ffd640 1 -- 172.21.15.46:0/3693874316 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fce1400faa0 con 0x7fce2c123920 2024-09-16T04:41:19.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.801+0000 7fce22ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:19.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.801+0000 7fce22ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:41:19.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.801+0000 7fce22ffd640 1 -- 172.21.15.46:0/3693874316 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c123920 msgr2=0x7fce2c123d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.801+0000 7fce22ffd640 1 --2- 172.21.15.46:0/3693874316 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c123920 0x7fce2c123d00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fce14006a90 tx=0x7fce14011cf0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.801+0000 7fce22ffd640 1 -- 172.21.15.46:0/3693874316 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c123920 msgr2=0x7fce2c123d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:41:19.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.801+0000 7fce22ffd640 1 --2- 172.21.15.46:0/3693874316 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c123920 0x7fce2c123d00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.807+0000 7fce30d03640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:41:19.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.807+0000 7fce30d03640 1 --2- 172.21.15.46:0/3693874316 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c124240 0x7fce2c11f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.807+0000 7fce30d03640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.807+0000 7fce30d03640 1 -- 172.21.15.46:0/3693874316 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fce2c067e20 con 0x7fce2c124240 2024-09-16T04:41:19.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.807+0000 7fce29d9b640 1 --2- 172.21.15.46:0/3693874316 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c124240 0x7fce2c11f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.807+0000 7fce29d9b640 10 monclient: get_auth_request con 0x7fce2c124240 auth_method 0 2024-09-16T04:41:19.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.807+0000 7fce29d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.807+0000 7fce29d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.807+0000 7fce29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.808+0000 7fce29d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.808+0000 7fce29d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.808+0000 7fce29d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.808+0000 7fce29d9b640 10 monclient(hunting): handle_auth_done global_id 24395 payload 824 2024-09-16T04:41:19.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.808+0000 7fce29d9b640 1 -- 172.21.15.46:0/3693874316 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fce2c1cc9c0 con 0x7fce2c124240 2024-09-16T04:41:19.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.808+0000 7fce29d9b640 1 --2- 172.21.15.46:0/3693874316 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c124240 0x7fce2c11f860 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fce14006a90 tx=0x7fce140046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:19.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.808+0000 7fce22ffd640 1 -- 172.21.15.46:0/3693874316 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce14014440 con 0x7fce2c124240 2024-09-16T04:41:19.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.808+0000 7fce22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:19.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.808+0000 7fce22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:19.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.808+0000 7fce22ffd640 10 monclient: dump: 2024-09-16T04:41:19.848 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:19.848 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.848 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.848 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.848 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:19.848 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.849 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:19.849 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:19.849 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:19.849 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.808+0000 7fce22ffd640 1 -- 172.21.15.46:0/3693874316 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7fce140273f0 con 0x7fce2c124240 2024-09-16T04:41:19.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.808+0000 7fce22ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:41:19.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.808+0000 7fce22ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:41:19.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.808+0000 7fce22ffd640 1 -- 172.21.15.46:0/3693874316 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c124240 msgr2=0x7fce2c11f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.808+0000 7fce22ffd640 1 --2- 172.21.15.46:0/3693874316 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c124240 0x7fce2c11f860 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fce14006a90 tx=0x7fce140046c0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.808+0000 7fce22ffd640 1 -- 172.21.15.46:0/3693874316 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c124240 msgr2=0x7fce2c11f860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:41:19.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.808+0000 7fce22ffd640 1 --2- 172.21.15.46:0/3693874316 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c124240 0x7fce2c11f860 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fce14006a90 tx=0x7fce140046c0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.850 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:41:19.850 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:41:19.850 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:41:19.850 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:41:19.850 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 26, 2024-09-16T04:41:19.850 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 55008, 2024-09-16T04:41:19.850 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:41:19.850 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:41:19.850 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:41:19.850 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:41:19.850 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:19.850 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 3, 2024-09-16T04:41:19.851 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:41:19.851 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:41:19.851 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:41:19.851 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:19.851 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:19.851 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:19.851 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:19.851 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:19.851 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:19.851 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:19.851 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:19.851 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:19.851 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:19.851 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:19.852 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:19.852 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:41:19.852 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:41:19.852 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:19.852 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:19.852 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:19.852 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:19.852 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:19.852 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:19.852 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:19.852 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:19.852 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:19.852 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:19.853 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:19.853 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:19.853 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.853 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:41:19.853 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:41:19.853 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:41:19.853 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:41:19.853 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:41:19.853 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:41:19.853 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:41:19.853 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:41:19.853 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:41:19.853 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:41:19.853 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:41:19.854 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:41:19.854 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:41:19.854 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:41:19.854 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:41:19.854 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:41:19.854 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:41:19.854 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:19.854 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:19.854 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:19.854 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:19.854 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:19.854 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:19.854 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:19.854 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:19.854 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:19.855 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:19.855 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:19.855 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:19.855 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:41:19.855 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.855 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:41:19.855 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.855 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:41:19.855 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:41:19.855 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:19.855 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:19.856 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.856 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:19.856 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:41:19.856 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:19.856 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.856 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.856 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:19.856 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:41:19.856 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:19.856 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:19.856 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:19.856 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.856 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:41:19.856 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:41:19.857 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:19.857 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:19.857 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:19.857 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.857 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.857 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:41:19.857 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:41:19.857 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:19.857 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:19.857 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.857 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:19.858 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:41:19.858 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:19.858 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.858 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.858 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:19.858 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:41:19.858 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:19.858 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:19.858 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:19.858 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.858 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:41:19.858 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:41:19.858 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:19.858 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:19.858 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:19.859 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.859 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.859 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:41:19.859 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:41:19.859 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:19.859 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:19.859 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.859 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:19.859 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:41:19.859 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:19.859 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.859 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.859 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:19.859 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:41:19.860 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:19.860 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:19.860 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:19.860 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.860 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:41:19.860 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:41:19.860 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:19.860 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:19.860 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:19.860 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:19.860 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:19.861 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.861 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:41:19.861 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:41:19.861 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.861 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:19.861 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:19.861 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:41:19.861 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:19.861 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:19.861 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:41:19.861 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.861 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:19.861 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:19.862 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:41:19.862 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:19.862 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:19.862 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:41:19.862 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:19.862 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:19.862 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:19.863 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:41:19.863 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:19.863 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:19.863 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:19.863 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:41:19.863 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:41:19.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.809+0000 7fce30d03640 1 -- 172.21.15.46:0/3693874316 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fce0c0383b0 msgr2=0x7fce0c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.809+0000 7fce30d03640 1 --2- 172.21.15.46:0/3693874316 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fce0c0383b0 0x7fce0c03a870 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fce1c034000 tx=0x7fce1c02f000 comp rx=0 tx=0).stop 2024-09-16T04:41:19.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.809+0000 7fce30d03640 10 monclient: shutdown 2024-09-16T04:41:19.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.809+0000 7fce30d03640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:19.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.809+0000 7fce30d03640 1 -- 172.21.15.46:0/3693874316 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c0a3c10 msgr2=0x7fce2c1cbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.809+0000 7fce30d03640 1 --2- 172.21.15.46:0/3693874316 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fce2c0a3c10 0x7fce2c1cbee0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fce2400ea00 tx=0x7fce2400eec0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.809+0000 7fce2ad9d640 1 -- 172.21.15.46:0/3693874316 reap_dead start 2024-09-16T04:41:19.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.809+0000 7fce2ad9d640 1 -- 172.21.15.46:0/3693874316 reap_dead start 2024-09-16T04:41:19.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.809+0000 7fce30d03640 1 -- 172.21.15.46:0/3693874316 shutdown_connections 2024-09-16T04:41:19.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.809+0000 7fce30d03640 1 -- 172.21.15.46:0/3693874316 >> 172.21.15.46:0/3693874316 conn(0x7fce2c0a6340 msgr2=0x7fce2c0a2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:19.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.809+0000 7fce30d03640 1 -- 172.21.15.46:0/3693874316 shutdown_connections 2024-09-16T04:41:19.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.809+0000 7fce30d03640 1 -- 172.21.15.46:0/3693874316 wait complete. 2024-09-16T04:41:19.868 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:41:19.868 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:41:19.869 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:41:19.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.969+0000 7f774be8d640 10 monclient: get_monmap_and_config 2024-09-16T04:41:19.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.969+0000 7f774be8d640 10 monclient: build_initial_monmap 2024-09-16T04:41:19.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.969+0000 7f774be8d640 10 monclient: monmap: 2024-09-16T04:41:19.971 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:19.971 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.971 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:41:19.970641+0000 2024-09-16T04:41:19.971 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:41:19.970641+0000 2024-09-16T04:41:19.971 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:19.971 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.972 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:41:19.972 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:41:19.972 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:41:19.972 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:41:19.972 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:41:19.972 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.969+0000 7f774be8d640 1 Processor -- start 2024-09-16T04:41:19.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.969+0000 7f774be8d640 1 -- start start 2024-09-16T04:41:19.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.969+0000 7f774be8d640 10 monclient: init 2024-09-16T04:41:19.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f774be8d640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:19.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f774be8d640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-16T04:41:19.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f774be8d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f77440a3c10 0x7f774409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f774be8d640 10 monclient(hunting): picked mon.noname-a con 0x7f77440a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:41:19.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f774be8d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f77440a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:41:19.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f774be8d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f774413be90 addr v1:172.21.15.46:6789/0 2024-09-16T04:41:19.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f774be8d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f774409ab70 con 0x7f774413be90 2024-09-16T04:41:19.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f774be8d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77440a4d60 con 0x7f77440a3ff0 2024-09-16T04:41:19.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f774be8d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f774be8d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f774409b030 con 0x7f77440a3c10 2024-09-16T04:41:19.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f774be8d640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:19.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f774be8d640 10 monclient(hunting): authenticate will time out at 878.386527s 2024-09-16T04:41:19.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f774a42a640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f774413be90 0x7f774413e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36052/0 (socket says 172.21.15.46:36052) 2024-09-16T04:41:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f7749c29640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f77440a3c10 0x7f774409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f774a42a640 1 -- 172.21.15.46:0/1318485870 learned_addr learned my addr 172.21.15.46:0/1318485870 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:41:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f7749c29640 10 monclient(hunting): get_auth_request con 0x7f77440a3c10 auth_method 0 2024-09-16T04:41:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f7749c29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f7749c29640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f7749c29640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f7749c29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f7749c29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f7749c29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f7749c29640 10 monclient(hunting): handle_auth_done global_id 34173 payload 274 2024-09-16T04:41:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f7749c29640 1 -- 172.21.15.46:0/1318485870 >> v1:172.21.15.114:6789/0 conn(0x7f77440a3ff0 legacy=0x7f77440a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f7749c29640 1 -- 172.21.15.46:0/1318485870 >> v1:172.21.15.46:6789/0 conn(0x7f774413be90 legacy=0x7f774413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f7749c29640 10 monclient: _finish_hunting 0 2024-09-16T04:41:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f7749c29640 1 monclient: found mon.noname-a 2024-09-16T04:41:19.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f7749c29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:19.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.971+0000 7f7749c29640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:41:19.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.971+0000 7f7749c29640 1 -- 172.21.15.46:0/1318485870 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f774409b310 con 0x7f77440a3c10 2024-09-16T04:41:19.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.970+0000 7f7748c27640 1 -- 172.21.15.46:0/1318485870 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1494775168 0 0) 0x7f774409ab70 con 0x7f774413be90 2024-09-16T04:41:19.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.971+0000 7f7748c27640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:41:19.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.971+0000 7f7749c29640 1 --2- 172.21.15.46:0/1318485870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77440a3c10 0x7f774409bcf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f77400097a0 tx=0x7f774002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7459346b47d02c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:19.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.971+0000 7f7748c27640 1 -- 172.21.15.46:0/1318485870 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7740037070 con 0x7f77440a3c10 2024-09-16T04:41:19.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.971+0000 7f7748c27640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:19.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.971+0000 7f7748c27640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:41:19.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.971+0000 7f7748c27640 10 monclient: dump: 2024-09-16T04:41:19.983 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:19.983 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.983 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.983 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:19.984 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.971+0000 7f7748c27640 10 monclient: _finish_auth 0 2024-09-16T04:41:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.971+0000 7f7748c27640 10 monclient: _check_auth_tickets 2024-09-16T04:41:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.971+0000 7f7748c27640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.971+0000 7f774be8d640 5 monclient: authenticate success, global_id 34173 2024-09-16T04:41:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.971+0000 7f7748c27640 1 -- 172.21.15.46:0/1318485870 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7740032440 con 0x7f77440a3c10 2024-09-16T04:41:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.971+0000 7f7748c27640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.971+0000 7f774be8d640 10 monclient: get_monmap_and_config success 2024-09-16T04:41:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.971+0000 7f774be8d640 4 set_mon_vals no callback set 2024-09-16T04:41:19.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.971+0000 7f7748c27640 1 -- 172.21.15.46:0/1318485870 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f774002ae60 con 0x7f77440a3c10 2024-09-16T04:41:19.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.971+0000 7f7748c27640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:19.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.971+0000 7f7748c27640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:41:19.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.971+0000 7f7748c27640 10 monclient: dump: 2024-09-16T04:41:19.985 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:19.985 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.985 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.985 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.985 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:19.985 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:19.986 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.972+0000 7f774be8d640 10 monclient: shutdown 2024-09-16T04:41:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.972+0000 7f774be8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.972+0000 7f774be8d640 1 -- 172.21.15.46:0/1318485870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77440a3c10 msgr2=0x7f774409bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.972+0000 7f774be8d640 1 --2- 172.21.15.46:0/1318485870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77440a3c10 0x7f774409bcf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f77400097a0 tx=0x7f774002a110 comp rx=0 tx=0).stop 2024-09-16T04:41:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.972+0000 7f774be8d640 1 -- 172.21.15.46:0/1318485870 shutdown_connections 2024-09-16T04:41:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.972+0000 7f774be8d640 1 --2- 172.21.15.46:0/1318485870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77440a3c10 0x7f774409bcf0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.972+0000 7f774be8d640 1 -- 172.21.15.46:0/1318485870 >> 172.21.15.46:0/1318485870 conn(0x7f77440a5d10 msgr2=0x7f77440a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.972+0000 7f774be8d640 1 -- 172.21.15.46:0/1318485870 shutdown_connections 2024-09-16T04:41:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.972+0000 7f774be8d640 1 -- 172.21.15.46:0/1318485870 wait complete. 2024-09-16T04:41:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.973+0000 7f774be8d640 10 monclient: build_initial_monmap 2024-09-16T04:41:19.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.973+0000 7f774be8d640 10 monclient: monmap: 2024-09-16T04:41:19.987 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:19.987 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:41:19.987 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:41:19.987 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:41:19.987 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:19.987 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.987 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:41:19.987 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:41:19.987 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:41:19.987 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.973+0000 7f774be8d640 1 Processor -- start 2024-09-16T04:41:19.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.973+0000 7f774be8d640 1 -- start start 2024-09-16T04:41:19.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.973+0000 7f774be8d640 10 monclient: init 2024-09-16T04:41:19.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.973+0000 7f774be8d640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:19.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.973+0000 7f774be8d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:41:19.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.973+0000 7f774be8d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77440a3c10 0x7f7744136670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.973+0000 7f774be8d640 10 monclient(hunting): picked mon.noname-c con 0x7f77440a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:19.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.973+0000 7f774be8d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77440a3ff0 0x7f7744136bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.973+0000 7f774be8d640 10 monclient(hunting): picked mon.noname-b con 0x7f77440a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:41:19.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.973+0000 7f774be8d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f774413be90 0x7f774413c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.973+0000 7f774be8d640 10 monclient(hunting): picked mon.noname-a con 0x7f774413be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:41:19.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.973+0000 7f774be8d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.973+0000 7f774be8d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f774406e550 con 0x7f774413be90 2024-09-16T04:41:19.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.973+0000 7f774be8d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.973+0000 7f774be8d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f774406e3d0 con 0x7f77440a3ff0 2024-09-16T04:41:19.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.973+0000 7f774be8d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.973+0000 7f774be8d640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f774413f9a0 con 0x7f77440a3c10 2024-09-16T04:41:19.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.973+0000 7f774be8d640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:19.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.973+0000 7f774be8d640 10 monclient(hunting): authenticate will time out at 878.389966s 2024-09-16T04:41:19.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774a42a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f774413be90 0x7f774413c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f7749c29640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77440a3c10 0x7f7744136670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774a42a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f774413be90 0x7f774413c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60746/0 (socket says 172.21.15.46:60746) 2024-09-16T04:41:19.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774a42a640 1 -- 172.21.15.46:0/2790204361 learned_addr learned my addr 172.21.15.46:0/2790204361 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:19.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774a42a640 10 monclient(hunting): get_auth_request con 0x7f774413be90 auth_method 0 2024-09-16T04:41:19.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774a42a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774a42a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774a42a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f7749c29640 10 monclient(hunting): get_auth_request con 0x7f77440a3c10 auth_method 0 2024-09-16T04:41:19.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f7749c29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f7749c29640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f7749c29640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f7749428640 1 --2- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77440a3ff0 0x7f7744136bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774a42a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774a42a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774a42a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f7749c29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f7749c29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f7749c29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f7749428640 10 monclient(hunting): get_auth_request con 0x7f77440a3ff0 auth_method 0 2024-09-16T04:41:19.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f7749428640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f7749428640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f7749428640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f7749428640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f7749428640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f7749428640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774a42a640 10 monclient(hunting): handle_auth_done global_id 34179 payload 824 2024-09-16T04:41:19.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774a42a640 1 -- 172.21.15.46:0/2790204361 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77440a3c10 msgr2=0x7f7744136670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774a42a640 1 --2- 172.21.15.46:0/2790204361 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77440a3c10 0x7f7744136670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774a42a640 1 -- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77440a3ff0 msgr2=0x7f7744136bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774a42a640 1 --2- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77440a3ff0 0x7f7744136bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774a42a640 10 monclient: _finish_hunting 0 2024-09-16T04:41:19.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774a42a640 1 monclient: found mon.noname-a 2024-09-16T04:41:19.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774a42a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:19.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774a42a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:41:19.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774a42a640 1 -- 172.21.15.46:0/2790204361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77440a0750 con 0x7f774413be90 2024-09-16T04:41:19.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f7749c29640 1 --2- 172.21.15.46:0/2790204361 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77440a3c10 0x7f7744136670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:41:19.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774a42a640 1 --2- 172.21.15.46:0/2790204361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f774413be90 0x7f774413c270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f773c00c800 tx=0x7f773c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:19.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f773a7fc640 1 -- 172.21.15.46:0/2790204361 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c018070 con 0x7f774413be90 2024-09-16T04:41:19.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f773a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:19.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f773a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:41:19.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f773a7fc640 10 monclient: dump: 2024-09-16T04:41:19.992 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:19.992 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.992 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.993 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.993 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:19.993 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.993 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:19.993 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:19.993 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:19.993 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f773a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:41:19.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f773a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:41:19.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f773a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:19.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f773a7fc640 1 -- 172.21.15.46:0/2790204361 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f773c004d80 con 0x7f774413be90 2024-09-16T04:41:19.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774be8d640 5 monclient: authenticate success, global_id 34179 2024-09-16T04:41:19.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f773a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:19.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f773a7fc640 1 -- 172.21.15.46:0/2790204361 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c00a820 con 0x7f774413be90 2024-09-16T04:41:19.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774be8d640 10 monclient: _renew_subs 2024-09-16T04:41:19.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774be8d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:41:19.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774be8d640 1 -- 172.21.15.46:0/2790204361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77440a0400 con 0x7f774413be90 2024-09-16T04:41:19.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f773a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:19.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f773a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:41:19.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f773bfff640 4 set_mon_vals no callback set 2024-09-16T04:41:19.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f773a7fc640 10 monclient: dump: 2024-09-16T04:41:19.994 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:19.994 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.994 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.995 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.995 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:19.995 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.995 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:19.995 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:19.995 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:19.995 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774be8d640 10 monclient: _renew_subs 2024-09-16T04:41:19.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774be8d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:41:19.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.974+0000 7f774be8d640 1 -- 172.21.15.46:0/2790204361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f774413e720 con 0x7f774413be90 2024-09-16T04:41:19.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.975+0000 7f774be8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:19.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.975+0000 7f774be8d640 1 --2- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f774413c7b0 0x7f774413cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.975+0000 7f774be8d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:19.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.975+0000 7f774be8d640 1 -- 172.21.15.46:0/2790204361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f774409bcf0 con 0x7f774413c7b0 2024-09-16T04:41:19.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.975+0000 7f773a7fc640 1 -- 172.21.15.46:0/2790204361 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f773c01b070 con 0x7f774413be90 2024-09-16T04:41:19.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.975+0000 7f773a7fc640 1 --2- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f77280383b0 0x7f772803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:19.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.975+0000 7f773a7fc640 1 -- 172.21.15.46:0/2790204361 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 4738+0+0 (secure 0 0 0) 0x7f773c04ad70 con 0x7f774413be90 2024-09-16T04:41:19.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.975+0000 7f7749c29640 1 --2- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f774413c7b0 0x7f774413cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.976+0000 7f7749c29640 10 monclient: get_auth_request con 0x7f774413c7b0 auth_method 0 2024-09-16T04:41:19.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.976+0000 7f7749c29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:19.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.976+0000 7f7749c29640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:19.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.976+0000 7f7749c29640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:19.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.976+0000 7f7749428640 1 --2- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f77280383b0 0x7f772803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:19.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.976+0000 7f7749c29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:19.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.976+0000 7f7749c29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:19.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.976+0000 7f7749c29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:19.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.976+0000 7f7749428640 10 monclient: get_auth_request con 0x7f77280383b0 auth_method 0 2024-09-16T04:41:19.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.976+0000 7f7749c29640 10 monclient(hunting): handle_auth_done global_id 5227 payload 824 2024-09-16T04:41:19.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.976+0000 7f7749c29640 1 -- 172.21.15.46:0/2790204361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7744069150 con 0x7f774413c7b0 2024-09-16T04:41:19.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.976+0000 7f7749428640 1 --2- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f77280383b0 0x7f772803a870 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7734000f60 tx=0x7f7734009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:19.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.976+0000 7f7749c29640 1 --2- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f774413c7b0 0x7f774413cb90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f774409b8d0 tx=0x7f77400046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:19.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.976+0000 7f773a7fc640 1 -- 172.21.15.46:0/2790204361 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7740041030 con 0x7f774413c7b0 2024-09-16T04:41:19.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.976+0000 7f773a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:19.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.976+0000 7f773a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:41:19.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.976+0000 7f773a7fc640 10 monclient: dump: 2024-09-16T04:41:19.997 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:19.997 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:19.997 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.997 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:19.997 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:19.997 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:19.998 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:19.998 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:19.998 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:19.998 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:19.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.977+0000 7f773a7fc640 1 -- 172.21.15.46:0/2790204361 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7740038040 con 0x7f774413c7b0 2024-09-16T04:41:19.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.977+0000 7f773a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:19.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.977+0000 7f773a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:41:19.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.977+0000 7f773a7fc640 1 -- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f774413c7b0 msgr2=0x7f774413cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:19.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.977+0000 7f773a7fc640 1 --2- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f774413c7b0 0x7f774413cb90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f774409b8d0 tx=0x7f77400046c0 comp rx=0 tx=0).stop 2024-09-16T04:41:19.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.977+0000 7f773a7fc640 1 -- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f774413c7b0 msgr2=0x7f774413cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:41:19.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.977+0000 7f773a7fc640 1 --2- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f774413c7b0 0x7f774413cb90 secure :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f774409b8d0 tx=0x7f77400046c0 comp rx=0 tx=0).stop 2024-09-16T04:41:20.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.983+0000 7f774be8d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:41:20.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.983+0000 7f774be8d640 1 --2- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f774413d0d0 0x7f774409d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:20.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.983+0000 7f774be8d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:20.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.983+0000 7f774be8d640 1 -- 172.21.15.46:0/2790204361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7744067e40 con 0x7f774413d0d0 2024-09-16T04:41:20.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.983+0000 7f7749c29640 1 --2- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f774413d0d0 0x7f774409d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:20.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.983+0000 7f7749c29640 10 monclient: get_auth_request con 0x7f774413d0d0 auth_method 0 2024-09-16T04:41:20.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.983+0000 7f7749c29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:20.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.983+0000 7f7749c29640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.983+0000 7f7749c29640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.983+0000 7f7749c29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.983+0000 7f7749c29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.983+0000 7f7749c29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.983+0000 7f7749c29640 10 monclient(hunting): handle_auth_done global_id 5230 payload 824 2024-09-16T04:41:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.983+0000 7f7749c29640 1 -- 172.21.15.46:0/2790204361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f77441370f0 con 0x7f774413d0d0 2024-09-16T04:41:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f7749c29640 1 --2- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f774413d0d0 0x7f774409d0f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7740041000 tx=0x7f7740009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f773a7fc640 1 -- 172.21.15.46:0/2790204361 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7740037070 con 0x7f774413d0d0 2024-09-16T04:41:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f773a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f773a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:41:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f773a7fc640 10 monclient: dump: 2024-09-16T04:41:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:20.002 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f773a7fc640 1 -- 172.21.15.46:0/2790204361 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f77400513f0 con 0x7f774413d0d0 2024-09-16T04:41:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f773a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:41:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f773a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:41:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f773a7fc640 1 -- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f774413d0d0 msgr2=0x7f774409d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f773a7fc640 1 --2- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f774413d0d0 0x7f774409d0f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7740041000 tx=0x7f7740009190 comp rx=0 tx=0).stop 2024-09-16T04:41:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f773a7fc640 1 -- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f774413d0d0 msgr2=0x7f774409d0f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:41:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f773a7fc640 1 --2- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f774413d0d0 0x7f774409d0f0 secure :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7740041000 tx=0x7f7740009190 comp rx=0 tx=0).stop 2024-09-16T04:41:20.003 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:41:20.003 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:41:20.003 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:41:20.003 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:41:20.003 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 26, 2024-09-16T04:41:20.003 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 165886, 2024-09-16T04:41:20.003 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:41:20.003 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:41:20.003 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:41:20.004 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:41:20.004 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:20.004 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 3, 2024-09-16T04:41:20.004 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:41:20.004 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:41:20.004 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:41:20.004 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:20.004 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:20.004 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:20.004 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:20.004 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:20.004 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:20.004 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:20.004 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:20.005 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:20.005 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:20.005 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:20.005 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:20.005 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:41:20.005 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:41:20.005 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:20.005 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:20.005 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:20.005 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:20.005 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:20.005 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:20.005 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:20.005 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:20.006 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:20.006 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:20.006 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:20.006 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:20.006 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:20.006 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:41:20.006 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:41:20.006 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:41:20.006 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:41:20.006 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:41:20.006 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:41:20.006 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:41:20.006 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:41:20.006 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:41:20.007 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:41:20.007 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:41:20.007 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:41:20.007 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:41:20.007 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:41:20.007 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:41:20.007 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:41:20.007 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:41:20.007 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:20.007 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:20.007 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:20.007 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:20.007 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:20.007 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:20.008 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:20.008 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:20.008 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:20.008 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:20.008 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:20.008 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:20.008 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:41:20.008 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:20.008 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:41:20.008 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:20.008 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:41:20.008 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:41:20.008 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:20.009 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:20.012 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:20.012 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:20.012 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:41:20.012 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:20.012 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:20.012 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:20.012 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:20.012 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:41:20.012 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:20.012 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:20.013 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:20.013 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:20.013 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:41:20.013 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:41:20.013 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:20.013 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:20.013 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:20.013 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:20.013 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:20.013 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:41:20.013 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:41:20.013 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:20.013 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:20.013 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:20.014 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:20.014 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:41:20.014 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:20.014 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:20.014 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:20.014 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:20.014 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:41:20.014 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:20.014 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:20.014 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:20.014 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:20.014 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:41:20.014 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:41:20.014 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:20.015 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:20.015 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:20.015 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:20.015 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:20.015 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:41:20.015 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:41:20.015 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:20.015 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:20.015 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:20.015 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:20.015 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:41:20.015 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:20.015 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:20.015 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:20.016 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:20.016 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:41:20.016 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:20.016 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:20.016 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:20.016 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:20.016 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:41:20.016 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:41:20.016 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:20.016 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:20.016 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:20.016 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:20.016 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:20.017 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:20.017 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:41:20.017 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:41:20.017 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:20.017 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:20.017 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:20.017 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:41:20.017 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:20.017 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:20.017 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:41:20.017 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:20.017 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:20.017 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:20.017 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4 2024-09-16T04:41:20.018 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:20.018 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:20.018 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:41:20.018 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:20.018 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:20.018 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:20.018 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5 2024-09-16T04:41:20.018 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:20.018 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:20.018 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:41:20.018 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:20.018 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:20.018 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:20.019 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:41:20.019 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:20.019 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:20.019 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:20.019 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:41:20.019 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:41:20.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f774be8d640 1 -- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f77280383b0 msgr2=0x7f772803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:20.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f774be8d640 1 --2- 172.21.15.46:0/2790204361 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f77280383b0 0x7f772803a870 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7734000f60 tx=0x7f7734009040 comp rx=0 tx=0).stop 2024-09-16T04:41:20.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f774be8d640 10 monclient: shutdown 2024-09-16T04:41:20.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f774be8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:20.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f774be8d640 1 -- 172.21.15.46:0/2790204361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f774413be90 msgr2=0x7f774413c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:20.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f774be8d640 1 --2- 172.21.15.46:0/2790204361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f774413be90 0x7f774413c270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f773c00c800 tx=0x7f773c00ccc0 comp rx=0 tx=0).stop 2024-09-16T04:41:20.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f774a42a640 1 -- 172.21.15.46:0/2790204361 reap_dead start 2024-09-16T04:41:20.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f774a42a640 1 -- 172.21.15.46:0/2790204361 reap_dead start 2024-09-16T04:41:20.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f774be8d640 1 -- 172.21.15.46:0/2790204361 shutdown_connections 2024-09-16T04:41:20.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f774be8d640 1 -- 172.21.15.46:0/2790204361 >> 172.21.15.46:0/2790204361 conn(0x7f77440a5d10 msgr2=0x7f77440a2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:20.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f774be8d640 1 -- 172.21.15.46:0/2790204361 shutdown_connections 2024-09-16T04:41:20.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:19.984+0000 7f774be8d640 1 -- 172.21.15.46:0/2790204361 wait complete. 2024-09-16T04:41:20.021 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:41:20.022 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T04:41:20.022 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:41:20.022 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T04:41:20.022 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:41:20.022 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T04:41:20.022 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:41:20.032 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T04:41:20.032 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:41:20.033 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:41:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.171+0000 7fc30a09b640 10 monclient: get_monmap_and_config 2024-09-16T04:41:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.171+0000 7fc30a09b640 10 monclient: build_initial_monmap 2024-09-16T04:41:20.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.171+0000 7fc30a09b640 10 monclient: monmap: 2024-09-16T04:41:20.174 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:20.174 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:20.175 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:41:20.173093+0000 2024-09-16T04:41:20.175 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:41:20.173093+0000 2024-09-16T04:41:20.175 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:20.175 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:20.175 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:41:20.175 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:41:20.175 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:41:20.175 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:41:20.175 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:41:20.175 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:20.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.172+0000 7fc30a09b640 1 Processor -- start 2024-09-16T04:41:20.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.172+0000 7fc30a09b640 1 -- start start 2024-09-16T04:41:20.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.172+0000 7fc30a09b640 10 monclient: init 2024-09-16T04:41:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.172+0000 7fc30a09b640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.172+0000 7fc30a09b640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T04:41:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.172+0000 7fc30a09b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc3040a3ba0 addr v1:172.21.15.46:6789/0 2024-09-16T04:41:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.172+0000 7fc30a09b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc3040a3f80 0x7fc3040a4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.172+0000 7fc30a09b640 10 monclient(hunting): picked mon.noname-b con 0x7fc3040a3f80 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.172+0000 7fc30a09b640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc30413bf00 0x7fc30413e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.172+0000 7fc30a09b640 10 monclient(hunting): picked mon.noname-c con 0x7fc30413bf00 addr v2:172.21.15.114:3300/0 2024-09-16T04:41:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.172+0000 7fc30a09b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc30409c390 con 0x7fc3040a3ba0 2024-09-16T04:41:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.172+0000 7fc30a09b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:20.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.172+0000 7fc30a09b640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fc30409b860 con 0x7fc30413bf00 2024-09-16T04:41:20.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.172+0000 7fc30a09b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:20.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.172+0000 7fc30a09b640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fc30409b2a0 con 0x7fc3040a3f80 2024-09-16T04:41:20.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.172+0000 7fc30a09b640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:20.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.172+0000 7fc30a09b640 10 monclient(hunting): authenticate will time out at 878.588954s 2024-09-16T04:41:20.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.172+0000 7fc3037fe640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7fc3040a3ba0 legacy=0x7fc30409bc80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:41:20.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc302ffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc3040a3f80 0x7fc3040a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:20.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc302ffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc3040a3f80 0x7fc3040a4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38192/0 (socket says 172.21.15.46:38192) 2024-09-16T04:41:20.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc302ffd640 1 -- 172.21.15.46:0/4099362786 learned_addr learned my addr 172.21.15.46:0/4099362786 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:20.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc302ffd640 10 monclient(hunting): get_auth_request con 0x7fc3040a3f80 auth_method 0 2024-09-16T04:41:20.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc302ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:20.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc302ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:20.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc302ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:20.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc302ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:20.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc302ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:20.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc302ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:20.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc303fff640 1 --2- 172.21.15.46:0/4099362786 >> v2:172.21.15.114:3300/0 conn(0x7fc30413bf00 0x7fc30413e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:20.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc303fff640 10 monclient(hunting): get_auth_request con 0x7fc30413bf00 auth_method 0 2024-09-16T04:41:20.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc303fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:20.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc303fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:20.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc303fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:20.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc302ffd640 10 monclient(hunting): handle_auth_done global_id 24401 payload 274 2024-09-16T04:41:20.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc302ffd640 1 -- 172.21.15.46:0/4099362786 >> v2:172.21.15.114:3300/0 conn(0x7fc30413bf00 msgr2=0x7fc30413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:41:20.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc302ffd640 1 --2- 172.21.15.46:0/4099362786 >> v2:172.21.15.114:3300/0 conn(0x7fc30413bf00 0x7fc30413e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:20.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc302ffd640 1 -- 172.21.15.46:0/4099362786 >> v1:172.21.15.46:6789/0 conn(0x7fc3040a3ba0 legacy=0x7fc30409bc80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:41:20.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc302ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:41:20.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc302ffd640 1 monclient: found mon.noname-b 2024-09-16T04:41:20.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc302ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:20.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc302ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:41:20.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.173+0000 7fc302ffd640 1 -- 172.21.15.46:0/4099362786 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30409afc0 con 0x7fc3040a3f80 2024-09-16T04:41:20.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc302ffd640 1 --2- 172.21.15.46:0/4099362786 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc3040a3f80 0x7fc3040a4360 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec009870 tx=0x7fc2ec02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9660c0e27aae0ec4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:20.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc3027fc640 1 -- 172.21.15.46:0/4099362786 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec037070 con 0x7fc3040a3f80 2024-09-16T04:41:20.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc3027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:20.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc3027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:41:20.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc3027fc640 10 monclient: dump: 2024-09-16T04:41:20.187 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:20.187 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:20.187 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:20.187 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:20.187 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:20.188 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:20.188 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:20.188 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:20.188 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:20.188 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:20.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc3027fc640 10 monclient: _finish_auth 0 2024-09-16T04:41:20.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc3027fc640 10 monclient: _check_auth_tickets 2024-09-16T04:41:20.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc3027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:20.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc3027fc640 1 -- 172.21.15.46:0/4099362786 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2ec032440 con 0x7fc3040a3f80 2024-09-16T04:41:20.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc30a09b640 5 monclient: authenticate success, global_id 24401 2024-09-16T04:41:20.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc30a09b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:41:20.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc3027fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:20.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc30a09b640 10 monclient: get_monmap_and_config success 2024-09-16T04:41:20.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc3027fc640 1 -- 172.21.15.46:0/4099362786 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec032810 con 0x7fc3040a3f80 2024-09-16T04:41:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc30a09b640 4 set_mon_vals no callback set 2024-09-16T04:41:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc3027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc3027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc3027fc640 10 monclient: dump: 2024-09-16T04:41:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:20.190 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:20.190 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:20.190 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:20.190 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:20.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc30a09b640 10 monclient: shutdown 2024-09-16T04:41:20.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc30a09b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:20.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc30a09b640 1 -- 172.21.15.46:0/4099362786 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc3040a3f80 msgr2=0x7fc3040a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:20.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc30a09b640 1 --2- 172.21.15.46:0/4099362786 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc3040a3f80 0x7fc3040a4360 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec009870 tx=0x7fc2ec02a180 comp rx=0 tx=0).stop 2024-09-16T04:41:20.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc30a09b640 1 -- 172.21.15.46:0/4099362786 shutdown_connections 2024-09-16T04:41:20.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc30a09b640 1 --2- 172.21.15.46:0/4099362786 >> v2:172.21.15.114:3300/0 conn(0x7fc30413bf00 0x7fc30413e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:20.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc30a09b640 1 --2- 172.21.15.46:0/4099362786 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc3040a3f80 0x7fc3040a4360 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:20.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc30a09b640 1 -- 172.21.15.46:0/4099362786 >> 172.21.15.46:0/4099362786 conn(0x7fc3040a60c0 msgr2=0x7fc3040a64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:20.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc30a09b640 1 -- 172.21.15.46:0/4099362786 shutdown_connections 2024-09-16T04:41:20.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.174+0000 7fc30a09b640 1 -- 172.21.15.46:0/4099362786 wait complete. 2024-09-16T04:41:20.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.175+0000 7fc30a09b640 10 monclient: build_initial_monmap 2024-09-16T04:41:20.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.175+0000 7fc30a09b640 10 monclient: monmap: 2024-09-16T04:41:20.191 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:20.191 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:41:20.191 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:41:20.191 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:41:20.191 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:20.191 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:20.191 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:41:20.191 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:41:20.191 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:41:20.191 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:20.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.175+0000 7fc30a09b640 1 Processor -- start 2024-09-16T04:41:20.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.175+0000 7fc30a09b640 1 -- start start 2024-09-16T04:41:20.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.175+0000 7fc30a09b640 10 monclient: init 2024-09-16T04:41:20.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc30a09b640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:20.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc30a09b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:41:20.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc30a09b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc3040a3ba0 0x7fc3041cbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:20.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc30a09b640 10 monclient(hunting): picked mon.noname-c con 0x7fc3040a3ba0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:20.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc30a09b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3040a3f80 0x7fc304137010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:20.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc30a09b640 10 monclient(hunting): picked mon.noname-a con 0x7fc3040a3f80 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:41:20.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc30a09b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc30413bf00 0x7fc30413a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:20.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc30a09b640 10 monclient(hunting): picked mon.noname-b con 0x7fc30413bf00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:41:20.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc30a09b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:20.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc30a09b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc3040a5540 con 0x7fc3040a3f80 2024-09-16T04:41:20.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc30a09b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:20.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc30a09b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc3040a53c0 con 0x7fc30413bf00 2024-09-16T04:41:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc30a09b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc30a09b640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fc30413e830 con 0x7fc3040a3ba0 2024-09-16T04:41:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc30a09b640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc30a09b640 10 monclient(hunting): authenticate will time out at 878.592411s 2024-09-16T04:41:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc302ffd640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3040a3f80 msgr2=0x7fc304137010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:41:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc302ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3040a3f80 0x7fc304137010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:41:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc3037fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc3040a3ba0 0x7fc3041cbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc3037fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc3040a3ba0 0x7fc3041cbd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38200/0 (socket says 172.21.15.46:38200) 2024-09-16T04:41:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc3037fe640 1 -- 172.21.15.46:0/2218298060 learned_addr learned my addr 172.21.15.46:0/2218298060 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc3037fe640 10 monclient(hunting): get_auth_request con 0x7fc3040a3ba0 auth_method 0 2024-09-16T04:41:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc3037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc3037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc3037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc3037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc3037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc3037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc303fff640 1 --2- 172.21.15.46:0/2218298060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc30413bf00 0x7fc30413a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc303fff640 10 monclient(hunting): get_auth_request con 0x7fc30413bf00 auth_method 0 2024-09-16T04:41:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc303fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc303fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc303fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc3037fe640 10 monclient(hunting): handle_auth_done global_id 24404 payload 824 2024-09-16T04:41:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc3037fe640 1 -- 172.21.15.46:0/2218298060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc30413bf00 msgr2=0x7fc30413a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc3037fe640 1 --2- 172.21.15.46:0/2218298060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc30413bf00 0x7fc30413a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc3037fe640 1 -- 172.21.15.46:0/2218298060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3040a3f80 msgr2=0x7fc304137010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:41:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc3037fe640 1 --2- 172.21.15.46:0/2218298060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3040a3f80 0x7fc304137010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc3037fe640 10 monclient: _finish_hunting 0 2024-09-16T04:41:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc3037fe640 1 monclient: found mon.noname-c 2024-09-16T04:41:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc3037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc3037fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:41:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc3037fe640 1 -- 172.21.15.46:0/2218298060 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30413c650 con 0x7fc3040a3ba0 2024-09-16T04:41:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.176+0000 7fc3037fe640 1 --2- 172.21.15.46:0/2218298060 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc3040a3ba0 0x7fc3041cbd10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc2f400b550 tx=0x7fc2f400ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.177+0000 7fc2fbfff640 1 -- 172.21.15.46:0/2218298060 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f4003dc0 con 0x7fc3040a3ba0 2024-09-16T04:41:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.177+0000 7fc2fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.177+0000 7fc2fbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:41:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.177+0000 7fc2fbfff640 10 monclient: dump: 2024-09-16T04:41:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:20.196 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:20.196 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:20.196 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:20.196 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:20.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.177+0000 7fc2fbfff640 10 monclient: _finish_auth 0 2024-09-16T04:41:20.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.177+0000 7fc2fbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:41:20.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.177+0000 7fc2fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:20.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.177+0000 7fc2fbfff640 1 -- 172.21.15.46:0/2218298060 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f4015070 con 0x7fc3040a3ba0 2024-09-16T04:41:20.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.177+0000 7fc30a09b640 5 monclient: authenticate success, global_id 24404 2024-09-16T04:41:20.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.177+0000 7fc30a09b640 10 monclient: _renew_subs 2024-09-16T04:41:20.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.177+0000 7fc30a09b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:20.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.177+0000 7fc30a09b640 1 -- 172.21.15.46:0/2218298060 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30413e5d0 con 0x7fc3040a3ba0 2024-09-16T04:41:20.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.177+0000 7fc2fbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:20.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.177+0000 7fc2fbfff640 1 -- 172.21.15.46:0/2218298060 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f40052b0 con 0x7fc3040a3ba0 2024-09-16T04:41:20.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.177+0000 7fc2fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:20.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.177+0000 7fc2fbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:20.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.177+0000 7fc2fbfff640 10 monclient: dump: 2024-09-16T04:41:20.197 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:20.197 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:20.197 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:20.197 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:20.197 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:20.197 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:20.197 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:20.197 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:20.197 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:20.197 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:20.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.177+0000 7fc30a09b640 10 monclient: _renew_subs 2024-09-16T04:41:20.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.177+0000 7fc30a09b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:20.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.177+0000 7fc3017fa640 4 set_mon_vals no callback set 2024-09-16T04:41:20.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.177+0000 7fc30a09b640 1 -- 172.21.15.46:0/2218298060 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc304120430 con 0x7fc3040a3ba0 2024-09-16T04:41:20.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.178+0000 7fc2fbfff640 1 -- 172.21.15.46:0/2218298060 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fc2f4005450 con 0x7fc3040a3ba0 2024-09-16T04:41:20.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.178+0000 7fc30a09b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:20.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.178+0000 7fc30a09b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:20.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.178+0000 7fc30a09b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:20.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.178+0000 7fc30a09b640 1 -- 172.21.15.46:0/2218298060 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3040a4360 con 0x7fc3040a3ba0 2024-09-16T04:41:20.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.178+0000 7fc2fbfff640 1 --2- 172.21.15.46:0/2218298060 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc2e4038360 0x7fc2e403a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:20.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.178+0000 7fc2fbfff640 1 -- 172.21.15.46:0/2218298060 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(112..112 src has 1..112) ==== 4738+0+0 (secure 0 0 0) 0x7fc2f404b550 con 0x7fc3040a3ba0 2024-09-16T04:41:20.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.178+0000 7fc302ffd640 1 --2- 172.21.15.46:0/2218298060 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc2e4038360 0x7fc2e403a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:20.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.178+0000 7fc302ffd640 10 monclient: get_auth_request con 0x7fc2e4038360 auth_method 0 2024-09-16T04:41:20.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.179+0000 7fc302ffd640 1 --2- 172.21.15.46:0/2218298060 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc2e4038360 0x7fc2e403a820 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec034000 tx=0x7fc2ec02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:20.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.183+0000 7fc2fbfff640 1 -- 172.21.15.46:0/2218298060 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc2f40193c0 con 0x7fc3040a3ba0 2024-09-16T04:41:20.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.183+0000 7fc2fbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:20.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.183+0000 7fc2fbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:41:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.428+0000 7fc30a09b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:41:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.428+0000 7fc30a09b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:41:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.428+0000 7fc30a09b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.428+0000 7fc30a09b640 1 -- 172.21.15.46:0/2218298060 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc3040a0390 con 0x7fc3040a3ba0 2024-09-16T04:41:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.429+0000 7fc2fbfff640 1 -- 172.21.15.46:0/2218298060 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fc2f400de10 con 0x7fc3040a3ba0 2024-09-16T04:41:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.429+0000 7fc2fbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:41:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.429+0000 7fc2fbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:41:20.431 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:41:20.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.431+0000 7fc30a09b640 1 -- 172.21.15.46:0/2218298060 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc2e4038360 msgr2=0x7fc2e403a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:20.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.431+0000 7fc30a09b640 1 --2- 172.21.15.46:0/2218298060 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc2e4038360 0x7fc2e403a820 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec034000 tx=0x7fc2ec02e000 comp rx=0 tx=0).stop 2024-09-16T04:41:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.431+0000 7fc30a09b640 10 monclient: shutdown 2024-09-16T04:41:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.431+0000 7fc30a09b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.431+0000 7fc30a09b640 1 -- 172.21.15.46:0/2218298060 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc3040a3ba0 msgr2=0x7fc3041cbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.431+0000 7fc30a09b640 1 --2- 172.21.15.46:0/2218298060 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc3040a3ba0 0x7fc3041cbd10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc2f400b550 tx=0x7fc2f400ba10 comp rx=0 tx=0).stop 2024-09-16T04:41:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.431+0000 7fc30a09b640 1 -- 172.21.15.46:0/2218298060 shutdown_connections 2024-09-16T04:41:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.431+0000 7fc30a09b640 1 --2- 172.21.15.46:0/2218298060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc30413bf00 0x7fc30413a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.431+0000 7fc30a09b640 1 --2- 172.21.15.46:0/2218298060 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc2e4038360 0x7fc2e403a820 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.431+0000 7fc30a09b640 1 --2- 172.21.15.46:0/2218298060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3040a3f80 0x7fc304137010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.431+0000 7fc30a09b640 1 --2- 172.21.15.46:0/2218298060 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc3040a3ba0 0x7fc3041cbd10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.431+0000 7fc30a09b640 1 -- 172.21.15.46:0/2218298060 >> 172.21.15.46:0/2218298060 conn(0x7fc3040a60c0 msgr2=0x7fc3040a2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.431+0000 7fc30a09b640 1 -- 172.21.15.46:0/2218298060 shutdown_connections 2024-09-16T04:41:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:20.431+0000 7fc30a09b640 1 -- 172.21.15.46:0/2218298060 wait complete. 2024-09-16T04:41:23.445 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:41:23.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.556+0000 7fb9b3265640 10 monclient: get_monmap_and_config 2024-09-16T04:41:23.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.556+0000 7fb9b3265640 10 monclient: build_initial_monmap 2024-09-16T04:41:23.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.557+0000 7fb9b3265640 10 monclient: monmap: 2024-09-16T04:41:23.559 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:23.559 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:23.559 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:41:23.558491+0000 2024-09-16T04:41:23.559 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:41:23.558491+0000 2024-09-16T04:41:23.559 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:23.559 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:23.560 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:41:23.560 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:41:23.560 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:41:23.560 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:41:23.560 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:41:23.560 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:23.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.557+0000 7fb9b3265640 1 Processor -- start 2024-09-16T04:41:23.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.558+0000 7fb9b3265640 1 -- start start 2024-09-16T04:41:23.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.558+0000 7fb9b3265640 10 monclient: init 2024-09-16T04:41:23.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.558+0000 7fb9b3265640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:23.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.558+0000 7fb9b3265640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-16T04:41:23.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.558+0000 7fb9b3265640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb9ac0a3c10 0x7fb9ac09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:23.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.558+0000 7fb9b3265640 10 monclient(hunting): picked mon.noname-a con 0x7fb9ac0a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:41:23.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.558+0000 7fb9b3265640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb9ac0a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:41:23.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.558+0000 7fb9b3265640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb9ac13be90 0x7fb9ac13e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:23.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.558+0000 7fb9b3265640 10 monclient(hunting): picked mon.noname-b con 0x7fb9ac13be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:23.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.558+0000 7fb9b3265640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9ac09b940 con 0x7fb9ac0a3ff0 2024-09-16T04:41:23.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.558+0000 7fb9b3265640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.558+0000 7fb9b3265640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb9ac09ab00 con 0x7fb9ac0a3c10 2024-09-16T04:41:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.558+0000 7fb9b3265640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.558+0000 7fb9b3265640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fb9ac09b030 con 0x7fb9ac13be90 2024-09-16T04:41:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.558+0000 7fb9b3265640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.558+0000 7fb9b3265640 10 monclient(hunting): authenticate will time out at 881.974642s 2024-09-16T04:41:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.558+0000 7fb9b0800640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7fb9ac0a3ff0 legacy=0x7fb9ac0a43d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:41:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.558+0000 7fb9b1001640 1 -- >> v2:172.21.15.46:3300/0 conn(0x7fb9ac0a3c10 msgr2=0x7fb9ac09bcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:41:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.558+0000 7fb9b1001640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb9ac0a3c10 0x7fb9ac09bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:41:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.559+0000 7fb9b1802640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb9ac13be90 0x7fb9ac13e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.559+0000 7fb9b1802640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb9ac13be90 0x7fb9ac13e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38216/0 (socket says 172.21.15.46:38216) 2024-09-16T04:41:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.559+0000 7fb9b1802640 1 -- 172.21.15.46:0/1189916988 learned_addr learned my addr 172.21.15.46:0/1189916988 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.559+0000 7fb9b1802640 10 monclient(hunting): get_auth_request con 0x7fb9ac13be90 auth_method 0 2024-09-16T04:41:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.559+0000 7fb9b1802640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:23.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.559+0000 7fb9b1802640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:23.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.559+0000 7fb9b1802640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:23.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.559+0000 7fb9b1802640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:23.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.559+0000 7fb9b1802640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:23.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.559+0000 7fb9b1802640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:23.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.559+0000 7fb9b1802640 10 monclient(hunting): handle_auth_done global_id 24407 payload 274 2024-09-16T04:41:23.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.559+0000 7fb9b1802640 1 -- 172.21.15.46:0/1189916988 >> v2:172.21.15.46:3300/0 conn(0x7fb9ac0a3c10 msgr2=0x7fb9ac09bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:41:23.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.559+0000 7fb9b1802640 1 --2- 172.21.15.46:0/1189916988 >> v2:172.21.15.46:3300/0 conn(0x7fb9ac0a3c10 0x7fb9ac09bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.559+0000 7fb9b1802640 1 -- 172.21.15.46:0/1189916988 >> v1:172.21.15.46:6789/0 conn(0x7fb9ac0a3ff0 legacy=0x7fb9ac0a43d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:41:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.559+0000 7fb9b1802640 10 monclient: _finish_hunting 0 2024-09-16T04:41:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.559+0000 7fb9b1802640 1 monclient: found mon.noname-b 2024-09-16T04:41:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.559+0000 7fb9b1802640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.559+0000 7fb9b1802640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:41:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.559+0000 7fb9b1802640 1 -- 172.21.15.46:0/1189916988 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9ac09b310 con 0x7fb9ac13be90 2024-09-16T04:41:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.560+0000 7fb9b1802640 1 --2- 172.21.15.46:0/1189916988 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb9ac13be90 0x7fb9ac13e280 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb9a4009e50 tx=0x7fb9a402a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=597e470ffcdf1d05 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.560+0000 7fb99ffff640 1 -- 172.21.15.46:0/1189916988 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a4036070 con 0x7fb9ac13be90 2024-09-16T04:41:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.560+0000 7fb99ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.560+0000 7fb99ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:41:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.560+0000 7fb99ffff640 10 monclient: dump: 2024-09-16T04:41:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:23.572 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:23.572 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:23.572 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:23.572 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:23.572 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:23.572 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:23.572 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:23.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.560+0000 7fb99ffff640 10 monclient: _finish_auth 0 2024-09-16T04:41:23.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.560+0000 7fb99ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:41:23.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.560+0000 7fb99ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:23.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.560+0000 7fb99ffff640 1 -- 172.21.15.46:0/1189916988 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9a4031600 con 0x7fb9ac13be90 2024-09-16T04:41:23.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.560+0000 7fb9b3265640 5 monclient: authenticate success, global_id 24407 2024-09-16T04:41:23.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.560+0000 7fb9b3265640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:41:23.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.560+0000 7fb99ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:23.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.560+0000 7fb9b3265640 10 monclient: get_monmap_and_config success 2024-09-16T04:41:23.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.560+0000 7fb99ffff640 1 -- 172.21.15.46:0/1189916988 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a40319d0 con 0x7fb9ac13be90 2024-09-16T04:41:23.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.560+0000 7fb99ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:23.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.560+0000 7fb9b3265640 4 set_mon_vals no callback set 2024-09-16T04:41:23.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.560+0000 7fb99ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:23.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.560+0000 7fb99ffff640 10 monclient: dump: 2024-09-16T04:41:23.573 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:23.573 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:23.573 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:23.573 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:23.573 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:23.573 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:23.573 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:23.573 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:23.574 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:23.574 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:23.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.561+0000 7fb9b3265640 10 monclient: shutdown 2024-09-16T04:41:23.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.561+0000 7fb9b3265640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:23.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.561+0000 7fb9b3265640 1 -- 172.21.15.46:0/1189916988 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb9ac13be90 msgr2=0x7fb9ac13e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:23.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.561+0000 7fb9b3265640 1 --2- 172.21.15.46:0/1189916988 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb9ac13be90 0x7fb9ac13e280 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb9a4009e50 tx=0x7fb9a402a5d0 comp rx=0 tx=0).stop 2024-09-16T04:41:23.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.561+0000 7fb9b3265640 1 -- 172.21.15.46:0/1189916988 shutdown_connections 2024-09-16T04:41:23.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.561+0000 7fb9b3265640 1 --2- 172.21.15.46:0/1189916988 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb9ac13be90 0x7fb9ac13e280 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:23.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.561+0000 7fb9b3265640 1 --2- 172.21.15.46:0/1189916988 >> v2:172.21.15.46:3300/0 conn(0x7fb9ac0a3c10 0x7fb9ac09bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:23.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.561+0000 7fb9b3265640 1 -- 172.21.15.46:0/1189916988 >> 172.21.15.46:0/1189916988 conn(0x7fb9ac0a6340 msgr2=0x7fb9ac06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:23.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.561+0000 7fb9b3265640 1 -- 172.21.15.46:0/1189916988 shutdown_connections 2024-09-16T04:41:23.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.561+0000 7fb9b3265640 1 -- 172.21.15.46:0/1189916988 wait complete. 2024-09-16T04:41:23.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.561+0000 7fb9b3265640 10 monclient: build_initial_monmap 2024-09-16T04:41:23.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.561+0000 7fb9b3265640 10 monclient: monmap: 2024-09-16T04:41:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:41:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:41:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:41:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:41:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:41:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:41:23.575 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b3265640 1 Processor -- start 2024-09-16T04:41:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b3265640 1 -- start start 2024-09-16T04:41:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b3265640 10 monclient: init 2024-09-16T04:41:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b3265640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b3265640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:41:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b3265640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ac0a3c10 0x7fb9ac133df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b3265640 10 monclient(hunting): picked mon.noname-a con 0x7fb9ac0a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:41:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b3265640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9ac0a3ff0 0x7fb9ac134330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b3265640 10 monclient(hunting): picked mon.noname-b con 0x7fb9ac0a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:41:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b3265640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb9ac13be90 0x7fb9ac1cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b3265640 10 monclient(hunting): picked mon.noname-c con 0x7fb9ac13be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b3265640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b3265640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb9ac0a6b40 con 0x7fb9ac0a3c10 2024-09-16T04:41:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b3265640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b3265640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb9ac0a69c0 con 0x7fb9ac0a3ff0 2024-09-16T04:41:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b3265640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b3265640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fb9ac13e7c0 con 0x7fb9ac13be90 2024-09-16T04:41:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b3265640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b3265640 10 monclient(hunting): authenticate will time out at 881.978729s 2024-09-16T04:41:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b1001640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ac0a3c10 msgr2=0x7fb9ac133df0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:41:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b1001640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ac0a3c10 0x7fb9ac133df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:41:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b1802640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb9ac13be90 0x7fb9ac1cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b1802640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb9ac13be90 0x7fb9ac1cbae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38228/0 (socket says 172.21.15.46:38228) 2024-09-16T04:41:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b1802640 1 -- 172.21.15.46:0/3370371560 learned_addr learned my addr 172.21.15.46:0/3370371560 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b1802640 10 monclient(hunting): get_auth_request con 0x7fb9ac13be90 auth_method 0 2024-09-16T04:41:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b1802640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b1802640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b1802640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b1802640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.562+0000 7fb9b1802640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b1802640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b0800640 1 --2- 172.21.15.46:0/3370371560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9ac0a3ff0 0x7fb9ac134330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:23.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b0800640 10 monclient(hunting): get_auth_request con 0x7fb9ac0a3ff0 auth_method 0 2024-09-16T04:41:23.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b0800640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:23.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b0800640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:23.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b0800640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:23.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b1802640 10 monclient(hunting): handle_auth_done global_id 24410 payload 824 2024-09-16T04:41:23.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b1802640 1 -- 172.21.15.46:0/3370371560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9ac0a3ff0 msgr2=0x7fb9ac134330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:23.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b1802640 1 --2- 172.21.15.46:0/3370371560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9ac0a3ff0 0x7fb9ac134330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:23.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b1802640 1 -- 172.21.15.46:0/3370371560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ac0a3c10 msgr2=0x7fb9ac133df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:41:23.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b1802640 1 --2- 172.21.15.46:0/3370371560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ac0a3c10 0x7fb9ac133df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:23.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b1802640 10 monclient: _finish_hunting 0 2024-09-16T04:41:23.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b1802640 1 monclient: found mon.noname-c 2024-09-16T04:41:23.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b1802640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:23.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b1802640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:41:23.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b1802640 1 -- 172.21.15.46:0/3370371560 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9ac09f140 con 0x7fb9ac13be90 2024-09-16T04:41:23.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b1802640 1 --2- 172.21.15.46:0/3370371560 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb9ac13be90 0x7fb9ac1cbae0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb9a4034040 tx=0x7fb9a4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:23.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb99dffb640 1 -- 172.21.15.46:0/3370371560 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a4041020 con 0x7fb9ac13be90 2024-09-16T04:41:23.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb99dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:23.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb99dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:41:23.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb99dffb640 10 monclient: dump: 2024-09-16T04:41:23.579 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:23.579 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:23.579 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:23.579 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:23.579 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:23.580 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:23.580 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:23.580 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:23.580 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:23.580 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:23.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb99dffb640 10 monclient: _finish_auth 0 2024-09-16T04:41:23.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb99dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:41:23.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb99dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:23.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b3265640 5 monclient: authenticate success, global_id 24410 2024-09-16T04:41:23.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb99dffb640 1 -- 172.21.15.46:0/3370371560 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9a40077a0 con 0x7fb9ac13be90 2024-09-16T04:41:23.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb99dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:23.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb99dffb640 1 -- 172.21.15.46:0/3370371560 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a4036040 con 0x7fb9ac13be90 2024-09-16T04:41:23.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b3265640 10 monclient: _renew_subs 2024-09-16T04:41:23.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b3265640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:23.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b3265640 1 -- 172.21.15.46:0/3370371560 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9ac13c550 con 0x7fb9ac13be90 2024-09-16T04:41:23.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb99dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:23.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb99dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:23.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb99f7fe640 4 set_mon_vals no callback set 2024-09-16T04:41:23.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb99dffb640 10 monclient: dump: 2024-09-16T04:41:23.582 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:23.582 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:23.582 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:23.582 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:23.583 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:23.583 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:23.583 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:23.583 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:23.583 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:23.583 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:23.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b3265640 10 monclient: _renew_subs 2024-09-16T04:41:23.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b3265640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:23.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.563+0000 7fb9b3265640 1 -- 172.21.15.46:0/3370371560 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9ac0a3480 con 0x7fb9ac13be90 2024-09-16T04:41:23.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.564+0000 7fb99dffb640 1 -- 172.21.15.46:0/3370371560 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fb9a4044020 con 0x7fb9ac13be90 2024-09-16T04:41:23.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.564+0000 7fb9b3265640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:23.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.564+0000 7fb9b3265640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:23.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.564+0000 7fb9b3265640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:23.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.564+0000 7fb9b3265640 1 -- 172.21.15.46:0/3370371560 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9ac09bd70 con 0x7fb9ac13be90 2024-09-16T04:41:23.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.565+0000 7fb99dffb640 1 --2- 172.21.15.46:0/3370371560 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb990038360 0x7fb99003a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:23.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.565+0000 7fb99dffb640 1 -- 172.21.15.46:0/3370371560 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(112..112 src has 1..112) ==== 4738+0+0 (secure 0 0 0) 0x7fb9a406c6a0 con 0x7fb9ac13be90 2024-09-16T04:41:23.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.565+0000 7fb9b1001640 1 --2- 172.21.15.46:0/3370371560 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb990038360 0x7fb99003a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:23.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.565+0000 7fb9b1001640 10 monclient: get_auth_request con 0x7fb990038360 auth_method 0 2024-09-16T04:41:23.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.565+0000 7fb9b1001640 1 --2- 172.21.15.46:0/3370371560 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb990038360 0x7fb99003a820 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8003980 tx=0x7fb9a8006d30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:23.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.569+0000 7fb99dffb640 1 -- 172.21.15.46:0/3370371560 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb9a404b030 con 0x7fb9ac13be90 2024-09-16T04:41:23.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.569+0000 7fb99dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:23.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.569+0000 7fb99dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:41:23.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.810+0000 7fb9b3265640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:41:23.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.810+0000 7fb9b3265640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:41:23.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.811+0000 7fb9b3265640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:23.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.811+0000 7fb9b3265640 1 -- 172.21.15.46:0/3370371560 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb9ac067e20 con 0x7fb9ac13be90 2024-09-16T04:41:23.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.811+0000 7fb99dffb640 1 -- 172.21.15.46:0/3370371560 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fb9a403dc70 con 0x7fb9ac13be90 2024-09-16T04:41:23.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.811+0000 7fb99dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:41:23.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.811+0000 7fb99dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:41:23.814 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:41:23.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.813+0000 7fb9b3265640 1 -- 172.21.15.46:0/3370371560 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb990038360 msgr2=0x7fb99003a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:23.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.813+0000 7fb9b3265640 1 --2- 172.21.15.46:0/3370371560 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb990038360 0x7fb99003a820 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8003980 tx=0x7fb9a8006d30 comp rx=0 tx=0).stop 2024-09-16T04:41:23.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.813+0000 7fb9b3265640 10 monclient: shutdown 2024-09-16T04:41:23.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.813+0000 7fb9b3265640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:23.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.813+0000 7fb9b3265640 1 -- 172.21.15.46:0/3370371560 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb9ac13be90 msgr2=0x7fb9ac1cbae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:23.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.813+0000 7fb9b3265640 1 --2- 172.21.15.46:0/3370371560 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb9ac13be90 0x7fb9ac1cbae0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb9a4034040 tx=0x7fb9a4004a40 comp rx=0 tx=0).stop 2024-09-16T04:41:23.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.813+0000 7fb9b3265640 1 -- 172.21.15.46:0/3370371560 shutdown_connections 2024-09-16T04:41:23.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.813+0000 7fb9b3265640 1 --2- 172.21.15.46:0/3370371560 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb9ac13be90 0x7fb9ac1cbae0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:23.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.813+0000 7fb9b3265640 1 --2- 172.21.15.46:0/3370371560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9ac0a3ff0 0x7fb9ac134330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:23.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.813+0000 7fb9b3265640 1 --2- 172.21.15.46:0/3370371560 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb990038360 0x7fb99003a820 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:23.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.813+0000 7fb9b3265640 1 --2- 172.21.15.46:0/3370371560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ac0a3c10 0x7fb9ac133df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:23.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.813+0000 7fb9b3265640 1 -- 172.21.15.46:0/3370371560 >> 172.21.15.46:0/3370371560 conn(0x7fb9ac0a6340 msgr2=0x7fb9ac13b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:23.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.813+0000 7fb9b3265640 1 -- 172.21.15.46:0/3370371560 shutdown_connections 2024-09-16T04:41:23.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:23.813+0000 7fb9b3265640 1 -- 172.21.15.46:0/3370371560 wait complete. 2024-09-16T04:41:26.826 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:41:26.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.937+0000 7f978bee6640 10 monclient: get_monmap_and_config 2024-09-16T04:41:26.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.937+0000 7f978bee6640 10 monclient: build_initial_monmap 2024-09-16T04:41:26.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.937+0000 7f978bee6640 10 monclient: monmap: 2024-09-16T04:41:26.940 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:26.940 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:26.940 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:41:26.938658+0000 2024-09-16T04:41:26.940 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:41:26.938658+0000 2024-09-16T04:41:26.940 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:26.940 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:26.940 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:41:26.940 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:41:26.940 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:41:26.940 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:41:26.940 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:41:26.940 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:26.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.937+0000 7f978bee6640 1 Processor -- start 2024-09-16T04:41:26.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.938+0000 7f978bee6640 1 -- start start 2024-09-16T04:41:26.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.938+0000 7f978bee6640 10 monclient: init 2024-09-16T04:41:26.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.938+0000 7f978bee6640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:26.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.938+0000 7f978bee6640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-16T04:41:26.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.938+0000 7f978bee6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97840a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:41:26.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.938+0000 7f978bee6640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f97840a3ff0 0x7f97840a4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:26.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.938+0000 7f978bee6640 10 monclient(hunting): picked mon.noname-c con 0x7f97840a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:41:26.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.938+0000 7f978bee6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f978413be90 addr v1:172.21.15.114:6789/0 2024-09-16T04:41:26.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.938+0000 7f978bee6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f978409ab70 con 0x7f97840a3c10 2024-09-16T04:41:26.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.938+0000 7f978bee6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97840a4d60 con 0x7f978413be90 2024-09-16T04:41:26.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.938+0000 7f978bee6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:26.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.938+0000 7f978bee6640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f978409b030 con 0x7f97840a3ff0 2024-09-16T04:41:26.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.938+0000 7f978bee6640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:26.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.938+0000 7f978bee6640 10 monclient(hunting): authenticate will time out at 885.354596s 2024-09-16T04:41:26.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.938+0000 7f9789c82640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f97840a3c10 legacy=0x7f978409bcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:41:26.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.938+0000 7f9789481640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f97840a3ff0 0x7f97840a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:26.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.939+0000 7f978a483640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f978413be90 0x7f978413e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.46:41996/0 (socket says 172.21.15.46:41996) 2024-09-16T04:41:26.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.939+0000 7f9789481640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f97840a3ff0 0x7f97840a4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.46:56180/0 (socket says 172.21.15.46:56180) 2024-09-16T04:41:26.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.939+0000 7f978a483640 1 -- 172.21.15.46:0/4294373896 learned_addr learned my addr 172.21.15.46:0/4294373896 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:41:26.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.939+0000 7f9789481640 10 monclient(hunting): get_auth_request con 0x7f97840a3ff0 auth_method 0 2024-09-16T04:41:26.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.939+0000 7f9789481640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:26.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.939+0000 7f9789481640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:26.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.939+0000 7f9789481640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:26.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.939+0000 7f9789481640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:26.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.939+0000 7f9789481640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:26.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.939+0000 7f9789481640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:26.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.939+0000 7f9789481640 10 monclient(hunting): handle_auth_done global_id 5242 payload 274 2024-09-16T04:41:26.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.939+0000 7f9789481640 1 -- 172.21.15.46:0/4294373896 >> v1:172.21.15.114:6789/0 conn(0x7f978413be90 legacy=0x7f978413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:26.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.939+0000 7f9789481640 1 -- 172.21.15.46:0/4294373896 >> v1:172.21.15.46:6789/0 conn(0x7f97840a3c10 legacy=0x7f978409bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:41:26.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.939+0000 7f9789481640 10 monclient: _finish_hunting 0 2024-09-16T04:41:26.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.939+0000 7f9789481640 1 monclient: found mon.noname-c 2024-09-16T04:41:26.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.939+0000 7f9789481640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:26.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.939+0000 7f9789481640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:41:26.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.939+0000 7f9789481640 1 -- 172.21.15.46:0/4294373896 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f978409b310 con 0x7f97840a3ff0 2024-09-16T04:41:26.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f9789481640 1 --2- 172.21.15.46:0/4294373896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97840a3ff0 0x7f97840a4400 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f97780097a0 tx=0x7f977802a0b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77a02c740b0df87b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:26.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f9788c80640 1 -- 172.21.15.46:0/4294373896 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9778037070 con 0x7f97840a3ff0 2024-09-16T04:41:26.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f9788c80640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:26.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f9788c80640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:41:26.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f9788c80640 10 monclient: dump: 2024-09-16T04:41:26.952 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:26.952 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:26.952 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:26.952 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:26.952 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:26.952 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:26.952 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:26.952 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:26.952 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:26.952 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:26.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f9788c80640 10 monclient: _finish_auth 0 2024-09-16T04:41:26.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f9788c80640 10 monclient: _check_auth_tickets 2024-09-16T04:41:26.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f9788c80640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:26.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f9788c80640 1 -- 172.21.15.46:0/4294373896 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f977802adc0 con 0x7f97840a3ff0 2024-09-16T04:41:26.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f978bee6640 5 monclient: authenticate success, global_id 5242 2024-09-16T04:41:26.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f978bee6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:41:26.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f9788c80640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:26.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f978bee6640 10 monclient: get_monmap_and_config success 2024-09-16T04:41:26.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f9788c80640 1 -- 172.21.15.46:0/4294373896 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9778032660 con 0x7f97840a3ff0 2024-09-16T04:41:26.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f978bee6640 4 set_mon_vals no callback set 2024-09-16T04:41:26.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f9788c80640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:26.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f9788c80640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:41:26.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f9788c80640 10 monclient: dump: 2024-09-16T04:41:26.953 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:26.954 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:26.954 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:26.954 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:26.954 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:26.954 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:26.954 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:26.954 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:26.954 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:26.954 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:26.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f978bee6640 10 monclient: shutdown 2024-09-16T04:41:26.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f978bee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:26.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f978bee6640 1 -- 172.21.15.46:0/4294373896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97840a3ff0 msgr2=0x7f97840a4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:26.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f978bee6640 1 --2- 172.21.15.46:0/4294373896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97840a3ff0 0x7f97840a4400 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f97780097a0 tx=0x7f977802a0b0 comp rx=0 tx=0).stop 2024-09-16T04:41:26.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f978bee6640 1 -- 172.21.15.46:0/4294373896 shutdown_connections 2024-09-16T04:41:26.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f978bee6640 1 --2- 172.21.15.46:0/4294373896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97840a3ff0 0x7f97840a4400 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:26.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.940+0000 7f978bee6640 1 -- 172.21.15.46:0/4294373896 >> 172.21.15.46:0/4294373896 conn(0x7f97840a5d10 msgr2=0x7f97840a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:26.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.941+0000 7f978bee6640 1 -- 172.21.15.46:0/4294373896 shutdown_connections 2024-09-16T04:41:26.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.941+0000 7f978bee6640 1 -- 172.21.15.46:0/4294373896 wait complete. 2024-09-16T04:41:26.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.941+0000 7f978bee6640 10 monclient: build_initial_monmap 2024-09-16T04:41:26.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.941+0000 7f978bee6640 10 monclient: monmap: 2024-09-16T04:41:26.955 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:26.955 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:41:26.955 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:41:26.955 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:41:26.955 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:26.955 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:26.955 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:41:26.956 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:41:26.956 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:41:26.956 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:26.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.941+0000 7f978bee6640 1 Processor -- start 2024-09-16T04:41:26.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.941+0000 7f978bee6640 1 -- start start 2024-09-16T04:41:26.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.941+0000 7f978bee6640 10 monclient: init 2024-09-16T04:41:26.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978bee6640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:26.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978bee6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:41:26.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978bee6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f97840a3c10 0x7f97841cba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:26.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978bee6640 10 monclient(hunting): picked mon.noname-c con 0x7f97840a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:26.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978bee6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97840a3ff0 0x7f9784136d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:26.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978bee6640 10 monclient(hunting): picked mon.noname-a con 0x7f97840a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:41:26.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978bee6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f978413be90 0x7f978413c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:26.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978bee6640 10 monclient(hunting): picked mon.noname-b con 0x7f978413be90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:41:26.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978bee6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:26.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978bee6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f978406e550 con 0x7f97840a3ff0 2024-09-16T04:41:26.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978bee6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:26.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978bee6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f978406e3d0 con 0x7f978413be90 2024-09-16T04:41:26.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978bee6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:26.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978bee6640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f978413f9a0 con 0x7f97840a3c10 2024-09-16T04:41:26.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978bee6640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:26.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978bee6640 10 monclient(hunting): authenticate will time out at 885.358458s 2024-09-16T04:41:26.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789481640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97840a3ff0 msgr2=0x7f9784136d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:41:26.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789481640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97840a3ff0 0x7f9784136d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:41:26.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789c82640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f97840a3c10 0x7f97841cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:26.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789c82640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f97840a3c10 0x7f97841cba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:46654/0 (socket says 172.21.15.46:46654) 2024-09-16T04:41:26.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789c82640 1 -- 172.21.15.46:0/2723390530 learned_addr learned my addr 172.21.15.46:0/2723390530 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:26.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789c82640 10 monclient(hunting): get_auth_request con 0x7f97840a3c10 auth_method 0 2024-09-16T04:41:26.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789c82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:26.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789c82640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:26.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789c82640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:26.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978a483640 1 --2- 172.21.15.46:0/2723390530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f978413be90 0x7f978413c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:26.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978a483640 10 monclient(hunting): get_auth_request con 0x7f978413be90 auth_method 0 2024-09-16T04:41:26.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978a483640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:26.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978a483640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:26.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978a483640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:26.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789c82640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:26.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789c82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:26.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789c82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:26.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978a483640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:26.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978a483640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:26.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f978a483640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:26.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789c82640 10 monclient(hunting): handle_auth_done global_id 24413 payload 824 2024-09-16T04:41:26.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789c82640 1 -- 172.21.15.46:0/2723390530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f978413be90 msgr2=0x7f978413c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:26.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789c82640 1 --2- 172.21.15.46:0/2723390530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f978413be90 0x7f978413c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:26.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789c82640 1 -- 172.21.15.46:0/2723390530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97840a3ff0 msgr2=0x7f9784136d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:41:26.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789c82640 1 --2- 172.21.15.46:0/2723390530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97840a3ff0 0x7f9784136d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:26.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789c82640 10 monclient: _finish_hunting 0 2024-09-16T04:41:26.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789c82640 1 monclient: found mon.noname-c 2024-09-16T04:41:26.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789c82640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:26.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789c82640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:41:26.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.942+0000 7f9789c82640 1 -- 172.21.15.46:0/2723390530 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9784120420 con 0x7f97840a3c10 2024-09-16T04:41:26.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f9789c82640 1 --2- 172.21.15.46:0/2723390530 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f97840a3c10 0x7f97841cba10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f977400ba40 tx=0x7f977400bf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:26.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f97827fc640 1 -- 172.21.15.46:0/2723390530 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977400cc00 con 0x7f97840a3c10 2024-09-16T04:41:26.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f97827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:26.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f97827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:41:26.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f97827fc640 10 monclient: dump: 2024-09-16T04:41:26.960 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:26.960 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:26.960 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:26.960 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:26.960 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:26.960 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:26.960 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:26.960 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:26.960 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:26.960 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:26.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f97827fc640 10 monclient: _finish_auth 0 2024-09-16T04:41:26.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f97827fc640 10 monclient: _check_auth_tickets 2024-09-16T04:41:26.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f97827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:26.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f97827fc640 1 -- 172.21.15.46:0/2723390530 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9774010070 con 0x7f97840a3c10 2024-09-16T04:41:26.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f978bee6640 5 monclient: authenticate success, global_id 24413 2024-09-16T04:41:26.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f978bee6640 10 monclient: _renew_subs 2024-09-16T04:41:26.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f978bee6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:26.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f978bee6640 1 -- 172.21.15.46:0/2723390530 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9784120700 con 0x7f97840a3c10 2024-09-16T04:41:26.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f97827fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:26.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f97827fc640 1 -- 172.21.15.46:0/2723390530 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9774015470 con 0x7f97840a3c10 2024-09-16T04:41:26.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f97827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:26.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f97827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:26.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f97827fc640 10 monclient: dump: 2024-09-16T04:41:26.961 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:26.961 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:26.961 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:26.962 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:26.962 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:26.962 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:26.962 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:26.962 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:26.962 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:26.962 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:26.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f978bee6640 10 monclient: _renew_subs 2024-09-16T04:41:26.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f9783fff640 4 set_mon_vals no callback set 2024-09-16T04:41:26.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f978bee6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:26.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.943+0000 7f978bee6640 1 -- 172.21.15.46:0/2723390530 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f978409d3c0 con 0x7f97840a3c10 2024-09-16T04:41:26.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.944+0000 7f97827fc640 1 -- 172.21.15.46:0/2723390530 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f977401e3f0 con 0x7f97840a3c10 2024-09-16T04:41:26.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.944+0000 7f978bee6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:26.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.944+0000 7f978bee6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:26.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.944+0000 7f978bee6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:26.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.944+0000 7f978bee6640 1 -- 172.21.15.46:0/2723390530 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97840a4400 con 0x7f97840a3c10 2024-09-16T04:41:26.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.944+0000 7f97827fc640 1 --2- 172.21.15.46:0/2723390530 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9770038360 0x7f977003a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:26.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.944+0000 7f97827fc640 1 -- 172.21.15.46:0/2723390530 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(112..112 src has 1..112) ==== 4738+0+0 (secure 0 0 0) 0x7f977404b950 con 0x7f97840a3c10 2024-09-16T04:41:26.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.944+0000 7f9789481640 1 --2- 172.21.15.46:0/2723390530 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9770038360 0x7f977003a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:26.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.944+0000 7f9789481640 10 monclient: get_auth_request con 0x7f9770038360 auth_method 0 2024-09-16T04:41:26.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.945+0000 7f9789481640 1 --2- 172.21.15.46:0/2723390530 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9770038360 0x7f977003a820 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9778034000 tx=0x7f977802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:26.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.949+0000 7f97827fc640 1 -- 172.21.15.46:0/2723390530 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f977401e8c0 con 0x7f97840a3c10 2024-09-16T04:41:26.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.949+0000 7f97827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:26.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:26.949+0000 7f97827fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:41:27.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:27.191+0000 7f978bee6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:41:27.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:27.191+0000 7f978bee6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:41:27.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:27.191+0000 7f978bee6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:27.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:27.191+0000 7f978bee6640 1 -- 172.21.15.46:0/2723390530 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f978413e830 con 0x7f97840a3c10 2024-09-16T04:41:27.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:27.192+0000 7f97827fc640 1 -- 172.21.15.46:0/2723390530 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f97740147d0 con 0x7f97840a3c10 2024-09-16T04:41:27.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:27.192+0000 7f97827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:41:27.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:27.192+0000 7f97827fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:41:27.195 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:41:27.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:27.193+0000 7f978bee6640 1 -- 172.21.15.46:0/2723390530 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9770038360 msgr2=0x7f977003a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:27.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:27.193+0000 7f978bee6640 1 --2- 172.21.15.46:0/2723390530 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9770038360 0x7f977003a820 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9778034000 tx=0x7f977802e000 comp rx=0 tx=0).stop 2024-09-16T04:41:27.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:27.193+0000 7f978bee6640 10 monclient: shutdown 2024-09-16T04:41:27.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:27.193+0000 7f978bee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:27.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:27.193+0000 7f978bee6640 1 -- 172.21.15.46:0/2723390530 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f97840a3c10 msgr2=0x7f97841cba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:27.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:27.193+0000 7f978bee6640 1 --2- 172.21.15.46:0/2723390530 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f97840a3c10 0x7f97841cba10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f977400ba40 tx=0x7f977400bf00 comp rx=0 tx=0).stop 2024-09-16T04:41:27.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:27.193+0000 7f978bee6640 1 -- 172.21.15.46:0/2723390530 shutdown_connections 2024-09-16T04:41:27.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:27.193+0000 7f978bee6640 1 --2- 172.21.15.46:0/2723390530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f978413be90 0x7f978413c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:27.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:27.193+0000 7f978bee6640 1 --2- 172.21.15.46:0/2723390530 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9770038360 0x7f977003a820 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:27.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:27.193+0000 7f978bee6640 1 --2- 172.21.15.46:0/2723390530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97840a3ff0 0x7f9784136d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:27.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:27.193+0000 7f978bee6640 1 --2- 172.21.15.46:0/2723390530 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f97840a3c10 0x7f97841cba10 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:27.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:27.194+0000 7f978bee6640 1 -- 172.21.15.46:0/2723390530 >> 172.21.15.46:0/2723390530 conn(0x7f97840a5d10 msgr2=0x7f978413b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:27.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:27.194+0000 7f978bee6640 1 -- 172.21.15.46:0/2723390530 shutdown_connections 2024-09-16T04:41:27.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:27.194+0000 7f978bee6640 1 -- 172.21.15.46:0/2723390530 wait complete. 2024-09-16T04:41:30.207 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:41:30.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.318+0000 7f2f369c4640 10 monclient: get_monmap_and_config 2024-09-16T04:41:30.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.318+0000 7f2f369c4640 10 monclient: build_initial_monmap 2024-09-16T04:41:30.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.318+0000 7f2f369c4640 10 monclient: monmap: 2024-09-16T04:41:30.321 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:30.321 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:30.321 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:41:30.320301+0000 2024-09-16T04:41:30.321 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:41:30.320301+0000 2024-09-16T04:41:30.321 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:30.321 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:30.321 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:41:30.321 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:41:30.321 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:41:30.322 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:41:30.322 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:41:30.322 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:30.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.319+0000 7f2f369c4640 1 Processor -- start 2024-09-16T04:41:30.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.319+0000 7f2f369c4640 1 -- start start 2024-09-16T04:41:30.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.319+0000 7f2f369c4640 10 monclient: init 2024-09-16T04:41:30.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f369c4640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:30.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f369c4640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T04:41:30.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f369c4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2f300a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:41:30.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f369c4640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f300a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:30.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f369c4640 10 monclient(hunting): picked mon.noname-b con 0x7f2f300a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:30.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f369c4640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 0x7f2f3013e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:30.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f369c4640 10 monclient(hunting): picked mon.noname-c con 0x7f2f3013be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:41:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f369c4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f3009b940 con 0x7f2f300a3c10 2024-09-16T04:41:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f369c4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f369c4640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f2f3009ab00 con 0x7f2f3013be90 2024-09-16T04:41:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f369c4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f369c4640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f2f3009b030 con 0x7f2f300a3ff0 2024-09-16T04:41:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f369c4640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f369c4640 10 monclient(hunting): authenticate will time out at 888.736421s 2024-09-16T04:41:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f2ffff640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f2f300a3c10 legacy=0x7f2f3009bcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:41:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f27fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f300a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f27fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f300a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:46666/0 (socket says 172.21.15.46:46666) 2024-09-16T04:41:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 learned_addr learned my addr 172.21.15.46:0/3771277853 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f27fff640 10 monclient(hunting): get_auth_request con 0x7f2f300a3ff0 auth_method 0 2024-09-16T04:41:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:30.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f27fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:30.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:30.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 msgr2=0x7f2f300a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:41:30.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 msgr2=0x7f2f300a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:41:30.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f300a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:41:30.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f300a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:41:30.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f34f61640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 0x7f2f3013e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:30.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f34f61640 10 monclient(hunting): get_auth_request con 0x7f2f3013be90 auth_method 0 2024-09-16T04:41:30.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f34f61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:30.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f34f61640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:30.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f34f61640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:30.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f34f61640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 msgr2=0x7f2f3013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:41:30.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.320+0000 7f2f34f61640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 msgr2=0x7f2f3013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:41:30.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.321+0000 7f2f34f61640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 0x7f2f3013e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:41:30.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.321+0000 7f2f34f61640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 0x7f2f3013e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:41:30.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.521+0000 7f2f2ffff640 1 -- 172.21.15.46:0/3771277853 >> v1:172.21.15.46:6789/0 conn(0x7f2f300a3c10 legacy=0x7f2f3009bcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:41:30.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.521+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f300a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:30.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.521+0000 7f2f27fff640 10 monclient(hunting): get_auth_request con 0x7f2f300a3ff0 auth_method 0 2024-09-16T04:41:30.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.521+0000 7f2f27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:30.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.521+0000 7f2f27fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:30.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.521+0000 7f2f27fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:41:30.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.521+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 msgr2=0x7f2f300a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:41:30.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.521+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 msgr2=0x7f2f300a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:41:30.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.521+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f300a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:41:30.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.521+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f300a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:41:30.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.522+0000 7f2f34f61640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 0x7f2f3013e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:30.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.522+0000 7f2f34f61640 10 monclient(hunting): get_auth_request con 0x7f2f3013be90 auth_method 0 2024-09-16T04:41:30.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.522+0000 7f2f34f61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:30.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.522+0000 7f2f34f61640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:30.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.522+0000 7f2f34f61640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:41:30.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.522+0000 7f2f34f61640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 msgr2=0x7f2f3013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:41:30.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.522+0000 7f2f34f61640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 msgr2=0x7f2f3013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:41:30.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.522+0000 7f2f34f61640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 0x7f2f3013e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:41:30.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.522+0000 7f2f34f61640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 0x7f2f3013e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:41:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.922+0000 7f2f2ffff640 1 -- 172.21.15.46:0/3771277853 >> v1:172.21.15.46:6789/0 conn(0x7f2f300a3c10 legacy=0x7f2f3009bcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:41:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.922+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f300a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.922+0000 7f2f27fff640 10 monclient(hunting): get_auth_request con 0x7f2f300a3ff0 auth_method 0 2024-09-16T04:41:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.922+0000 7f2f27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.922+0000 7f2f27fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.922+0000 7f2f27fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:41:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.922+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 msgr2=0x7f2f300a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:41:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.922+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 msgr2=0x7f2f300a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:41:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.922+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f300a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:41:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.922+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f300a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:41:30.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.923+0000 7f2f34f61640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 0x7f2f3013e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:30.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.923+0000 7f2f34f61640 10 monclient(hunting): get_auth_request con 0x7f2f3013be90 auth_method 0 2024-09-16T04:41:30.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.923+0000 7f2f34f61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:30.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.923+0000 7f2f34f61640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:30.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.923+0000 7f2f34f61640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:41:30.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.923+0000 7f2f34f61640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 msgr2=0x7f2f3013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:41:30.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.923+0000 7f2f34f61640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 msgr2=0x7f2f3013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:41:30.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.923+0000 7f2f34f61640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 0x7f2f3013e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:41:30.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:30.923+0000 7f2f34f61640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 0x7f2f3013e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:41:31.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:31.723+0000 7f2f2ffff640 1 -- 172.21.15.46:0/3771277853 >> v1:172.21.15.46:6789/0 conn(0x7f2f300a3c10 legacy=0x7f2f3009bcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:41:31.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:31.723+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f300a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:31.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:31.723+0000 7f2f27fff640 10 monclient(hunting): get_auth_request con 0x7f2f300a3ff0 auth_method 0 2024-09-16T04:41:31.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:31.723+0000 7f2f27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:31.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:31.723+0000 7f2f27fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:31.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:31.723+0000 7f2f27fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:41:31.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:31.723+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 msgr2=0x7f2f300a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:41:31.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:31.723+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 msgr2=0x7f2f300a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:41:31.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:31.723+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f300a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:41:31.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:31.723+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f300a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:41:31.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:31.724+0000 7f2f34f61640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 0x7f2f3013e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:31.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:31.724+0000 7f2f34f61640 10 monclient(hunting): get_auth_request con 0x7f2f3013be90 auth_method 0 2024-09-16T04:41:31.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:31.724+0000 7f2f34f61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:31.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:31.724+0000 7f2f34f61640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:31.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:31.724+0000 7f2f34f61640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:41:31.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:31.724+0000 7f2f34f61640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 msgr2=0x7f2f3013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:41:31.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:31.724+0000 7f2f34f61640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 msgr2=0x7f2f3013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:41:31.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:31.724+0000 7f2f34f61640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 0x7f2f3013e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:41:31.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:31.724+0000 7f2f34f61640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 0x7f2f3013e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:41:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.321+0000 7f2f2e7fc640 10 monclient(hunting): tick 2024-09-16T04:41:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.321+0000 7f2f2e7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:41:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.321+0000 7f2f2e7fc640 1 monclient(hunting): continuing hunt 2024-09-16T04:41:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.321+0000 7f2f2e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:41:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.321+0000 7f2f2e7fc640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 msgr2=0x7f2f300a43d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:41:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.321+0000 7f2f2e7fc640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f300a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.321+0000 7f2f2e7fc640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 msgr2=0x7f2f3013e280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:41:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.321+0000 7f2f2e7fc640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f3013be90 0x7f2f3013e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.321+0000 7f2f2e7fc640 1 -- 172.21.15.46:0/3771277853 >> v1:172.21.15.46:6789/0 conn(0x7f2f300a3c10 legacy=0x7f2f3009bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:41:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.321+0000 7f2f2e7fc640 10 monclient: _add_conns ranks=[1,3] 2024-09-16T04:41:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.321+0000 7f2f2e7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2f100036c0 addr v1:172.21.15.114:6789/0 2024-09-16T04:41:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.321+0000 7f2f2e7fc640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10007330 0x7f2f100097f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.321+0000 7f2f2e7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f2f10007330 addr v2:172.21.15.46:3300/0 2024-09-16T04:41:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.321+0000 7f2f2e7fc640 1 -- 172.21.15.46:0/3771277853 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f3009b310 con 0x7f2f100036c0 2024-09-16T04:41:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.321+0000 7f2f2e7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.321+0000 7f2f2e7fc640 1 -- 172.21.15.46:0/3771277853 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2f3009ab00 con 0x7f2f10007330 2024-09-16T04:41:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.321+0000 7f2f2e7fc640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.321+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10007330 msgr2=0x7f2f100097f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:41:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.321+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10007330 0x7f2f100097f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:41:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.522+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10007330 msgr2=0x7f2f100097f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:41:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.522+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10007330 0x7f2f100097f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:41:33.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.923+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10007330 msgr2=0x7f2f100097f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:41:33.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:33.923+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10007330 0x7f2f100097f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:41:34.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:34.724+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10007330 msgr2=0x7f2f100097f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:41:34.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:34.724+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10007330 0x7f2f100097f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:41:36.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2e7fc640 10 monclient(hunting): tick 2024-09-16T04:41:36.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2e7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:41:36.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2e7fc640 1 monclient(hunting): continuing hunt 2024-09-16T04:41:36.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:41:36.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2e7fc640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10007330 msgr2=0x7f2f100097f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:41:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2e7fc640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10007330 0x7f2f100097f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2e7fc640 1 -- 172.21.15.46:0/3771277853 >> v1:172.21.15.114:6789/0 conn(0x7f2f100036c0 legacy=0x7f2f10005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2e7fc640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-16T04:41:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f34f61640 1 -- 172.21.15.46:0/3771277853 reap_dead start 2024-09-16T04:41:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2e7fc640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f1000a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2e7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f2f300a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2e7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2f1000a550 addr v1:172.21.15.46:6789/0 2024-09-16T04:41:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2e7fc640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10009890 0x7f2f1000a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2e7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f2f10009890 addr v2:172.21.15.46:3300/0 2024-09-16T04:41:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2e7fc640 1 -- 172.21.15.46:0/3771277853 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f3009b940 con 0x7f2f1000a550 2024-09-16T04:41:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2e7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2e7fc640 1 -- 172.21.15.46:0/3771277853 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2f3009ab00 con 0x7f2f10009890 2024-09-16T04:41:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2e7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2e7fc640 1 -- 172.21.15.46:0/3771277853 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f2f3009b030 con 0x7f2f300a3ff0 2024-09-16T04:41:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2e7fc640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 >> v1:172.21.15.46:6789/0 conn(0x7f2f1000a550 legacy=0x7f2f10006230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:41:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f34f61640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10009890 msgr2=0x7f2f1000a930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:41:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f34f61640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10009890 0x7f2f1000a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:41:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2ffff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f1000a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2ffff640 10 monclient(hunting): get_auth_request con 0x7f2f300a3ff0 auth_method 0 2024-09-16T04:41:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2ffff640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 msgr2=0x7f2f1000a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:41:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2ffff640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 msgr2=0x7f2f1000a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:41:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2ffff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f1000a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:41:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.321+0000 7f2f2ffff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f1000a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:41:36.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.522+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 >> v1:172.21.15.46:6789/0 conn(0x7f2f1000a550 legacy=0x7f2f10006230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:41:36.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.522+0000 7f2f34f61640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10009890 msgr2=0x7f2f1000a930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:41:36.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.522+0000 7f2f34f61640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10009890 0x7f2f1000a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:41:36.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.522+0000 7f2f2ffff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f1000a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:36.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.522+0000 7f2f2ffff640 10 monclient(hunting): get_auth_request con 0x7f2f300a3ff0 auth_method 0 2024-09-16T04:41:36.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.522+0000 7f2f2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:36.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.522+0000 7f2f2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:36.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.522+0000 7f2f2ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:41:36.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.522+0000 7f2f2ffff640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 msgr2=0x7f2f1000a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:41:36.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.522+0000 7f2f2ffff640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 msgr2=0x7f2f1000a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:41:36.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.522+0000 7f2f2ffff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f1000a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:41:36.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.522+0000 7f2f2ffff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f1000a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:41:36.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.923+0000 7f2f34f61640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10009890 msgr2=0x7f2f1000a930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:41:36.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.923+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 >> v1:172.21.15.46:6789/0 conn(0x7f2f1000a550 legacy=0x7f2f10006230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:41:36.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.923+0000 7f2f34f61640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10009890 0x7f2f1000a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:41:36.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.923+0000 7f2f2ffff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f1000a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:36.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.923+0000 7f2f2ffff640 10 monclient(hunting): get_auth_request con 0x7f2f300a3ff0 auth_method 0 2024-09-16T04:41:36.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.923+0000 7f2f2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:36.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.923+0000 7f2f2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:36.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.923+0000 7f2f2ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:41:36.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.923+0000 7f2f2ffff640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 msgr2=0x7f2f1000a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:41:36.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.923+0000 7f2f2ffff640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 msgr2=0x7f2f1000a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:41:36.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.923+0000 7f2f2ffff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f1000a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:41:36.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:36.923+0000 7f2f2ffff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f1000a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:41:37.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:37.724+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 >> v1:172.21.15.46:6789/0 conn(0x7f2f1000a550 legacy=0x7f2f10006230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:41:37.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:37.724+0000 7f2f34f61640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10009890 msgr2=0x7f2f1000a930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:41:37.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:37.724+0000 7f2f34f61640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10009890 0x7f2f1000a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:41:37.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:37.724+0000 7f2f2ffff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f1000a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:37.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:37.724+0000 7f2f2ffff640 10 monclient(hunting): get_auth_request con 0x7f2f300a3ff0 auth_method 0 2024-09-16T04:41:37.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:37.724+0000 7f2f2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:37.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:37.724+0000 7f2f2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:37.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:37.724+0000 7f2f2ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:41:37.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:37.724+0000 7f2f2ffff640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 msgr2=0x7f2f1000a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:41:37.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:37.724+0000 7f2f2ffff640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 msgr2=0x7f2f1000a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:41:37.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:37.724+0000 7f2f2ffff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f1000a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:41:37.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:37.724+0000 7f2f2ffff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f1000a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:41:39.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f2e7fc640 10 monclient(hunting): tick 2024-09-16T04:41:39.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f2e7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:41:39.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f2e7fc640 1 monclient(hunting): continuing hunt 2024-09-16T04:41:39.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f2e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:41:39.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f2e7fc640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 msgr2=0x7f2f1000a010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:41:39.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f2e7fc640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f300a3ff0 0x7f2f1000a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f2e7fc640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10009890 msgr2=0x7f2f1000a930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:41:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f2e7fc640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.46:3300/0 conn(0x7f2f10009890 0x7f2f1000a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f2e7fc640 1 -- 172.21.15.46:0/3771277853 >> v1:172.21.15.46:6789/0 conn(0x7f2f1000a550 legacy=0x7f2f10006230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:41:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f2e7fc640 10 monclient: _add_conns ranks=[1,4] 2024-09-16T04:41:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f2e7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2f10005db0 addr v1:172.21.15.114:6789/0 2024-09-16T04:41:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f2e7fc640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f10004ba0 0x7f2f10004fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f2e7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f2f10004ba0 addr v2:172.21.15.114:3300/0 2024-09-16T04:41:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f2e7fc640 1 -- 172.21.15.46:0/3771277853 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f10005b10 con 0x7f2f10005db0 2024-09-16T04:41:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f2e7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f2e7fc640 1 -- 172.21.15.46:0/3771277853 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f2f3009ab00 con 0x7f2f10004ba0 2024-09-16T04:41:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f2e7fc640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f10004ba0 0x7f2f10004fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f27fff640 10 monclient(hunting): get_auth_request con 0x7f2f10004ba0 auth_method 0 2024-09-16T04:41:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f27fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f10004ba0 msgr2=0x7f2f10004fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:41:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f10004ba0 msgr2=0x7f2f10004fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:41:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f10004ba0 0x7f2f10004fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:41:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.322+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f10004ba0 0x7f2f10004fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:41:39.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.523+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f10004ba0 0x7f2f10004fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:39.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.523+0000 7f2f27fff640 10 monclient(hunting): get_auth_request con 0x7f2f10004ba0 auth_method 0 2024-09-16T04:41:39.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.523+0000 7f2f27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:39.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.523+0000 7f2f27fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:39.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.523+0000 7f2f27fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:41:39.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.523+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f10004ba0 msgr2=0x7f2f10004fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:41:39.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.523+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f10004ba0 msgr2=0x7f2f10004fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:41:39.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.523+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f10004ba0 0x7f2f10004fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:41:39.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.523+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f10004ba0 0x7f2f10004fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:41:39.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.563+0000 7f2f2f7fe640 1 -- 172.21.15.46:0/3771277853 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3047162130 0 0) 0x7f2f10005b10 con 0x7f2f10005db0 2024-09-16T04:41:39.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.563+0000 7f2f2f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:41:39.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.563+0000 7f2f2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:39.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.563+0000 7f2f2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:39.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.563+0000 7f2f2f7fe640 10 monclient(hunting): my global_id is 5248 2024-09-16T04:41:39.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.563+0000 7f2f2f7fe640 1 -- 172.21.15.46:0/3771277853 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f1c003520 con 0x7f2f10005db0 2024-09-16T04:41:39.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 1 -- 172.21.15.46:0/3771277853 <== mon.1 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3454001240 0 0) 0x7f2f1c003520 con 0x7f2f10005db0 2024-09-16T04:41:39.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 1 -- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f10004ba0 msgr2=0x7f2f10004fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:41:39.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 1 --2- 172.21.15.46:0/3771277853 >> v2:172.21.15.114:3300/0 conn(0x7f2f10004ba0 0x7f2f10004fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:41:39.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 1 monclient: found mon.noname-c-legacy 2024-09-16T04:41:39.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:39.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.114:6789/0 2024-09-16T04:41:39.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 1 -- 172.21.15.46:0/3771277853 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f3009b940 con 0x7f2f10005db0 2024-09-16T04:41:39.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:41:39.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:41:39.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:39.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 1 -- 172.21.15.46:0/3771277853 <== mon.1 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f2f20009890 con 0x7f2f10005db0 2024-09-16T04:41:39.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:39.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T04:41:39.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 10 monclient: dump: 2024-09-16T04:41:39.580 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:39.580 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:39.580 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.580 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.580 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:39.580 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:39.581 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:39.581 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:39.581 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:39.581 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:39.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T04:41:39.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:39.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 1 -- 172.21.15.46:0/3771277853 >> v1:172.21.15.114:6789/0 conn(0x7f2f10005db0 legacy=0x7f2f10006b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:39.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f34f61640 1 -- 172.21.15.46:0/3771277853 reap_dead start 2024-09-16T04:41:39.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:41:39.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f1c0039b0 0x7f2f1c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:39.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 10 monclient(hunting): picked mon.b con 0x7f2f1c0039b0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:41:39.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f1c004800 0x7f2f1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:39.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 10 monclient(hunting): picked mon.a con 0x7f2f1c004800 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:41:39.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f1c005210 0x7f2f1c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:39.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 10 monclient(hunting): picked mon.c con 0x7f2f1c005210 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:39.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:39.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 1 -- 172.21.15.46:0/3771277853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3009ab00 con 0x7f2f1c004800 2024-09-16T04:41:39.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:39.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 1 -- 172.21.15.46:0/3771277853 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c00c140 con 0x7f2f1c0039b0 2024-09-16T04:41:39.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:39.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 1 -- 172.21.15.46:0/3771277853 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f2f1c00c340 con 0x7f2f1c005210 2024-09-16T04:41:39.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:39.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:41:39.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f27fff640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f1c004800 msgr2=0x7f2f1c004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:41:39.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f27fff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f1c004800 0x7f2f1c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:41:39.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2ffff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f1c0039b0 0x7f2f1c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:39.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f34f61640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f1c005210 0x7f2f1c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2ffff640 10 monclient(hunting): get_auth_request con 0x7f2f1c0039b0 auth_method 0 2024-09-16T04:41:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:41:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f34f61640 10 monclient(hunting): get_auth_request con 0x7f2f1c005210 auth_method 0 2024-09-16T04:41:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f34f61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f34f61640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f34f61640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:41:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.565+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.566+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:41:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.566+0000 7f2f34f61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.566+0000 7f2f34f61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:39.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.566+0000 7f2f34f61640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:41:39.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.566+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_done global_id 5248 payload 293 2024-09-16T04:41:39.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.566+0000 7f2f2ffff640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f1c005210 msgr2=0x7f2f1c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:41:39.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.566+0000 7f2f2ffff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f1c005210 0x7f2f1c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.566+0000 7f2f2ffff640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f1c004800 msgr2=0x7f2f1c004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:41:39.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.566+0000 7f2f2ffff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f1c004800 0x7f2f1c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.566+0000 7f2f2ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:41:39.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.566+0000 7f2f2ffff640 1 monclient: found mon.b 2024-09-16T04:41:39.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.566+0000 7f2f2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:39.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.566+0000 7f2f2ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:41:39.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.566+0000 7f2f2ffff640 1 -- 172.21.15.46:0/3771277853 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f1c003520 con 0x7f2f1c0039b0 2024-09-16T04:41:39.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.566+0000 7f2f2ffff640 10 monclient: _finish_auth 0 2024-09-16T04:41:39.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.566+0000 7f2f2ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:41:39.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.566+0000 7f2f2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:39.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.566+0000 7f2f369c4640 5 monclient: authenticate success, global_id 5248 2024-09-16T04:41:39.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.566+0000 7f2f369c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:41:39.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.567+0000 7f2f2ffff640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f1c0039b0 0x7f2f1c003db0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2f20009420 tx=0x7f2f1c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a5b2c57260ca904 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:39.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.567+0000 7f2f2f7fe640 1 -- 172.21.15.46:0/3771277853 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2000b490 con 0x7f2f1c0039b0 2024-09-16T04:41:39.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.567+0000 7f2f2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:39.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.567+0000 7f2f2f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:41:39.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.567+0000 7f2f2f7fe640 10 monclient: dump: 2024-09-16T04:41:39.585 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:39.585 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:39.585 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.585 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.586 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:39.586 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:39.586 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:39.586 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:39.586 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:39.586 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:39.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.567+0000 7f2f2f7fe640 1 -- 172.21.15.46:0/3771277853 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f2000b0f0 con 0x7f2f1c0039b0 2024-09-16T04:41:39.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.567+0000 7f2f369c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:41:39.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.567+0000 7f2f2f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:39.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.567+0000 7f2f369c4640 10 monclient: get_monmap_and_config success 2024-09-16T04:41:39.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.567+0000 7f2f369c4640 4 set_mon_vals no callback set 2024-09-16T04:41:39.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.567+0000 7f2f2f7fe640 1 -- 172.21.15.46:0/3771277853 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f20035420 con 0x7f2f1c0039b0 2024-09-16T04:41:39.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.567+0000 7f2f2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:39.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.567+0000 7f2f2f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:41:39.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.567+0000 7f2f2f7fe640 10 monclient: dump: 2024-09-16T04:41:39.587 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:39.587 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:39.587 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.587 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.587 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:39.587 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:39.587 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:39.587 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:39.587 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:39.587 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:39.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.568+0000 7f2f369c4640 10 monclient: shutdown 2024-09-16T04:41:39.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.568+0000 7f2f369c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:39.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.568+0000 7f2f369c4640 1 -- 172.21.15.46:0/3771277853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f1c0039b0 msgr2=0x7f2f1c003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:39.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.568+0000 7f2f369c4640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f1c0039b0 0x7f2f1c003db0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2f20009420 tx=0x7f2f1c005830 comp rx=0 tx=0).stop 2024-09-16T04:41:39.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.568+0000 7f2f369c4640 1 -- 172.21.15.46:0/3771277853 shutdown_connections 2024-09-16T04:41:39.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.568+0000 7f2f369c4640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f1c005210 0x7f2f1c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.568+0000 7f2f369c4640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f1c004800 0x7f2f1c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.568+0000 7f2f369c4640 1 --2- 172.21.15.46:0/3771277853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f1c0039b0 0x7f2f1c003db0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.568+0000 7f2f369c4640 1 -- 172.21.15.46:0/3771277853 >> 172.21.15.46:0/3771277853 conn(0x7f2f300a6340 msgr2=0x7f2f3006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.568+0000 7f2f369c4640 1 -- 172.21.15.46:0/3771277853 shutdown_connections 2024-09-16T04:41:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.568+0000 7f2f369c4640 1 -- 172.21.15.46:0/3771277853 wait complete. 2024-09-16T04:41:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.569+0000 7f2f369c4640 10 monclient: build_initial_monmap 2024-09-16T04:41:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.569+0000 7f2f369c4640 10 monclient: monmap: 2024-09-16T04:41:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:41:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:41:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:41:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:41:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:41:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:41:39.589 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.569+0000 7f2f369c4640 1 Processor -- start 2024-09-16T04:41:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.569+0000 7f2f369c4640 1 -- start start 2024-09-16T04:41:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.569+0000 7f2f369c4640 10 monclient: init 2024-09-16T04:41:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f369c4640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f369c4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:41:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f369c4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f1c0039b0 0x7f2f3013a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f369c4640 10 monclient(hunting): picked mon.noname-b con 0x7f2f1c0039b0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:41:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f369c4640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f1c004800 0x7f2f3009d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f369c4640 10 monclient(hunting): picked mon.noname-c con 0x7f2f1c004800 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f369c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f1c005210 0x7f2f3009d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f369c4640 10 monclient(hunting): picked mon.noname-a con 0x7f2f1c005210 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:41:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f369c4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f369c4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3009be70 con 0x7f2f1c005210 2024-09-16T04:41:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f369c4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f369c4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3009bcf0 con 0x7f2f1c0039b0 2024-09-16T04:41:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f369c4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f369c4640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f2f3009bff0 con 0x7f2f1c004800 2024-09-16T04:41:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f369c4640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f369c4640 10 monclient(hunting): authenticate will time out at 897.986298s 2024-09-16T04:41:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f34f61640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f1c005210 msgr2=0x7f2f3009d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:41:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f34f61640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f1c005210 0x7f2f3009d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:41:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f27fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f1c004800 0x7f2f3009d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f2ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f1c0039b0 0x7f2f3013a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f27fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f1c004800 0x7f2f3009d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:37486/0 (socket says 172.21.15.46:37486) 2024-09-16T04:41:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f2ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f1c0039b0 0x7f2f3013a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.46:60656/0 (socket says 172.21.15.46:60656) 2024-09-16T04:41:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f27fff640 1 -- 172.21.15.46:0/387914722 learned_addr learned my addr 172.21.15.46:0/387914722 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f27fff640 10 monclient(hunting): get_auth_request con 0x7f2f1c004800 auth_method 0 2024-09-16T04:41:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f27fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f2ffff640 10 monclient(hunting): get_auth_request con 0x7f2f1c0039b0 auth_method 0 2024-09-16T04:41:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_done global_id 5251 payload 824 2024-09-16T04:41:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f2ffff640 1 -- 172.21.15.46:0/387914722 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f1c004800 msgr2=0x7f2f3009d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f2ffff640 1 --2- 172.21.15.46:0/387914722 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f1c004800 0x7f2f3009d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f2ffff640 1 -- 172.21.15.46:0/387914722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f1c005210 msgr2=0x7f2f3009d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:41:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f2ffff640 1 --2- 172.21.15.46:0/387914722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f1c005210 0x7f2f3009d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f2ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:41:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f2ffff640 1 monclient: found mon.noname-b 2024-09-16T04:41:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f2ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3300/0 2024-09-16T04:41:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.570+0000 7f2f2ffff640 1 -- 172.21.15.46:0/387914722 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f300a09f0 con 0x7f2f1c0039b0 2024-09-16T04:41:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f2ffff640 1 --2- 172.21.15.46:0/387914722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f1c0039b0 0x7f2f3013a400 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c00c310 tx=0x7f2f20005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f2d7fa640 1 -- 172.21.15.46:0/387914722 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f20044020 con 0x7f2f1c0039b0 2024-09-16T04:41:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f2d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:41:39.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f2d7fa640 10 monclient: dump: 2024-09-16T04:41:39.594 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:39.594 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:39.594 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.594 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.594 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:39.594 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:39.594 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:39.594 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:39.594 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:39.594 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:39.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f2d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:41:39.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f2d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:41:39.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f2d7fa640 1 -- 172.21.15.46:0/387914722 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f20009a70 con 0x7f2f1c0039b0 2024-09-16T04:41:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f369c4640 5 monclient: authenticate success, global_id 5251 2024-09-16T04:41:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f2d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f2d7fa640 1 -- 172.21.15.46:0/387914722 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2003f460 con 0x7f2f1c0039b0 2024-09-16T04:41:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f369c4640 10 monclient: _renew_subs 2024-09-16T04:41:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f369c4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:41:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f369c4640 1 -- 172.21.15.46:0/387914722 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f300a0400 con 0x7f2f1c0039b0 2024-09-16T04:41:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f2d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:41:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f2d7fa640 10 monclient: dump: 2024-09-16T04:41:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:39.596 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f2effd640 4 set_mon_vals no callback set 2024-09-16T04:41:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f369c4640 10 monclient: _renew_subs 2024-09-16T04:41:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f369c4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:41:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.571+0000 7f2f369c4640 1 -- 172.21.15.46:0/387914722 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f3011f900 con 0x7f2f1c0039b0 2024-09-16T04:41:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.572+0000 7f2f2d7fa640 1 -- 172.21.15.46:0/387914722 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f2f20047020 con 0x7f2f1c0039b0 2024-09-16T04:41:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.572+0000 7f2f2d7fa640 1 --2- 172.21.15.46:0/387914722 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2f10039840 0x7f2f1003bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.572+0000 7f2f369c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.572+0000 7f2f369c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.572+0000 7f2f2d7fa640 1 -- 172.21.15.46:0/387914722 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 4738+0+0 (secure 0 0 0) 0x7f2f2006c770 con 0x7f2f1c0039b0 2024-09-16T04:41:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.572+0000 7f2f369c4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:41:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.572+0000 7f2f369c4640 1 -- 172.21.15.46:0/387914722 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f3013ae30 con 0x7f2f1c0039b0 2024-09-16T04:41:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.572+0000 7f2f27fff640 1 --2- 172.21.15.46:0/387914722 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2f10039840 0x7f2f1003bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.572+0000 7f2f27fff640 10 monclient: get_auth_request con 0x7f2f10039840 auth_method 0 2024-09-16T04:41:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.573+0000 7f2f27fff640 1 --2- 172.21.15.46:0/387914722 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2f10039840 0x7f2f1003bd00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2f3009ff20 tx=0x7f2f18008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.578+0000 7f2f2d7fa640 1 -- 172.21.15.46:0/387914722 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2f2004e070 con 0x7f2f1c0039b0 2024-09-16T04:41:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.578+0000 7f2f2d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.578+0000 7f2f2d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:41:39.623 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36899]: starting. 2024-09-16T04:41:39.623 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36899]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-36897 2024-09-16T04:41:39.623 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[36899]: shutting down. 2024-09-16T04:41:39.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.830+0000 7f2f369c4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:41:39.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.830+0000 7f2f369c4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:41:39.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.830+0000 7f2f369c4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:41:39.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.830+0000 7f2f369c4640 1 -- 172.21.15.46:0/387914722 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2f300a43d0 con 0x7f2f1c0039b0 2024-09-16T04:41:39.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.830+0000 7f2f2d7fa640 1 -- 172.21.15.46:0/387914722 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f2f2003fe00 con 0x7f2f1c0039b0 2024-09-16T04:41:39.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.830+0000 7f2f2d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:41:39.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.830+0000 7f2f2d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:41:39.833 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":30,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:41:39.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.832+0000 7f2f369c4640 1 -- 172.21.15.46:0/387914722 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2f10039840 msgr2=0x7f2f1003bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:39.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.832+0000 7f2f369c4640 1 --2- 172.21.15.46:0/387914722 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2f10039840 0x7f2f1003bd00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2f3009ff20 tx=0x7f2f18008040 comp rx=0 tx=0).stop 2024-09-16T04:41:39.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.832+0000 7f2f369c4640 10 monclient: shutdown 2024-09-16T04:41:39.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.832+0000 7f2f369c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:39.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.832+0000 7f2f369c4640 1 -- 172.21.15.46:0/387914722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f1c0039b0 msgr2=0x7f2f3013a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:39.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.832+0000 7f2f369c4640 1 --2- 172.21.15.46:0/387914722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f1c0039b0 0x7f2f3013a400 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c00c310 tx=0x7f2f20005040 comp rx=0 tx=0).stop 2024-09-16T04:41:39.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.832+0000 7f2f369c4640 1 -- 172.21.15.46:0/387914722 shutdown_connections 2024-09-16T04:41:39.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.832+0000 7f2f369c4640 1 --2- 172.21.15.46:0/387914722 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2f10039840 0x7f2f1003bd00 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.832+0000 7f2f369c4640 1 --2- 172.21.15.46:0/387914722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f1c005210 0x7f2f3009d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.832+0000 7f2f369c4640 1 --2- 172.21.15.46:0/387914722 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2f1c004800 0x7f2f3009d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.832+0000 7f2f369c4640 1 --2- 172.21.15.46:0/387914722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f1c0039b0 0x7f2f3013a400 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.832+0000 7f2f369c4640 1 -- 172.21.15.46:0/387914722 >> 172.21.15.46:0/387914722 conn(0x7f2f300a6340 msgr2=0x7f2f30137010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:39.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.832+0000 7f2f369c4640 1 -- 172.21.15.46:0/387914722 shutdown_connections 2024-09-16T04:41:39.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.832+0000 7f2f369c4640 1 -- 172.21.15.46:0/387914722 wait complete. 2024-09-16T04:41:39.846 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:41:39.846 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T04:41:39.846 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:41:39.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.947+0000 7f032030a640 10 monclient: get_monmap_and_config 2024-09-16T04:41:39.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.947+0000 7f032030a640 10 monclient: build_initial_monmap 2024-09-16T04:41:39.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.947+0000 7f032030a640 10 monclient: monmap: 2024-09-16T04:41:39.950 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:39.950 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:39.950 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:41:39.948731+0000 2024-09-16T04:41:39.950 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:41:39.948731+0000 2024-09-16T04:41:39.950 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:39.950 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:39.950 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:41:39.951 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:41:39.951 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:41:39.951 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:41:39.951 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:41:39.951 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:39.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.947+0000 7f032030a640 1 Processor -- start 2024-09-16T04:41:39.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.947+0000 7f032030a640 1 -- start start 2024-09-16T04:41:39.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.947+0000 7f032030a640 10 monclient: init 2024-09-16T04:41:39.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f032030a640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:39.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f032030a640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-16T04:41:39.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f032030a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f03180a3c10 0x7f031809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:39.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f032030a640 10 monclient(hunting): picked mon.noname-a con 0x7f03180a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:41:39.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f032030a640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03180a3ff0 0x7f03180a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:39.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f032030a640 10 monclient(hunting): picked mon.noname-b con 0x7f03180a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:39.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f032030a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f03181369d0 addr v1:172.21.15.46:6789/0 2024-09-16T04:41:39.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f032030a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f031809b940 con 0x7f03181369d0 2024-09-16T04:41:39.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f032030a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:39.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f032030a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f031809ab00 con 0x7f03180a3c10 2024-09-16T04:41:39.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f032030a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:39.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f032030a640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f031809b030 con 0x7f03180a3ff0 2024-09-16T04:41:39.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f032030a640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:39.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f032030a640 10 monclient(hunting): authenticate will time out at 898.364593s 2024-09-16T04:41:39.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f031e8a7640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f03181369d0 legacy=0x7f0318136e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:41:39.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f031e0a6640 1 -- >> v2:172.21.15.46:3300/0 conn(0x7f03180a3c10 msgr2=0x7f031809bcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:41:39.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f031e0a6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f03180a3c10 0x7f031809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:41:39.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f031d8a5640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03180a3ff0 0x7f03180a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:39.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f031d8a5640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03180a3ff0 0x7f03180a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:37498/0 (socket says 172.21.15.46:37498) 2024-09-16T04:41:39.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f031d8a5640 1 -- 172.21.15.46:0/2436827683 learned_addr learned my addr 172.21.15.46:0/2436827683 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:39.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f031d8a5640 10 monclient(hunting): get_auth_request con 0x7f03180a3ff0 auth_method 0 2024-09-16T04:41:39.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f031d8a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:39.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f031d8a5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:39.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f031d8a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:39.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f031d8a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f031d8a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f031d8a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f031d8a5640 10 monclient(hunting): handle_auth_done global_id 24437 payload 274 2024-09-16T04:41:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.948+0000 7f031d8a5640 1 -- 172.21.15.46:0/2436827683 >> v2:172.21.15.46:3300/0 conn(0x7f03180a3c10 msgr2=0x7f031809bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:41:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f031d8a5640 1 --2- 172.21.15.46:0/2436827683 >> v2:172.21.15.46:3300/0 conn(0x7f03180a3c10 0x7f031809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f031d8a5640 1 -- 172.21.15.46:0/2436827683 >> v1:172.21.15.46:6789/0 conn(0x7f03181369d0 legacy=0x7f0318136e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:41:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f031d8a5640 10 monclient: _finish_hunting 0 2024-09-16T04:41:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f031d8a5640 1 monclient: found mon.noname-b 2024-09-16T04:41:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f031d8a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f031d8a5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:41:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f031d8a5640 1 -- 172.21.15.46:0/2436827683 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f031809b310 con 0x7f03180a3ff0 2024-09-16T04:41:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f031d8a5640 1 --2- 172.21.15.46:0/2436827683 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03180a3ff0 0x7f03180a43d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0308009870 tx=0x7f030802a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=475cfa4934f503e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f031d0a4640 1 -- 172.21.15.46:0/2436827683 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0308037070 con 0x7f03180a3ff0 2024-09-16T04:41:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f031d0a4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f031d0a4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:41:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f031d0a4640 10 monclient: dump: 2024-09-16T04:41:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:39.965 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:39.965 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:39.965 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:39.965 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:39.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f031d0a4640 10 monclient: _finish_auth 0 2024-09-16T04:41:39.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f031d0a4640 10 monclient: _check_auth_tickets 2024-09-16T04:41:39.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f031d0a4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:39.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f031d0a4640 1 -- 172.21.15.46:0/2436827683 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f030802ae20 con 0x7f03180a3ff0 2024-09-16T04:41:39.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f032030a640 5 monclient: authenticate success, global_id 24437 2024-09-16T04:41:39.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f032030a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:41:39.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f031d0a4640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:39.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f032030a640 10 monclient: get_monmap_and_config success 2024-09-16T04:41:39.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f031d0a4640 1 -- 172.21.15.46:0/2436827683 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0308032660 con 0x7f03180a3ff0 2024-09-16T04:41:39.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f031d0a4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:39.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f032030a640 4 set_mon_vals no callback set 2024-09-16T04:41:39.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f031d0a4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:39.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.949+0000 7f031d0a4640 10 monclient: dump: 2024-09-16T04:41:39.966 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:39.966 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:39.966 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.966 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.966 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:39.966 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:39.966 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:39.966 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:39.966 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:39.966 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:39.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.950+0000 7f032030a640 10 monclient: shutdown 2024-09-16T04:41:39.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.950+0000 7f032030a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:39.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.950+0000 7f032030a640 1 -- 172.21.15.46:0/2436827683 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03180a3ff0 msgr2=0x7f03180a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:39.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.950+0000 7f032030a640 1 --2- 172.21.15.46:0/2436827683 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03180a3ff0 0x7f03180a43d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0308009870 tx=0x7f030802a110 comp rx=0 tx=0).stop 2024-09-16T04:41:39.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.950+0000 7f032030a640 1 -- 172.21.15.46:0/2436827683 shutdown_connections 2024-09-16T04:41:39.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.950+0000 7f032030a640 1 --2- 172.21.15.46:0/2436827683 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03180a3ff0 0x7f03180a43d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.950+0000 7f032030a640 1 --2- 172.21.15.46:0/2436827683 >> v2:172.21.15.46:3300/0 conn(0x7f03180a3c10 0x7f031809bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.950+0000 7f032030a640 1 -- 172.21.15.46:0/2436827683 >> 172.21.15.46:0/2436827683 conn(0x7f03180a6340 msgr2=0x7f031806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:39.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.950+0000 7f032030a640 1 -- 172.21.15.46:0/2436827683 shutdown_connections 2024-09-16T04:41:39.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.950+0000 7f032030a640 1 -- 172.21.15.46:0/2436827683 wait complete. 2024-09-16T04:41:39.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.951+0000 7f032030a640 10 monclient: build_initial_monmap 2024-09-16T04:41:39.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.951+0000 7f032030a640 10 monclient: monmap: 2024-09-16T04:41:39.967 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:39.967 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:41:39.967 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:41:39.968 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:41:39.968 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:39.968 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:39.968 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:41:39.968 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:41:39.968 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:41:39.968 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:39.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.951+0000 7f032030a640 1 Processor -- start 2024-09-16T04:41:39.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.951+0000 7f032030a640 1 -- start start 2024-09-16T04:41:39.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.951+0000 7f032030a640 10 monclient: init 2024-09-16T04:41:39.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.951+0000 7f032030a640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:39.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f032030a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:41:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f032030a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03180a3c10 0x7f03181cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f032030a640 10 monclient(hunting): picked mon.noname-b con 0x7f03180a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:41:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f032030a640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03180a3ff0 0x7f0318135ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f032030a640 10 monclient(hunting): picked mon.noname-c con 0x7f03180a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f032030a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03181369d0 0x7f03181361e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f032030a640 10 monclient(hunting): picked mon.noname-a con 0x7f03181369d0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:41:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f032030a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f032030a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f03180a4910 con 0x7f03181369d0 2024-09-16T04:41:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f032030a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f032030a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f03180a69c0 con 0x7f03180a3c10 2024-09-16T04:41:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f032030a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f032030a640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f03180a0400 con 0x7f03180a3ff0 2024-09-16T04:41:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f032030a640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f032030a640 10 monclient(hunting): authenticate will time out at 898.368127s 2024-09-16T04:41:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031e8a7640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03181369d0 msgr2=0x7f03181361e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:41:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031e8a7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03181369d0 0x7f03181361e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:41:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031d8a5640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03180a3ff0 0x7f0318135ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031d8a5640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03180a3ff0 0x7f0318135ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:37508/0 (socket says 172.21.15.46:37508) 2024-09-16T04:41:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031d8a5640 1 -- 172.21.15.46:0/4277865974 learned_addr learned my addr 172.21.15.46:0/4277865974 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031d8a5640 10 monclient(hunting): get_auth_request con 0x7f03180a3ff0 auth_method 0 2024-09-16T04:41:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031d8a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031d8a5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031d8a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031d8a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031d8a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031d8a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031e0a6640 1 --2- 172.21.15.46:0/4277865974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03180a3c10 0x7f03181cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031d8a5640 10 monclient(hunting): handle_auth_done global_id 24440 payload 824 2024-09-16T04:41:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031d8a5640 1 -- 172.21.15.46:0/4277865974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03180a3c10 msgr2=0x7f03181cbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031d8a5640 1 --2- 172.21.15.46:0/4277865974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03180a3c10 0x7f03181cbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031d8a5640 1 -- 172.21.15.46:0/4277865974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03181369d0 msgr2=0x7f03181361e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:41:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031d8a5640 1 --2- 172.21.15.46:0/4277865974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03181369d0 0x7f03181361e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031d8a5640 10 monclient: _finish_hunting 0 2024-09-16T04:41:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031d8a5640 1 monclient: found mon.noname-c 2024-09-16T04:41:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031d8a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031d8a5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:41:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031d8a5640 1 -- 172.21.15.46:0/4277865974 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0318125a90 con 0x7f03180a3ff0 2024-09-16T04:41:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031e0a6640 10 monclient: get_auth_request con 0x7f03180a3c10 auth_method 0 2024-09-16T04:41:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031e0a6640 1 --2- 172.21.15.46:0/4277865974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03180a3c10 0x7f03181cbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:41:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031d8a5640 1 --2- 172.21.15.46:0/4277865974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03180a3ff0 0x7f0318135ca0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0308009870 tx=0x7f03080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f030effd640 1 -- 172.21.15.46:0/4277865974 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0308042070 con 0x7f03180a3ff0 2024-09-16T04:41:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f030effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f030effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:41:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f030effd640 10 monclient: dump: 2024-09-16T04:41:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:39.973 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:39.973 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:39.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f030effd640 10 monclient: _finish_auth 0 2024-09-16T04:41:39.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f030effd640 10 monclient: _check_auth_tickets 2024-09-16T04:41:39.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f030effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:39.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f030effd640 1 -- 172.21.15.46:0/4277865974 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f030803e670 con 0x7f03180a3ff0 2024-09-16T04:41:39.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f030effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:39.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f030effd640 1 -- 172.21.15.46:0/4277865974 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0308037040 con 0x7f03180a3ff0 2024-09-16T04:41:39.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f030effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:39.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f030effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:39.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f030effd640 10 monclient: dump: 2024-09-16T04:41:39.973 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:39.973 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:39.973 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.974 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.974 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:39.974 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:39.974 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:39.974 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:39.974 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:39.974 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:39.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f032030a640 5 monclient: authenticate success, global_id 24440 2024-09-16T04:41:39.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f032030a640 10 monclient: _renew_subs 2024-09-16T04:41:39.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f032030a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:39.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f032030a640 1 -- 172.21.15.46:0/4277865974 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03180a3170 con 0x7f03180a3ff0 2024-09-16T04:41:39.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.952+0000 7f031c8a3640 4 set_mon_vals no callback set 2024-09-16T04:41:39.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.953+0000 7f032030a640 10 monclient: _renew_subs 2024-09-16T04:41:39.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.953+0000 7f032030a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:39.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.953+0000 7f032030a640 1 -- 172.21.15.46:0/4277865974 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f03181276a0 con 0x7f03180a3ff0 2024-09-16T04:41:39.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.953+0000 7f032030a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:39.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.953+0000 7f032030a640 1 --2- 172.21.15.46:0/4277865974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03181237d0 0x7f0318123bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:39.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.953+0000 7f030effd640 1 -- 172.21.15.46:0/4277865974 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f0308045070 con 0x7f03180a3ff0 2024-09-16T04:41:39.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.953+0000 7f032030a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:39.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.953+0000 7f032030a640 1 -- 172.21.15.46:0/4277865974 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f03180a4a90 con 0x7f03181237d0 2024-09-16T04:41:39.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.953+0000 7f031e0a6640 1 --2- 172.21.15.46:0/4277865974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03181237d0 0x7f0318123bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:39.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.953+0000 7f031e0a6640 10 monclient: get_auth_request con 0x7f03181237d0 auth_method 0 2024-09-16T04:41:39.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.953+0000 7f031e0a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:39.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.953+0000 7f031e0a6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:39.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.953+0000 7f031e0a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:39.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.953+0000 7f030effd640 1 --2- 172.21.15.46:0/4277865974 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f02fc038360 0x7f02fc03a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:39.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.953+0000 7f031e0a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:39.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.953+0000 7f031e0a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:39.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.953+0000 7f031e0a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:39.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.954+0000 7f030effd640 1 -- 172.21.15.46:0/4277865974 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(113..113 src has 1..113) ==== 4346+0+0 (secure 0 0 0) 0x7f030806be90 con 0x7f03180a3ff0 2024-09-16T04:41:39.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.954+0000 7f031e0a6640 10 monclient(hunting): handle_auth_done global_id 24443 payload 824 2024-09-16T04:41:39.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.954+0000 7f031e0a6640 1 -- 172.21.15.46:0/4277865974 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f03180a0590 con 0x7f03181237d0 2024-09-16T04:41:39.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.954+0000 7f031e8a7640 1 --2- 172.21.15.46:0/4277865974 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f02fc038360 0x7f02fc03a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:39.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.954+0000 7f031e8a7640 10 monclient: get_auth_request con 0x7f02fc038360 auth_method 0 2024-09-16T04:41:39.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.954+0000 7f031e0a6640 1 --2- 172.21.15.46:0/4277865974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03181237d0 0x7f0318123bb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f031400c6f0 tx=0x7f031400cbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:39.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.954+0000 7f030effd640 1 -- 172.21.15.46:0/4277865974 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0314005c40 con 0x7f03181237d0 2024-09-16T04:41:39.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.954+0000 7f030effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:39.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.954+0000 7f030effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:39.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.954+0000 7f030effd640 10 monclient: dump: 2024-09-16T04:41:39.977 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:39.977 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:39.977 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.977 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.977 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:39.977 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:39.977 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:39.978 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:39.978 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:39.978 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:39.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.954+0000 7f031e8a7640 1 --2- 172.21.15.46:0/4277865974 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f02fc038360 0x7f02fc03a820 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f031809c9d0 tx=0x7f0310009270 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:39.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.954+0000 7f030effd640 1 -- 172.21.15.46:0/4277865974 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0314005c40 con 0x7f03181237d0 2024-09-16T04:41:39.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.954+0000 7f030effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:39.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.954+0000 7f030effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:41:39.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.954+0000 7f030effd640 1 -- 172.21.15.46:0/4277865974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03181237d0 msgr2=0x7f0318123bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:39.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.954+0000 7f030effd640 1 --2- 172.21.15.46:0/4277865974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03181237d0 0x7f0318123bb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f031400c6f0 tx=0x7f031400cbb0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.954+0000 7f030effd640 1 -- 172.21.15.46:0/4277865974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03181237d0 msgr2=0x7f0318123bb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:41:39.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.954+0000 7f030effd640 1 --2- 172.21.15.46:0/4277865974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03181237d0 0x7f0318123bb0 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f031400c6f0 tx=0x7f031400cbb0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.960+0000 7f032030a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:41:39.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.960+0000 7f032030a640 1 --2- 172.21.15.46:0/4277865974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03181240f0 0x7f031811f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:39.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.960+0000 7f032030a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:39.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.960+0000 7f032030a640 1 -- 172.21.15.46:0/4277865974 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f031809bd70 con 0x7f03181240f0 2024-09-16T04:41:39.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f031e0a6640 1 --2- 172.21.15.46:0/4277865974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03181240f0 0x7f031811f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:39.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f031e0a6640 10 monclient: get_auth_request con 0x7f03181240f0 auth_method 0 2024-09-16T04:41:39.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f031e0a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:39.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f031e0a6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:39.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f031e0a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:39.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f031e0a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:39.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f031e0a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:39.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f031e0a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:39.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f031e0a6640 10 monclient(hunting): handle_auth_done global_id 24446 payload 824 2024-09-16T04:41:39.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f031e0a6640 1 -- 172.21.15.46:0/4277865974 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0318127300 con 0x7f03181240f0 2024-09-16T04:41:39.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f031e0a6640 1 --2- 172.21.15.46:0/4277865974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03181240f0 0x7f031811f860 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f031400c6f0 tx=0x7f03140059d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:39.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f030effd640 1 -- 172.21.15.46:0/4277865974 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031401cce0 con 0x7f03181240f0 2024-09-16T04:41:39.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f030effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:39.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f030effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:39.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f030effd640 10 monclient: dump: 2024-09-16T04:41:39.982 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:39.982 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:39.982 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.982 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:39.982 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:39.982 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:39.982 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:39.982 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:39.982 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:39.982 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:39.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f030effd640 1 -- 172.21.15.46:0/4277865974 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4700 (secure 0 0 0) 0x7f031401cce0 con 0x7f03181240f0 2024-09-16T04:41:39.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f030effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:41:39.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f030effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:41:39.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f030effd640 1 -- 172.21.15.46:0/4277865974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03181240f0 msgr2=0x7f031811f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:39.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f030effd640 1 --2- 172.21.15.46:0/4277865974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03181240f0 0x7f031811f860 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f031400c6f0 tx=0x7f03140059d0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f030effd640 1 -- 172.21.15.46:0/4277865974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03181240f0 msgr2=0x7f031811f860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:41:39.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.961+0000 7f030effd640 1 --2- 172.21.15.46:0/4277865974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03181240f0 0x7f031811f860 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f031400c6f0 tx=0x7f03140059d0 comp rx=0 tx=0).stop 2024-09-16T04:41:39.983 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:41:39.984 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:41:39.984 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:41:39.984 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:41:39.984 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 30, 2024-09-16T04:41:39.984 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 75161, 2024-09-16T04:41:39.984 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:41:39.984 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:41:39.984 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:41:39.984 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:39.984 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:41:39.985 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:41:39.985 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:41:39.985 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:41:39.985 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:39.985 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:39.985 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:39.985 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:39.986 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:39.986 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:39.986 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:39.986 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:39.986 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:39.986 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:39.986 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:39.986 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:39.986 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:41:39.986 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:41:39.986 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:39.987 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:39.987 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:39.987 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:39.987 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:39.987 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:39.987 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:39.987 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:39.987 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:39.987 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:39.987 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:39.988 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:39.988 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:39.988 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:41:39.988 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:41:39.988 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:41:39.988 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:41:39.988 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:41:39.988 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:41:39.988 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:41:39.988 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:41:39.988 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:41:39.989 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:41:39.989 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:41:39.989 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:41:39.989 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:41:39.989 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:41:39.989 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:41:39.990 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:41:39.990 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:41:39.990 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:39.990 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:39.990 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:39.990 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:39.990 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:39.990 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:39.990 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:39.990 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:39.990 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:39.990 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:39.991 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:39.991 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:39.991 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:41:39.991 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:39.991 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:41:39.991 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:39.991 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:41:39.991 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:41:39.992 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:39.992 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:39.992 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:39.992 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:39.992 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:41:39.992 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:39.992 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:39.992 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:39.992 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:39.992 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:41:39.992 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:39.992 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:39.992 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:39.993 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:39.993 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:41:39.993 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:41:39.993 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:39.993 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:39.993 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:39.993 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:39.993 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:39.993 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:41:39.993 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:41:39.993 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:39.993 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:39.993 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:39.993 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:39.994 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:41:39.994 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:39.994 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:39.994 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:39.994 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:39.994 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:41:39.994 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:39.994 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:39.994 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:39.994 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:39.994 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:41:39.994 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:41:39.994 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:39.994 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:39.995 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:39.995 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:39.995 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:39.995 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:41:39.995 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:41:39.995 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:39.995 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:39.995 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:39.995 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:39.995 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:41:39.995 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:39.995 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:39.995 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:39.996 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:39.996 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:41:39.996 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:39.996 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:39.996 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:39.996 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:39.996 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:41:39.996 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:41:39.996 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:39.996 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:39.996 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:39.996 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:39.996 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:39.996 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:39.997 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:41:39.997 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:41:39.997 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:39.997 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:39.997 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:39.997 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:41:39.997 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:39.997 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:39.997 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:41:39.997 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:39.997 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:39.997 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:39.997 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:41:39.997 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:39.998 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:39.998 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:41:39.998 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:39.998 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:39.998 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:39.998 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5 2024-09-16T04:41:39.998 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:39.998 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:39.998 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:39.998 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:41:39.998 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:41:39.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.962+0000 7f032030a640 1 -- 172.21.15.46:0/4277865974 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f02fc038360 msgr2=0x7f02fc03a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.962+0000 7f032030a640 1 --2- 172.21.15.46:0/4277865974 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f02fc038360 0x7f02fc03a820 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f031809c9d0 tx=0x7f0310009270 comp rx=0 tx=0).stop 2024-09-16T04:41:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.962+0000 7f032030a640 10 monclient: shutdown 2024-09-16T04:41:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.962+0000 7f032030a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.962+0000 7f032030a640 1 -- 172.21.15.46:0/4277865974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03180a3ff0 msgr2=0x7f0318135ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.962+0000 7f032030a640 1 --2- 172.21.15.46:0/4277865974 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f03180a3ff0 0x7f0318135ca0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0308009870 tx=0x7f03080046c0 comp rx=0 tx=0).stop 2024-09-16T04:41:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.962+0000 7f031e8a7640 1 -- 172.21.15.46:0/4277865974 reap_dead start 2024-09-16T04:41:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.962+0000 7f031e8a7640 1 -- 172.21.15.46:0/4277865974 reap_dead start 2024-09-16T04:41:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.962+0000 7f032030a640 1 -- 172.21.15.46:0/4277865974 shutdown_connections 2024-09-16T04:41:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.962+0000 7f032030a640 1 -- 172.21.15.46:0/4277865974 >> 172.21.15.46:0/4277865974 conn(0x7f03180a6340 msgr2=0x7f03180a1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.962+0000 7f032030a640 1 -- 172.21.15.46:0/4277865974 shutdown_connections 2024-09-16T04:41:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:39.962+0000 7f032030a640 1 -- 172.21.15.46:0/4277865974 wait complete. 2024-09-16T04:41:40.001 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:41:40.001 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:41:40.001 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:41:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.100+0000 7fd0eda77640 10 monclient: get_monmap_and_config 2024-09-16T04:41:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.100+0000 7fd0eda77640 10 monclient: build_initial_monmap 2024-09-16T04:41:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.100+0000 7fd0eda77640 10 monclient: monmap: 2024-09-16T04:41:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:41:40.102299+0000 2024-09-16T04:41:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:41:40.102299+0000 2024-09-16T04:41:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:41:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:41:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:41:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:41:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:41:40.104 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:40.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.101+0000 7fd0eda77640 1 Processor -- start 2024-09-16T04:41:40.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.101+0000 7fd0eda77640 1 -- start start 2024-09-16T04:41:40.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.101+0000 7fd0eda77640 10 monclient: init 2024-09-16T04:41:40.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0eda77640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:40.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0eda77640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-16T04:41:40.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0eda77640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0e80a3c10 0x7fd0e809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:40.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0eda77640 10 monclient(hunting): picked mon.noname-b con 0x7fd0e80a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:40.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0eda77640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd0e80a3ff0 0x7fd0e80a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:40.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0eda77640 10 monclient(hunting): picked mon.noname-c con 0x7fd0e80a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:41:40.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0eda77640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd0e81369d0 0x7fd0e8136e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:40.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0eda77640 10 monclient(hunting): picked mon.noname-a con 0x7fd0e81369d0 addr v2:172.21.15.46:3300/0 2024-09-16T04:41:40.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0eda77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:40.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0eda77640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd0e809ab70 con 0x7fd0e81369d0 2024-09-16T04:41:40.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0eda77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:40.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0eda77640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fd0e80a4d60 con 0x7fd0e80a3ff0 2024-09-16T04:41:40.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0eda77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:40.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0eda77640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd0e809b030 con 0x7fd0e80a3c10 2024-09-16T04:41:40.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0eda77640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0eda77640 10 monclient(hunting): authenticate will time out at 898.518100s 2024-09-16T04:41:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e7fff640 1 -- >> v2:172.21.15.46:3300/0 conn(0x7fd0e81369d0 msgr2=0x7fd0e8136e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:41:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd0e81369d0 0x7fd0e8136e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:41:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e77fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0e80a3c10 0x7fd0e809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e77fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0e80a3c10 0x7fd0e809bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:37536/0 (socket says 172.21.15.46:37536) 2024-09-16T04:41:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e77fe640 1 -- 172.21.15.46:0/3452300899 learned_addr learned my addr 172.21.15.46:0/3452300899 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e77fe640 10 monclient(hunting): get_auth_request con 0x7fd0e80a3c10 auth_method 0 2024-09-16T04:41:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e6ffd640 1 --2- 172.21.15.46:0/3452300899 >> v2:172.21.15.114:3300/0 conn(0x7fd0e80a3ff0 0x7fd0e80a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e6ffd640 10 monclient(hunting): get_auth_request con 0x7fd0e80a3ff0 auth_method 0 2024-09-16T04:41:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e77fe640 10 monclient(hunting): handle_auth_done global_id 24449 payload 274 2024-09-16T04:41:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e77fe640 1 -- 172.21.15.46:0/3452300899 >> v2:172.21.15.114:3300/0 conn(0x7fd0e80a3ff0 msgr2=0x7fd0e80a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:41:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e77fe640 1 --2- 172.21.15.46:0/3452300899 >> v2:172.21.15.114:3300/0 conn(0x7fd0e80a3ff0 0x7fd0e80a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e77fe640 1 -- 172.21.15.46:0/3452300899 >> v2:172.21.15.46:3300/0 conn(0x7fd0e81369d0 msgr2=0x7fd0e8136e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:41:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e77fe640 1 --2- 172.21.15.46:0/3452300899 >> v2:172.21.15.46:3300/0 conn(0x7fd0e81369d0 0x7fd0e8136e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:41:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e77fe640 1 monclient: found mon.noname-b 2024-09-16T04:41:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:41:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.102+0000 7fd0e77fe640 1 -- 172.21.15.46:0/3452300899 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0e809b310 con 0x7fd0e80a3c10 2024-09-16T04:41:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.103+0000 7fd0e77fe640 1 --2- 172.21.15.46:0/3452300899 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0e80a3c10 0x7fd0e809bcf0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd0e806e8e0 tx=0x7fd0d802a160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2914eff216d6df64 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.103+0000 7fd0e67fc640 1 -- 172.21.15.46:0/3452300899 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8037070 con 0x7fd0e80a3c10 2024-09-16T04:41:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.103+0000 7fd0e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.103+0000 7fd0e67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:41:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.103+0000 7fd0e67fc640 10 monclient: dump: 2024-09-16T04:41:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:40.117 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.103+0000 7fd0e67fc640 10 monclient: _finish_auth 0 2024-09-16T04:41:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.103+0000 7fd0e67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:41:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.103+0000 7fd0e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.103+0000 7fd0e67fc640 1 -- 172.21.15.46:0/3452300899 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0d8032440 con 0x7fd0e80a3c10 2024-09-16T04:41:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.103+0000 7fd0e67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.103+0000 7fd0e67fc640 1 -- 172.21.15.46:0/3452300899 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8032810 con 0x7fd0e80a3c10 2024-09-16T04:41:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.103+0000 7fd0eda77640 5 monclient: authenticate success, global_id 24449 2024-09-16T04:41:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.103+0000 7fd0eda77640 10 monclient: get_monmap_and_config success 2024-09-16T04:41:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.103+0000 7fd0eda77640 4 set_mon_vals no callback set 2024-09-16T04:41:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.103+0000 7fd0e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.103+0000 7fd0e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:40.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.103+0000 7fd0e67fc640 10 monclient: dump: 2024-09-16T04:41:40.118 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:40.118 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:40.118 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:40.118 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:40.118 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:40.118 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:40.118 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:40.118 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:40.118 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:40.118 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:40.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.103+0000 7fd0eda77640 10 monclient: shutdown 2024-09-16T04:41:40.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.103+0000 7fd0eda77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:40.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.103+0000 7fd0eda77640 1 -- 172.21.15.46:0/3452300899 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0e80a3c10 msgr2=0x7fd0e809bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:40.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.103+0000 7fd0eda77640 1 --2- 172.21.15.46:0/3452300899 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0e80a3c10 0x7fd0e809bcf0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd0e806e8e0 tx=0x7fd0d802a160 comp rx=0 tx=0).stop 2024-09-16T04:41:40.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.104+0000 7fd0eda77640 1 -- 172.21.15.46:0/3452300899 shutdown_connections 2024-09-16T04:41:40.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.104+0000 7fd0eda77640 1 --2- 172.21.15.46:0/3452300899 >> v2:172.21.15.46:3300/0 conn(0x7fd0e81369d0 0x7fd0e8136e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:40.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.104+0000 7fd0eda77640 1 --2- 172.21.15.46:0/3452300899 >> v2:172.21.15.114:3300/0 conn(0x7fd0e80a3ff0 0x7fd0e80a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:40.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.104+0000 7fd0eda77640 1 --2- 172.21.15.46:0/3452300899 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0e80a3c10 0x7fd0e809bcf0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:40.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.104+0000 7fd0eda77640 1 -- 172.21.15.46:0/3452300899 >> 172.21.15.46:0/3452300899 conn(0x7fd0e80a5d10 msgr2=0x7fd0e80a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:40.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.104+0000 7fd0eda77640 1 -- 172.21.15.46:0/3452300899 shutdown_connections 2024-09-16T04:41:40.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.104+0000 7fd0eda77640 1 -- 172.21.15.46:0/3452300899 wait complete. 2024-09-16T04:41:40.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.104+0000 7fd0eda77640 10 monclient: build_initial_monmap 2024-09-16T04:41:40.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.104+0000 7fd0eda77640 10 monclient: monmap: 2024-09-16T04:41:40.120 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:41:40.120 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:41:40.120 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:41:40.120 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:41:40.120 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:41:40.120 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:40.120 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:41:40.120 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:41:40.120 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:41:40.120 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:40.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0eda77640 1 Processor -- start 2024-09-16T04:41:40.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0eda77640 1 -- start start 2024-09-16T04:41:40.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0eda77640 10 monclient: init 2024-09-16T04:41:40.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0eda77640 10 monclient: _reopen_session rank -1 2024-09-16T04:41:40.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0eda77640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:41:40.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0eda77640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0e80a3c10 0x7fd0e81cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:40.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0eda77640 10 monclient(hunting): picked mon.noname-b con 0x7fd0e80a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:41:40.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0eda77640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0e80a3ff0 0x7fd0e81cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:40.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0eda77640 10 monclient(hunting): picked mon.noname-a con 0x7fd0e80a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:41:40.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0eda77640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0e81369d0 0x7fd0e8123210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:40.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0eda77640 10 monclient(hunting): picked mon.noname-c con 0x7fd0e81369d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:41:40.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0eda77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:40.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0eda77640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e80a4910 con 0x7fd0e80a3ff0 2024-09-16T04:41:40.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0eda77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:40.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0eda77640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e806e3d0 con 0x7fd0e80a3c10 2024-09-16T04:41:40.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0eda77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:40.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0eda77640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd0e80a0490 con 0x7fd0e81369d0 2024-09-16T04:41:40.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0eda77640 10 monclient(hunting): _renew_subs 2024-09-16T04:41:40.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0eda77640 10 monclient(hunting): authenticate will time out at 898.521755s 2024-09-16T04:41:40.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0e6ffd640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0e80a3ff0 msgr2=0x7fd0e81cc420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:41:40.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0e6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0e80a3ff0 0x7fd0e81cc420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:41:40.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0e7fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0e81369d0 0x7fd0e8123210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:40.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0e7fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0e81369d0 0x7fd0e8123210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:37540/0 (socket says 172.21.15.46:37540) 2024-09-16T04:41:40.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0e7fff640 1 -- 172.21.15.46:0/3895235997 learned_addr learned my addr 172.21.15.46:0/3895235997 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:41:40.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0e7fff640 10 monclient(hunting): get_auth_request con 0x7fd0e81369d0 auth_method 0 2024-09-16T04:41:40.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:40.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:40.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:40.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:40.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.105+0000 7fd0e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0e77fe640 1 --2- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0e80a3c10 0x7fd0e81cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0e7fff640 10 monclient(hunting): handle_auth_done global_id 24452 payload 824 2024-09-16T04:41:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0e7fff640 1 -- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0e80a3c10 msgr2=0x7fd0e81cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0e7fff640 1 --2- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0e80a3c10 0x7fd0e81cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0e7fff640 1 -- 172.21.15.46:0/3895235997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0e80a3ff0 msgr2=0x7fd0e81cc420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:41:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0e7fff640 1 --2- 172.21.15.46:0/3895235997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0e80a3ff0 0x7fd0e81cc420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0e7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:41:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0e7fff640 1 monclient: found mon.noname-c 2024-09-16T04:41:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:41:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0e7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:41:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0e7fff640 1 -- 172.21.15.46:0/3895235997 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0e8136100 con 0x7fd0e81369d0 2024-09-16T04:41:40.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0e77fe640 10 monclient: get_auth_request con 0x7fd0e80a3c10 auth_method 0 2024-09-16T04:41:40.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0e77fe640 1 --2- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0e80a3c10 0x7fd0e81cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:41:40.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0e7fff640 1 --2- 172.21.15.46:0/3895235997 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0e81369d0 0x7fd0e8123210 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd0e000be60 tx=0x7fd0e000bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:40.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0cffff640 1 -- 172.21.15.46:0/3895235997 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0016430 con 0x7fd0e81369d0 2024-09-16T04:41:40.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:40.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0cffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:41:40.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0cffff640 10 monclient: dump: 2024-09-16T04:41:40.125 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:40.126 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:40.126 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:40.126 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:40.126 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:40.126 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:40.126 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:40.126 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:40.126 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:40.126 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:40.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0cffff640 10 monclient: _finish_auth 0 2024-09-16T04:41:40.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0cffff640 10 monclient: _check_auth_tickets 2024-09-16T04:41:40.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:41:40.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0eda77640 5 monclient: authenticate success, global_id 24452 2024-09-16T04:41:40.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0cffff640 1 -- 172.21.15.46:0/3895235997 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0e0010030 con 0x7fd0e81369d0 2024-09-16T04:41:40.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0eda77640 10 monclient: _renew_subs 2024-09-16T04:41:40.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0eda77640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:40.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0eda77640 1 -- 172.21.15.46:0/3895235997 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0e8126e80 con 0x7fd0e81369d0 2024-09-16T04:41:40.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0cffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:41:40.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0cffff640 1 -- 172.21.15.46:0/3895235997 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0016810 con 0x7fd0e81369d0 2024-09-16T04:41:40.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:40.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0cffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:41:40.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0cffff640 10 monclient: dump: 2024-09-16T04:41:40.128 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:40.128 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:40.128 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:40.128 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:40.128 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:40.128 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:40.128 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:40.128 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:40.129 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:40.129 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:40.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0eda77640 10 monclient: _renew_subs 2024-09-16T04:41:40.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0eda77640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:41:40.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0eda77640 1 -- 172.21.15.46:0/3895235997 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0e809f8c0 con 0x7fd0e81369d0 2024-09-16T04:41:40.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.106+0000 7fd0e5ffb640 4 set_mon_vals no callback set 2024-09-16T04:41:40.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.107+0000 7fd0eda77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:40.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.107+0000 7fd0eda77640 1 --2- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0e8123750 0x7fd0e8123b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:40.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.107+0000 7fd0eda77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:40.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.107+0000 7fd0eda77640 1 -- 172.21.15.46:0/3895235997 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e80a8af0 con 0x7fd0e8123750 2024-09-16T04:41:40.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.107+0000 7fd0cffff640 1 -- 172.21.15.46:0/3895235997 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fd0e00153e0 con 0x7fd0e81369d0 2024-09-16T04:41:40.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.107+0000 7fd0e77fe640 1 --2- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0e8123750 0x7fd0e8123b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:40.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.107+0000 7fd0cffff640 1 --2- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd0d4038360 0x7fd0d403a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:40.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.107+0000 7fd0e77fe640 10 monclient: get_auth_request con 0x7fd0e8123750 auth_method 0 2024-09-16T04:41:40.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.107+0000 7fd0e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:40.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.107+0000 7fd0e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:40.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.107+0000 7fd0e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:40.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.107+0000 7fd0cffff640 1 -- 172.21.15.46:0/3895235997 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(113..113 src has 1..113) ==== 4346+0+0 (secure 0 0 0) 0x7fd0e004ba20 con 0x7fd0e81369d0 2024-09-16T04:41:40.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.107+0000 7fd0e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.107+0000 7fd0e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.107+0000 7fd0e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.107+0000 7fd0e6ffd640 1 --2- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd0d4038360 0x7fd0d403a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.107+0000 7fd0e6ffd640 10 monclient: get_auth_request con 0x7fd0d4038360 auth_method 0 2024-09-16T04:41:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.108+0000 7fd0e77fe640 10 monclient(hunting): handle_auth_done global_id 5278 payload 824 2024-09-16T04:41:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.108+0000 7fd0e77fe640 1 -- 172.21.15.46:0/3895235997 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd0e809bcf0 con 0x7fd0e8123750 2024-09-16T04:41:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.108+0000 7fd0e77fe640 1 --2- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0e8123750 0x7fd0e8123b30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd0e809d0c0 tx=0x7fd0d80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.108+0000 7fd0e6ffd640 1 --2- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd0d4038360 0x7fd0d403a820 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8124a70 tx=0x7fd0dc0092e0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.108+0000 7fd0cffff640 1 -- 172.21.15.46:0/3895235997 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8037040 con 0x7fd0e8123750 2024-09-16T04:41:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.108+0000 7fd0cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.108+0000 7fd0cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:41:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.108+0000 7fd0cffff640 10 monclient: dump: 2024-09-16T04:41:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:40.132 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.108+0000 7fd0cffff640 1 -- 172.21.15.46:0/3895235997 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd0d803c870 con 0x7fd0e8123750 2024-09-16T04:41:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.108+0000 7fd0cffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:41:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.108+0000 7fd0cffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:41:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.108+0000 7fd0cffff640 1 -- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0e8123750 msgr2=0x7fd0e8123b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.108+0000 7fd0cffff640 1 --2- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0e8123750 0x7fd0e8123b30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd0e809d0c0 tx=0x7fd0d80046c0 comp rx=0 tx=0).stop 2024-09-16T04:41:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.108+0000 7fd0cffff640 1 -- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0e8123750 msgr2=0x7fd0e8123b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:41:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.108+0000 7fd0cffff640 1 --2- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0e8123750 0x7fd0e8123b30 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.114+0000 7fd0eda77640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:41:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.114+0000 7fd0eda77640 1 --2- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0e8124070 0x7fd0e811f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:41:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.114+0000 7fd0eda77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:41:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.114+0000 7fd0eda77640 1 -- 172.21.15.46:0/3895235997 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e8067e40 con 0x7fd0e8124070 2024-09-16T04:41:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.115+0000 7fd0e77fe640 1 --2- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0e8124070 0x7fd0e811f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:41:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.115+0000 7fd0e77fe640 10 monclient: get_auth_request con 0x7fd0e8124070 auth_method 0 2024-09-16T04:41:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.115+0000 7fd0e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:41:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.115+0000 7fd0e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:41:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.115+0000 7fd0e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:41:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.115+0000 7fd0e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:41:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.115+0000 7fd0e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:41:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.115+0000 7fd0e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:41:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.115+0000 7fd0e77fe640 10 monclient(hunting): handle_auth_done global_id 5281 payload 824 2024-09-16T04:41:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.115+0000 7fd0e77fe640 1 -- 172.21.15.46:0/3895235997 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd0e81cc960 con 0x7fd0e8124070 2024-09-16T04:41:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.115+0000 7fd0e77fe640 1 --2- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0e8124070 0x7fd0e811f860 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8000c00 tx=0x7fd0d8008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:41:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.115+0000 7fd0cffff640 1 -- 172.21.15.46:0/3895235997 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8037040 con 0x7fd0e8124070 2024-09-16T04:41:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.115+0000 7fd0cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:41:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.115+0000 7fd0cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:41:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.115+0000 7fd0cffff640 10 monclient: dump: 2024-09-16T04:41:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:41:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:41:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:41:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:41:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:41:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:41:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:41:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:41:40.137 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:41:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.116+0000 7fd0cffff640 1 -- 172.21.15.46:0/3895235997 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4870 (secure 0 0 0) 0x7fd0d8009490 con 0x7fd0e8124070 2024-09-16T04:41:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.116+0000 7fd0cffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:41:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.116+0000 7fd0cffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:41:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.116+0000 7fd0cffff640 1 -- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0e8124070 msgr2=0x7fd0e811f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.116+0000 7fd0cffff640 1 --2- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0e8124070 0x7fd0e811f860 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8000c00 tx=0x7fd0d8008030 comp rx=0 tx=0).stop 2024-09-16T04:41:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.116+0000 7fd0cffff640 1 -- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0e8124070 msgr2=0x7fd0e811f860 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:41:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.116+0000 7fd0cffff640 1 --2- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0e8124070 0x7fd0e811f860 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:40.138 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:41:40.138 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:41:40.138 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:41:40.138 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:41:40.138 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 30, 2024-09-16T04:41:40.138 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 186018, 2024-09-16T04:41:40.138 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:41:40.138 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:41:40.138 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:41:40.138 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:40.139 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:41:40.139 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:41:40.139 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:41:40.139 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:41:40.139 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:40.139 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:40.139 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:40.139 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:40.139 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:40.139 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:40.139 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:40.140 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:40.140 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:40.140 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:40.140 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:40.140 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:40.140 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:41:40.140 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:41:40.140 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:40.140 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:40.140 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:40.140 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:40.141 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:40.142 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:40.142 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:40.142 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:40.142 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:40.142 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:40.142 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:40.142 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:40.142 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:40.142 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:41:40.142 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:41:40.142 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:41:40.142 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:41:40.143 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:41:40.143 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:41:40.143 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:41:40.143 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:41:40.143 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:41:40.143 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:41:40.143 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:41:40.143 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:41:40.143 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:41:40.143 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:41:40.143 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:41:40.143 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:41:40.143 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:41:40.143 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:41:40.144 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:41:40.144 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:41:40.144 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:41:40.144 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:41:40.144 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:41:40.144 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:41:40.144 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:41:40.144 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:41:40.144 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:41:40.144 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:41:40.144 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:40.144 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:41:40.144 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:40.145 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:41:40.145 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:40.145 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:41:40.145 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:41:40.145 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:40.145 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:40.145 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:40.145 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:40.145 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:41:40.145 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:40.145 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:40.145 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:40.145 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:40.146 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:41:40.146 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:40.146 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:40.146 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:40.146 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:40.146 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:41:40.146 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:41:40.146 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:40.146 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:40.146 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:40.146 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:40.146 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:40.146 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:41:40.146 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:41:40.146 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:40.147 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:40.147 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:40.147 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:40.147 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:41:40.147 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:40.147 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:40.147 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:40.147 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:40.147 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:41:40.147 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:40.147 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:40.147 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:40.147 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:40.148 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:41:40.148 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:41:40.148 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:40.148 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:40.148 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:40.148 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:40.148 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:40.148 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:41:40.148 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:41:40.148 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:41:40.148 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:41:40.148 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:40.148 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:41:40.148 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:41:40.149 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:40.149 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:40.149 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:40.149 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:41:40.149 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:41:40.149 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:41:40.149 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:40.149 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:40.149 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:40.149 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:41:40.149 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:41:40.149 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:41:40.149 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:41:40.150 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:41:40.150 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:40.150 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:40.150 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:40.150 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:41:40.150 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:41:40.150 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:40.150 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:40.150 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:40.150 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:41:40.150 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:40.150 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:40.150 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:41:40.151 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:40.151 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:40.151 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:40.151 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4 2024-09-16T04:41:40.151 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:40.151 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:40.151 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:41:40.151 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:40.151 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:40.151 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:40.151 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5 2024-09-16T04:41:40.151 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:40.151 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:41:40.151 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:41:40.152 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:41:40.152 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:41:40.152 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:41:40.152 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:41:40.152 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:41:40.152 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:41:40.152 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:41:40.152 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:41:40.152 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:41:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.116+0000 7fd0eda77640 1 -- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd0d4038360 msgr2=0x7fd0d403a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.116+0000 7fd0eda77640 1 --2- 172.21.15.46:0/3895235997 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd0d4038360 0x7fd0d403a820 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8124a70 tx=0x7fd0dc0092e0 comp rx=0 tx=0).stop 2024-09-16T04:41:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.116+0000 7fd0eda77640 10 monclient: shutdown 2024-09-16T04:41:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.116+0000 7fd0eda77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:41:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.116+0000 7fd0eda77640 1 -- 172.21.15.46:0/3895235997 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0e81369d0 msgr2=0x7fd0e8123210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:41:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.116+0000 7fd0eda77640 1 --2- 172.21.15.46:0/3895235997 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0e81369d0 0x7fd0e8123210 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd0e000be60 tx=0x7fd0e000bf50 comp rx=0 tx=0).stop 2024-09-16T04:41:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.116+0000 7fd0e7fff640 1 -- 172.21.15.46:0/3895235997 reap_dead start 2024-09-16T04:41:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.116+0000 7fd0eda77640 1 -- 172.21.15.46:0/3895235997 shutdown_connections 2024-09-16T04:41:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.116+0000 7fd0eda77640 1 --2- 172.21.15.46:0/3895235997 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0e81369d0 0x7fd0e8123210 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:41:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.116+0000 7fd0eda77640 1 -- 172.21.15.46:0/3895235997 >> 172.21.15.46:0/3895235997 conn(0x7fd0e80a5d10 msgr2=0x7fd0e809e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:41:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.116+0000 7fd0eda77640 1 -- 172.21.15.46:0/3895235997 shutdown_connections 2024-09-16T04:41:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:41:40.116+0000 7fd0eda77640 1 -- 172.21.15.46:0/3895235997 wait complete. 2024-09-16T04:41:40.155 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:41:40.155 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:41:41.721 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:41.721 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36932]: starting. 2024-09-16T04:41:41.721 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36932]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-36897 2024-09-16T04:41:41.721 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36932]: created object 0... 2024-09-16T04:41:41.721 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36932]: created object 25... 2024-09-16T04:41:41.721 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36932]: created object 49... 2024-09-16T04:41:41.721 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36932]: finishing. 2024-09-16T04:41:41.721 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[36932]: shutting down. 2024-09-16T04:41:41.733 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:41.733 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36934]: starting. 2024-09-16T04:41:41.733 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36934]: listing objects. 2024-09-16T04:41:41.734 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36934]: listed object 0... 2024-09-16T04:41:41.734 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36934]: listed object 25... 2024-09-16T04:41:41.734 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36934]: saw 50 objects 2024-09-16T04:41:41.735 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[36934]: shutting down. 2024-09-16T04:41:42.639 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:42.639 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36933]: starting. 2024-09-16T04:41:42.639 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36933]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-36897 2024-09-16T04:41:42.639 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[36933]: shutting down. 2024-09-16T04:41:42.640 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:42.640 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:42.641 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:41:42.643 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:41:44.743 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37003]: starting. 2024-09-16T04:41:44.743 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37003]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-37002 2024-09-16T04:41:44.743 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37003]: created object 0... 2024-09-16T04:41:44.744 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37003]: created object 25... 2024-09-16T04:41:44.744 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37003]: created object 49... 2024-09-16T04:41:44.744 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37003]: finishing. 2024-09-16T04:41:44.744 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37003]: shutting down. 2024-09-16T04:41:45.659 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37004]: starting. 2024-09-16T04:41:45.660 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37004]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-37002 2024-09-16T04:41:45.660 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37004]: shutting down. 2024-09-16T04:41:47.757 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:47.757 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37039]: starting. 2024-09-16T04:41:47.758 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37039]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-37002 2024-09-16T04:41:47.758 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37039]: created object 0... 2024-09-16T04:41:47.758 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37039]: created object 25... 2024-09-16T04:41:47.758 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37039]: created object 49... 2024-09-16T04:41:47.758 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37039]: finishing. 2024-09-16T04:41:47.758 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37039]: shutting down. 2024-09-16T04:41:47.770 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:47.770 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37041]: starting. 2024-09-16T04:41:47.770 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37041]: listing objects. 2024-09-16T04:41:47.770 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37041]: listed object 0... 2024-09-16T04:41:47.770 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37041]: listed object 25... 2024-09-16T04:41:47.770 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37041]: saw 50 objects 2024-09-16T04:41:47.770 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37041]: shutting down. 2024-09-16T04:41:48.674 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:48.674 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37040]: starting. 2024-09-16T04:41:48.675 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37040]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-37002 2024-09-16T04:41:48.675 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37040]: shutting down. 2024-09-16T04:41:48.676 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:48.676 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:48.676 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:41:48.678 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:41:50.217 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37110]: starting. 2024-09-16T04:41:50.217 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37110]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-37109 2024-09-16T04:41:50.217 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37110]: created object 0... 2024-09-16T04:41:50.217 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37110]: created object 25... 2024-09-16T04:41:50.217 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37110]: created object 49... 2024-09-16T04:41:50.217 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37110]: finishing. 2024-09-16T04:41:50.217 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37110]: shutting down. 2024-09-16T04:41:51.139 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37111]: starting. 2024-09-16T04:41:51.140 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37111]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-37109 2024-09-16T04:41:51.140 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37111]: shutting down. 2024-09-16T04:41:53.245 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:53.245 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37146]: starting. 2024-09-16T04:41:53.245 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37146]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-37109 2024-09-16T04:41:53.245 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37146]: created object 0... 2024-09-16T04:41:53.245 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37146]: created object 25... 2024-09-16T04:41:53.245 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37146]: created object 49... 2024-09-16T04:41:53.245 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37146]: finishing. 2024-09-16T04:41:53.246 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37146]: shutting down. 2024-09-16T04:41:53.258 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:53.258 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37148]: starting. 2024-09-16T04:41:53.259 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37148]: listing objects. 2024-09-16T04:41:53.259 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37148]: listed object 0... 2024-09-16T04:41:53.259 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37148]: listed object 25... 2024-09-16T04:41:53.259 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37148]: saw 50 objects 2024-09-16T04:41:53.259 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37148]: shutting down. 2024-09-16T04:41:54.168 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:54.168 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37147]: starting. 2024-09-16T04:41:54.168 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37147]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-37109 2024-09-16T04:41:54.168 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37147]: shutting down. 2024-09-16T04:41:54.169 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:54.169 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:54.169 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:41:54.172 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:41:56.268 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37217]: starting. 2024-09-16T04:41:56.268 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37217]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-37216 2024-09-16T04:41:56.268 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37217]: created object 0... 2024-09-16T04:41:56.268 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37217]: created object 25... 2024-09-16T04:41:56.268 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37217]: created object 49... 2024-09-16T04:41:56.268 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37217]: finishing. 2024-09-16T04:41:56.269 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37217]: shutting down. 2024-09-16T04:41:57.180 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37218]: starting. 2024-09-16T04:41:57.181 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37218]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-37216 2024-09-16T04:41:57.181 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37218]: shutting down. 2024-09-16T04:41:59.219 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:59.219 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37253]: starting. 2024-09-16T04:41:59.220 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37253]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-37216 2024-09-16T04:41:59.220 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37253]: created object 0... 2024-09-16T04:41:59.220 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37253]: created object 25... 2024-09-16T04:41:59.220 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37253]: created object 49... 2024-09-16T04:41:59.220 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37253]: finishing. 2024-09-16T04:41:59.220 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37253]: shutting down. 2024-09-16T04:41:59.232 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:41:59.232 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37255]: starting. 2024-09-16T04:41:59.232 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37255]: listing objects. 2024-09-16T04:41:59.232 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37255]: listed object 0... 2024-09-16T04:41:59.232 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37255]: listed object 25... 2024-09-16T04:41:59.232 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37255]: saw 50 objects 2024-09-16T04:41:59.232 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37255]: shutting down. 2024-09-16T04:42:00.139 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:00.139 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37254]: starting. 2024-09-16T04:42:00.139 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37254]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-37216 2024-09-16T04:42:00.139 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37254]: shutting down. 2024-09-16T04:42:00.140 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:00.140 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:00.140 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:42:00.143 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:42:00.156 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:42:00.156 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T04:42:00.156 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T04:42:00.157 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T04:42:00.160 INFO:tasks.ceph.mon.a:Started 2024-09-16T04:42:00.160 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:42:00.160 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:42:00.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.273+0000 7f6e5a45e640 10 monclient: get_monmap_and_config 2024-09-16T04:42:00.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.273+0000 7f6e5a45e640 10 monclient: build_initial_monmap 2024-09-16T04:42:00.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.273+0000 7f6e5a45e640 10 monclient: monmap: 2024-09-16T04:42:00.275 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:00.275 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:00.275 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:00.274067+0000 2024-09-16T04:42:00.275 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:00.274067+0000 2024-09-16T04:42:00.275 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:00.275 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:00.275 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:00.275 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:00.276 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:00.276 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:00.276 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:00.276 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:00.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.274+0000 7f6e5a45e640 1 Processor -- start 2024-09-16T04:42:00.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.274+0000 7f6e5a45e640 1 -- start start 2024-09-16T04:42:00.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.274+0000 7f6e5a45e640 10 monclient: init 2024-09-16T04:42:00.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.274+0000 7f6e5a45e640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:00.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.274+0000 7f6e5a45e640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-16T04:42:00.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e5a45e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6e540a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:42:00.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e5a45e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6e540a3ff0 0x7f6e540a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:00.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e5a45e640 10 monclient(hunting): picked mon.noname-b con 0x7f6e540a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:00.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e5a45e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e5413be90 addr v1:172.21.15.46:6789/0 2024-09-16T04:42:00.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e5a45e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e5409ab70 con 0x7f6e5413be90 2024-09-16T04:42:00.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e5a45e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e540a4d60 con 0x7f6e540a3c10 2024-09-16T04:42:00.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e5a45e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:00.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e5a45e640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f6e5409b030 con 0x7f6e540a3ff0 2024-09-16T04:42:00.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e5a45e640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:00.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e5a45e640 10 monclient(hunting): authenticate will time out at 918.690149s 2024-09-16T04:42:00.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e537fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6e540a3ff0 0x7f6e540a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:00.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e537fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6e540a3ff0 0x7f6e540a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38338/0 (socket says 172.21.15.46:38338) 2024-09-16T04:42:00.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e537fe640 1 -- 172.21.15.46:0/249478959 learned_addr learned my addr 172.21.15.46:0/249478959 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:00.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e589fb640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6e5413be90 0x7f6e5413e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50660/0 (socket says 172.21.15.46:50660) 2024-09-16T04:42:00.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e537fe640 10 monclient(hunting): get_auth_request con 0x7f6e540a3ff0 auth_method 0 2024-09-16T04:42:00.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:00.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:00.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:00.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:00.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:00.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:00.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e537fe640 10 monclient(hunting): handle_auth_done global_id 24539 payload 274 2024-09-16T04:42:00.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e537fe640 1 -- 172.21.15.46:0/249478959 >> v1:172.21.15.114:6789/0 conn(0x7f6e540a3c10 legacy=0x7f6e5409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:00.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e537fe640 1 -- 172.21.15.46:0/249478959 >> v1:172.21.15.46:6789/0 conn(0x7f6e5413be90 legacy=0x7f6e5413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:00.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e537fe640 10 monclient: _finish_hunting 0 2024-09-16T04:42:00.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e537fe640 1 monclient: found mon.noname-b 2024-09-16T04:42:00.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:00.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e537fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:42:00.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.275+0000 7f6e537fe640 1 -- 172.21.15.46:0/249478959 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e5409b310 con 0x7f6e540a3ff0 2024-09-16T04:42:00.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.276+0000 7f6e537fe640 1 --2- 172.21.15.46:0/249478959 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6e540a3ff0 0x7f6e540a43d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c0098a0 tx=0x7f6e3c02a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6eb4632c4f12c90 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:00.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.276+0000 7f6e52ffd640 1 -- 172.21.15.46:0/249478959 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c037070 con 0x7f6e540a3ff0 2024-09-16T04:42:00.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.276+0000 7f6e52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:00.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.276+0000 7f6e52ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:42:00.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.276+0000 7f6e52ffd640 10 monclient: dump: 2024-09-16T04:42:00.288 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:00.288 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:00.288 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:00.288 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:00.288 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:00.288 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:00.288 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:00.288 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:00.288 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:00.288 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:00.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.276+0000 7f6e52ffd640 10 monclient: _finish_auth 0 2024-09-16T04:42:00.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.276+0000 7f6e52ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:42:00.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.276+0000 7f6e52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:00.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.276+0000 7f6e52ffd640 1 -- 172.21.15.46:0/249478959 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e3c032440 con 0x7f6e540a3ff0 2024-09-16T04:42:00.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.276+0000 7f6e5a45e640 5 monclient: authenticate success, global_id 24539 2024-09-16T04:42:00.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.276+0000 7f6e5a45e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:00.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.276+0000 7f6e52ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:00.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.276+0000 7f6e5a45e640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:00.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.276+0000 7f6e52ffd640 1 -- 172.21.15.46:0/249478959 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c032810 con 0x7f6e540a3ff0 2024-09-16T04:42:00.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.276+0000 7f6e5a45e640 4 set_mon_vals no callback set 2024-09-16T04:42:00.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.276+0000 7f6e52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:00.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.276+0000 7f6e52ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:00.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.276+0000 7f6e52ffd640 10 monclient: dump: 2024-09-16T04:42:00.289 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:00.290 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:00.290 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:00.290 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:00.290 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:00.290 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:00.290 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:00.290 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:00.290 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:00.290 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:00.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.277+0000 7f6e5a45e640 10 monclient: shutdown 2024-09-16T04:42:00.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.277+0000 7f6e5a45e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:00.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.277+0000 7f6e5a45e640 1 -- 172.21.15.46:0/249478959 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6e540a3ff0 msgr2=0x7f6e540a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:00.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.277+0000 7f6e5a45e640 1 --2- 172.21.15.46:0/249478959 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6e540a3ff0 0x7f6e540a43d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c0098a0 tx=0x7f6e3c02a1c0 comp rx=0 tx=0).stop 2024-09-16T04:42:00.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.277+0000 7f6e5a45e640 1 -- 172.21.15.46:0/249478959 shutdown_connections 2024-09-16T04:42:00.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.277+0000 7f6e5a45e640 1 --2- 172.21.15.46:0/249478959 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6e540a3ff0 0x7f6e540a43d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:00.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.277+0000 7f6e5a45e640 1 -- 172.21.15.46:0/249478959 >> 172.21.15.46:0/249478959 conn(0x7f6e540a5d10 msgr2=0x7f6e540a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:00.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.277+0000 7f6e5a45e640 1 -- 172.21.15.46:0/249478959 shutdown_connections 2024-09-16T04:42:00.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.277+0000 7f6e5a45e640 1 -- 172.21.15.46:0/249478959 wait complete. 2024-09-16T04:42:00.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.277+0000 7f6e5a45e640 10 monclient: build_initial_monmap 2024-09-16T04:42:00.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.277+0000 7f6e5a45e640 10 monclient: monmap: 2024-09-16T04:42:00.291 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:00.291 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:00.291 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:00.291 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:00.291 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:00.291 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:00.292 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:00.292 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:00.292 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:00.292 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:00.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e5a45e640 1 Processor -- start 2024-09-16T04:42:00.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e5a45e640 1 -- start start 2024-09-16T04:42:00.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e5a45e640 10 monclient: init 2024-09-16T04:42:00.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e5a45e640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:00.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e5a45e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:42:00.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e5a45e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e540a3c10 0x7f6e541cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:00.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e5a45e640 10 monclient(hunting): picked mon.noname-b con 0x7f6e540a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:00.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e5a45e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6e540a3ff0 0x7f6e541cc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:00.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e5a45e640 10 monclient(hunting): picked mon.noname-c con 0x7f6e540a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:00.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e5a45e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5413be90 0x7f6e540a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:00.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e5a45e640 10 monclient(hunting): picked mon.noname-a con 0x7f6e5413be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:00.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e5a45e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:00.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e5a45e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5406e550 con 0x7f6e5413be90 2024-09-16T04:42:00.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e5a45e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:00.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e5a45e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5406e3d0 con 0x7f6e540a3c10 2024-09-16T04:42:00.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e5a45e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:00.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e5a45e640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f6e5413f9a0 con 0x7f6e540a3ff0 2024-09-16T04:42:00.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e5a45e640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:00.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e5a45e640 10 monclient(hunting): authenticate will time out at 918.693852s 2024-09-16T04:42:00.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e589fb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5413be90 0x7f6e540a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:00.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e537fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6e540a3ff0 0x7f6e541cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:00.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e537fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6e540a3ff0 0x7f6e541cc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38350/0 (socket says 172.21.15.46:38350) 2024-09-16T04:42:00.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.278+0000 7f6e537fe640 1 -- 172.21.15.46:0/2997089569 learned_addr learned my addr 172.21.15.46:0/2997089569 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:00.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e537fe640 10 monclient(hunting): get_auth_request con 0x7f6e540a3ff0 auth_method 0 2024-09-16T04:42:00.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:00.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:00.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:00.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e589fb640 10 monclient(hunting): get_auth_request con 0x7f6e5413be90 auth_method 0 2024-09-16T04:42:00.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e589fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:00.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e589fb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:00.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e589fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:00.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e53fff640 1 --2- 172.21.15.46:0/2997089569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e540a3c10 0x7f6e541cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:00.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e53fff640 10 monclient(hunting): get_auth_request con 0x7f6e540a3c10 auth_method 0 2024-09-16T04:42:00.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:00.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e53fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:00.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:00.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:00.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:00.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:00.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e589fb640 1 -- 172.21.15.46:0/2997089569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5413be90 msgr2=0x7f6e540a0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:42:00.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e589fb640 1 -- 172.21.15.46:0/2997089569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5413be90 msgr2=0x7f6e540a0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:42:00.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e589fb640 1 --2- 172.21.15.46:0/2997089569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5413be90 0x7f6e540a0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:00.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:00.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:00.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e589fb640 1 --2- 172.21.15.46:0/2997089569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5413be90 0x7f6e540a0400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:00.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:00.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e537fe640 10 monclient(hunting): handle_auth_done global_id 24542 payload 824 2024-09-16T04:42:00.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e537fe640 1 -- 172.21.15.46:0/2997089569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e540a3c10 msgr2=0x7f6e541cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:00.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e537fe640 1 --2- 172.21.15.46:0/2997089569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e540a3c10 0x7f6e541cbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:00.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e537fe640 1 -- 172.21.15.46:0/2997089569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5413be90 msgr2=0x7f6e540a0400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:42:00.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e537fe640 1 --2- 172.21.15.46:0/2997089569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5413be90 0x7f6e540a0400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:00.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e537fe640 10 monclient: _finish_hunting 0 2024-09-16T04:42:00.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e537fe640 1 monclient: found mon.noname-c 2024-09-16T04:42:00.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:00.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e537fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:42:00.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e537fe640 1 -- 172.21.15.46:0/2997089569 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e54120740 con 0x7f6e540a3ff0 2024-09-16T04:42:00.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e537fe640 1 --2- 172.21.15.46:0/2997089569 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6e540a3ff0 0x7f6e541cc300 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c0098a0 tx=0x7f6e3c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:00.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e50ff9640 1 -- 172.21.15.46:0/2997089569 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c037040 con 0x7f6e540a3ff0 2024-09-16T04:42:00.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e50ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:00.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e50ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:42:00.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e50ff9640 10 monclient: dump: 2024-09-16T04:42:00.297 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:00.297 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:00.297 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:00.297 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:00.297 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:00.297 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:00.297 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:00.297 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:00.298 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:00.298 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:00.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e50ff9640 10 monclient: _finish_auth 0 2024-09-16T04:42:00.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e50ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:42:00.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e50ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:00.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e50ff9640 1 -- 172.21.15.46:0/2997089569 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e3c030770 con 0x7f6e540a3ff0 2024-09-16T04:42:00.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e5a45e640 5 monclient: authenticate success, global_id 24542 2024-09-16T04:42:00.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e50ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:00.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e5a45e640 10 monclient: _renew_subs 2024-09-16T04:42:00.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e50ff9640 1 -- 172.21.15.46:0/2997089569 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c030b40 con 0x7f6e540a3ff0 2024-09-16T04:42:00.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e5a45e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:00.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e5a45e640 1 -- 172.21.15.46:0/2997089569 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e540a2270 con 0x7f6e540a3ff0 2024-09-16T04:42:00.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e50ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:00.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e50ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:00.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e50ff9640 10 monclient: dump: 2024-09-16T04:42:00.299 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:00.299 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:00.299 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:00.299 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:00.299 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:00.299 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:00.299 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:00.299 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:00.300 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:00.300 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:00.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e527fc640 4 set_mon_vals no callback set 2024-09-16T04:42:00.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e5a45e640 10 monclient: _renew_subs 2024-09-16T04:42:00.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e5a45e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:00.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.279+0000 7f6e5a45e640 1 -- 172.21.15.46:0/2997089569 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e54120460 con 0x7f6e540a3ff0 2024-09-16T04:42:00.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.280+0000 7f6e50ff9640 1 -- 172.21.15.46:0/2997089569 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f6e3c044020 con 0x7f6e540a3ff0 2024-09-16T04:42:00.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.280+0000 7f6e5a45e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:00.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.280+0000 7f6e5a45e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:00.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.280+0000 7f6e5a45e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:00.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.280+0000 7f6e5a45e640 1 -- 172.21.15.46:0/2997089569 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e540a43d0 con 0x7f6e540a3ff0 2024-09-16T04:42:00.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.280+0000 7f6e50ff9640 1 --2- 172.21.15.46:0/2997089569 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6e340383b0 0x7f6e3403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:00.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.281+0000 7f6e50ff9640 1 -- 172.21.15.46:0/2997089569 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(134..134 src has 1..134) ==== 4346+0+0 (secure 0 0 0) 0x7f6e3c06ba50 con 0x7f6e540a3ff0 2024-09-16T04:42:00.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.281+0000 7f6e53fff640 1 --2- 172.21.15.46:0/2997089569 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6e340383b0 0x7f6e3403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:00.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.281+0000 7f6e53fff640 10 monclient: get_auth_request con 0x7f6e340383b0 auth_method 0 2024-09-16T04:42:00.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.281+0000 7f6e53fff640 1 --2- 172.21.15.46:0/2997089569 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6e340383b0 0x7f6e3403a870 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6e540a1e20 tx=0x7f6e44009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:00.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.285+0000 7f6e50ff9640 1 -- 172.21.15.46:0/2997089569 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6e3c04b020 con 0x7f6e540a3ff0 2024-09-16T04:42:00.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.286+0000 7f6e50ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:00.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.286+0000 7f6e50ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:00.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.528+0000 7f6e5a45e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:42:00.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.529+0000 7f6e5a45e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:00.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.529+0000 7f6e5a45e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:00.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.529+0000 7f6e5a45e640 1 -- 172.21.15.46:0/2997089569 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6e54069150 con 0x7f6e540a3ff0 2024-09-16T04:42:00.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.529+0000 7f6e50ff9640 1 -- 172.21.15.46:0/2997089569 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f6e3c03cbe0 con 0x7f6e540a3ff0 2024-09-16T04:42:00.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.529+0000 7f6e50ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:00.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.529+0000 7f6e50ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:00.531 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":30,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:42:00.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.531+0000 7f6e5a45e640 1 -- 172.21.15.46:0/2997089569 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6e340383b0 msgr2=0x7f6e3403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:00.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.531+0000 7f6e5a45e640 1 --2- 172.21.15.46:0/2997089569 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6e340383b0 0x7f6e3403a870 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6e540a1e20 tx=0x7f6e44009040 comp rx=0 tx=0).stop 2024-09-16T04:42:00.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.531+0000 7f6e5a45e640 10 monclient: shutdown 2024-09-16T04:42:00.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.531+0000 7f6e5a45e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.531+0000 7f6e5a45e640 1 -- 172.21.15.46:0/2997089569 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6e540a3ff0 msgr2=0x7f6e541cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.531+0000 7f6e5a45e640 1 --2- 172.21.15.46:0/2997089569 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6e540a3ff0 0x7f6e541cc300 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c0098a0 tx=0x7f6e3c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:42:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.531+0000 7f6e5a45e640 1 -- 172.21.15.46:0/2997089569 shutdown_connections 2024-09-16T04:42:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.531+0000 7f6e5a45e640 1 --2- 172.21.15.46:0/2997089569 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6e340383b0 0x7f6e3403a870 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.531+0000 7f6e5a45e640 1 --2- 172.21.15.46:0/2997089569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5413be90 0x7f6e540a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.531+0000 7f6e5a45e640 1 --2- 172.21.15.46:0/2997089569 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6e540a3ff0 0x7f6e541cc300 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.531+0000 7f6e5a45e640 1 --2- 172.21.15.46:0/2997089569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e540a3c10 0x7f6e541cbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.531+0000 7f6e5a45e640 1 -- 172.21.15.46:0/2997089569 >> 172.21.15.46:0/2997089569 conn(0x7f6e540a5d10 msgr2=0x7f6e5409f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.531+0000 7f6e5a45e640 1 -- 172.21.15.46:0/2997089569 shutdown_connections 2024-09-16T04:42:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:00.531+0000 7f6e5a45e640 1 -- 172.21.15.46:0/2997089569 wait complete. 2024-09-16T04:42:03.544 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:42:03.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.656+0000 7f538fac9640 10 monclient: get_monmap_and_config 2024-09-16T04:42:03.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.656+0000 7f538fac9640 10 monclient: build_initial_monmap 2024-09-16T04:42:03.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.656+0000 7f538fac9640 10 monclient: monmap: 2024-09-16T04:42:03.657 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:03.657 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:03.657 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:03.656557+0000 2024-09-16T04:42:03.657 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:03.656557+0000 2024-09-16T04:42:03.657 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:03.657 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:03.658 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:03.658 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:03.658 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:03.658 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:03.658 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:03.658 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:03.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.656+0000 7f538fac9640 1 Processor -- start 2024-09-16T04:42:03.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.656+0000 7f538fac9640 1 -- start start 2024-09-16T04:42:03.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.656+0000 7f538fac9640 10 monclient: init 2024-09-16T04:42:03.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538fac9640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:03.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538fac9640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-16T04:42:03.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538fac9640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 0x7f538809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:03.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538fac9640 10 monclient(hunting): picked mon.noname-c con 0x7f53880a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:42:03.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538fac9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f53880a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:42:03.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538fac9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 0x7f538813e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:03.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538fac9640 10 monclient(hunting): picked mon.noname-a con 0x7f538813be90 addr v2:172.21.15.46:3300/0 2024-09-16T04:42:03.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538fac9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f538809b940 con 0x7f53880a3ff0 2024-09-16T04:42:03.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538fac9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:03.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538fac9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f538809ab00 con 0x7f538813be90 2024-09-16T04:42:03.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538fac9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:03.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538fac9640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f538809b030 con 0x7f53880a3c10 2024-09-16T04:42:03.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538fac9640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:03.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538fac9640 10 monclient(hunting): authenticate will time out at 922.072466s 2024-09-16T04:42:03.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538e066640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 0x7f538813e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:03.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538d064640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f53880a3ff0 0x7f53880a43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50668/0 (socket says 172.21.15.46:50668) 2024-09-16T04:42:03.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538d064640 1 -- 172.21.15.46:0/1421863117 learned_addr learned my addr 172.21.15.46:0/1421863117 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:42:03.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538e066640 10 monclient(hunting): get_auth_request con 0x7f538813be90 auth_method 0 2024-09-16T04:42:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538e066640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538e066640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538e066640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538e066640 1 -- 172.21.15.46:0/1421863117 >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 msgr2=0x7f538813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:42:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538e066640 1 -- 172.21.15.46:0/1421863117 >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 msgr2=0x7f538813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:42:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538e066640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 0x7f538813e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538e066640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 0x7f538813e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538d865640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 0x7f538809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538d865640 10 monclient(hunting): get_auth_request con 0x7f53880a3c10 auth_method 0 2024-09-16T04:42:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538d865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538d865640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538d865640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538d865640 1 -- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 msgr2=0x7f538809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:42:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538d865640 1 -- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 msgr2=0x7f538809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:42:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538d865640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 0x7f538809bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:03.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.657+0000 7f538d865640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 0x7f538809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.858+0000 7f538e066640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 0x7f538813e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.858+0000 7f538e066640 10 monclient(hunting): get_auth_request con 0x7f538813be90 auth_method 0 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.858+0000 7f538e066640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.858+0000 7f538e066640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.858+0000 7f538e066640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.858+0000 7f538e066640 1 -- 172.21.15.46:0/1421863117 >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 msgr2=0x7f538813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.858+0000 7f538e066640 1 -- 172.21.15.46:0/1421863117 >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 msgr2=0x7f538813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.858+0000 7f538e066640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 0x7f538813e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.858+0000 7f538e066640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 0x7f538813e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.858+0000 7f538d865640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 0x7f538809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.858+0000 7f538d865640 10 monclient(hunting): get_auth_request con 0x7f53880a3c10 auth_method 0 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.858+0000 7f538d865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.858+0000 7f538d865640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.858+0000 7f538d865640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.858+0000 7f538d865640 1 -- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 msgr2=0x7f538809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.858+0000 7f538d865640 1 -- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 msgr2=0x7f538809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.858+0000 7f538d865640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 0x7f538809bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:03.858+0000 7f538d865640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 0x7f538809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:42:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:04.259+0000 7f538e066640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 0x7f538813e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:04.259+0000 7f538e066640 10 monclient(hunting): get_auth_request con 0x7f538813be90 auth_method 0 2024-09-16T04:42:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:04.259+0000 7f538e066640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:04.259+0000 7f538e066640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:04.259+0000 7f538e066640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:04.259+0000 7f538e066640 1 -- 172.21.15.46:0/1421863117 >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 msgr2=0x7f538813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:42:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:04.259+0000 7f538e066640 1 -- 172.21.15.46:0/1421863117 >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 msgr2=0x7f538813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:42:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:04.259+0000 7f538e066640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 0x7f538813e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:04.259+0000 7f538e066640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 0x7f538813e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:42:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:04.259+0000 7f538d865640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 0x7f538809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:04.259+0000 7f538d865640 10 monclient(hunting): get_auth_request con 0x7f53880a3c10 auth_method 0 2024-09-16T04:42:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:04.259+0000 7f538d865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:04.259+0000 7f538d865640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:04.259+0000 7f538d865640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:04.259+0000 7f538d865640 1 -- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 msgr2=0x7f538809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:42:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:04.259+0000 7f538d865640 1 -- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 msgr2=0x7f538809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:42:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:04.259+0000 7f538d865640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 0x7f538809bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:04.259+0000 7f538d865640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 0x7f538809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:42:05.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:05.060+0000 7f538e066640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 0x7f538813e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:05.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:05.060+0000 7f538e066640 10 monclient(hunting): get_auth_request con 0x7f538813be90 auth_method 0 2024-09-16T04:42:05.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:05.060+0000 7f538e066640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:05.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:05.060+0000 7f538e066640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:05.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:05.060+0000 7f538e066640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:05.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:05.060+0000 7f538e066640 1 -- 172.21.15.46:0/1421863117 >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 msgr2=0x7f538813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:42:05.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:05.060+0000 7f538e066640 1 -- 172.21.15.46:0/1421863117 >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 msgr2=0x7f538813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:42:05.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:05.060+0000 7f538e066640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 0x7f538813e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:05.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:05.060+0000 7f538e066640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 0x7f538813e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:42:05.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:05.060+0000 7f538d865640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 0x7f538809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:05.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:05.060+0000 7f538d865640 10 monclient(hunting): get_auth_request con 0x7f53880a3c10 auth_method 0 2024-09-16T04:42:05.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:05.060+0000 7f538d865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:05.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:05.060+0000 7f538d865640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:05.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:05.060+0000 7f538d865640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:05.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:05.061+0000 7f538d865640 1 -- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 msgr2=0x7f538809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:42:05.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:05.061+0000 7f538d865640 1 -- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 msgr2=0x7f538809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:42:05.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:05.061+0000 7f538d865640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 0x7f538809bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:05.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:05.061+0000 7f538d865640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 0x7f538809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:42:06.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 1 -- 172.21.15.46:0/1421863117 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3366412756 0 0) 0x7f538809b940 con 0x7f53880a3ff0 2024-09-16T04:42:06.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 10 monclient(hunting): _negotiate 2024-09-16T04:42:06.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:06.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:06.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 10 monclient(hunting): my global_id is 44097 2024-09-16T04:42:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 1 -- 172.21.15.46:0/1421863117 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5374003520 con 0x7f53880a3ff0 2024-09-16T04:42:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 1 -- 172.21.15.46:0/1421863117 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2286859582 0 0) 0x7f5374003520 con 0x7f53880a3ff0 2024-09-16T04:42:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 1 -- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 msgr2=0x7f538809bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:42:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.114:3300/0 conn(0x7f53880a3c10 0x7f538809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 1 -- 172.21.15.46:0/1421863117 >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 msgr2=0x7f538813e280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:42:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 1 --2- 172.21.15.46:0/1421863117 >> v2:172.21.15.46:3300/0 conn(0x7f538813be90 0x7f538813e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 10 monclient: _finish_hunting 0 2024-09-16T04:42:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:42:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:42:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 1 -- 172.21.15.46:0/1421863117 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f538809b310 con 0x7f53880a3ff0 2024-09-16T04:42:06.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 10 monclient: _finish_auth 0 2024-09-16T04:42:06.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 10 monclient: _check_auth_tickets 2024-09-16T04:42:06.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:06.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 1 -- 172.21.15.46:0/1421863117 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f5384001670 con 0x7f53880a3ff0 2024-09-16T04:42:06.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:06.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:42:06.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 10 monclient: dump: 2024-09-16T04:42:06.489 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:06.489 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:06.489 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.489 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.489 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:06.489 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:06.490 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:06.490 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:06.490 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:06.490 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:06.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:42:06.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:06.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.483+0000 7f538c863640 1 -- 172.21.15.46:0/1421863117 >> v1:172.21.15.46:6789/0 conn(0x7f53880a3ff0 legacy=0x7f53880a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:06.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538c863640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:42:06.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538c863640 1 --2- 172.21.15.46:0/1421863117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5374003a00 0x7f5374003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:06.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538c863640 10 monclient(hunting): picked mon.a con 0x7f5374003a00 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:06.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538c863640 1 --2- 172.21.15.46:0/1421863117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5374004800 0x7f5374004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:06.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538c863640 10 monclient(hunting): picked mon.b con 0x7f5374004800 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:06.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538c863640 1 --2- 172.21.15.46:0/1421863117 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53740052e0 0x7f537400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:06.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538c863640 10 monclient(hunting): picked mon.c con 0x7f53740052e0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:06.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538c863640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:06.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538c863640 1 -- 172.21.15.46:0/1421863117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f538809ab00 con 0x7f5374003a00 2024-09-16T04:42:06.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538c863640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:06.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538c863640 1 -- 172.21.15.46:0/1421863117 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f538809b030 con 0x7f5374004800 2024-09-16T04:42:06.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538c863640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:06.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538c863640 1 -- 172.21.15.46:0/1421863117 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f537400c230 con 0x7f53740052e0 2024-09-16T04:42:06.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538c863640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:06.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538c863640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:42:06.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538d865640 1 --2- 172.21.15.46:0/1421863117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5374003a00 0x7f5374003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:06.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538e066640 1 --2- 172.21.15.46:0/1421863117 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53740052e0 0x7f537400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:06.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538d865640 10 monclient(hunting): get_auth_request con 0x7f5374003a00 auth_method 0 2024-09-16T04:42:06.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538d865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:06.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538d865640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:06.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538d865640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:06.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538d064640 1 --2- 172.21.15.46:0/1421863117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5374004800 0x7f5374004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:06.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538e066640 10 monclient(hunting): get_auth_request con 0x7f53740052e0 auth_method 0 2024-09-16T04:42:06.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538e066640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:06.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538e066640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:06.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538e066640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:06.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538d064640 10 monclient(hunting): get_auth_request con 0x7f5374004800 auth_method 0 2024-09-16T04:42:06.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538d064640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:06.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538d064640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:06.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538d064640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:06.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538e066640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:06.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538e066640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:06.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538e066640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:42:06.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538d865640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:06.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538d865640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:06.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538d865640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:42:06.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538d064640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:06.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538d064640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:06.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538d064640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:42:06.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538e066640 10 monclient(hunting): handle_auth_done global_id 44097 payload 293 2024-09-16T04:42:06.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538e066640 1 -- 172.21.15.46:0/1421863117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5374004800 msgr2=0x7f5374004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:06.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538e066640 1 --2- 172.21.15.46:0/1421863117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5374004800 0x7f5374004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:06.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538e066640 1 -- 172.21.15.46:0/1421863117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5374003a00 msgr2=0x7f5374003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:06.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538e066640 1 --2- 172.21.15.46:0/1421863117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5374003a00 0x7f5374003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:06.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538e066640 10 monclient: _finish_hunting 0 2024-09-16T04:42:06.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538e066640 1 monclient: found mon.c 2024-09-16T04:42:06.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538e066640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:06.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538e066640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:06.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538e066640 1 -- 172.21.15.46:0/1421863117 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5374003520 con 0x7f53740052e0 2024-09-16T04:42:06.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538e066640 10 monclient: _finish_auth 0 2024-09-16T04:42:06.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538e066640 10 monclient: _check_auth_tickets 2024-09-16T04:42:06.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538e066640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:06.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538fac9640 5 monclient: authenticate success, global_id 44097 2024-09-16T04:42:06.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538fac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:06.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.484+0000 7f538d865640 1 --2- 172.21.15.46:0/1421863117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5374003a00 0x7f5374003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:42:06.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.485+0000 7f538e066640 1 -- 172.21.15.46:0/1421863117 reap_dead start 2024-09-16T04:42:06.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.485+0000 7f538e066640 1 --2- 172.21.15.46:0/1421863117 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53740052e0 0x7f537400b9c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5380004360 tx=0x7f5374005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc088cab7a77de43 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:06.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.485+0000 7f538c863640 1 -- 172.21.15.46:0/1421863117 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c011790 con 0x7f53740052e0 2024-09-16T04:42:06.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.485+0000 7f538c863640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:06.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.485+0000 7f538c863640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:06.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.485+0000 7f538c863640 10 monclient: dump: 2024-09-16T04:42:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:06.496 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.485+0000 7f538c863640 1 -- 172.21.15.46:0/1421863117 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f538002ba80 con 0x7f53740052e0 2024-09-16T04:42:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.485+0000 7f538c863640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.485+0000 7f538c863640 1 -- 172.21.15.46:0/1421863117 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5380031070 con 0x7f53740052e0 2024-09-16T04:42:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.485+0000 7f538fac9640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.485+0000 7f538fac9640 4 set_mon_vals no callback set 2024-09-16T04:42:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.485+0000 7f538c863640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.485+0000 7f538c863640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.485+0000 7f538c863640 10 monclient: dump: 2024-09-16T04:42:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:06.498 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:06.498 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:06.498 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:06.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.487+0000 7f538fac9640 10 monclient: shutdown 2024-09-16T04:42:06.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.487+0000 7f538fac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:06.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.487+0000 7f538fac9640 1 -- 172.21.15.46:0/1421863117 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53740052e0 msgr2=0x7f537400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:06.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.487+0000 7f538fac9640 1 --2- 172.21.15.46:0/1421863117 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53740052e0 0x7f537400b9c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5380004360 tx=0x7f5374005830 comp rx=0 tx=0).stop 2024-09-16T04:42:06.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.487+0000 7f538fac9640 1 -- 172.21.15.46:0/1421863117 shutdown_connections 2024-09-16T04:42:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.487+0000 7f538fac9640 1 --2- 172.21.15.46:0/1421863117 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f53740052e0 0x7f537400b9c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.487+0000 7f538fac9640 1 -- 172.21.15.46:0/1421863117 >> 172.21.15.46:0/1421863117 conn(0x7f53880a6340 msgr2=0x7f538806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.487+0000 7f538fac9640 1 -- 172.21.15.46:0/1421863117 shutdown_connections 2024-09-16T04:42:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.487+0000 7f538fac9640 1 -- 172.21.15.46:0/1421863117 wait complete. 2024-09-16T04:42:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.488+0000 7f538fac9640 10 monclient: build_initial_monmap 2024-09-16T04:42:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.488+0000 7f538fac9640 10 monclient: monmap: 2024-09-16T04:42:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:06.505 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.488+0000 7f538fac9640 1 Processor -- start 2024-09-16T04:42:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538fac9640 1 -- start start 2024-09-16T04:42:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538fac9640 10 monclient: init 2024-09-16T04:42:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538fac9640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538fac9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:42:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538fac9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53740052e0 0x7f53881333a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538fac9640 10 monclient(hunting): picked mon.noname-b con 0x7f53740052e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538fac9640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f538809d110 0x7f538809d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538fac9640 10 monclient(hunting): picked mon.noname-c con 0x7f538809d110 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538fac9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f538809da90 0x7f53881d2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538fac9640 10 monclient(hunting): picked mon.noname-a con 0x7f538809da90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538fac9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538fac9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f538809be70 con 0x7f538809da90 2024-09-16T04:42:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538fac9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538fac9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f538809bcf0 con 0x7f53740052e0 2024-09-16T04:42:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538fac9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538fac9640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f538809bff0 con 0x7f538809d110 2024-09-16T04:42:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538fac9640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538fac9640 10 monclient(hunting): authenticate will time out at 924.904649s 2024-09-16T04:42:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538d064640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f538809d110 0x7f538809d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538d064640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f538809d110 0x7f538809d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39406/0 (socket says 172.21.15.46:39406) 2024-09-16T04:42:06.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538d064640 1 -- 172.21.15.46:0/749003581 learned_addr learned my addr 172.21.15.46:0/749003581 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:06.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538d064640 10 monclient(hunting): get_auth_request con 0x7f538809d110 auth_method 0 2024-09-16T04:42:06.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538d064640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:06.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538d064640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:06.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538d064640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:06.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538e066640 1 --2- 172.21.15.46:0/749003581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f538809da90 0x7f53881d2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:06.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538d064640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:06.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538d064640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:06.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538d865640 1 --2- 172.21.15.46:0/749003581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53740052e0 0x7f53881333a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:06.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538d064640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:06.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538d865640 10 monclient(hunting): get_auth_request con 0x7f53740052e0 auth_method 0 2024-09-16T04:42:06.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538d865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:06.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538d865640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:06.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538d865640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:06.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538e066640 10 monclient(hunting): get_auth_request con 0x7f538809da90 auth_method 0 2024-09-16T04:42:06.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538e066640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:06.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538e066640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:06.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.489+0000 7f538e066640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:06.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538d865640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:06.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538d865640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:06.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538d865640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:06.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538e066640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:06.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538e066640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:06.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538e066640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:06.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538d064640 10 monclient(hunting): handle_auth_done global_id 24545 payload 824 2024-09-16T04:42:06.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538d064640 1 -- 172.21.15.46:0/749003581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53740052e0 msgr2=0x7f53881333a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:06.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538d064640 1 --2- 172.21.15.46:0/749003581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53740052e0 0x7f53881333a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:06.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538d064640 1 -- 172.21.15.46:0/749003581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f538809da90 msgr2=0x7f53881d2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:06.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538d064640 1 --2- 172.21.15.46:0/749003581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f538809da90 0x7f53881d2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:06.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538d064640 10 monclient: _finish_hunting 0 2024-09-16T04:42:06.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538d064640 1 monclient: found mon.noname-c 2024-09-16T04:42:06.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538d064640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:06.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538d064640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:42:06.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538d064640 1 -- 172.21.15.46:0/749003581 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f538809fca0 con 0x7f538809d110 2024-09-16T04:42:06.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538d064640 1 --2- 172.21.15.46:0/749003581 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f538809d110 0x7f538809d550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5384004fa0 tx=0x7f5384007430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:06.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f537a7fc640 1 -- 172.21.15.46:0/749003581 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538400f720 con 0x7f538809d110 2024-09-16T04:42:06.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f537a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:06.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f537a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:42:06.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f537a7fc640 10 monclient: dump: 2024-09-16T04:42:06.509 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:06.510 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:06.510 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.510 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.510 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:06.510 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:06.510 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:06.510 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:06.510 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:06.510 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:06.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f537a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:42:06.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f537a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:42:06.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f537a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:06.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f537a7fc640 1 -- 172.21.15.46:0/749003581 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5384018070 con 0x7f538809d110 2024-09-16T04:42:06.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538fac9640 5 monclient: authenticate success, global_id 24545 2024-09-16T04:42:06.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538fac9640 10 monclient: _renew_subs 2024-09-16T04:42:06.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538fac9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:06.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538fac9640 1 -- 172.21.15.46:0/749003581 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f538811fab0 con 0x7f538809d110 2024-09-16T04:42:06.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f537a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:06.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f537a7fc640 1 -- 172.21.15.46:0/749003581 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538400fb00 con 0x7f538809d110 2024-09-16T04:42:06.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f537a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:06.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f537a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:06.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f537a7fc640 10 monclient: dump: 2024-09-16T04:42:06.511 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:06.511 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:06.511 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.511 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.512 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:06.512 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:06.512 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:06.512 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:06.512 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:06.512 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:06.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538fac9640 10 monclient: _renew_subs 2024-09-16T04:42:06.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f537bfff640 4 set_mon_vals no callback set 2024-09-16T04:42:06.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538fac9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:06.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.490+0000 7f538fac9640 1 -- 172.21.15.46:0/749003581 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f538809df10 con 0x7f538809d110 2024-09-16T04:42:06.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.491+0000 7f538fac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:06.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.491+0000 7f538fac9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:06.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.491+0000 7f538fac9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.491+0000 7f538fac9640 1 -- 172.21.15.46:0/749003581 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53880a43d0 con 0x7f538809d110 2024-09-16T04:42:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.491+0000 7f537a7fc640 1 -- 172.21.15.46:0/749003581 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f538400fca0 con 0x7f538809d110 2024-09-16T04:42:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.492+0000 7f537a7fc640 1 --2- 172.21.15.46:0/749003581 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f536c038400 0x7f536c03a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.492+0000 7f537a7fc640 1 -- 172.21.15.46:0/749003581 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(134..134 src has 1..134) ==== 4346+0+0 (secure 0 0 0) 0x7f538404b440 con 0x7f538809d110 2024-09-16T04:42:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.492+0000 7f538d865640 1 --2- 172.21.15.46:0/749003581 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f536c038400 0x7f536c03a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.492+0000 7f538d865640 10 monclient: get_auth_request con 0x7f536c038400 auth_method 0 2024-09-16T04:42:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.493+0000 7f538d865640 1 --2- 172.21.15.46:0/749003581 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f536c038400 0x7f536c03a8c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f537c006fd0 tx=0x7f537c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.497+0000 7f537a7fc640 1 -- 172.21.15.46:0/749003581 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f538401b360 con 0x7f538809d110 2024-09-16T04:42:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.497+0000 7f537a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.497+0000 7f537a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.746+0000 7f538fac9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:42:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.747+0000 7f538fac9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.747+0000 7f538fac9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.747+0000 7f538fac9640 1 -- 172.21.15.46:0/749003581 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f538809e250 con 0x7f538809d110 2024-09-16T04:42:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.747+0000 7f537a7fc640 1 -- 172.21.15.46:0/749003581 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f538401b540 con 0x7f538809d110 2024-09-16T04:42:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.747+0000 7f537a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.747+0000 7f537a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:06.749 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:42:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.749+0000 7f538fac9640 1 -- 172.21.15.46:0/749003581 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f536c038400 msgr2=0x7f536c03a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.749+0000 7f538fac9640 1 --2- 172.21.15.46:0/749003581 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f536c038400 0x7f536c03a8c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f537c006fd0 tx=0x7f537c008040 comp rx=0 tx=0).stop 2024-09-16T04:42:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.749+0000 7f538fac9640 10 monclient: shutdown 2024-09-16T04:42:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.749+0000 7f538fac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.749+0000 7f538fac9640 1 -- 172.21.15.46:0/749003581 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f538809d110 msgr2=0x7f538809d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.749+0000 7f538fac9640 1 --2- 172.21.15.46:0/749003581 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f538809d110 0x7f538809d550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5384004fa0 tx=0x7f5384007430 comp rx=0 tx=0).stop 2024-09-16T04:42:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.749+0000 7f538fac9640 1 -- 172.21.15.46:0/749003581 shutdown_connections 2024-09-16T04:42:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.749+0000 7f538fac9640 1 --2- 172.21.15.46:0/749003581 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f536c038400 0x7f536c03a8c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.749+0000 7f538fac9640 1 --2- 172.21.15.46:0/749003581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f538809da90 0x7f53881d2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.749+0000 7f538fac9640 1 --2- 172.21.15.46:0/749003581 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f538809d110 0x7f538809d550 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.749+0000 7f538fac9640 1 --2- 172.21.15.46:0/749003581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53740052e0 0x7f53881333a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.749+0000 7f538fac9640 1 -- 172.21.15.46:0/749003581 >> 172.21.15.46:0/749003581 conn(0x7f53880a6340 msgr2=0x7f538813b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.749+0000 7f538fac9640 1 -- 172.21.15.46:0/749003581 shutdown_connections 2024-09-16T04:42:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.749+0000 7f538fac9640 1 -- 172.21.15.46:0/749003581 wait complete. 2024-09-16T04:42:06.761 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:42:06.761 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:42:06.761 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:42:06.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.862+0000 7fe98a4d9640 10 monclient: get_monmap_and_config 2024-09-16T04:42:06.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.862+0000 7fe98a4d9640 10 monclient: build_initial_monmap 2024-09-16T04:42:06.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.862+0000 7fe98a4d9640 10 monclient: monmap: 2024-09-16T04:42:06.863 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:06.864 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:06.864 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:06.862622+0000 2024-09-16T04:42:06.864 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:06.862622+0000 2024-09-16T04:42:06.864 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:06.864 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:06.864 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:06.864 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:06.864 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:06.864 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:06.864 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:06.864 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:06.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.862+0000 7fe98a4d9640 1 Processor -- start 2024-09-16T04:42:06.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe98a4d9640 1 -- start start 2024-09-16T04:42:06.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe98a4d9640 10 monclient: init 2024-09-16T04:42:06.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe98a4d9640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:06.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe98a4d9640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-16T04:42:06.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe98a4d9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9840a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:42:06.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe98a4d9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe9840a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:42:06.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe98a4d9640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe98413cea0 0x7fe98413f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:06.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe98a4d9640 10 monclient(hunting): picked mon.noname-c con 0x7fe98413cea0 addr v2:172.21.15.114:3300/0 2024-09-16T04:42:06.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe98a4d9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe98409ab70 con 0x7fe9840a3ff0 2024-09-16T04:42:06.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe98a4d9640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9840a4d60 con 0x7fe9840a3c10 2024-09-16T04:42:06.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe98a4d9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:06.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe98a4d9640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fe98409b030 con 0x7fe98413cea0 2024-09-16T04:42:06.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe98a4d9640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:06.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe98a4d9640 10 monclient(hunting): authenticate will time out at 925.278563s 2024-09-16T04:42:06.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe97bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe9840a3ff0 0x7fe9840a4400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51646/0 (socket says 172.21.15.46:51646) 2024-09-16T04:42:06.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe97bfff640 1 -- 172.21.15.46:0/3191407280 learned_addr learned my addr 172.21.15.46:0/3191407280 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:42:06.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe988a76640 1 --2- 172.21.15.46:0/3191407280 >> v2:172.21.15.114:3300/0 conn(0x7fe98413cea0 0x7fe98413f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:06.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe9837fe640 1 -- 172.21.15.46:0/3191407280 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 27602695 0 0) 0x7fe98409ab70 con 0x7fe9840a3ff0 2024-09-16T04:42:06.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe9837fe640 10 monclient(hunting): _negotiate 2024-09-16T04:42:06.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe9837fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:06.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe9837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:06.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe9837fe640 10 monclient(hunting): my global_id is 44103 2024-09-16T04:42:06.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe9837fe640 1 -- 172.21.15.46:0/3191407280 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe970003520 con 0x7fe9840a3ff0 2024-09-16T04:42:06.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe988a76640 10 monclient(hunting): get_auth_request con 0x7fe98413cea0 auth_method 0 2024-09-16T04:42:06.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe988a76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:06.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe988a76640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:06.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.863+0000 7fe988a76640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:06.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe988a76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:06.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe988a76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:06.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe988a76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:06.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 1 -- 172.21.15.46:0/3191407280 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2798861647 0 0) 0x7fe970003520 con 0x7fe9840a3ff0 2024-09-16T04:42:06.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 1 -- 172.21.15.46:0/3191407280 >> v2:172.21.15.114:3300/0 conn(0x7fe98413cea0 msgr2=0x7fe98413f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:06.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 1 --2- 172.21.15.46:0/3191407280 >> v2:172.21.15.114:3300/0 conn(0x7fe98413cea0 0x7fe98413f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:06.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 1 -- 172.21.15.46:0/3191407280 >> v1:172.21.15.114:6789/0 conn(0x7fe9840a3c10 legacy=0x7fe98409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:06.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 10 monclient: _finish_hunting 0 2024-09-16T04:42:06.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:42:06.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:06.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:42:06.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 1 -- 172.21.15.46:0/3191407280 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe98409b310 con 0x7fe9840a3ff0 2024-09-16T04:42:06.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 10 monclient: _finish_auth 0 2024-09-16T04:42:06.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 10 monclient: _check_auth_tickets 2024-09-16T04:42:06.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:06.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 1 -- 172.21.15.46:0/3191407280 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7fe96c001670 con 0x7fe9840a3ff0 2024-09-16T04:42:06.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe98a4d9640 5 monclient: authenticate success, global_id 44103 2024-09-16T04:42:06.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe98a4d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:06.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:06.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:42:06.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 10 monclient: dump: 2024-09-16T04:42:06.879 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:06.879 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:06.879 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.879 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.880 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:06.880 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:06.880 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:06.880 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:06.880 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:06.880 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:06.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:42:06.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:06.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 1 -- 172.21.15.46:0/3191407280 >> v1:172.21.15.46:6789/0 conn(0x7fe9840a3ff0 legacy=0x7fe9840a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:06.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:42:06.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 1 --2- 172.21.15.46:0/3191407280 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9700039b0 0x7fe970003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:06.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 10 monclient(hunting): picked mon.c con 0x7fe9700039b0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:06.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 1 --2- 172.21.15.46:0/3191407280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe970004800 0x7fe970004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:06.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 10 monclient(hunting): picked mon.b con 0x7fe970004800 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:06.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 1 --2- 172.21.15.46:0/3191407280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9700052e0 0x7fe97000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:06.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 10 monclient(hunting): picked mon.a con 0x7fe9700052e0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:06.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:06.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 1 -- 172.21.15.46:0/3191407280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe98409b030 con 0x7fe9700052e0 2024-09-16T04:42:06.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:06.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 1 -- 172.21.15.46:0/3191407280 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe97000c1b0 con 0x7fe970004800 2024-09-16T04:42:06.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:06.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 1 -- 172.21.15.46:0/3191407280 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe97000c3b0 con 0x7fe9700039b0 2024-09-16T04:42:06.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:06.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe9837fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:42:06.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe98a4d9640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:06.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe983fff640 1 --2- 172.21.15.46:0/3191407280 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9700039b0 0x7fe970003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:06.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe988a76640 1 --2- 172.21.15.46:0/3191407280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9700052e0 0x7fe97000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:06.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe983fff640 10 monclient(hunting): get_auth_request con 0x7fe9700039b0 auth_method 0 2024-09-16T04:42:06.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe983fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:06.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe983fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:06.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe983fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:06.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe988a76640 10 monclient(hunting): get_auth_request con 0x7fe9700052e0 auth_method 0 2024-09-16T04:42:06.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe988a76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:06.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe988a76640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:06.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe988a76640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:06.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe97bfff640 1 --2- 172.21.15.46:0/3191407280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe970004800 0x7fe970004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:06.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe983fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:06.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe983fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:06.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe983fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:42:06.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe988a76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:06.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe988a76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:06.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe988a76640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:42:06.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe97bfff640 10 monclient(hunting): get_auth_request con 0x7fe970004800 auth_method 0 2024-09-16T04:42:06.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe97bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:06.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe97bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:06.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe97bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:06.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe97bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:06.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe97bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:06.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe97bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:42:06.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe983fff640 10 monclient(hunting): handle_auth_done global_id 44103 payload 293 2024-09-16T04:42:06.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe983fff640 1 -- 172.21.15.46:0/3191407280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe970004800 msgr2=0x7fe970004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:06.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe983fff640 1 --2- 172.21.15.46:0/3191407280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe970004800 0x7fe970004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:06.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe983fff640 1 -- 172.21.15.46:0/3191407280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9700052e0 msgr2=0x7fe97000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:06.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe983fff640 1 --2- 172.21.15.46:0/3191407280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9700052e0 0x7fe97000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:06.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe983fff640 10 monclient: _finish_hunting 0 2024-09-16T04:42:06.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe983fff640 1 monclient: found mon.c 2024-09-16T04:42:06.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe983fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:06.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe983fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:06.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe983fff640 1 -- 172.21.15.46:0/3191407280 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe970003520 con 0x7fe9700039b0 2024-09-16T04:42:06.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe983fff640 10 monclient: _finish_auth 0 2024-09-16T04:42:06.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe983fff640 10 monclient: _check_auth_tickets 2024-09-16T04:42:06.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe983fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:06.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe988a76640 1 --2- 172.21.15.46:0/3191407280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9700052e0 0x7fe97000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:42:06.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.864+0000 7fe988a76640 1 -- 172.21.15.46:0/3191407280 reap_dead start 2024-09-16T04:42:06.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.865+0000 7fe983fff640 1 --2- 172.21.15.46:0/3191407280 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9700039b0 0x7fe970003dd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe974001720 tx=0x7fe970005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b0ecac0640b11b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:06.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.865+0000 7fe9837fe640 1 -- 172.21.15.46:0/3191407280 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe974001370 con 0x7fe9700039b0 2024-09-16T04:42:06.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.865+0000 7fe9837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:06.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.865+0000 7fe9837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:06.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.865+0000 7fe9837fe640 10 monclient: dump: 2024-09-16T04:42:06.885 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:06.885 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:06.885 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.885 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.885 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:06.885 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:06.885 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:06.885 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:06.886 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:06.886 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:06.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.865+0000 7fe9837fe640 1 -- 172.21.15.46:0/3191407280 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9740052d0 con 0x7fe9700039b0 2024-09-16T04:42:06.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.865+0000 7fe98a4d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:06.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.865+0000 7fe9837fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:06.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.865+0000 7fe98a4d9640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:06.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.865+0000 7fe9837fe640 1 -- 172.21.15.46:0/3191407280 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97402ab90 con 0x7fe9700039b0 2024-09-16T04:42:06.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.865+0000 7fe9837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:06.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.865+0000 7fe98a4d9640 4 set_mon_vals no callback set 2024-09-16T04:42:06.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.865+0000 7fe9837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:06.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.865+0000 7fe9837fe640 10 monclient: dump: 2024-09-16T04:42:06.888 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:06.888 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:06.888 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.888 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.888 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:06.889 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:06.889 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:06.889 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:06.889 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:06.890 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.866+0000 7fe98a4d9640 10 monclient: shutdown 2024-09-16T04:42:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.866+0000 7fe98a4d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.866+0000 7fe98a4d9640 1 -- 172.21.15.46:0/3191407280 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9700039b0 msgr2=0x7fe970003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.866+0000 7fe98a4d9640 1 --2- 172.21.15.46:0/3191407280 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9700039b0 0x7fe970003dd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe974001720 tx=0x7fe970005830 comp rx=0 tx=0).stop 2024-09-16T04:42:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.866+0000 7fe98a4d9640 1 -- 172.21.15.46:0/3191407280 shutdown_connections 2024-09-16T04:42:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.866+0000 7fe98a4d9640 1 --2- 172.21.15.46:0/3191407280 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9700039b0 0x7fe970003dd0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.866+0000 7fe98a4d9640 1 -- 172.21.15.46:0/3191407280 >> 172.21.15.46:0/3191407280 conn(0x7fe9840a5d10 msgr2=0x7fe9840a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.866+0000 7fe98a4d9640 1 -- 172.21.15.46:0/3191407280 shutdown_connections 2024-09-16T04:42:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.866+0000 7fe98a4d9640 1 -- 172.21.15.46:0/3191407280 wait complete. 2024-09-16T04:42:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.866+0000 7fe98a4d9640 10 monclient: build_initial_monmap 2024-09-16T04:42:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.866+0000 7fe98a4d9640 10 monclient: monmap: 2024-09-16T04:42:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:06.891 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:06.891 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:06.891 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:06.891 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:06.891 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:06.891 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:06.891 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:06.891 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:06.891 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:06.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe98a4d9640 1 Processor -- start 2024-09-16T04:42:06.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe98a4d9640 1 -- start start 2024-09-16T04:42:06.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe98a4d9640 10 monclient: init 2024-09-16T04:42:06.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe98a4d9640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:06.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe98a4d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:42:06.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe98a4d9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9700039b0 0x7fe98411f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:06.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe98a4d9640 10 monclient(hunting): picked mon.noname-b con 0x7fe9700039b0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:06.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe98a4d9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98411fda0 0x7fe984120200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:06.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe98a4d9640 10 monclient(hunting): picked mon.noname-a con 0x7fe98411fda0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:06.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe98a4d9640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9841cc030 0x7fe9841cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:06.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe98a4d9640 10 monclient(hunting): picked mon.noname-c con 0x7fe9841cc030 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:06.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe98a4d9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:06.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe98a4d9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe98409be70 con 0x7fe98411fda0 2024-09-16T04:42:06.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe98a4d9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:06.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe98a4d9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe98409bcf0 con 0x7fe9700039b0 2024-09-16T04:42:06.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe98a4d9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:06.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe98a4d9640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe98409bff0 con 0x7fe9841cc030 2024-09-16T04:42:06.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe98a4d9640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:06.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe98a4d9640 10 monclient(hunting): authenticate will time out at 925.282764s 2024-09-16T04:42:06.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe988a76640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9841cc030 0x7fe9841cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:06.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe97bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98411fda0 0x7fe984120200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:06.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe97bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98411fda0 0x7fe984120200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49486/0 (socket says 172.21.15.46:49486) 2024-09-16T04:42:06.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe988a76640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9841cc030 0x7fe9841cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39422/0 (socket says 172.21.15.46:39422) 2024-09-16T04:42:06.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe97bfff640 1 -- 172.21.15.46:0/3229646614 learned_addr learned my addr 172.21.15.46:0/3229646614 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:06.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe97bfff640 10 monclient(hunting): get_auth_request con 0x7fe98411fda0 auth_method 0 2024-09-16T04:42:06.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe97bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:06.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe97bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:06.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe97bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:06.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe988a76640 10 monclient(hunting): get_auth_request con 0x7fe9841cc030 auth_method 0 2024-09-16T04:42:06.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe988a76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:06.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe988a76640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:06.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe988a76640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:06.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe97bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:06.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe97bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:06.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe97bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:06.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe988a76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:06.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.867+0000 7fe988a76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:06.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe988a76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:06.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe983fff640 1 --2- 172.21.15.46:0/3229646614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9700039b0 0x7fe98411f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:06.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe97bfff640 10 monclient(hunting): handle_auth_done global_id 44106 payload 824 2024-09-16T04:42:06.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe97bfff640 1 -- 172.21.15.46:0/3229646614 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9841cc030 msgr2=0x7fe9841cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:06.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe97bfff640 1 --2- 172.21.15.46:0/3229646614 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9841cc030 0x7fe9841cc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:06.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe97bfff640 1 -- 172.21.15.46:0/3229646614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9700039b0 msgr2=0x7fe98411f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:06.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe97bfff640 1 --2- 172.21.15.46:0/3229646614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9700039b0 0x7fe98411f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:06.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe97bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:42:06.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe97bfff640 1 monclient: found mon.noname-a 2024-09-16T04:42:06.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe97bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:06.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe97bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:06.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe97bfff640 1 -- 172.21.15.46:0/3229646614 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe984122b40 con 0x7fe98411fda0 2024-09-16T04:42:06.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe988a76640 1 --2- 172.21.15.46:0/3229646614 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9841cc030 0x7fe9841cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:42:06.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe983fff640 10 monclient: get_auth_request con 0x7fe9700039b0 auth_method 0 2024-09-16T04:42:06.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe983fff640 1 --2- 172.21.15.46:0/3229646614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9700039b0 0x7fe98411f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:42:06.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe97bfff640 1 --2- 172.21.15.46:0/3229646614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98411fda0 0x7fe984120200 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe96c0072f0 tx=0x7fe96c0077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:06.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe9817fa640 1 -- 172.21.15.46:0/3229646614 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c01a070 con 0x7fe98411fda0 2024-09-16T04:42:06.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe9817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:06.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe9817fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:06.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe9817fa640 10 monclient: dump: 2024-09-16T04:42:06.896 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:06.896 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:06.896 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:06.897 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe9817fa640 10 monclient: _finish_auth 0 2024-09-16T04:42:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe9817fa640 10 monclient: _check_auth_tickets 2024-09-16T04:42:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe9817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe9817fa640 1 -- 172.21.15.46:0/3229646614 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe96c001660 con 0x7fe98411fda0 2024-09-16T04:42:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe98a4d9640 5 monclient: authenticate success, global_id 44106 2024-09-16T04:42:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe9817fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe98a4d9640 10 monclient: _renew_subs 2024-09-16T04:42:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe9817fa640 1 -- 172.21.15.46:0/3229646614 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c00fb60 con 0x7fe98411fda0 2024-09-16T04:42:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe98a4d9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe98a4d9640 1 -- 172.21.15.46:0/3229646614 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe984122e20 con 0x7fe98411fda0 2024-09-16T04:42:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe9817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe9817fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe982ffd640 4 set_mon_vals no callback set 2024-09-16T04:42:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe9817fa640 10 monclient: dump: 2024-09-16T04:42:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:06.899 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe98a4d9640 10 monclient: _renew_subs 2024-09-16T04:42:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe98a4d9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.868+0000 7fe98a4d9640 1 -- 172.21.15.46:0/3229646614 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9840a0400 con 0x7fe98411fda0 2024-09-16T04:42:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.869+0000 7fe9817fa640 1 -- 172.21.15.46:0/3229646614 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fe96c01d070 con 0x7fe98411fda0 2024-09-16T04:42:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.869+0000 7fe98a4d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.869+0000 7fe98a4d9640 1 --2- 172.21.15.46:0/3229646614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98402d7c0 0x7fe9841cca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.869+0000 7fe98a4d9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.869+0000 7fe98a4d9640 1 -- 172.21.15.46:0/3229646614 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe98402dba0 con 0x7fe98402d7c0 2024-09-16T04:42:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.869+0000 7fe983fff640 1 --2- 172.21.15.46:0/3229646614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98402d7c0 0x7fe9841cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.869+0000 7fe9817fa640 1 --2- 172.21.15.46:0/3229646614 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe9640383b0 0x7fe96403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.869+0000 7fe983fff640 10 monclient: get_auth_request con 0x7fe98402d7c0 auth_method 0 2024-09-16T04:42:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.869+0000 7fe983fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.869+0000 7fe983fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.869+0000 7fe983fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.869+0000 7fe9817fa640 1 -- 172.21.15.46:0/3229646614 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 4346+0+0 (secure 0 0 0) 0x7fe96c04b0b0 con 0x7fe98411fda0 2024-09-16T04:42:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.869+0000 7fe983fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.869+0000 7fe983fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.869+0000 7fe983fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.869+0000 7fe983fff640 10 monclient(hunting): handle_auth_done global_id 44109 payload 824 2024-09-16T04:42:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.869+0000 7fe983fff640 1 -- 172.21.15.46:0/3229646614 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe984123160 con 0x7fe98402d7c0 2024-09-16T04:42:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.869+0000 7fe988a76640 1 --2- 172.21.15.46:0/3229646614 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe9640383b0 0x7fe96403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.869+0000 7fe988a76640 10 monclient: get_auth_request con 0x7fe9640383b0 auth_method 0 2024-09-16T04:42:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.869+0000 7fe983fff640 1 --2- 172.21.15.46:0/3229646614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98402d7c0 0x7fe9841cca20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe9700057e0 tx=0x7fe974001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.870+0000 7fe9817fa640 1 -- 172.21.15.46:0/3229646614 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe974040020 con 0x7fe98402d7c0 2024-09-16T04:42:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.870+0000 7fe9817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.870+0000 7fe9817fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.870+0000 7fe9817fa640 10 monclient: dump: 2024-09-16T04:42:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:06.902 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.870+0000 7fe988a76640 1 --2- 172.21.15.46:0/3229646614 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe9640383b0 0x7fe96403a870 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe98406ef00 tx=0x7fe97c00b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.870+0000 7fe9817fa640 1 -- 172.21.15.46:0/3229646614 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe974007020 con 0x7fe98402d7c0 2024-09-16T04:42:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.870+0000 7fe9817fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.870+0000 7fe9817fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.870+0000 7fe9817fa640 1 -- 172.21.15.46:0/3229646614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98402d7c0 msgr2=0x7fe9841cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.870+0000 7fe9817fa640 1 --2- 172.21.15.46:0/3229646614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98402d7c0 0x7fe9841cca20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe9700057e0 tx=0x7fe974001120 comp rx=0 tx=0).stop 2024-09-16T04:42:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.870+0000 7fe9817fa640 1 -- 172.21.15.46:0/3229646614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98402d7c0 msgr2=0x7fe9841cca20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.870+0000 7fe9817fa640 1 --2- 172.21.15.46:0/3229646614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98402d7c0 0x7fe9841cca20 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe9700057e0 tx=0x7fe974001120 comp rx=0 tx=0).stop 2024-09-16T04:42:06.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.876+0000 7fe98a4d9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:42:06.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.876+0000 7fe98a4d9640 1 --2- 172.21.15.46:0/3229646614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe984062df0 0x7fe984063200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:06.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.876+0000 7fe98a4d9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:06.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.876+0000 7fe98a4d9640 1 -- 172.21.15.46:0/3229646614 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe9840a4400 con 0x7fe984062df0 2024-09-16T04:42:06.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.876+0000 7fe983fff640 1 --2- 172.21.15.46:0/3229646614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe984062df0 0x7fe984063200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:06.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.876+0000 7fe983fff640 10 monclient: get_auth_request con 0x7fe984062df0 auth_method 0 2024-09-16T04:42:06.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.876+0000 7fe983fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:06.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.876+0000 7fe983fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.876+0000 7fe983fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.876+0000 7fe983fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.876+0000 7fe983fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.876+0000 7fe983fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.876+0000 7fe983fff640 10 monclient(hunting): handle_auth_done global_id 44112 payload 824 2024-09-16T04:42:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.876+0000 7fe983fff640 1 -- 172.21.15.46:0/3229646614 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe9840a1300 con 0x7fe984062df0 2024-09-16T04:42:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe983fff640 1 --2- 172.21.15.46:0/3229646614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe984062df0 0x7fe984063200 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe974000c00 tx=0x7fe974009030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe9817fa640 1 -- 172.21.15.46:0/3229646614 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe974040070 con 0x7fe984062df0 2024-09-16T04:42:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe9817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe9817fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe9817fa640 10 monclient: dump: 2024-09-16T04:42:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:06.907 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe9817fa640 1 -- 172.21.15.46:0/3229646614 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7fe9740035f0 con 0x7fe984062df0 2024-09-16T04:42:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe9817fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:42:06.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe9817fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:06.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe9817fa640 1 -- 172.21.15.46:0/3229646614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe984062df0 msgr2=0x7fe984063200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:06.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe9817fa640 1 --2- 172.21.15.46:0/3229646614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe984062df0 0x7fe984063200 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe974000c00 tx=0x7fe974009030 comp rx=0 tx=0).stop 2024-09-16T04:42:06.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe9817fa640 1 -- 172.21.15.46:0/3229646614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe984062df0 msgr2=0x7fe984063200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:06.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe9817fa640 1 --2- 172.21.15.46:0/3229646614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe984062df0 0x7fe984063200 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:06.908 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:42:06.908 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:42:06.908 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:42:06.908 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:42:06.908 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 38, 2024-09-16T04:42:06.909 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 6624, 2024-09-16T04:42:06.909 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:42:06.909 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:42:06.909 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:42:06.909 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:42:06.909 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:06.909 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:42:06.909 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:06.909 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:42:06.909 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:42:06.909 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:06.909 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:06.909 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:06.910 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:06.910 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:06.910 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:06.910 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:06.910 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:06.910 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:06.910 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:06.910 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:06.910 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:06.910 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:42:06.910 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:42:06.910 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:06.910 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:06.911 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:06.911 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:06.911 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:06.911 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:06.911 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:06.911 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:06.911 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:06.911 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:06.911 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:06.911 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:06.911 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:06.911 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:42:06.911 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:42:06.911 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:42:06.912 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:42:06.912 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:42:06.912 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:42:06.912 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:06.912 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:06.912 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:42:06.912 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:42:06.912 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:42:06.912 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:42:06.912 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:42:06.912 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:42:06.912 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:42:06.912 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:06.913 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:42:06.913 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:06.913 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:06.913 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:06.913 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:06.913 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:06.913 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:06.913 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:06.913 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:06.913 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:06.913 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:06.913 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:06.913 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:06.913 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:42:06.914 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:06.914 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:42:06.914 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:06.914 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:42:06.914 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:42:06.914 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:06.914 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:06.914 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:06.914 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:06.914 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:42:06.914 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:06.914 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:06.915 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:06.915 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:06.915 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:42:06.915 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:06.915 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:06.915 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:06.915 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:06.915 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:42:06.915 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:42:06.915 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:06.915 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:06.915 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:06.915 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:06.915 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:06.916 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:42:06.916 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:42:06.916 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:06.916 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:06.916 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:06.916 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:06.916 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:42:06.916 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:06.916 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:06.916 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:06.916 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:06.916 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:42:06.916 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:06.917 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:06.917 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:06.917 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:06.917 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:42:06.917 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:42:06.917 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:06.917 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:06.917 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:06.917 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:06.917 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:06.917 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:42:06.917 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:42:06.917 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:06.918 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:06.918 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:06.918 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:06.918 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:42:06.918 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:06.918 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:06.918 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:06.918 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:06.918 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:42:06.918 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:06.918 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:06.918 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:06.918 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:06.918 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:42:06.919 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:42:06.919 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:06.919 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:06.919 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:06.919 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:06.919 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:06.919 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:06.919 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:42:06.919 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:42:06.919 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:06.919 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:06.919 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:06.919 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:42:06.919 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:06.920 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:06.920 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:42:06.920 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:06.920 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:06.920 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:06.920 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:42:06.920 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:06.920 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:06.920 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:06.920 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:42:06.920 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:42:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe98a4d9640 1 -- 172.21.15.46:0/3229646614 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe9640383b0 msgr2=0x7fe96403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe98a4d9640 1 --2- 172.21.15.46:0/3229646614 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe9640383b0 0x7fe96403a870 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe98406ef00 tx=0x7fe97c00b040 comp rx=0 tx=0).stop 2024-09-16T04:42:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe98a4d9640 10 monclient: shutdown 2024-09-16T04:42:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe98a4d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe98a4d9640 1 -- 172.21.15.46:0/3229646614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98411fda0 msgr2=0x7fe984120200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe98a4d9640 1 --2- 172.21.15.46:0/3229646614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98411fda0 0x7fe984120200 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe96c0072f0 tx=0x7fe96c0077b0 comp rx=0 tx=0).stop 2024-09-16T04:42:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe988a76640 1 -- 172.21.15.46:0/3229646614 reap_dead start 2024-09-16T04:42:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe988a76640 1 -- 172.21.15.46:0/3229646614 reap_dead start 2024-09-16T04:42:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe98a4d9640 1 -- 172.21.15.46:0/3229646614 shutdown_connections 2024-09-16T04:42:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe98a4d9640 1 -- 172.21.15.46:0/3229646614 >> 172.21.15.46:0/3229646614 conn(0x7fe9840a5d10 msgr2=0x7fe98409fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe98a4d9640 1 -- 172.21.15.46:0/3229646614 shutdown_connections 2024-09-16T04:42:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:06.877+0000 7fe98a4d9640 1 -- 172.21.15.46:0/3229646614 wait complete. 2024-09-16T04:42:06.923 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:42:06.923 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:42:06.923 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:42:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.023+0000 7fae183b9640 10 monclient: get_monmap_and_config 2024-09-16T04:42:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.023+0000 7fae183b9640 10 monclient: build_initial_monmap 2024-09-16T04:42:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.023+0000 7fae183b9640 10 monclient: monmap: 2024-09-16T04:42:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:07.024283+0000 2024-09-16T04:42:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:07.024283+0000 2024-09-16T04:42:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:07.026 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:07.026 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:07.026 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:07.026 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:07.026 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.024+0000 7fae183b9640 1 Processor -- start 2024-09-16T04:42:07.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.024+0000 7fae183b9640 1 -- start start 2024-09-16T04:42:07.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.024+0000 7fae183b9640 10 monclient: init 2024-09-16T04:42:07.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae183b9640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:07.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae183b9640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-16T04:42:07.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae183b9640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae100a3c10 0x7fae1009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:07.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae183b9640 10 monclient(hunting): picked mon.noname-b con 0x7fae100a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:07.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae183b9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fae100a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:42:07.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae183b9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fae1013be90 addr v1:172.21.15.46:6789/0 2024-09-16T04:42:07.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae183b9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae1009b940 con 0x7fae1013be90 2024-09-16T04:42:07.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae183b9640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae1009ab00 con 0x7fae100a3ff0 2024-09-16T04:42:07.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae183b9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:07.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae183b9640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fae1009b030 con 0x7fae100a3c10 2024-09-16T04:42:07.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae183b9640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:07.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae183b9640 10 monclient(hunting): authenticate will time out at 925.440126s 2024-09-16T04:42:07.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16155640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae100a3c10 0x7fae1009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:07.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16956640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fae1013be90 0x7fae1013e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51648/0 (socket says 172.21.15.46:51648) 2024-09-16T04:42:07.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16956640 1 -- 172.21.15.46:0/3282947850 learned_addr learned my addr 172.21.15.46:0/3282947850 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:42:07.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16155640 10 monclient(hunting): get_auth_request con 0x7fae100a3c10 auth_method 0 2024-09-16T04:42:07.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16155640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:07.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16155640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:07.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16155640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:07.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16155640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:07.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16155640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:07.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16155640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:07.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16956640 1 -- 172.21.15.46:0/3282947850 >> v1:172.21.15.46:6789/0 conn(0x7fae1013be90 legacy=0x7fae1013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-16T04:42:07.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16956640 1 -- 172.21.15.46:0/3282947850 >> v1:172.21.15.46:6789/0 conn(0x7fae1013be90 legacy=0x7fae1013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:42:07.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16956640 1 --1- 172.21.15.46:0/3282947850 >> v1:172.21.15.46:6789/0 conn(0x7fae1013be90 0x7fae1013e280 :-1 s=OPENED pgs=23 cs=1 l=1).handle_message read tag failed 2024-09-16T04:42:07.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16956640 1 --1- 172.21.15.46:0/3282947850 >> v1:172.21.15.46:6789/0 conn(0x7fae1013be90 0x7fae1013e280 :-1 s=OPENED pgs=23 cs=1 l=1).fault on lossy channel, failing 2024-09-16T04:42:07.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae15153640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.46:6789/0 2024-09-16T04:42:07.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16155640 10 monclient(hunting): handle_auth_done global_id 24551 payload 274 2024-09-16T04:42:07.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16155640 1 -- 172.21.15.46:0/3282947850 >> v1:172.21.15.114:6789/0 conn(0x7fae100a3ff0 legacy=0x7fae100a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:07.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16155640 1 -- 172.21.15.46:0/3282947850 >> v1:172.21.15.46:6789/0 conn(0x7fae1013be90 legacy=0x7fae1013e280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:07.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16155640 10 monclient: _finish_hunting 0 2024-09-16T04:42:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16155640 1 monclient: found mon.noname-b 2024-09-16T04:42:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16155640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16155640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:42:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.025+0000 7fae16155640 1 -- 172.21.15.46:0/3282947850 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae1009b310 con 0x7fae100a3c10 2024-09-16T04:42:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae16155640 1 --2- 172.21.15.46:0/3282947850 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae100a3c10 0x7fae1009bcf0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fae100689b0 tx=0x7fae0802a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df263fd613bcace4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae15153640 1 -- 172.21.15.46:0/3282947850 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae08037070 con 0x7fae100a3c10 2024-09-16T04:42:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae15153640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae15153640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:42:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae15153640 10 monclient: dump: 2024-09-16T04:42:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.039 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:07.039 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:07.039 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:07.039 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae15153640 10 monclient: _finish_auth 0 2024-09-16T04:42:07.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae15153640 10 monclient: _check_auth_tickets 2024-09-16T04:42:07.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae15153640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:07.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae183b9640 5 monclient: authenticate success, global_id 24551 2024-09-16T04:42:07.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae183b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:07.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae15153640 1 -- 172.21.15.46:0/3282947850 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae0802ad10 con 0x7fae100a3c10 2024-09-16T04:42:07.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae15153640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:07.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae183b9640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:07.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae183b9640 4 set_mon_vals no callback set 2024-09-16T04:42:07.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae15153640 1 -- 172.21.15.46:0/3282947850 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae08032660 con 0x7fae100a3c10 2024-09-16T04:42:07.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae15153640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:07.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae15153640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:07.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae15153640 10 monclient: dump: 2024-09-16T04:42:07.040 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:07.040 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:07.040 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.040 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.040 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:07.040 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.040 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:07.040 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:07.040 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:07.040 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae183b9640 10 monclient: shutdown 2024-09-16T04:42:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae183b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae183b9640 1 -- 172.21.15.46:0/3282947850 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae100a3c10 msgr2=0x7fae1009bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.026+0000 7fae183b9640 1 --2- 172.21.15.46:0/3282947850 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae100a3c10 0x7fae1009bcf0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fae100689b0 tx=0x7fae0802a000 comp rx=0 tx=0).stop 2024-09-16T04:42:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.027+0000 7fae183b9640 1 -- 172.21.15.46:0/3282947850 shutdown_connections 2024-09-16T04:42:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.027+0000 7fae183b9640 1 --2- 172.21.15.46:0/3282947850 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae100a3c10 0x7fae1009bcf0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.027+0000 7fae183b9640 1 -- 172.21.15.46:0/3282947850 >> 172.21.15.46:0/3282947850 conn(0x7fae100a6340 msgr2=0x7fae1006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.027+0000 7fae183b9640 1 -- 172.21.15.46:0/3282947850 shutdown_connections 2024-09-16T04:42:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.027+0000 7fae183b9640 1 -- 172.21.15.46:0/3282947850 wait complete. 2024-09-16T04:42:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.027+0000 7fae183b9640 10 monclient: build_initial_monmap 2024-09-16T04:42:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.027+0000 7fae183b9640 10 monclient: monmap: 2024-09-16T04:42:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:07.042 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.027+0000 7fae183b9640 1 Processor -- start 2024-09-16T04:42:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.027+0000 7fae183b9640 1 -- start start 2024-09-16T04:42:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.027+0000 7fae183b9640 10 monclient: init 2024-09-16T04:42:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae183b9640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae183b9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:42:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae183b9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae100a3c10 0x7fae101cba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae183b9640 10 monclient(hunting): picked mon.noname-b con 0x7fae100a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae183b9640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae100a3ff0 0x7fae10136e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae183b9640 10 monclient(hunting): picked mon.noname-c con 0x7fae100a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae183b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae1013be90 0x7fae1009d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae183b9640 10 monclient(hunting): picked mon.noname-a con 0x7fae1013be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae183b9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae183b9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fae100a6b40 con 0x7fae1013be90 2024-09-16T04:42:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae183b9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae183b9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fae100a69c0 con 0x7fae100a3c10 2024-09-16T04:42:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae183b9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae183b9640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fae1013f9a0 con 0x7fae100a3ff0 2024-09-16T04:42:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae183b9640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae183b9640 10 monclient(hunting): authenticate will time out at 925.443562s 2024-09-16T04:42:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae15954640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae100a3ff0 0x7fae10136e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:07.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae15954640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae100a3ff0 0x7fae10136e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39446/0 (socket says 172.21.15.46:39446) 2024-09-16T04:42:07.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae16956640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae1013be90 0x7fae1009d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:07.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae15954640 1 -- 172.21.15.46:0/1319976129 learned_addr learned my addr 172.21.15.46:0/1319976129 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:07.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae15954640 10 monclient(hunting): get_auth_request con 0x7fae100a3ff0 auth_method 0 2024-09-16T04:42:07.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae15954640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:07.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae15954640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:07.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae15954640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:07.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae16155640 1 --2- 172.21.15.46:0/1319976129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae100a3c10 0x7fae101cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:07.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae16956640 10 monclient(hunting): get_auth_request con 0x7fae1013be90 auth_method 0 2024-09-16T04:42:07.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae16956640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:07.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae16956640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:07.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae16956640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:07.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae16155640 10 monclient(hunting): get_auth_request con 0x7fae100a3c10 auth_method 0 2024-09-16T04:42:07.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae16155640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae16155640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae16155640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae15954640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae15954640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae15954640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae16956640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae16956640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae16956640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae16155640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae16155640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae16155640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae15954640 10 monclient(hunting): handle_auth_done global_id 24554 payload 824 2024-09-16T04:42:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae15954640 1 -- 172.21.15.46:0/1319976129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae100a3c10 msgr2=0x7fae101cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae15954640 1 --2- 172.21.15.46:0/1319976129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae100a3c10 0x7fae101cba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae15954640 1 -- 172.21.15.46:0/1319976129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae1013be90 msgr2=0x7fae1009d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae15954640 1 --2- 172.21.15.46:0/1319976129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae1013be90 0x7fae1009d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae15954640 10 monclient: _finish_hunting 0 2024-09-16T04:42:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.028+0000 7fae15954640 1 monclient: found mon.noname-c 2024-09-16T04:42:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae15954640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae15954640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:42:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae15954640 1 -- 172.21.15.46:0/1319976129 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae100a0ac0 con 0x7fae100a3ff0 2024-09-16T04:42:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae16956640 1 --2- 172.21.15.46:0/1319976129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae1013be90 0x7fae1009d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:42:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae15954640 1 --2- 172.21.15.46:0/1319976129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae100a3ff0 0x7fae10136e00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fae0400c430 tx=0x7fae0400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae0effd640 1 -- 172.21.15.46:0/1319976129 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae04011790 con 0x7fae100a3ff0 2024-09-16T04:42:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae0effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:42:07.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae0effd640 10 monclient: dump: 2024-09-16T04:42:07.047 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:07.047 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:07.047 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.047 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.047 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:07.047 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.047 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:07.047 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:07.047 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:07.047 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae0effd640 10 monclient: _finish_auth 0 2024-09-16T04:42:07.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae0effd640 10 monclient: _check_auth_tickets 2024-09-16T04:42:07.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae0effd640 1 -- 172.21.15.46:0/1319976129 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae04017070 con 0x7fae100a3ff0 2024-09-16T04:42:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae183b9640 5 monclient: authenticate success, global_id 24554 2024-09-16T04:42:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae183b9640 10 monclient: _renew_subs 2024-09-16T04:42:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae183b9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae183b9640 1 -- 172.21.15.46:0/1319976129 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae101201c0 con 0x7fae100a3ff0 2024-09-16T04:42:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae0effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae0effd640 1 -- 172.21.15.46:0/1319976129 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae04011b70 con 0x7fae100a3ff0 2024-09-16T04:42:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae0effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae0effd640 10 monclient: dump: 2024-09-16T04:42:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:07.049 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae183b9640 10 monclient: _renew_subs 2024-09-16T04:42:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae14952640 4 set_mon_vals no callback set 2024-09-16T04:42:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae183b9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.029+0000 7fae183b9640 1 -- 172.21.15.46:0/1319976129 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fae1009f9b0 con 0x7fae100a3ff0 2024-09-16T04:42:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae183b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae183b9640 1 --2- 172.21.15.46:0/1319976129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae10137340 0x7fae1009d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae183b9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae183b9640 1 -- 172.21.15.46:0/1319976129 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fae1009eee0 con 0x7fae10137340 2024-09-16T04:42:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae0effd640 1 -- 172.21.15.46:0/1319976129 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fae04011d10 con 0x7fae100a3ff0 2024-09-16T04:42:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae16155640 1 --2- 172.21.15.46:0/1319976129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae10137340 0x7fae1009d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae16155640 10 monclient: get_auth_request con 0x7fae10137340 auth_method 0 2024-09-16T04:42:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae16155640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae16155640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae16155640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae16155640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae16155640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae16155640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae0effd640 1 --2- 172.21.15.46:0/1319976129 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fadfc0383e0 0x7fadfc03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae0effd640 1 -- 172.21.15.46:0/1319976129 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(134..134 src has 1..134) ==== 4346+0+0 (secure 0 0 0) 0x7fae0404b3c0 con 0x7fae100a3ff0 2024-09-16T04:42:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae16155640 10 monclient(hunting): handle_auth_done global_id 24557 payload 824 2024-09-16T04:42:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae16155640 1 -- 172.21.15.46:0/1319976129 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fae1009bd70 con 0x7fae10137340 2024-09-16T04:42:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae16956640 1 --2- 172.21.15.46:0/1319976129 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fadfc0383e0 0x7fadfc03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:07.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae16956640 10 monclient: get_auth_request con 0x7fadfc0383e0 auth_method 0 2024-09-16T04:42:07.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae16155640 1 --2- 172.21.15.46:0/1319976129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae10137340 0x7fae1009d820 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fae10068160 tx=0x7fae080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:07.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae0effd640 1 -- 172.21.15.46:0/1319976129 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae08037040 con 0x7fae10137340 2024-09-16T04:42:07.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:07.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae0effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:07.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.030+0000 7fae0effd640 10 monclient: dump: 2024-09-16T04:42:07.051 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:07.051 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:07.051 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.051 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.051 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:07.051 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.051 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:07.051 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:07.051 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:07.052 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.031+0000 7fae16956640 1 --2- 172.21.15.46:0/1319976129 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fadfc0383e0 0x7fadfc03a8a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fae000012e0 tx=0x7fae00009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.031+0000 7fae0effd640 1 -- 172.21.15.46:0/1319976129 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fae08046640 con 0x7fae10137340 2024-09-16T04:42:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.031+0000 7fae0effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.031+0000 7fae0effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.031+0000 7fae0effd640 1 -- 172.21.15.46:0/1319976129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae10137340 msgr2=0x7fae1009d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.031+0000 7fae0effd640 1 --2- 172.21.15.46:0/1319976129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae10137340 0x7fae1009d820 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fae10068160 tx=0x7fae080046c0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.031+0000 7fae0effd640 1 -- 172.21.15.46:0/1319976129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae10137340 msgr2=0x7fae1009d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.031+0000 7fae0effd640 1 --2- 172.21.15.46:0/1319976129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae10137340 0x7fae1009d820 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.037+0000 7fae183b9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:42:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.037+0000 7fae183b9640 1 --2- 172.21.15.46:0/1319976129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae1009dd60 0x7fae100a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.037+0000 7fae183b9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.037+0000 7fae183b9640 1 -- 172.21.15.46:0/1319976129 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fae1009e170 con 0x7fae1009dd60 2024-09-16T04:42:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.037+0000 7fae16155640 1 --2- 172.21.15.46:0/1319976129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae1009dd60 0x7fae100a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.037+0000 7fae16155640 10 monclient: get_auth_request con 0x7fae1009dd60 auth_method 0 2024-09-16T04:42:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.037+0000 7fae16155640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.037+0000 7fae16155640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.037+0000 7fae16155640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.037+0000 7fae16155640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.037+0000 7fae16155640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.037+0000 7fae16155640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.037+0000 7fae16155640 10 monclient(hunting): handle_auth_done global_id 24560 payload 824 2024-09-16T04:42:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.037+0000 7fae16155640 1 -- 172.21.15.46:0/1319976129 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fae10067e20 con 0x7fae1009dd60 2024-09-16T04:42:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.037+0000 7fae16155640 1 --2- 172.21.15.46:0/1319976129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae1009dd60 0x7fae100a20f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fae0802a040 tx=0x7fae08008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.037+0000 7fae0effd640 1 -- 172.21.15.46:0/1319976129 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0804f070 con 0x7fae1009dd60 2024-09-16T04:42:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.037+0000 7fae0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.037+0000 7fae0effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.037+0000 7fae0effd640 10 monclient: dump: 2024-09-16T04:42:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:07.056 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.038+0000 7fae0effd640 1 -- 172.21.15.46:0/1319976129 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7fae080093c0 con 0x7fae1009dd60 2024-09-16T04:42:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.038+0000 7fae0effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:42:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.038+0000 7fae0effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.038+0000 7fae0effd640 1 -- 172.21.15.46:0/1319976129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae1009dd60 msgr2=0x7fae100a20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.038+0000 7fae0effd640 1 --2- 172.21.15.46:0/1319976129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae1009dd60 0x7fae100a20f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fae0802a040 tx=0x7fae08008030 comp rx=0 tx=0).stop 2024-09-16T04:42:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.038+0000 7fae0effd640 1 -- 172.21.15.46:0/1319976129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae1009dd60 msgr2=0x7fae100a20f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.038+0000 7fae0effd640 1 --2- 172.21.15.46:0/1319976129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae1009dd60 0x7fae100a20f0 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fae0802a040 tx=0x7fae08008030 comp rx=0 tx=0).stop 2024-09-16T04:42:07.057 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:42:07.057 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:42:07.057 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:42:07.057 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:42:07.057 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 38, 2024-09-16T04:42:07.057 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 102236, 2024-09-16T04:42:07.058 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:42:07.058 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:42:07.058 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:42:07.058 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:42:07.058 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:07.058 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:42:07.058 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:07.058 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:42:07.058 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:42:07.058 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:07.058 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:07.058 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:07.058 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:07.059 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:07.059 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:07.059 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:07.059 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:07.059 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:07.059 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:07.059 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:07.059 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:07.059 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:42:07.059 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:42:07.059 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:07.059 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:07.059 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:07.059 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:07.060 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:07.060 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:07.060 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:07.060 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:07.060 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:07.060 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:07.060 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:07.060 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:07.060 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.060 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:42:07.060 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:42:07.060 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:42:07.060 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:42:07.061 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:42:07.061 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:42:07.061 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:07.061 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:07.061 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:42:07.061 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:42:07.061 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:42:07.061 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:42:07.061 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:42:07.061 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:42:07.061 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:42:07.061 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:07.062 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:42:07.062 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:07.062 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:07.062 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:07.062 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:07.062 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:07.062 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:07.062 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:07.062 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:07.062 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:07.062 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:07.062 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:07.062 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:07.063 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:42:07.063 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.063 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:42:07.063 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.063 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:42:07.063 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:42:07.063 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:07.063 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:07.063 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.063 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:07.063 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:42:07.063 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:07.063 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.063 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.064 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:07.064 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:42:07.064 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:07.064 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:07.064 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:07.064 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.064 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:42:07.064 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:42:07.064 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:07.064 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:07.064 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:07.064 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.064 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.065 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:42:07.065 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:42:07.065 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:07.065 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:07.065 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.065 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:07.065 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:42:07.065 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:07.065 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.065 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.065 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:07.065 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:42:07.065 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:07.066 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:07.066 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:07.066 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.066 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:42:07.066 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:42:07.066 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:07.066 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:07.066 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:07.066 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.066 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.066 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:42:07.066 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:42:07.066 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:07.066 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:07.067 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.067 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:07.067 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:42:07.067 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:07.067 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.067 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.067 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:07.067 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:42:07.067 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:07.067 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:07.067 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:07.067 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.068 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:42:07.068 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:42:07.068 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:07.068 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:07.068 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:07.068 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:07.068 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:07.068 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.068 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:42:07.068 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:42:07.068 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.068 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:07.068 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:07.069 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:42:07.069 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:07.069 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:07.069 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:42:07.069 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.069 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:07.069 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:07.069 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:42:07.069 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:07.069 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:07.069 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:42:07.069 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.070 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:07.070 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:07.070 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:42:07.070 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:07.070 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:07.070 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.070 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:42:07.070 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:42:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.038+0000 7fae183b9640 1 -- 172.21.15.46:0/1319976129 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fadfc0383e0 msgr2=0x7fadfc03a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.038+0000 7fae183b9640 1 --2- 172.21.15.46:0/1319976129 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fadfc0383e0 0x7fadfc03a8a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fae000012e0 tx=0x7fae00009040 comp rx=0 tx=0).stop 2024-09-16T04:42:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.038+0000 7fae183b9640 10 monclient: shutdown 2024-09-16T04:42:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.038+0000 7fae183b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.038+0000 7fae183b9640 1 -- 172.21.15.46:0/1319976129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae100a3ff0 msgr2=0x7fae10136e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.038+0000 7fae183b9640 1 --2- 172.21.15.46:0/1319976129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae100a3ff0 0x7fae10136e00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fae0400c430 tx=0x7fae0400c8f0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.038+0000 7fae16956640 1 -- 172.21.15.46:0/1319976129 reap_dead start 2024-09-16T04:42:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.038+0000 7fae183b9640 1 -- 172.21.15.46:0/1319976129 shutdown_connections 2024-09-16T04:42:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.038+0000 7fae183b9640 1 --2- 172.21.15.46:0/1319976129 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fae100a3ff0 0x7fae10136e00 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.038+0000 7fae183b9640 1 -- 172.21.15.46:0/1319976129 >> 172.21.15.46:0/1319976129 conn(0x7fae100a6340 msgr2=0x7fae100a82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.038+0000 7fae183b9640 1 -- 172.21.15.46:0/1319976129 shutdown_connections 2024-09-16T04:42:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.038+0000 7fae183b9640 1 -- 172.21.15.46:0/1319976129 wait complete. 2024-09-16T04:42:07.073 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:42:07.073 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:42:07.073 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:42:07.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.174+0000 7f045f92c640 10 monclient: get_monmap_and_config 2024-09-16T04:42:07.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.174+0000 7f045f92c640 10 monclient: build_initial_monmap 2024-09-16T04:42:07.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.174+0000 7f045f92c640 10 monclient: monmap: 2024-09-16T04:42:07.175 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:07.175 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:07.175 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:07.174789+0000 2024-09-16T04:42:07.176 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:07.174789+0000 2024-09-16T04:42:07.176 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:07.176 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.176 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:07.176 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:07.176 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:07.176 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:07.176 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:07.176 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.174+0000 7f045f92c640 1 Processor -- start 2024-09-16T04:42:07.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045f92c640 1 -- start start 2024-09-16T04:42:07.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045f92c640 10 monclient: init 2024-09-16T04:42:07.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045f92c640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:07.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045f92c640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-16T04:42:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045f92c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f04580a3c10 0x7f045809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045f92c640 10 monclient(hunting): picked mon.noname-c con 0x7f04580a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:42:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045f92c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f04580a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:42:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045f92c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f045813be90 0x7f045813e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045f92c640 10 monclient(hunting): picked mon.noname-b con 0x7f045813be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045f92c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f045809b940 con 0x7f04580a3ff0 2024-09-16T04:42:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045f92c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045f92c640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f045809ab00 con 0x7f04580a3c10 2024-09-16T04:42:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045f92c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045f92c640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f045809b030 con 0x7f045813be90 2024-09-16T04:42:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045f92c640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045f92c640 10 monclient(hunting): authenticate will time out at 925.590725s 2024-09-16T04:42:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045dec9640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f045813be90 0x7f045813e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045dec9640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f045813be90 0x7f045813e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39472/0 (socket says 172.21.15.46:39472) 2024-09-16T04:42:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045dec9640 1 -- 172.21.15.46:0/706625831 learned_addr learned my addr 172.21.15.46:0/706625831 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045dec9640 10 monclient(hunting): get_auth_request con 0x7f045813be90 auth_method 0 2024-09-16T04:42:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045dec9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045dec9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045dec9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045d6c8640 1 --2- 172.21.15.46:0/706625831 >> v2:172.21.15.114:3300/0 conn(0x7f04580a3c10 0x7f045809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045dec9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045dec9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.175+0000 7f045dec9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.176+0000 7f045d6c8640 10 monclient(hunting): get_auth_request con 0x7f04580a3c10 auth_method 0 2024-09-16T04:42:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.176+0000 7f045d6c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.176+0000 7f045d6c8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.176+0000 7f045d6c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.176+0000 7f045d6c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.176+0000 7f045d6c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.176+0000 7f045d6c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.176+0000 7f045dec9640 10 monclient(hunting): handle_auth_done global_id 24563 payload 274 2024-09-16T04:42:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.176+0000 7f045dec9640 1 -- 172.21.15.46:0/706625831 >> v2:172.21.15.114:3300/0 conn(0x7f04580a3c10 msgr2=0x7f045809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.176+0000 7f045dec9640 1 --2- 172.21.15.46:0/706625831 >> v2:172.21.15.114:3300/0 conn(0x7f04580a3c10 0x7f045809bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.176+0000 7f045dec9640 1 -- 172.21.15.46:0/706625831 >> v1:172.21.15.114:6789/0 conn(0x7f04580a3ff0 legacy=0x7f04580a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.176+0000 7f045dec9640 10 monclient: _finish_hunting 0 2024-09-16T04:42:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.176+0000 7f045dec9640 1 monclient: found mon.noname-b 2024-09-16T04:42:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.176+0000 7f045dec9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.176+0000 7f045dec9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:42:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.176+0000 7f045dec9640 1 -- 172.21.15.46:0/706625831 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045809b310 con 0x7f045813be90 2024-09-16T04:42:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.176+0000 7f045dec9640 1 --2- 172.21.15.46:0/706625831 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f045813be90 0x7f045813e280 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0450009e50 tx=0x7f045002a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7735b8e0f3245690 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f044ffff640 1 -- 172.21.15.46:0/706625831 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0450036070 con 0x7f045813be90 2024-09-16T04:42:07.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f044ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:07.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f044ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:42:07.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f044ffff640 10 monclient: dump: 2024-09-16T04:42:07.188 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:07.188 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:07.188 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.188 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.188 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:07.188 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.188 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:07.189 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:07.189 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:07.189 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f044ffff640 10 monclient: _finish_auth 0 2024-09-16T04:42:07.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f044ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:42:07.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f044ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:07.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f044ffff640 1 -- 172.21.15.46:0/706625831 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0450031600 con 0x7f045813be90 2024-09-16T04:42:07.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f044ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:07.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f044ffff640 1 -- 172.21.15.46:0/706625831 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04500319d0 con 0x7f045813be90 2024-09-16T04:42:07.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f045f92c640 5 monclient: authenticate success, global_id 24563 2024-09-16T04:42:07.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f045f92c640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:07.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f044ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:07.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f045f92c640 4 set_mon_vals no callback set 2024-09-16T04:42:07.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f044ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:07.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f044ffff640 10 monclient: dump: 2024-09-16T04:42:07.190 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:07.190 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:07.190 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.190 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.190 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:07.190 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.190 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:07.191 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:07.191 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:07.191 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f045f92c640 10 monclient: shutdown 2024-09-16T04:42:07.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f045f92c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:07.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f045f92c640 1 -- 172.21.15.46:0/706625831 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f045813be90 msgr2=0x7f045813e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:07.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f045f92c640 1 --2- 172.21.15.46:0/706625831 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f045813be90 0x7f045813e280 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0450009e50 tx=0x7f045002a5f0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f045f92c640 1 -- 172.21.15.46:0/706625831 shutdown_connections 2024-09-16T04:42:07.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f045f92c640 1 --2- 172.21.15.46:0/706625831 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f045813be90 0x7f045813e280 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f045f92c640 1 --2- 172.21.15.46:0/706625831 >> v2:172.21.15.114:3300/0 conn(0x7f04580a3c10 0x7f045809bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f045f92c640 1 -- 172.21.15.46:0/706625831 >> 172.21.15.46:0/706625831 conn(0x7f04580a6340 msgr2=0x7f045806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:07.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f045f92c640 1 -- 172.21.15.46:0/706625831 shutdown_connections 2024-09-16T04:42:07.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.177+0000 7f045f92c640 1 -- 172.21.15.46:0/706625831 wait complete. 2024-09-16T04:42:07.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.178+0000 7f045f92c640 10 monclient: build_initial_monmap 2024-09-16T04:42:07.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.178+0000 7f045f92c640 10 monclient: monmap: 2024-09-16T04:42:07.192 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:07.192 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:07.193 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:07.193 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:07.193 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:07.193 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.193 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:07.193 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:07.193 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:07.193 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.178+0000 7f045f92c640 1 Processor -- start 2024-09-16T04:42:07.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.178+0000 7f045f92c640 1 -- start start 2024-09-16T04:42:07.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.178+0000 7f045f92c640 10 monclient: init 2024-09-16T04:42:07.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045f92c640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:07.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045f92c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:42:07.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045f92c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04580a3c10 0x7f0458139800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:07.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045f92c640 10 monclient(hunting): picked mon.noname-b con 0x7f04580a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:07.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045f92c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f04580a3ff0 0x7f0458137e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:07.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045f92c640 10 monclient(hunting): picked mon.noname-c con 0x7f04580a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:07.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045f92c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f045813be90 0x7f04581383b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:07.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045f92c640 10 monclient(hunting): picked mon.noname-a con 0x7f045813be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:07.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045f92c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:07.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045f92c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f04580a6b40 con 0x7f045813be90 2024-09-16T04:42:07.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045f92c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:07.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045f92c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f04580a69c0 con 0x7f04580a3c10 2024-09-16T04:42:07.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045f92c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:07.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045f92c640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f045813e7c0 con 0x7f04580a3ff0 2024-09-16T04:42:07.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045f92c640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:07.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045f92c640 10 monclient(hunting): authenticate will time out at 925.594119s 2024-09-16T04:42:07.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045cec7640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f04580a3ff0 0x7f0458137e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:07.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045cec7640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f04580a3ff0 0x7f0458137e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39486/0 (socket says 172.21.15.46:39486) 2024-09-16T04:42:07.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045cec7640 1 -- 172.21.15.46:0/4184334902 learned_addr learned my addr 172.21.15.46:0/4184334902 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:07.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045cec7640 10 monclient(hunting): get_auth_request con 0x7f04580a3ff0 auth_method 0 2024-09-16T04:42:07.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045cec7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:07.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045cec7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:07.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045cec7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:07.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045dec9640 1 --2- 172.21.15.46:0/4184334902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f045813be90 0x7f04581383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:07.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045dec9640 10 monclient(hunting): get_auth_request con 0x7f045813be90 auth_method 0 2024-09-16T04:42:07.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045dec9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:07.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045dec9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:07.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045dec9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:07.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045cec7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:07.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045cec7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:07.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045d6c8640 1 --2- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04580a3c10 0x7f0458139800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:07.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045cec7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:07.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045d6c8640 10 monclient(hunting): get_auth_request con 0x7f04580a3c10 auth_method 0 2024-09-16T04:42:07.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045d6c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:07.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045d6c8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:07.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045d6c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:07.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045dec9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:07.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045dec9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:07.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045dec9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:07.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045d6c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:07.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045d6c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:07.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045d6c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:07.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045cec7640 10 monclient(hunting): handle_auth_done global_id 24566 payload 824 2024-09-16T04:42:07.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045cec7640 1 -- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04580a3c10 msgr2=0x7f0458139800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:07.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045cec7640 1 --2- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04580a3c10 0x7f0458139800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045cec7640 1 -- 172.21.15.46:0/4184334902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f045813be90 msgr2=0x7f04581383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:07.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045cec7640 1 --2- 172.21.15.46:0/4184334902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f045813be90 0x7f04581383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045cec7640 10 monclient: _finish_hunting 0 2024-09-16T04:42:07.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045cec7640 1 monclient: found mon.noname-c 2024-09-16T04:42:07.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045cec7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:07.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045cec7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:42:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045cec7640 1 -- 172.21.15.46:0/4184334902 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045813a530 con 0x7f04580a3ff0 2024-09-16T04:42:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045cec7640 1 --2- 172.21.15.46:0/4184334902 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f04580a3ff0 0x7f0458137e70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f044800c480 tx=0x7f044800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f044dffb640 1 -- 172.21.15.46:0/4184334902 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04480052d0 con 0x7f04580a3ff0 2024-09-16T04:42:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f044dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f044dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:42:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f044dffb640 10 monclient: dump: 2024-09-16T04:42:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:07.199 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:07.199 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f044dffb640 10 monclient: _finish_auth 0 2024-09-16T04:42:07.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f044dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:42:07.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f044dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:07.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f044dffb640 1 -- 172.21.15.46:0/4184334902 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0448005470 con 0x7f04580a3ff0 2024-09-16T04:42:07.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045f92c640 5 monclient: authenticate success, global_id 24566 2024-09-16T04:42:07.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f044dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:07.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045f92c640 10 monclient: _renew_subs 2024-09-16T04:42:07.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f044dffb640 1 -- 172.21.15.46:0/4184334902 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0448012870 con 0x7f04580a3ff0 2024-09-16T04:42:07.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045f92c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:07.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f045f92c640 1 -- 172.21.15.46:0/4184334902 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f045813a810 con 0x7f04580a3ff0 2024-09-16T04:42:07.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f044f7fe640 4 set_mon_vals no callback set 2024-09-16T04:42:07.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f044dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:07.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f044dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:07.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.179+0000 7f044dffb640 10 monclient: dump: 2024-09-16T04:42:07.200 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:07.200 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:07.200 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.200 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.200 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:07.200 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.200 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:07.201 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:07.201 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:07.201 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.180+0000 7f045f92c640 10 monclient: _renew_subs 2024-09-16T04:42:07.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.180+0000 7f045f92c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:07.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.180+0000 7f045f92c640 1 -- 172.21.15.46:0/4184334902 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f045813b820 con 0x7f04580a3ff0 2024-09-16T04:42:07.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.180+0000 7f044dffb640 1 -- 172.21.15.46:0/4184334902 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f0448012a10 con 0x7f04580a3ff0 2024-09-16T04:42:07.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.180+0000 7f045f92c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:07.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.180+0000 7f045f92c640 1 --2- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04581388f0 0x7f04580a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:07.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.180+0000 7f045f92c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:07.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.180+0000 7f045f92c640 1 -- 172.21.15.46:0/4184334902 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0458138cd0 con 0x7f04581388f0 2024-09-16T04:42:07.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.180+0000 7f044dffb640 1 --2- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f043c0383b0 0x7f043c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.181+0000 7f044dffb640 1 -- 172.21.15.46:0/4184334902 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(134..134 src has 1..134) ==== 4346+0+0 (secure 0 0 0) 0x7f044804b1f0 con 0x7f04580a3ff0 2024-09-16T04:42:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.181+0000 7f045d6c8640 1 --2- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04581388f0 0x7f04580a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.181+0000 7f045dec9640 1 --2- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f043c0383b0 0x7f043c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.181+0000 7f045d6c8640 10 monclient: get_auth_request con 0x7f04581388f0 auth_method 0 2024-09-16T04:42:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.181+0000 7f045d6c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.181+0000 7f045d6c8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.181+0000 7f045d6c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.181+0000 7f045dec9640 10 monclient: get_auth_request con 0x7f043c0383b0 auth_method 0 2024-09-16T04:42:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.181+0000 7f045d6c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.181+0000 7f045d6c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.181+0000 7f045d6c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.181+0000 7f045d6c8640 10 monclient(hunting): handle_auth_done global_id 5425 payload 824 2024-09-16T04:42:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.181+0000 7f045d6c8640 1 -- 172.21.15.46:0/4184334902 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f045809bcf0 con 0x7f04581388f0 2024-09-16T04:42:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.181+0000 7f045d6c8640 1 --2- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04581388f0 0x7f04580a0610 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0454011890 tx=0x7f0454011d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.181+0000 7f045dec9640 1 --2- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f043c0383b0 0x7f043c03a870 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0450033000 tx=0x7f0450009000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.181+0000 7f044dffb640 1 -- 172.21.15.46:0/4184334902 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0454019070 con 0x7f04581388f0 2024-09-16T04:42:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.181+0000 7f044dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.181+0000 7f044dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:42:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.181+0000 7f044dffb640 10 monclient: dump: 2024-09-16T04:42:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:07.203 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.182+0000 7f044dffb640 1 -- 172.21.15.46:0/4184334902 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0454004240 con 0x7f04581388f0 2024-09-16T04:42:07.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.182+0000 7f044dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:07.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.182+0000 7f044dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:07.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.182+0000 7f044dffb640 1 -- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04581388f0 msgr2=0x7f04580a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:07.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.182+0000 7f044dffb640 1 --2- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04581388f0 0x7f04580a0610 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0454011890 tx=0x7f0454011d50 comp rx=0 tx=0).stop 2024-09-16T04:42:07.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.182+0000 7f044dffb640 1 -- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04581388f0 msgr2=0x7f04580a0610 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:07.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.182+0000 7f044dffb640 1 --2- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04581388f0 0x7f04580a0610 secure :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0454011890 tx=0x7f0454011d50 comp rx=0 tx=0).stop 2024-09-16T04:42:07.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.188+0000 7f045f92c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:42:07.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.188+0000 7f045f92c640 1 --2- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04580a0b50 0x7f04580a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:07.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.188+0000 7f045f92c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:07.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.188+0000 7f045f92c640 1 -- 172.21.15.46:0/4184334902 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0458073000 con 0x7f04580a0b50 2024-09-16T04:42:07.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.188+0000 7f045d6c8640 1 --2- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04580a0b50 0x7f04580a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:07.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.188+0000 7f045d6c8640 10 monclient: get_auth_request con 0x7f04580a0b50 auth_method 0 2024-09-16T04:42:07.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.188+0000 7f045d6c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:07.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.188+0000 7f045d6c8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:07.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.188+0000 7f045d6c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:07.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.188+0000 7f045d6c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:07.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.188+0000 7f045d6c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:07.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.188+0000 7f045d6c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:07.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.188+0000 7f045d6c8640 10 monclient(hunting): handle_auth_done global_id 5428 payload 824 2024-09-16T04:42:07.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.188+0000 7f045d6c8640 1 -- 172.21.15.46:0/4184334902 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0458067e20 con 0x7f04580a0b50 2024-09-16T04:42:07.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.189+0000 7f045d6c8640 1 --2- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04580a0b50 0x7f04580a0f60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0454011890 tx=0x7f04540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:07.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.189+0000 7f044dffb640 1 -- 172.21.15.46:0/4184334902 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0454019040 con 0x7f04580a0b50 2024-09-16T04:42:07.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.189+0000 7f044dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:07.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.189+0000 7f044dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:42:07.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.189+0000 7f044dffb640 10 monclient: dump: 2024-09-16T04:42:07.208 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:07.208 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:07.208 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.208 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.208 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:07.208 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.208 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:07.208 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:07.209 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:07.209 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.189+0000 7f044dffb640 1 -- 172.21.15.46:0/4184334902 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f04540044d0 con 0x7f04580a0b50 2024-09-16T04:42:07.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.189+0000 7f044dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:42:07.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.189+0000 7f044dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:07.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.189+0000 7f044dffb640 1 -- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04580a0b50 msgr2=0x7f04580a0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:07.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.189+0000 7f044dffb640 1 --2- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04580a0b50 0x7f04580a0f60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0454011890 tx=0x7f04540046c0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.189+0000 7f044dffb640 1 -- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04580a0b50 msgr2=0x7f04580a0f60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:07.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.189+0000 7f044dffb640 1 --2- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04580a0b50 0x7f04580a0f60 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.209 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:42:07.210 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:42:07.210 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:42:07.210 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:42:07.210 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 38, 2024-09-16T04:42:07.210 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 213091, 2024-09-16T04:42:07.210 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:42:07.210 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:42:07.210 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:42:07.210 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:42:07.210 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:07.210 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:42:07.210 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:07.210 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:42:07.211 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:42:07.211 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:07.211 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:07.211 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:07.211 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:07.211 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:07.211 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:07.211 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:07.211 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:07.211 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:07.211 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:07.211 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:07.211 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:07.211 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:42:07.211 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:42:07.212 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:07.212 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:07.212 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:07.212 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:07.212 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:07.212 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:07.212 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:07.212 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:07.212 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:07.212 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:07.212 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:07.212 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:07.212 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.213 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:42:07.213 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:42:07.213 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:42:07.213 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:42:07.213 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:42:07.213 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:42:07.213 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:07.213 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:07.213 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:42:07.213 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:42:07.213 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:42:07.213 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:42:07.213 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:42:07.213 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:42:07.214 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:42:07.214 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:07.214 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:42:07.214 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:07.214 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:07.214 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:07.214 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:07.214 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:07.214 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:07.214 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:07.214 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:07.214 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:07.214 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:07.214 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:07.215 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:07.215 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:42:07.215 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.215 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:42:07.215 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.215 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:42:07.215 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:42:07.215 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:07.215 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:07.215 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.215 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:07.215 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:42:07.215 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:07.215 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.216 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.216 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:07.216 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:42:07.216 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:07.216 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:07.216 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:07.216 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.216 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:42:07.216 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:42:07.216 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:07.216 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:07.216 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:07.216 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.217 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.217 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:42:07.217 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:42:07.217 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:07.217 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:07.217 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.217 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:07.217 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:42:07.217 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:07.217 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.217 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.217 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:07.217 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:42:07.218 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:07.218 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:07.218 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:07.218 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.218 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:42:07.218 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:42:07.218 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:07.218 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:07.218 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:07.218 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.218 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.219 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:42:07.219 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:42:07.219 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:07.219 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:07.219 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.219 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:07.219 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:42:07.219 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:07.219 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.219 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.219 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:07.219 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:42:07.219 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:07.219 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:07.220 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:07.220 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.220 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:42:07.220 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:42:07.220 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:07.220 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:07.220 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:07.220 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:07.220 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:07.220 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.220 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:42:07.220 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:42:07.220 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.221 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:07.221 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:07.221 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:42:07.221 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:07.221 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:07.221 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:42:07.221 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.221 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:07.221 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:07.221 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4 2024-09-16T04:42:07.222 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:07.222 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:07.222 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:42:07.222 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.222 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:07.222 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:07.222 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 7 2024-09-16T04:42:07.222 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:07.222 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:07.222 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:42:07.222 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:07.222 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:07.222 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:07.222 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:42:07.223 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:07.223 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:07.223 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:07.223 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:42:07.223 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:42:07.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.189+0000 7f045f92c640 1 -- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f043c0383b0 msgr2=0x7f043c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:07.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.189+0000 7f045f92c640 1 --2- 172.21.15.46:0/4184334902 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f043c0383b0 0x7f043c03a870 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0450033000 tx=0x7f0450009000 comp rx=0 tx=0).stop 2024-09-16T04:42:07.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.189+0000 7f045f92c640 10 monclient: shutdown 2024-09-16T04:42:07.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.189+0000 7f045f92c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.189+0000 7f045f92c640 1 -- 172.21.15.46:0/4184334902 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f04580a3ff0 msgr2=0x7f0458137e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.189+0000 7f045f92c640 1 --2- 172.21.15.46:0/4184334902 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f04580a3ff0 0x7f0458137e70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f044800c480 tx=0x7f044800c940 comp rx=0 tx=0).stop 2024-09-16T04:42:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.189+0000 7f045dec9640 1 -- 172.21.15.46:0/4184334902 reap_dead start 2024-09-16T04:42:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.189+0000 7f045dec9640 1 -- 172.21.15.46:0/4184334902 reap_dead start 2024-09-16T04:42:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.190+0000 7f045f92c640 1 -- 172.21.15.46:0/4184334902 shutdown_connections 2024-09-16T04:42:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.190+0000 7f045f92c640 1 -- 172.21.15.46:0/4184334902 >> 172.21.15.46:0/4184334902 conn(0x7f04580a6340 msgr2=0x7f045813d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.190+0000 7f045f92c640 1 -- 172.21.15.46:0/4184334902 shutdown_connections 2024-09-16T04:42:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.190+0000 7f045f92c640 1 -- 172.21.15.46:0/4184334902 wait complete. 2024-09-16T04:42:07.225 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:42:07.225 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:42:07.226 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:42:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.325+0000 7f1071f5f640 10 monclient: get_monmap_and_config 2024-09-16T04:42:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.325+0000 7f1071f5f640 10 monclient: build_initial_monmap 2024-09-16T04:42:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.325+0000 7f1071f5f640 10 monclient: monmap: 2024-09-16T04:42:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:07.325682+0000 2024-09-16T04:42:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:07.325682+0000 2024-09-16T04:42:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:07.327 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.325+0000 7f1071f5f640 1 Processor -- start 2024-09-16T04:42:07.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f1071f5f640 1 -- start start 2024-09-16T04:42:07.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f1071f5f640 10 monclient: init 2024-09-16T04:42:07.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f1071f5f640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:07.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f1071f5f640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-16T04:42:07.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f1071f5f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f106c073760 addr v1:172.21.15.46:6789/0 2024-09-16T04:42:07.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f1071f5f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f106c139be0 addr v1:172.21.15.114:6789/0 2024-09-16T04:42:07.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f1071f5f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f106c13d6f0 0x7f106c13fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:07.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f1071f5f640 10 monclient(hunting): picked mon.noname-a con 0x7f106c13d6f0 addr v2:172.21.15.46:3300/0 2024-09-16T04:42:07.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f1071f5f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f106c074250 con 0x7f106c073760 2024-09-16T04:42:07.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f1071f5f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f106c09ef20 con 0x7f106c139be0 2024-09-16T04:42:07.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f1071f5f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:07.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f1071f5f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f106c06e070 con 0x7f106c13d6f0 2024-09-16T04:42:07.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f1071f5f640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:07.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f1071f5f640 10 monclient(hunting): authenticate will time out at 925.741564s 2024-09-16T04:42:07.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f106bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f106c13d6f0 0x7f106c13fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:07.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f106bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f106c13d6f0 0x7f106c13fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49526/0 (socket says 172.21.15.46:49526) 2024-09-16T04:42:07.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f106bfff640 1 -- 172.21.15.46:0/3881697699 learned_addr learned my addr 172.21.15.46:0/3881697699 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:07.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f106b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f106c073760 0x7f106c073b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51664/0 (socket says 172.21.15.46:51664) 2024-09-16T04:42:07.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f106bfff640 10 monclient(hunting): get_auth_request con 0x7f106c13d6f0 auth_method 0 2024-09-16T04:42:07.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f106bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:07.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f106bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f106bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f106bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f106bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f106bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f106bfff640 10 monclient(hunting): handle_auth_done global_id 44121 payload 274 2024-09-16T04:42:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f106bfff640 1 -- 172.21.15.46:0/3881697699 >> v1:172.21.15.114:6789/0 conn(0x7f106c139be0 legacy=0x7f106c13bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f106bfff640 1 -- 172.21.15.46:0/3881697699 >> v1:172.21.15.46:6789/0 conn(0x7f106c073760 legacy=0x7f106c073b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f106bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:42:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f106bfff640 1 monclient: found mon.noname-a 2024-09-16T04:42:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f106bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f106bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.327+0000 7f106bfff640 1 -- 172.21.15.46:0/3881697699 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f106c0a36e0 con 0x7f106c13d6f0 2024-09-16T04:42:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.326+0000 7f106a7fc640 1 -- 172.21.15.46:0/3881697699 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4094578804 0 0) 0x7f106c074250 con 0x7f106c073760 2024-09-16T04:42:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.327+0000 7f106a7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:42:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.327+0000 7f106bfff640 1 --2- 172.21.15.46:0/3881697699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f106c13d6f0 0x7f106c13fae0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1064009e50 tx=0x7f106402a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=827dbdb790700ef0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.327+0000 7f106a7fc640 1 -- 172.21.15.46:0/3881697699 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1064036070 con 0x7f106c13d6f0 2024-09-16T04:42:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.327+0000 7f106a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.327+0000 7f106a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.327+0000 7f106a7fc640 10 monclient: dump: 2024-09-16T04:42:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:07.338 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.327+0000 7f106a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:42:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.327+0000 7f106a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:42:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.327+0000 7f106a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.327+0000 7f1071f5f640 5 monclient: authenticate success, global_id 44121 2024-09-16T04:42:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.327+0000 7f1071f5f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.327+0000 7f106a7fc640 1 -- 172.21.15.46:0/3881697699 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1064031600 con 0x7f106c13d6f0 2024-09-16T04:42:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.327+0000 7f106a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.327+0000 7f1071f5f640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.327+0000 7f1071f5f640 4 set_mon_vals no callback set 2024-09-16T04:42:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.327+0000 7f106a7fc640 1 -- 172.21.15.46:0/3881697699 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10640319d0 con 0x7f106c13d6f0 2024-09-16T04:42:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.327+0000 7f106a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.327+0000 7f106a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.327+0000 7f106a7fc640 10 monclient: dump: 2024-09-16T04:42:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:07.339 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.328+0000 7f1071f5f640 10 monclient: shutdown 2024-09-16T04:42:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.328+0000 7f1071f5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.328+0000 7f1071f5f640 1 -- 172.21.15.46:0/3881697699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f106c13d6f0 msgr2=0x7f106c13fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.328+0000 7f1071f5f640 1 --2- 172.21.15.46:0/3881697699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f106c13d6f0 0x7f106c13fae0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1064009e50 tx=0x7f106402a640 comp rx=0 tx=0).stop 2024-09-16T04:42:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.328+0000 7f1071f5f640 1 -- 172.21.15.46:0/3881697699 shutdown_connections 2024-09-16T04:42:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.328+0000 7f1071f5f640 1 --2- 172.21.15.46:0/3881697699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f106c13d6f0 0x7f106c13fae0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.328+0000 7f1071f5f640 1 -- 172.21.15.46:0/3881697699 >> 172.21.15.46:0/3881697699 conn(0x7f106c0a3b10 msgr2=0x7f106c0a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.328+0000 7f1071f5f640 1 -- 172.21.15.46:0/3881697699 shutdown_connections 2024-09-16T04:42:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.328+0000 7f1071f5f640 1 -- 172.21.15.46:0/3881697699 wait complete. 2024-09-16T04:42:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.329+0000 7f1071f5f640 10 monclient: build_initial_monmap 2024-09-16T04:42:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.329+0000 7f1071f5f640 10 monclient: monmap: 2024-09-16T04:42:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:07.340 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:07.341 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.329+0000 7f1071f5f640 1 Processor -- start 2024-09-16T04:42:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.329+0000 7f1071f5f640 1 -- start start 2024-09-16T04:42:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.329+0000 7f1071f5f640 10 monclient: init 2024-09-16T04:42:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.329+0000 7f1071f5f640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.329+0000 7f1071f5f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:42:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.329+0000 7f1071f5f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f106c073760 0x7f106c121a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.329+0000 7f1071f5f640 10 monclient(hunting): picked mon.noname-a con 0x7f106c073760 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.329+0000 7f1071f5f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c139be0 0x7f106c121fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.329+0000 7f1071f5f640 10 monclient(hunting): picked mon.noname-b con 0x7f106c139be0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.329+0000 7f1071f5f640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f106c13d6f0 0x7f106c1224f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.329+0000 7f1071f5f640 10 monclient(hunting): picked mon.noname-c con 0x7f106c13d6f0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.329+0000 7f1071f5f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.329+0000 7f1071f5f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f106c0a2c90 con 0x7f106c073760 2024-09-16T04:42:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.329+0000 7f1071f5f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.329+0000 7f1071f5f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f106c09df80 con 0x7f106c139be0 2024-09-16T04:42:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.329+0000 7f1071f5f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.329+0000 7f1071f5f640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f106c0a2e10 con 0x7f106c13d6f0 2024-09-16T04:42:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.329+0000 7f1071f5f640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1071f5f640 10 monclient(hunting): authenticate will time out at 925.745009s 2024-09-16T04:42:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f106c073760 0x7f106c121a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106bfff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f106c13d6f0 0x7f106c1224f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f106c073760 0x7f106c121a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49542/0 (socket says 172.21.15.46:49542) 2024-09-16T04:42:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106b7fe640 1 -- 172.21.15.46:0/287230144 learned_addr learned my addr 172.21.15.46:0/287230144 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106b7fe640 10 monclient(hunting): get_auth_request con 0x7f106c073760 auth_method 0 2024-09-16T04:42:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106bfff640 10 monclient(hunting): get_auth_request con 0x7f106c13d6f0 auth_method 0 2024-09-16T04:42:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106affd640 1 --2- 172.21.15.46:0/287230144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c139be0 0x7f106c121fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106affd640 10 monclient(hunting): get_auth_request con 0x7f106c139be0 auth_method 0 2024-09-16T04:42:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106b7fe640 10 monclient(hunting): handle_auth_done global_id 44127 payload 824 2024-09-16T04:42:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106b7fe640 1 -- 172.21.15.46:0/287230144 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f106c13d6f0 msgr2=0x7f106c1224f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106b7fe640 1 --2- 172.21.15.46:0/287230144 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f106c13d6f0 0x7f106c1224f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106b7fe640 1 -- 172.21.15.46:0/287230144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c139be0 msgr2=0x7f106c121fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106b7fe640 1 --2- 172.21.15.46:0/287230144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c139be0 0x7f106c121fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:42:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106b7fe640 1 monclient: found mon.noname-a 2024-09-16T04:42:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106b7fe640 1 -- 172.21.15.46:0/287230144 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f106c124020 con 0x7f106c073760 2024-09-16T04:42:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106bfff640 1 --2- 172.21.15.46:0/287230144 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f106c13d6f0 0x7f106c1224f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:42:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f106b7fe640 1 --2- 172.21.15.46:0/287230144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f106c073760 0x7f106c121a70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f105c00c460 tx=0x7f105c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1053fff640 1 -- 172.21.15.46:0/287230144 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c0074e0 con 0x7f106c073760 2024-09-16T04:42:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1053fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1053fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1053fff640 10 monclient: dump: 2024-09-16T04:42:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:07.348 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1053fff640 10 monclient: _finish_auth 0 2024-09-16T04:42:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1053fff640 10 monclient: _check_auth_tickets 2024-09-16T04:42:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1053fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1053fff640 1 -- 172.21.15.46:0/287230144 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f105c0047e0 con 0x7f106c073760 2024-09-16T04:42:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1071f5f640 5 monclient: authenticate success, global_id 44127 2024-09-16T04:42:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1071f5f640 10 monclient: _renew_subs 2024-09-16T04:42:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1071f5f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1071f5f640 1 -- 172.21.15.46:0/287230144 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f106c1245e0 con 0x7f106c073760 2024-09-16T04:42:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1053fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1053fff640 1 -- 172.21.15.46:0/287230144 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c004bd0 con 0x7f106c073760 2024-09-16T04:42:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1053fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1053fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1053fff640 10 monclient: dump: 2024-09-16T04:42:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:07.350 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1071f5f640 10 monclient: _renew_subs 2024-09-16T04:42:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1071f5f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1071f5f640 1 -- 172.21.15.46:0/287230144 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f106c123d40 con 0x7f106c073760 2024-09-16T04:42:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.330+0000 7f1069ffb640 4 set_mon_vals no callback set 2024-09-16T04:42:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.331+0000 7f1053fff640 1 -- 172.21.15.46:0/287230144 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f105c011070 con 0x7f106c073760 2024-09-16T04:42:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.331+0000 7f1071f5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.331+0000 7f1071f5f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.331+0000 7f1071f5f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.331+0000 7f1071f5f640 1 -- 172.21.15.46:0/287230144 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f106c09fa20 con 0x7f106c073760 2024-09-16T04:42:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.331+0000 7f1053fff640 1 --2- 172.21.15.46:0/287230144 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f10580383b0 0x7f105803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.331+0000 7f1053fff640 1 -- 172.21.15.46:0/287230144 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 4346+0+0 (secure 0 0 0) 0x7f105c04b210 con 0x7f106c073760 2024-09-16T04:42:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.332+0000 7f106affd640 1 --2- 172.21.15.46:0/287230144 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f10580383b0 0x7f105803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.332+0000 7f106affd640 10 monclient: get_auth_request con 0x7f10580383b0 auth_method 0 2024-09-16T04:42:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.332+0000 7f106affd640 1 --2- 172.21.15.46:0/287230144 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f10580383b0 0x7f105803a870 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f106c1cc930 tx=0x7f1060007e30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.336+0000 7f1053fff640 1 -- 172.21.15.46:0/287230144 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f105c011360 con 0x7f106c073760 2024-09-16T04:42:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.336+0000 7f1053fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.336+0000 7f1053fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:07.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.584+0000 7f1071f5f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:42:07.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.584+0000 7f1071f5f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:42:07.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.584+0000 7f1071f5f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:07.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.584+0000 7f1071f5f640 1 -- 172.21.15.46:0/287230144 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f106c1243c0 con 0x7f106c073760 2024-09-16T04:42:07.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.585+0000 7f1053fff640 1 -- 172.21.15.46:0/287230144 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f105c01cd70 con 0x7f106c073760 2024-09-16T04:42:07.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.585+0000 7f1053fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:42:07.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.585+0000 7f1053fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:07.587 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.585+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.587 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.585+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=11,kv=12,logm=64} crc {auth=395621168,config=3541143443,health=1096256309,kv=4017288519,logm=511870749}) 2024-09-16T04:42:07.587 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.585+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=11,kv=12,logm=64} crc {auth=395621168,config=3541143443,health=3584242354,kv=4017288519,logm=1049851386}) 2024-09-16T04:42:07.587 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.585+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.587 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.585+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=11,kv=12,logm=64} crc {auth=395621168,config=3541143443,health=1096256309,kv=4017288519,logm=511870749}) 2024-09-16T04:42:07.587 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.585+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=11,config=1,health=11,kv=11,logm=66} crc {auth=395621168,config=555582061,health=1647462659,kv=4017288519,logm=2472914040}) 2024-09-16T04:42:07.587 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.586+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.588 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.586+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477682024}) 2024-09-16T04:42:07.588 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.586+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2795991183}) 2024-09-16T04:42:07.588 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.586+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.588 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.586+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477682024}) 2024-09-16T04:42:07.588 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.586+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2782612988}) 2024-09-16T04:42:07.588 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.587+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.588 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.587+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799923897}) 2024-09-16T04:42:07.589 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.587+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3877728489}) 2024-09-16T04:42:07.589 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.587+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.589 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.587+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799923897}) 2024-09-16T04:42:07.589 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.587+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1949165354}) 2024-09-16T04:42:07.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.587+0000 7f1071f5f640 1 -- 172.21.15.46:0/287230144 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f10580383b0 msgr2=0x7f105803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:07.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.587+0000 7f1071f5f640 1 --2- 172.21.15.46:0/287230144 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f10580383b0 0x7f105803a870 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f106c1cc930 tx=0x7f1060007e30 comp rx=0 tx=0).stop 2024-09-16T04:42:07.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.587+0000 7f1071f5f640 10 monclient: shutdown 2024-09-16T04:42:07.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.587+0000 7f1071f5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:07.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.587+0000 7f1071f5f640 1 -- 172.21.15.46:0/287230144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f106c073760 msgr2=0x7f106c121a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:07.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.587+0000 7f1071f5f640 1 --2- 172.21.15.46:0/287230144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f106c073760 0x7f106c121a70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f105c00c460 tx=0x7f105c00c920 comp rx=0 tx=0).stop 2024-09-16T04:42:07.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.587+0000 7f1071f5f640 1 -- 172.21.15.46:0/287230144 shutdown_connections 2024-09-16T04:42:07.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.587+0000 7f1071f5f640 1 --2- 172.21.15.46:0/287230144 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f106c13d6f0 0x7f106c1224f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.587+0000 7f1071f5f640 1 --2- 172.21.15.46:0/287230144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c139be0 0x7f106c121fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.587+0000 7f1071f5f640 1 --2- 172.21.15.46:0/287230144 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f10580383b0 0x7f105803a870 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.587+0000 7f1071f5f640 1 --2- 172.21.15.46:0/287230144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f106c073760 0x7f106c121a70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:07.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.587+0000 7f1071f5f640 1 -- 172.21.15.46:0/287230144 >> 172.21.15.46:0/287230144 conn(0x7f106c0a3b10 msgr2=0x7f106c06cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:07.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.587+0000 7f1071f5f640 1 -- 172.21.15.46:0/287230144 shutdown_connections 2024-09-16T04:42:07.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:07.587+0000 7f1071f5f640 1 -- 172.21.15.46:0/287230144 wait complete. 2024-09-16T04:42:07.596 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.587+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.596 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.587+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1575675616}) 2024-09-16T04:42:07.596 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.587+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=286810896}) 2024-09-16T04:42:07.596 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.587+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.596 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.587+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1575675616}) 2024-09-16T04:42:07.596 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.588+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=696009473}) 2024-09-16T04:42:07.596 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.588+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.596 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.588+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2068240702}) 2024-09-16T04:42:07.596 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.588+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=661940776}) 2024-09-16T04:42:07.596 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.588+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.597 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.588+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2068240702}) 2024-09-16T04:42:07.597 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.588+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3814877565}) 2024-09-16T04:42:07.597 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.589+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.597 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.589+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251262791}) 2024-09-16T04:42:07.597 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.589+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4007167502}) 2024-09-16T04:42:07.597 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.589+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.597 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.589+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251262791}) 2024-09-16T04:42:07.597 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.589+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1465313607}) 2024-09-16T04:42:07.597 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.590+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.597 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.590+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2869982705}) 2024-09-16T04:42:07.597 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.590+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2729173206}) 2024-09-16T04:42:07.597 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.590+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.597 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.590+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2869982705}) 2024-09-16T04:42:07.597 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.590+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4055951860}) 2024-09-16T04:42:07.598 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.590+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.598 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.591+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1608091430}) 2024-09-16T04:42:07.598 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.591+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=264839270}) 2024-09-16T04:42:07.598 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.591+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.598 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.591+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1608091430}) 2024-09-16T04:42:07.598 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.591+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2911990457}) 2024-09-16T04:42:07.598 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.591+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.598 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.591+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=785442523}) 2024-09-16T04:42:07.598 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.591+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3178214837}) 2024-09-16T04:42:07.598 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.591+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.598 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.591+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=785442523}) 2024-09-16T04:42:07.598 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.591+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1628398052}) 2024-09-16T04:42:07.598 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.592+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.598 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.592+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=76,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=2} crc {logm=735986295,mdsmap=3956149183,mgr=677368822,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=1227489364,mon_config_key=2072804365}) 2024-09-16T04:42:07.599 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.592+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=75,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=3} crc {logm=3448233015,mdsmap=571966421,mgr=1081907435,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=1227489364,mon_config_key=3136301670}) 2024-09-16T04:42:07.599 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.592+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.599 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.592+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=76,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=2} crc {logm=735986295,mdsmap=3956149183,mgr=677368822,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=1227489364,mon_config_key=2072804365}) 2024-09-16T04:42:07.599 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.592+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=70,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=9} crc {logm=1253733868,mdsmap=3956149183,mgr=1081907435,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=315671301,mon_config_key=104982079}) 2024-09-16T04:42:07.599 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.593+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.599 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.593+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=12,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=77} crc {mon_config_key=3667200259,monmap=3088247691,osd_metadata=2543215506,osd_pg_creating=2048278026,osd_snap=723727393,osdmap=2011613217}) 2024-09-16T04:42:07.599 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.593+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=12,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=76} crc {mon_config_key=3667200259,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=2048278026,osd_snap=723727393,osdmap=4188604161}) 2024-09-16T04:42:07.599 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.593+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.599 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.593+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=12,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=77} crc {mon_config_key=3667200259,monmap=3088247691,osd_metadata=2543215506,osd_pg_creating=2048278026,osd_snap=723727393,osdmap=2011613217}) 2024-09-16T04:42:07.599 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.593+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=12,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=77} crc {mon_config_key=3667200259,monmap=3088247691,osd_metadata=2257658339,osd_pg_creating=2048278027,osd_snap=723727393,osdmap=2901842707}) 2024-09-16T04:42:07.599 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.595+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.599 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.595+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2058812002}) 2024-09-16T04:42:07.599 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.595+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1218601195}) 2024-09-16T04:42:07.599 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.595+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.600 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.595+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2058812002}) 2024-09-16T04:42:07.600 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.595+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3177824967}) 2024-09-16T04:42:07.600 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.597+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.600 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.597+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=33} crc {osdmap=691598388}) 2024-09-16T04:42:07.601 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.597+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=34} crc {osdmap=3933401031}) 2024-09-16T04:42:07.601 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.597+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:07.601 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.597+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=33} crc {osdmap=691598388}) 2024-09-16T04:42:07.601 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:07.597+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=30} crc {osdmap=2598737896}) 2024-09-16T04:42:07.602 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:42:08.578 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37324]: starting. 2024-09-16T04:42:08.578 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37324]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-37323 2024-09-16T04:42:08.578 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37324]: created object 0... 2024-09-16T04:42:08.578 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37324]: created object 25... 2024-09-16T04:42:08.579 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37324]: created object 49... 2024-09-16T04:42:08.579 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37324]: finishing. 2024-09-16T04:42:08.579 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37324]: shutting down. 2024-09-16T04:42:08.602 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:42:08.602 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:42:08.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.704+0000 7f1d5985e640 10 monclient: get_monmap_and_config 2024-09-16T04:42:08.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.705+0000 7f1d5985e640 10 monclient: build_initial_monmap 2024-09-16T04:42:08.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.705+0000 7f1d5985e640 10 monclient: monmap: 2024-09-16T04:42:08.706 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:08.707 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:08.707 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:08.705494+0000 2024-09-16T04:42:08.707 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:08.705494+0000 2024-09-16T04:42:08.707 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:08.707 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:08.707 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:08.707 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:08.707 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:08.707 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:08.707 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:08.707 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:08.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.705+0000 7f1d5985e640 1 Processor -- start 2024-09-16T04:42:08.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.705+0000 7f1d5985e640 1 -- start start 2024-09-16T04:42:08.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.705+0000 7f1d5985e640 10 monclient: init 2024-09-16T04:42:08.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d5985e640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:08.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d5985e640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-16T04:42:08.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d5985e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1d540a3c10 0x7f1d5409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:08.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d5985e640 10 monclient(hunting): picked mon.noname-a con 0x7f1d540a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:42:08.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d5985e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1d540a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:42:08.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d5985e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d5413be90 0x7f1d5413e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:08.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d5985e640 10 monclient(hunting): picked mon.noname-b con 0x7f1d5413be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:08.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d5985e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d5409b940 con 0x7f1d540a3ff0 2024-09-16T04:42:08.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d5985e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:08.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d5985e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f1d5409ab00 con 0x7f1d540a3c10 2024-09-16T04:42:08.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d5985e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:08.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d5985e640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f1d5409b030 con 0x7f1d5413be90 2024-09-16T04:42:08.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d5985e640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:08.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d5985e640 10 monclient(hunting): authenticate will time out at 927.121199s 2024-09-16T04:42:08.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d537fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d5413be90 0x7f1d5413e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:08.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d52ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1d540a3c10 0x7f1d5409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:08.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d527fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1d540a3ff0 0x7f1d540a43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51674/0 (socket says 172.21.15.46:51674) 2024-09-16T04:42:08.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d52ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1d540a3c10 0x7f1d5409bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49552/0 (socket says 172.21.15.46:49552) 2024-09-16T04:42:08.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d537fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d5413be90 0x7f1d5413e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39508/0 (socket says 172.21.15.46:39508) 2024-09-16T04:42:08.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d527fc640 1 -- 172.21.15.46:0/3445081712 learned_addr learned my addr 172.21.15.46:0/3445081712 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:42:08.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d52ffd640 10 monclient(hunting): get_auth_request con 0x7f1d540a3c10 auth_method 0 2024-09-16T04:42:08.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:08.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:08.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d537fe640 10 monclient(hunting): get_auth_request con 0x7f1d5413be90 auth_method 0 2024-09-16T04:42:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d51ffb640 1 -- 172.21.15.46:0/3445081712 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3961506921 0 0) 0x7f1d5409b940 con 0x7f1d540a3ff0 2024-09-16T04:42:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d51ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:42:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d51ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d51ffb640 10 monclient(hunting): my global_id is 44133 2024-09-16T04:42:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d51ffb640 1 -- 172.21.15.46:0/3445081712 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d40003520 con 0x7f1d540a3ff0 2024-09-16T04:42:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d52ffd640 10 monclient(hunting): handle_auth_done global_id 44130 payload 274 2024-09-16T04:42:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d52ffd640 1 -- 172.21.15.46:0/3445081712 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d5413be90 msgr2=0x7f1d5413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d52ffd640 1 --2- 172.21.15.46:0/3445081712 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d5413be90 0x7f1d5413e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d52ffd640 1 -- 172.21.15.46:0/3445081712 >> v1:172.21.15.46:6789/0 conn(0x7f1d540a3ff0 legacy=0x7f1d540a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d52ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:42:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d52ffd640 1 monclient: found mon.noname-a 2024-09-16T04:42:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d52ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d52ffd640 1 -- 172.21.15.46:0/3445081712 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d5409b310 con 0x7f1d540a3c10 2024-09-16T04:42:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.706+0000 7f1d537fe640 1 --2- 172.21.15.46:0/3445081712 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d5413be90 0x7f1d5413e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:42:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.707+0000 7f1d52ffd640 1 --2- 172.21.15.46:0/3445081712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d540a3c10 0x7f1d5409bcf0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c0097a0 tx=0x7f1d3c02a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8281424e5e4d65f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.707+0000 7f1d51ffb640 1 -- 172.21.15.46:0/3445081712 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c037070 con 0x7f1d540a3c10 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.707+0000 7f1d51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.707+0000 7f1d51ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.707+0000 7f1d51ffb640 10 monclient: dump: 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:08.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.707+0000 7f1d51ffb640 10 monclient: _finish_auth 0 2024-09-16T04:42:08.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.707+0000 7f1d51ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:42:08.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.707+0000 7f1d51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:08.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.707+0000 7f1d5985e640 5 monclient: authenticate success, global_id 44130 2024-09-16T04:42:08.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.707+0000 7f1d5985e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:08.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.707+0000 7f1d51ffb640 1 -- 172.21.15.46:0/3445081712 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d3c02adb0 con 0x7f1d540a3c10 2024-09-16T04:42:08.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.707+0000 7f1d51ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:08.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.707+0000 7f1d5985e640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:08.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.707+0000 7f1d5985e640 4 set_mon_vals no callback set 2024-09-16T04:42:08.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.707+0000 7f1d51ffb640 1 -- 172.21.15.46:0/3445081712 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c032660 con 0x7f1d540a3c10 2024-09-16T04:42:08.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.707+0000 7f1d51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:08.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.707+0000 7f1d51ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.707+0000 7f1d51ffb640 10 monclient: dump: 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.708+0000 7f1d5985e640 10 monclient: shutdown 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.708+0000 7f1d5985e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.708+0000 7f1d5985e640 1 -- 172.21.15.46:0/3445081712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d540a3c10 msgr2=0x7f1d5409bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.708+0000 7f1d5985e640 1 --2- 172.21.15.46:0/3445081712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d540a3c10 0x7f1d5409bcf0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c0097a0 tx=0x7f1d3c02a060 comp rx=0 tx=0).stop 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.708+0000 7f1d5985e640 1 -- 172.21.15.46:0/3445081712 shutdown_connections 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.708+0000 7f1d5985e640 1 --2- 172.21.15.46:0/3445081712 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d5413be90 0x7f1d5413e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.708+0000 7f1d5985e640 1 --2- 172.21.15.46:0/3445081712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d540a3c10 0x7f1d5409bcf0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.708+0000 7f1d5985e640 1 -- 172.21.15.46:0/3445081712 >> 172.21.15.46:0/3445081712 conn(0x7f1d540a6340 msgr2=0x7f1d5406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.708+0000 7f1d5985e640 1 -- 172.21.15.46:0/3445081712 shutdown_connections 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.708+0000 7f1d5985e640 1 -- 172.21.15.46:0/3445081712 wait complete. 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.708+0000 7f1d5985e640 10 monclient: build_initial_monmap 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.708+0000 7f1d5985e640 10 monclient: monmap: 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:08.724 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d5985e640 1 Processor -- start 2024-09-16T04:42:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d5985e640 1 -- start start 2024-09-16T04:42:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d5985e640 10 monclient: init 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d5985e640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d5985e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d5985e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d540a3c10 0x7f1d54133d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d5985e640 10 monclient(hunting): picked mon.noname-b con 0x7f1d540a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d5985e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d540a3ff0 0x7f1d541342c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d5985e640 10 monclient(hunting): picked mon.noname-a con 0x7f1d540a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d5985e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d5413be90 0x7f1d541cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d5985e640 10 monclient(hunting): picked mon.noname-c con 0x7f1d5413be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d5985e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d5985e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1d540a6b40 con 0x7f1d540a3ff0 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d5985e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d5985e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1d540a69c0 con 0x7f1d540a3c10 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d5985e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d5985e640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f1d5413e7c0 con 0x7f1d5413be90 2024-09-16T04:42:08.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d5985e640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:08.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d5985e640 10 monclient(hunting): authenticate will time out at 927.124805s 2024-09-16T04:42:08.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d527fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d540a3ff0 0x7f1d541342c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:08.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d537fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d5413be90 0x7f1d541cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:08.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d527fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d540a3ff0 0x7f1d541342c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49562/0 (socket says 172.21.15.46:49562) 2024-09-16T04:42:08.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d527fc640 1 -- 172.21.15.46:0/2016775109 learned_addr learned my addr 172.21.15.46:0/2016775109 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:08.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d527fc640 10 monclient(hunting): get_auth_request con 0x7f1d540a3ff0 auth_method 0 2024-09-16T04:42:08.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d537fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d5413be90 0x7f1d541cb9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39522/0 (socket says 172.21.15.46:39522) 2024-09-16T04:42:08.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:08.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:08.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:08.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d537fe640 10 monclient(hunting): get_auth_request con 0x7f1d5413be90 auth_method 0 2024-09-16T04:42:08.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:08.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.709+0000 7f1d537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d527fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d527fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d527fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d52ffd640 1 --2- 172.21.15.46:0/2016775109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d540a3c10 0x7f1d54133d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d52ffd640 10 monclient(hunting): get_auth_request con 0x7f1d540a3c10 auth_method 0 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d527fc640 10 monclient(hunting): handle_auth_done global_id 44136 payload 824 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d527fc640 1 -- 172.21.15.46:0/2016775109 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d5413be90 msgr2=0x7f1d541cb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d527fc640 1 --2- 172.21.15.46:0/2016775109 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d5413be90 0x7f1d541cb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d527fc640 1 -- 172.21.15.46:0/2016775109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d540a3c10 msgr2=0x7f1d54133d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d527fc640 1 --2- 172.21.15.46:0/2016775109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d540a3c10 0x7f1d54133d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d527fc640 10 monclient: _finish_hunting 0 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d527fc640 1 monclient: found mon.noname-a 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d527fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d527fc640 1 -- 172.21.15.46:0/2016775109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d54138e70 con 0x7f1d540a3ff0 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d527fc640 1 --2- 172.21.15.46:0/2016775109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d540a3ff0 0x7f1d541342c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1d4400c430 tx=0x7f1d4400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d4bfff640 1 -- 172.21.15.46:0/2016775109 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d44017070 con 0x7f1d540a3ff0 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d4bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d4bfff640 10 monclient: dump: 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d4bfff640 10 monclient: _finish_auth 0 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d4bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d4bfff640 1 -- 172.21.15.46:0/2016775109 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d44005ce0 con 0x7f1d540a3ff0 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d5985e640 5 monclient: authenticate success, global_id 44136 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d5985e640 10 monclient: _renew_subs 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d5985e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d5985e640 1 -- 172.21.15.46:0/2016775109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d5413c470 con 0x7f1d540a3ff0 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d4bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d4bfff640 1 -- 172.21.15.46:0/2016775109 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d440072f0 con 0x7f1d540a3ff0 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d4bfff640 10 monclient: dump: 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:08.731 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:08.731 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:08.731 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:08.731 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:08.731 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:08.731 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:08.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d5985e640 10 monclient: _renew_subs 2024-09-16T04:42:08.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d50ff9640 4 set_mon_vals no callback set 2024-09-16T04:42:08.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d5985e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:08.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.710+0000 7f1d5985e640 1 -- 172.21.15.46:0/2016775109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d5413a320 con 0x7f1d540a3ff0 2024-09-16T04:42:08.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.711+0000 7f1d4bfff640 1 -- 172.21.15.46:0/2016775109 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f1d4401a070 con 0x7f1d540a3ff0 2024-09-16T04:42:08.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.711+0000 7f1d5985e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:08.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.711+0000 7f1d5985e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:08.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.711+0000 7f1d5985e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:08.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.711+0000 7f1d5985e640 1 -- 172.21.15.46:0/2016775109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d5409bcf0 con 0x7f1d540a3ff0 2024-09-16T04:42:08.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.711+0000 7f1d4bfff640 1 --2- 172.21.15.46:0/2016775109 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1d340383b0 0x7f1d3403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:08.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.711+0000 7f1d4bfff640 1 -- 172.21.15.46:0/2016775109 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 4738+0+0 (secure 0 0 0) 0x7f1d4404ad10 con 0x7f1d540a3ff0 2024-09-16T04:42:08.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.711+0000 7f1d52ffd640 1 --2- 172.21.15.46:0/2016775109 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1d340383b0 0x7f1d3403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:08.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.711+0000 7f1d52ffd640 10 monclient: get_auth_request con 0x7f1d340383b0 auth_method 0 2024-09-16T04:42:08.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.712+0000 7f1d52ffd640 1 --2- 172.21.15.46:0/2016775109 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1d340383b0 0x7f1d3403a870 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c034000 tx=0x7f1d3c02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:08.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.716+0000 7f1d4bfff640 1 -- 172.21.15.46:0/2016775109 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1d44019e10 con 0x7f1d540a3ff0 2024-09-16T04:42:08.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.716+0000 7f1d4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:08.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.716+0000 7f1d4bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.961+0000 7f1d5985e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:42:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.961+0000 7f1d5985e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.961+0000 7f1d5985e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.961+0000 7f1d5985e640 1 -- 172.21.15.46:0/2016775109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1d54138bf0 con 0x7f1d540a3ff0 2024-09-16T04:42:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.962+0000 7f1d4bfff640 1 -- 172.21.15.46:0/2016775109 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f1d44010b90 con 0x7f1d540a3ff0 2024-09-16T04:42:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.962+0000 7f1d4bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.962+0000 7f1d4bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:08.964 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:42:08.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.964+0000 7f1d5985e640 1 -- 172.21.15.46:0/2016775109 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1d340383b0 msgr2=0x7f1d3403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:08.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.964+0000 7f1d5985e640 1 --2- 172.21.15.46:0/2016775109 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1d340383b0 0x7f1d3403a870 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c034000 tx=0x7f1d3c02e000 comp rx=0 tx=0).stop 2024-09-16T04:42:08.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.964+0000 7f1d5985e640 10 monclient: shutdown 2024-09-16T04:42:08.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.964+0000 7f1d5985e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:08.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.964+0000 7f1d5985e640 1 -- 172.21.15.46:0/2016775109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d540a3ff0 msgr2=0x7f1d541342c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:08.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.964+0000 7f1d5985e640 1 --2- 172.21.15.46:0/2016775109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d540a3ff0 0x7f1d541342c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1d4400c430 tx=0x7f1d4400c8f0 comp rx=0 tx=0).stop 2024-09-16T04:42:08.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.964+0000 7f1d5985e640 1 -- 172.21.15.46:0/2016775109 shutdown_connections 2024-09-16T04:42:08.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.964+0000 7f1d5985e640 1 --2- 172.21.15.46:0/2016775109 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1d5413be90 0x7f1d541cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:08.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.964+0000 7f1d5985e640 1 --2- 172.21.15.46:0/2016775109 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1d340383b0 0x7f1d3403a870 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:08.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.964+0000 7f1d5985e640 1 --2- 172.21.15.46:0/2016775109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d540a3ff0 0x7f1d541342c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:08.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.964+0000 7f1d5985e640 1 --2- 172.21.15.46:0/2016775109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d540a3c10 0x7f1d54133d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:08.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.964+0000 7f1d5985e640 1 -- 172.21.15.46:0/2016775109 >> 172.21.15.46:0/2016775109 conn(0x7f1d540a6340 msgr2=0x7f1d540a7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:08.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.964+0000 7f1d5985e640 1 -- 172.21.15.46:0/2016775109 shutdown_connections 2024-09-16T04:42:08.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:08.964+0000 7f1d5985e640 1 -- 172.21.15.46:0/2016775109 wait complete. 2024-09-16T04:42:08.976 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:42:08.976 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:42:08.976 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:42:08.976 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:42:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.077+0000 7fc006cd0640 10 monclient: get_monmap_and_config 2024-09-16T04:42:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.077+0000 7fc006cd0640 10 monclient: build_initial_monmap 2024-09-16T04:42:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.077+0000 7fc006cd0640 10 monclient: monmap: 2024-09-16T04:42:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:09.077700+0000 2024-09-16T04:42:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:09.077700+0000 2024-09-16T04:42:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:09.079 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.077+0000 7fc006cd0640 1 Processor -- start 2024-09-16T04:42:09.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc006cd0640 1 -- start start 2024-09-16T04:42:09.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc006cd0640 10 monclient: init 2024-09-16T04:42:09.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc006cd0640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:09.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc006cd0640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-16T04:42:09.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc006cd0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc0000a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:42:09.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc006cd0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc0000a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:42:09.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc006cd0640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc00013cea0 0x7fc00013f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc006cd0640 10 monclient(hunting): picked mon.noname-b con 0x7fc00013cea0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc006cd0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc00009ab70 con 0x7fc0000a3ff0 2024-09-16T04:42:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc006cd0640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0000a4d60 con 0x7fc0000a3c10 2024-09-16T04:42:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc006cd0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc006cd0640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fc00009b030 con 0x7fc00013cea0 2024-09-16T04:42:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc006cd0640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc006cd0640 10 monclient(hunting): authenticate will time out at 927.493630s 2024-09-16T04:42:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc00526d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc00013cea0 0x7fc00013f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fbffffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc0000a3ff0 0x7fc0000a4400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51676/0 (socket says 172.21.15.46:51676) 2024-09-16T04:42:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fbffffff640 1 -- 172.21.15.46:0/2849511795 learned_addr learned my addr 172.21.15.46:0/2849511795 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:42:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc00526d640 10 monclient(hunting): get_auth_request con 0x7fc00013cea0 auth_method 0 2024-09-16T04:42:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc00526d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc00526d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc00526d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc00526d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc00526d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.078+0000 7fc00526d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:09.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fc00526d640 10 monclient(hunting): handle_auth_done global_id 24578 payload 274 2024-09-16T04:42:09.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fc00526d640 1 -- 172.21.15.46:0/2849511795 >> v1:172.21.15.114:6789/0 conn(0x7fc0000a3c10 legacy=0x7fc00009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fbfff7fe640 1 -- 172.21.15.46:0/2849511795 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2345092699 0 0) 0x7fc00009ab70 con 0x7fc0000a3ff0 2024-09-16T04:42:09.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fc00526d640 1 -- 172.21.15.46:0/2849511795 >> v1:172.21.15.46:6789/0 conn(0x7fc0000a3ff0 legacy=0x7fc0000a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fc00526d640 10 monclient: _finish_hunting 0 2024-09-16T04:42:09.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fc00526d640 1 monclient: found mon.noname-b 2024-09-16T04:42:09.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fc00526d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:09.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fc00526d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:42:09.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fc00526d640 1 -- 172.21.15.46:0/2849511795 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00009b310 con 0x7fc00013cea0 2024-09-16T04:42:09.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fbfff7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:42:09.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fc00526d640 1 --2- 172.21.15.46:0/2849511795 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc00013cea0 0x7fc00013f290 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbff8009dc0 tx=0x7fbff802a5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0a678ab1fe11468 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:09.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fbfff7fe640 1 -- 172.21.15.46:0/2849511795 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8007740 con 0x7fc00013cea0 2024-09-16T04:42:09.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fbfff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fbfff7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:42:09.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fbfff7fe640 10 monclient: dump: 2024-09-16T04:42:09.091 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.091 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.091 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.091 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.091 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.091 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.091 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.091 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.091 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.092 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fbfff7fe640 10 monclient: _finish_auth 0 2024-09-16T04:42:09.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fbfff7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:42:09.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fbfff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:09.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fbfff7fe640 1 -- 172.21.15.46:0/2849511795 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbff80078e0 con 0x7fc00013cea0 2024-09-16T04:42:09.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fc006cd0640 5 monclient: authenticate success, global_id 24578 2024-09-16T04:42:09.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fc006cd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:09.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fbfff7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:09.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fc006cd0640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:09.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fc006cd0640 4 set_mon_vals no callback set 2024-09-16T04:42:09.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fbfff7fe640 1 -- 172.21.15.46:0/2849511795 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8007cb0 con 0x7fc00013cea0 2024-09-16T04:42:09.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fbfff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fbfff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:09.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.079+0000 7fbfff7fe640 10 monclient: dump: 2024-09-16T04:42:09.093 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.093 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.093 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.093 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.093 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.093 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.093 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.093 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.093 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.093 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.080+0000 7fc006cd0640 10 monclient: shutdown 2024-09-16T04:42:09.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.080+0000 7fc006cd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:09.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.080+0000 7fc006cd0640 1 -- 172.21.15.46:0/2849511795 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc00013cea0 msgr2=0x7fc00013f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.080+0000 7fc006cd0640 1 --2- 172.21.15.46:0/2849511795 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc00013cea0 0x7fc00013f290 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbff8009dc0 tx=0x7fbff802a5b0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.080+0000 7fc006cd0640 1 -- 172.21.15.46:0/2849511795 shutdown_connections 2024-09-16T04:42:09.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.080+0000 7fc006cd0640 1 --2- 172.21.15.46:0/2849511795 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc00013cea0 0x7fc00013f290 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.080+0000 7fc006cd0640 1 -- 172.21.15.46:0/2849511795 >> 172.21.15.46:0/2849511795 conn(0x7fc0000a5d10 msgr2=0x7fc0000a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:09.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.080+0000 7fc006cd0640 1 -- 172.21.15.46:0/2849511795 shutdown_connections 2024-09-16T04:42:09.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.080+0000 7fc006cd0640 1 -- 172.21.15.46:0/2849511795 wait complete. 2024-09-16T04:42:09.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.081+0000 7fc006cd0640 10 monclient: build_initial_monmap 2024-09-16T04:42:09.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.081+0000 7fc006cd0640 10 monclient: monmap: 2024-09-16T04:42:09.094 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:09.094 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:09.094 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:09.094 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:09.094 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:09.094 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.094 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:09.095 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:09.095 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:09.095 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.081+0000 7fc006cd0640 1 Processor -- start 2024-09-16T04:42:09.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.081+0000 7fc006cd0640 1 -- start start 2024-09-16T04:42:09.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.081+0000 7fc006cd0640 10 monclient: init 2024-09-16T04:42:09.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.081+0000 7fc006cd0640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:09.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.081+0000 7fc006cd0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:42:09.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc006cd0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0000a3c10 0x7fc000136b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc006cd0640 10 monclient(hunting): picked mon.noname-a con 0x7fc0000a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:09.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc006cd0640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc0000a3ff0 0x7fc000139cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc006cd0640 10 monclient(hunting): picked mon.noname-c con 0x7fc0000a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:09.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc006cd0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc00013cea0 0x7fc00013a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc006cd0640 10 monclient(hunting): picked mon.noname-b con 0x7fc00013cea0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:09.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc006cd0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc006cd0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc00013f950 con 0x7fc0000a3c10 2024-09-16T04:42:09.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc006cd0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc006cd0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc00013f7d0 con 0x7fc00013cea0 2024-09-16T04:42:09.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc006cd0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc006cd0640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fc00013fad0 con 0x7fc0000a3ff0 2024-09-16T04:42:09.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc006cd0640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:09.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc006cd0640 10 monclient(hunting): authenticate will time out at 927.497089s 2024-09-16T04:42:09.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc004a6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0000a3c10 0x7fc000136b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc0000a3ff0 0x7fc000139cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc004a6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0000a3c10 0x7fc000136b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49574/0 (socket says 172.21.15.46:49574) 2024-09-16T04:42:09.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc0000a3ff0 0x7fc000139cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39546/0 (socket says 172.21.15.46:39546) 2024-09-16T04:42:09.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc004a6c640 1 -- 172.21.15.46:0/4159893128 learned_addr learned my addr 172.21.15.46:0/4159893128 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:09.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc004a6c640 10 monclient(hunting): get_auth_request con 0x7fc0000a3c10 auth_method 0 2024-09-16T04:42:09.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc004a6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc004a6c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc004a6c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffffff640 10 monclient(hunting): get_auth_request con 0x7fc0000a3ff0 auth_method 0 2024-09-16T04:42:09.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc004a6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc004a6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc004a6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:09.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:09.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc00526d640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc00013cea0 0x7fc00013a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc00526d640 10 monclient(hunting): get_auth_request con 0x7fc00013cea0 auth_method 0 2024-09-16T04:42:09.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc00526d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc00526d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc00526d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc004a6c640 10 monclient(hunting): handle_auth_done global_id 44142 payload 824 2024-09-16T04:42:09.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc004a6c640 1 -- 172.21.15.46:0/4159893128 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc0000a3ff0 msgr2=0x7fc000139cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc004a6c640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc0000a3ff0 0x7fc000139cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc004a6c640 1 -- 172.21.15.46:0/4159893128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc00013cea0 msgr2=0x7fc00013a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc004a6c640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc00013cea0 0x7fc00013a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc004a6c640 10 monclient: _finish_hunting 0 2024-09-16T04:42:09.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc004a6c640 1 monclient: found mon.noname-a 2024-09-16T04:42:09.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc004a6c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:09.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc004a6c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:09.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc004a6c640 1 -- 172.21.15.46:0/4159893128 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00011ffc0 con 0x7fc0000a3c10 2024-09-16T04:42:09.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc00526d640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc00013cea0 0x7fc00013a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:42:09.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffffff640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc0000a3ff0 0x7fc000139cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:42:09.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc004a6c640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0000a3c10 0x7fc000136b30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800c450 tx=0x7fbfe800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:09.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffd7fa640 1 -- 172.21.15.46:0/4159893128 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe80052d0 con 0x7fc0000a3c10 2024-09-16T04:42:09.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:09.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffd7fa640 10 monclient: dump: 2024-09-16T04:42:09.099 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.099 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.099 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.099 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.099 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.100 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.100 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.100 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.100 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.100 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffd7fa640 10 monclient: _finish_auth 0 2024-09-16T04:42:09.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffd7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:42:09.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:09.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffd7fa640 1 -- 172.21.15.46:0/4159893128 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfe8005470 con 0x7fc0000a3c10 2024-09-16T04:42:09.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc006cd0640 5 monclient: authenticate success, global_id 44142 2024-09-16T04:42:09.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc006cd0640 10 monclient: _renew_subs 2024-09-16T04:42:09.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc006cd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:09.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fc006cd0640 1 -- 172.21.15.46:0/4159893128 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc00013d280 con 0x7fc0000a3c10 2024-09-16T04:42:09.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:09.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffd7fa640 1 -- 172.21.15.46:0/4159893128 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8012870 con 0x7fc0000a3c10 2024-09-16T04:42:09.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:09.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffd7fa640 10 monclient: dump: 2024-09-16T04:42:09.101 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.101 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.101 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.101 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.101 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.101 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.101 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.101 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.101 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.102 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.082+0000 7fbffeffd640 4 set_mon_vals no callback set 2024-09-16T04:42:09.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.083+0000 7fc006cd0640 10 monclient: _renew_subs 2024-09-16T04:42:09.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.083+0000 7fc006cd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:09.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.083+0000 7fc006cd0640 1 -- 172.21.15.46:0/4159893128 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0001202a0 con 0x7fc0000a3c10 2024-09-16T04:42:09.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.083+0000 7fbffd7fa640 1 -- 172.21.15.46:0/4159893128 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fbfe8012a10 con 0x7fc0000a3c10 2024-09-16T04:42:09.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.083+0000 7fc006cd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:09.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.083+0000 7fc006cd0640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc000137070 0x7fc00002c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.083+0000 7fc006cd0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.083+0000 7fc006cd0640 1 -- 172.21.15.46:0/4159893128 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc0000a5730 con 0x7fc000137070 2024-09-16T04:42:09.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.083+0000 7fbffffff640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc000137070 0x7fc00002c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.083+0000 7fbffffff640 10 monclient: get_auth_request con 0x7fc000137070 auth_method 0 2024-09-16T04:42:09.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.083+0000 7fbffffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.083+0000 7fbffffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.083+0000 7fbffffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.083+0000 7fbffffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.083+0000 7fbffffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.083+0000 7fbffd7fa640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fbfe00383e0 0x7fbfe003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.083+0000 7fbffffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:09.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.084+0000 7fbffd7fa640 1 -- 172.21.15.46:0/4159893128 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 4738+0+0 (secure 0 0 0) 0x7fbfe804b320 con 0x7fc0000a3c10 2024-09-16T04:42:09.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.084+0000 7fbffffff640 10 monclient(hunting): handle_auth_done global_id 44145 payload 824 2024-09-16T04:42:09.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.084+0000 7fbffffff640 1 -- 172.21.15.46:0/4159893128 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc00009bcf0 con 0x7fc000137070 2024-09-16T04:42:09.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.084+0000 7fbffffff640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc000137070 0x7fc00002c640 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc0001201d0 tx=0x7fbff0001d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:09.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.084+0000 7fc00526d640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fbfe00383e0 0x7fbfe003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.084+0000 7fc00526d640 10 monclient: get_auth_request con 0x7fbfe00383e0 auth_method 0 2024-09-16T04:42:09.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.084+0000 7fbffd7fa640 1 -- 172.21.15.46:0/4159893128 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff000fa40 con 0x7fc000137070 2024-09-16T04:42:09.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.084+0000 7fbffd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.084+0000 7fbffd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:09.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.084+0000 7fbffd7fa640 10 monclient: dump: 2024-09-16T04:42:09.104 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.104 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.104 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.104 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.104 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.104 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.104 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.104 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.104 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.104 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.084+0000 7fbffd7fa640 1 -- 172.21.15.46:0/4159893128 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbff0018070 con 0x7fc000137070 2024-09-16T04:42:09.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.084+0000 7fbffd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:09.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.084+0000 7fbffd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:09.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.084+0000 7fbffd7fa640 1 -- 172.21.15.46:0/4159893128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc000137070 msgr2=0x7fc00002c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.084+0000 7fbffd7fa640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc000137070 0x7fc00002c640 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc0001201d0 tx=0x7fbff0001d20 comp rx=0 tx=0).stop 2024-09-16T04:42:09.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.084+0000 7fbffd7fa640 1 -- 172.21.15.46:0/4159893128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc000137070 msgr2=0x7fc00002c640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:09.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.084+0000 7fbffd7fa640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc000137070 0x7fc00002c640 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.084+0000 7fc00526d640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fbfe00383e0 0x7fbfe003a8a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbff8034000 tx=0x7fbff802f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:09.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.090+0000 7fc006cd0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:42:09.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.090+0000 7fc006cd0640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc00002cb80 0x7fc00002cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.090+0000 7fc006cd0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.090+0000 7fc006cd0640 1 -- 172.21.15.46:0/4159893128 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc00009e800 con 0x7fc00002cb80 2024-09-16T04:42:09.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffffff640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc00002cb80 0x7fc00002cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffffff640 10 monclient: get_auth_request con 0x7fc00002cb80 auth_method 0 2024-09-16T04:42:09.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:09.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffffff640 10 monclient(hunting): handle_auth_done global_id 44148 payload 824 2024-09-16T04:42:09.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffffff640 1 -- 172.21.15.46:0/4159893128 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc000069150 con 0x7fc00002cb80 2024-09-16T04:42:09.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffffff640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc00002cb80 0x7fc00002cf90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc0000a3220 tx=0x7fbff0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:09.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffd7fa640 1 -- 172.21.15.46:0/4159893128 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff0013950 con 0x7fc00002cb80 2024-09-16T04:42:09.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:09.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffd7fa640 10 monclient: dump: 2024-09-16T04:42:09.108 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.108 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.108 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.108 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.108 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.108 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.108 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.108 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.108 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.108 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffd7fa640 1 -- 172.21.15.46:0/4159893128 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7fbff0013950 con 0x7fc00002cb80 2024-09-16T04:42:09.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:42:09.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:09.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffd7fa640 1 -- 172.21.15.46:0/4159893128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc00002cb80 msgr2=0x7fc00002cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffd7fa640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc00002cb80 0x7fc00002cf90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc0000a3220 tx=0x7fbff0000ed0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffd7fa640 1 -- 172.21.15.46:0/4159893128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc00002cb80 msgr2=0x7fc00002cf90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:09.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.091+0000 7fbffd7fa640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc00002cb80 0x7fc00002cf90 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.109 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:42:09.109 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:42:09.109 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:42:09.110 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:42:09.110 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 38, 2024-09-16T04:42:09.110 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 8838, 2024-09-16T04:42:09.110 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:42:09.110 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:42:09.110 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:42:09.110 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:42:09.110 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:09.110 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 2, 2024-09-16T04:42:09.110 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:09.110 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:42:09.110 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:42:09.110 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:09.110 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:09.111 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:09.111 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:09.111 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:09.111 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:09.111 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:09.111 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:09.111 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:09.111 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:09.111 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:09.111 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:09.111 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:42:09.111 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:42:09.111 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:09.112 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:09.112 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:09.112 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:09.112 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:09.112 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:09.112 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:09.112 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:09.112 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:09.112 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:09.112 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:09.112 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:09.112 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.112 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:42:09.113 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:42:09.113 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:42:09.113 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:42:09.113 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:42:09.113 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:42:09.113 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:09.113 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:09.113 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:42:09.113 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:42:09.113 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:42:09.113 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:42:09.113 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:42:09.113 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:42:09.114 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:42:09.114 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:09.114 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:42:09.114 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:09.114 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:09.114 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:09.114 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:09.114 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:09.114 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:09.114 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:09.114 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:09.114 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:09.114 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:09.114 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:09.115 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:09.115 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:42:09.115 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.115 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:42:09.115 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.115 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:42:09.115 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:42:09.115 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:09.115 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:09.115 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.115 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:09.115 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:42:09.115 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:09.115 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.116 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.116 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:09.116 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:42:09.116 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:09.116 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:09.116 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:09.116 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.116 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:42:09.116 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:42:09.116 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:09.116 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:09.116 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:09.116 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.116 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.117 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:42:09.117 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:42:09.117 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:09.117 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:09.117 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.117 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:09.117 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:42:09.117 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:09.117 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.117 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.117 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:09.117 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:42:09.117 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:09.118 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:09.118 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:09.118 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.118 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:42:09.118 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:42:09.118 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:09.118 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:09.118 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:09.118 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.118 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.118 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:42:09.118 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:42:09.118 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:09.118 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:09.118 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.119 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:09.119 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:42:09.119 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:09.119 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.119 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.119 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:09.119 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:42:09.119 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:09.119 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:09.119 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:09.119 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.119 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:42:09.119 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:42:09.119 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:09.120 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:09.120 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:09.120 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:09.120 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:09.120 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.120 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:42:09.120 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:42:09.120 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.120 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:09.120 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:09.120 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:42:09.120 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:09.120 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:09.120 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:42:09.121 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.121 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:09.121 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:09.121 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:42:09.121 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:09.121 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:09.121 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.121 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:42:09.121 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:42:09.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.092+0000 7fc006cd0640 1 -- 172.21.15.46:0/4159893128 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fbfe00383e0 msgr2=0x7fbfe003a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.092+0000 7fc006cd0640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fbfe00383e0 0x7fbfe003a8a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbff8034000 tx=0x7fbff802f000 comp rx=0 tx=0).stop 2024-09-16T04:42:09.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.092+0000 7fc006cd0640 10 monclient: shutdown 2024-09-16T04:42:09.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.092+0000 7fc006cd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:09.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.092+0000 7fc006cd0640 1 -- 172.21.15.46:0/4159893128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0000a3c10 msgr2=0x7fc000136b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.092+0000 7fc006cd0640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0000a3c10 0x7fc000136b30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800c450 tx=0x7fbfe800c910 comp rx=0 tx=0).stop 2024-09-16T04:42:09.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.092+0000 7fc00526d640 1 -- 172.21.15.46:0/4159893128 reap_dead start 2024-09-16T04:42:09.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.092+0000 7fc006cd0640 1 -- 172.21.15.46:0/4159893128 shutdown_connections 2024-09-16T04:42:09.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.092+0000 7fc006cd0640 1 --2- 172.21.15.46:0/4159893128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0000a3c10 0x7fc000136b30 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.092+0000 7fc006cd0640 1 -- 172.21.15.46:0/4159893128 >> 172.21.15.46:0/4159893128 conn(0x7fc0000a5d10 msgr2=0x7fc0000a2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:09.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.092+0000 7fc006cd0640 1 -- 172.21.15.46:0/4159893128 shutdown_connections 2024-09-16T04:42:09.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.092+0000 7fc006cd0640 1 -- 172.21.15.46:0/4159893128 wait complete. 2024-09-16T04:42:09.124 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:42:09.124 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:42:09.124 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:42:09.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.225+0000 7f90ccb49640 10 monclient: get_monmap_and_config 2024-09-16T04:42:09.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.225+0000 7f90ccb49640 10 monclient: build_initial_monmap 2024-09-16T04:42:09.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.225+0000 7f90ccb49640 10 monclient: monmap: 2024-09-16T04:42:09.226 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:09.226 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.226 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:09.225747+0000 2024-09-16T04:42:09.226 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:09.225747+0000 2024-09-16T04:42:09.227 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:09.227 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.227 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:09.227 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:09.227 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:09.227 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:09.227 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:09.227 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.225+0000 7f90ccb49640 1 Processor -- start 2024-09-16T04:42:09.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.225+0000 7f90ccb49640 1 -- start start 2024-09-16T04:42:09.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.225+0000 7f90ccb49640 10 monclient: init 2024-09-16T04:42:09.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90ccb49640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:09.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90ccb49640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-16T04:42:09.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90ccb49640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f90c80a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:42:09.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90ccb49640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f90c80a3ff0 0x7f90c80a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90ccb49640 10 monclient(hunting): picked mon.noname-c con 0x7f90c80a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:42:09.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90ccb49640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f90c813be90 addr v1:172.21.15.114:6789/0 2024-09-16T04:42:09.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90ccb49640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90c809b940 con 0x7f90c80a3c10 2024-09-16T04:42:09.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90ccb49640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90c809ab00 con 0x7f90c813be90 2024-09-16T04:42:09.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90ccb49640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90ccb49640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f90c809b030 con 0x7f90c80a3ff0 2024-09-16T04:42:09.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90ccb49640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:09.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90ccb49640 10 monclient(hunting): authenticate will time out at 927.641564s 2024-09-16T04:42:09.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90c659c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f90c80a3c10 0x7f90c809bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51690/0 (socket says 172.21.15.46:51690) 2024-09-16T04:42:09.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90c659c640 1 -- 172.21.15.46:0/2975695737 learned_addr learned my addr 172.21.15.46:0/2975695737 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:42:09.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90c5d9b640 1 --2- 172.21.15.46:0/2975695737 >> v2:172.21.15.114:3300/0 conn(0x7f90c80a3ff0 0x7f90c80a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90c559a640 1 -- 172.21.15.46:0/2975695737 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1225853938 0 0) 0x7f90c809b940 con 0x7f90c80a3c10 2024-09-16T04:42:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90c559a640 10 monclient(hunting): _negotiate 2024-09-16T04:42:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90c559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90c559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90c559a640 10 monclient(hunting): my global_id is 44151 2024-09-16T04:42:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90c559a640 1 -- 172.21.15.46:0/2975695737 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90b0003520 con 0x7f90c80a3c10 2024-09-16T04:42:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90c5d9b640 10 monclient(hunting): get_auth_request con 0x7f90c80a3ff0 auth_method 0 2024-09-16T04:42:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.226+0000 7f90c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:09.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 1 -- 172.21.15.46:0/2975695737 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3294736733 0 0) 0x7f90c809ab00 con 0x7f90c813be90 2024-09-16T04:42:09.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 10 monclient(hunting): _negotiate 2024-09-16T04:42:09.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 10 monclient(hunting): my global_id is 5443 2024-09-16T04:42:09.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 1 -- 172.21.15.46:0/2975695737 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90c809b940 con 0x7f90c813be90 2024-09-16T04:42:09.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 1 -- 172.21.15.46:0/2975695737 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3891512283 0 0) 0x7f90b0003520 con 0x7f90c80a3c10 2024-09-16T04:42:09.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 1 -- 172.21.15.46:0/2975695737 >> v2:172.21.15.114:3300/0 conn(0x7f90c80a3ff0 msgr2=0x7f90c80a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:09.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 1 --2- 172.21.15.46:0/2975695737 >> v2:172.21.15.114:3300/0 conn(0x7f90c80a3ff0 0x7f90c80a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 1 -- 172.21.15.46:0/2975695737 >> v1:172.21.15.114:6789/0 conn(0x7f90c813be90 legacy=0x7f90c813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 10 monclient: _finish_hunting 0 2024-09-16T04:42:09.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:42:09.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:09.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:42:09.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 1 -- 172.21.15.46:0/2975695737 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90c809b310 con 0x7f90c80a3c10 2024-09-16T04:42:09.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 10 monclient: _finish_auth 0 2024-09-16T04:42:09.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 10 monclient: _check_auth_tickets 2024-09-16T04:42:09.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:09.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90ccb49640 5 monclient: authenticate success, global_id 44151 2024-09-16T04:42:09.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 1 -- 172.21.15.46:0/2975695737 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f90b8001670 con 0x7f90c80a3c10 2024-09-16T04:42:09.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90ccb49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:09.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:42:09.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 10 monclient: dump: 2024-09-16T04:42:09.249 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.249 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.249 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.249 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.249 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.249 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.249 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.250 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.250 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.250 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:42:09.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:09.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 1 -- 172.21.15.46:0/2975695737 >> v1:172.21.15.46:6789/0 conn(0x7f90c80a3c10 legacy=0x7f90c809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:42:09.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 1 --2- 172.21.15.46:0/2975695737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90b0003b90 0x7f90b0003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 10 monclient(hunting): picked mon.b con 0x7f90b0003b90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:09.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 1 --2- 172.21.15.46:0/2975695737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90b0004800 0x7f90b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 10 monclient(hunting): picked mon.a con 0x7f90b0004800 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:09.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 1 --2- 172.21.15.46:0/2975695737 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90b0005210 0x7f90b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 10 monclient(hunting): picked mon.c con 0x7f90b0005210 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:09.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 1 -- 172.21.15.46:0/2975695737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f90c809b030 con 0x7f90b0004800 2024-09-16T04:42:09.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 1 -- 172.21.15.46:0/2975695737 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f90b000c140 con 0x7f90b0003b90 2024-09-16T04:42:09.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 1 -- 172.21.15.46:0/2975695737 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f90b000c340 con 0x7f90b0005210 2024-09-16T04:42:09.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:09.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c559a640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:42:09.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90ccb49640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:09.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 1 --2- 172.21.15.46:0/2975695737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90b0004800 0x7f90b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 10 monclient(hunting): get_auth_request con 0x7f90b0004800 auth_method 0 2024-09-16T04:42:09.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:09.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c659c640 1 --2- 172.21.15.46:0/2975695737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90b0003b90 0x7f90b0003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c6d9d640 1 --2- 172.21.15.46:0/2975695737 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90b0005210 0x7f90b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c659c640 10 monclient(hunting): get_auth_request con 0x7f90b0003b90 auth_method 0 2024-09-16T04:42:09.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:09.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c6d9d640 10 monclient(hunting): get_auth_request con 0x7f90b0005210 auth_method 0 2024-09-16T04:42:09.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:09.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 10 monclient(hunting): handle_auth_done global_id 44151 payload 293 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 1 -- 172.21.15.46:0/2975695737 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90b0005210 msgr2=0x7f90b000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 1 --2- 172.21.15.46:0/2975695737 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90b0005210 0x7f90b000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 1 -- 172.21.15.46:0/2975695737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90b0003b90 msgr2=0x7f90b0003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 1 --2- 172.21.15.46:0/2975695737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90b0003b90 0x7f90b0003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 1 monclient: found mon.a 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 1 -- 172.21.15.46:0/2975695737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b0003520 con 0x7f90b0004800 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c6d9d640 1 -- 172.21.15.46:0/2975695737 reap_dead start 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 10 monclient: _finish_auth 0 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.227+0000 7f90c5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.228+0000 7f90c5d9b640 1 --2- 172.21.15.46:0/2975695737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90b0004800 0x7f90b0004c80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f90bc009ac0 tx=0x7f90b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1759eece45330ccc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.228+0000 7f90c559a640 1 -- 172.21.15.46:0/2975695737 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc036070 con 0x7f90b0004800 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.228+0000 7f90c559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.228+0000 7f90c559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.228+0000 7f90c559a640 10 monclient: dump: 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.228+0000 7f90ccb49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.228+0000 7f90c559a640 1 -- 172.21.15.46:0/2975695737 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90bc031400 con 0x7f90b0004800 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.228+0000 7f90c559a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.228+0000 7f90ccb49640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.228+0000 7f90ccb49640 4 set_mon_vals no callback set 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.228+0000 7f90c559a640 1 -- 172.21.15.46:0/2975695737 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc009690 con 0x7f90b0004800 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.228+0000 7f90c559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.228+0000 7f90c559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.228+0000 7f90c559a640 10 monclient: dump: 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.229+0000 7f90ccb49640 10 monclient: shutdown 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.229+0000 7f90ccb49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.229+0000 7f90ccb49640 1 -- 172.21.15.46:0/2975695737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90b0004800 msgr2=0x7f90b0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.229+0000 7f90ccb49640 1 --2- 172.21.15.46:0/2975695737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90b0004800 0x7f90b0004c80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f90bc009ac0 tx=0x7f90b0005830 comp rx=0 tx=0).stop 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.229+0000 7f90ccb49640 1 -- 172.21.15.46:0/2975695737 shutdown_connections 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.229+0000 7f90ccb49640 1 --2- 172.21.15.46:0/2975695737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90b0004800 0x7f90b0004c80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.229+0000 7f90ccb49640 1 -- 172.21.15.46:0/2975695737 >> 172.21.15.46:0/2975695737 conn(0x7f90c80a6340 msgr2=0x7f90c806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.229+0000 7f90ccb49640 1 -- 172.21.15.46:0/2975695737 shutdown_connections 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.229+0000 7f90ccb49640 1 -- 172.21.15.46:0/2975695737 wait complete. 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90ccb49640 10 monclient: build_initial_monmap 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90ccb49640 10 monclient: monmap: 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90ccb49640 1 Processor -- start 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90ccb49640 1 -- start start 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90ccb49640 10 monclient: init 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90ccb49640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90ccb49640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90ccb49640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90b0004800 0x7f90c81cc030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90ccb49640 10 monclient(hunting): picked mon.noname-b con 0x7f90b0004800 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90ccb49640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90c81cc570 0x7f90c81cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90ccb49640 10 monclient(hunting): picked mon.noname-c con 0x7f90c81cc570 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90ccb49640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90c811fce0 0x7f90c8120160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90ccb49640 10 monclient(hunting): picked mon.noname-a con 0x7f90c811fce0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90ccb49640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90ccb49640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f90c809be70 con 0x7f90c811fce0 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90ccb49640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90ccb49640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f90c809bcf0 con 0x7f90b0004800 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90ccb49640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90ccb49640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f90c809bff0 con 0x7f90c81cc570 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90ccb49640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90ccb49640 10 monclient(hunting): authenticate will time out at 927.645879s 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90c5d9b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90c81cc570 0x7f90c81cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90c6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90c811fce0 0x7f90c8120160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90c5d9b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90c81cc570 0x7f90c81cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39570/0 (socket says 172.21.15.46:39570) 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c6d9d640 10 monclient(hunting): get_auth_request con 0x7f90c811fce0 auth_method 0 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.230+0000 7f90c5d9b640 1 -- 172.21.15.46:0/1666859688 learned_addr learned my addr 172.21.15.46:0/1666859688 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c5d9b640 10 monclient(hunting): get_auth_request con 0x7f90c81cc570 auth_method 0 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c659c640 1 --2- 172.21.15.46:0/1666859688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90b0004800 0x7f90c81cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c659c640 10 monclient(hunting): get_auth_request con 0x7f90b0004800 auth_method 0 2024-09-16T04:42:09.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c5d9b640 10 monclient(hunting): handle_auth_done global_id 24584 payload 824 2024-09-16T04:42:09.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c5d9b640 1 -- 172.21.15.46:0/1666859688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90b0004800 msgr2=0x7f90c81cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c5d9b640 1 --2- 172.21.15.46:0/1666859688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90b0004800 0x7f90c81cc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c5d9b640 1 -- 172.21.15.46:0/1666859688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90c811fce0 msgr2=0x7f90c8120160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c5d9b640 1 --2- 172.21.15.46:0/1666859688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90c811fce0 0x7f90c8120160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c5d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c5d9b640 1 monclient: found mon.noname-c 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c5d9b640 1 -- 172.21.15.46:0/1666859688 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90c8122260 con 0x7f90c81cc570 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c659c640 1 --2- 172.21.15.46:0/1666859688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90b0004800 0x7f90c81cc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c6d9d640 1 --2- 172.21.15.46:0/1666859688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90c811fce0 0x7f90c8120160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90c5d9b640 1 --2- 172.21.15.46:0/1666859688 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90c81cc570 0x7f90c81cc9d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f90bc034040 tx=0x7f90bc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90b6ffd640 1 -- 172.21.15.46:0/1666859688 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc03f020 con 0x7f90c81cc570 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90b6ffd640 10 monclient: dump: 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90b6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90b6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90b6ffd640 1 -- 172.21.15.46:0/1666859688 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90bc037040 con 0x7f90c81cc570 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90ccb49640 5 monclient: authenticate success, global_id 24584 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90ccb49640 10 monclient: _renew_subs 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90ccb49640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90ccb49640 1 -- 172.21.15.46:0/1666859688 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90c80a28a0 con 0x7f90c81cc570 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90b6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90b6ffd640 1 -- 172.21.15.46:0/1666859688 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc036070 con 0x7f90c81cc570 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90b6ffd640 10 monclient: dump: 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90ccb49640 10 monclient: _renew_subs 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90b7fff640 4 set_mon_vals no callback set 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90ccb49640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.231+0000 7f90ccb49640 1 -- 172.21.15.46:0/1666859688 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f90c80a0400 con 0x7f90c81cc570 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.232+0000 7f90ccb49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.232+0000 7f90ccb49640 1 --2- 172.21.15.46:0/1666859688 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90c81235c0 0x7f90c81206a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.232+0000 7f90ccb49640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.232+0000 7f90ccb49640 1 -- 172.21.15.46:0/1666859688 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f90c8122700 con 0x7f90c81235c0 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.232+0000 7f90b6ffd640 1 -- 172.21.15.46:0/1666859688 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f90bc042020 con 0x7f90c81cc570 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.232+0000 7f90c659c640 1 --2- 172.21.15.46:0/1666859688 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90c81235c0 0x7f90c81206a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.232+0000 7f90c659c640 10 monclient: get_auth_request con 0x7f90c81235c0 auth_method 0 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.232+0000 7f90c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.232+0000 7f90c659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.232+0000 7f90c659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.232+0000 7f90c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.232+0000 7f90c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.232+0000 7f90c659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.232+0000 7f90b6ffd640 1 --2- 172.21.15.46:0/1666859688 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f90a80383e0 0x7f90a803a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.232+0000 7f90c659c640 10 monclient(hunting): handle_auth_done global_id 24587 payload 824 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.232+0000 7f90b6ffd640 1 -- 172.21.15.46:0/1666859688 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(136..136 src has 1..136) ==== 4738+0+0 (secure 0 0 0) 0x7f90bc06b920 con 0x7f90c81cc570 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.232+0000 7f90c659c640 1 -- 172.21.15.46:0/1666859688 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f90c80a43d0 con 0x7f90c81235c0 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.233+0000 7f90c659c640 1 --2- 172.21.15.46:0/1666859688 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90c81235c0 0x7f90c81206a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f90b8001eb0 tx=0x7f90b8007430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.233+0000 7f90c6d9d640 1 --2- 172.21.15.46:0/1666859688 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f90a80383e0 0x7f90a803a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.233+0000 7f90b6ffd640 1 -- 172.21.15.46:0/1666859688 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b8011820 con 0x7f90c81235c0 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.233+0000 7f90b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.233+0000 7f90b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.233+0000 7f90b6ffd640 10 monclient: dump: 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.271 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.271 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.271 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.271 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.271 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.271 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.271 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.233+0000 7f90c6d9d640 10 monclient: get_auth_request con 0x7f90a80383e0 auth_method 0 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.233+0000 7f90b6ffd640 1 -- 172.21.15.46:0/1666859688 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f90b801a070 con 0x7f90c81235c0 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.233+0000 7f90b6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.233+0000 7f90b6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.233+0000 7f90b6ffd640 1 -- 172.21.15.46:0/1666859688 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90c81235c0 msgr2=0x7f90c81206a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.233+0000 7f90b6ffd640 1 --2- 172.21.15.46:0/1666859688 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90c81235c0 0x7f90c81206a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f90b8001eb0 tx=0x7f90b8007430 comp rx=0 tx=0).stop 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.233+0000 7f90b6ffd640 1 -- 172.21.15.46:0/1666859688 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90c81235c0 msgr2=0x7f90c81206a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.233+0000 7f90b6ffd640 1 --2- 172.21.15.46:0/1666859688 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90c81235c0 0x7f90c81206a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.233+0000 7f90c6d9d640 1 --2- 172.21.15.46:0/1666859688 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f90a80383e0 0x7f90a803a8a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f90c8067ce0 tx=0x7f90c0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.239+0000 7f90ccb49640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.239+0000 7f90ccb49640 1 --2- 172.21.15.46:0/1666859688 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90c8120be0 0x7f90c80a7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.239+0000 7f90ccb49640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.239+0000 7f90ccb49640 1 -- 172.21.15.46:0/1666859688 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f90c8067e20 con 0x7f90c8120be0 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.239+0000 7f90c659c640 1 --2- 172.21.15.46:0/1666859688 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90c8120be0 0x7f90c80a7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.239+0000 7f90c659c640 10 monclient: get_auth_request con 0x7f90c8120be0 auth_method 0 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.239+0000 7f90c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.239+0000 7f90c659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.239+0000 7f90c659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.239+0000 7f90c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.239+0000 7f90c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.239+0000 7f90c659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.240+0000 7f90c659c640 10 monclient(hunting): handle_auth_done global_id 24590 payload 824 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.240+0000 7f90c659c640 1 -- 172.21.15.46:0/1666859688 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f90c8120ff0 con 0x7f90c8120be0 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.240+0000 7f90c659c640 1 --2- 172.21.15.46:0/1666859688 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90c8120be0 0x7f90c80a7010 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f90b8019000 tx=0x7f90b8001da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.240+0000 7f90b6ffd640 1 -- 172.21.15.46:0/1666859688 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b801fe10 con 0x7f90c8120be0 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.240+0000 7f90b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.240+0000 7f90b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.240+0000 7f90b6ffd640 10 monclient: dump: 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.240+0000 7f90b6ffd640 1 -- 172.21.15.46:0/1666859688 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f90b8015680 con 0x7f90c8120be0 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.240+0000 7f90b6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.240+0000 7f90b6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.240+0000 7f90b6ffd640 1 -- 172.21.15.46:0/1666859688 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90c8120be0 msgr2=0x7f90c80a7010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.240+0000 7f90b6ffd640 1 --2- 172.21.15.46:0/1666859688 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90c8120be0 0x7f90c80a7010 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f90b8019000 tx=0x7f90b8001da0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.240+0000 7f90b6ffd640 1 -- 172.21.15.46:0/1666859688 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90c8120be0 msgr2=0x7f90c80a7010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:09.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.240+0000 7f90b6ffd640 1 --2- 172.21.15.46:0/1666859688 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90c8120be0 0x7f90c80a7010 secure :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f90b8019000 tx=0x7f90b8001da0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.276 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:42:09.276 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:42:09.276 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:42:09.276 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 38, 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 104439, 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 2, 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:42:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:42:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:42:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:09.291 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:09.291 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:42:09.291 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:09.291 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:09.291 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.291 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:42:09.291 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:42:09.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.241+0000 7f90ccb49640 1 -- 172.21.15.46:0/1666859688 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f90a80383e0 msgr2=0x7f90a803a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.241+0000 7f90ccb49640 1 --2- 172.21.15.46:0/1666859688 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f90a80383e0 0x7f90a803a8a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f90c8067ce0 tx=0x7f90c0009040 comp rx=0 tx=0).stop 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.241+0000 7f90ccb49640 10 monclient: shutdown 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.241+0000 7f90ccb49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.241+0000 7f90ccb49640 1 -- 172.21.15.46:0/1666859688 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90c81cc570 msgr2=0x7f90c81cc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.241+0000 7f90ccb49640 1 --2- 172.21.15.46:0/1666859688 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f90c81cc570 0x7f90c81cc9d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f90bc034040 tx=0x7f90bc004730 comp rx=0 tx=0).stop 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.241+0000 7f90c6d9d640 1 -- 172.21.15.46:0/1666859688 reap_dead start 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.241+0000 7f90c6d9d640 1 -- 172.21.15.46:0/1666859688 reap_dead start 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.241+0000 7f90ccb49640 1 -- 172.21.15.46:0/1666859688 shutdown_connections 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.241+0000 7f90ccb49640 1 -- 172.21.15.46:0/1666859688 >> 172.21.15.46:0/1666859688 conn(0x7f90c80a6340 msgr2=0x7f90c80a2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.241+0000 7f90ccb49640 1 -- 172.21.15.46:0/1666859688 shutdown_connections 2024-09-16T04:42:09.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.241+0000 7f90ccb49640 1 -- 172.21.15.46:0/1666859688 wait complete. 2024-09-16T04:42:09.295 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:42:09.295 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:42:09.295 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:42:09.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.396+0000 7fe710a77640 10 monclient: get_monmap_and_config 2024-09-16T04:42:09.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.396+0000 7fe710a77640 10 monclient: build_initial_monmap 2024-09-16T04:42:09.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.396+0000 7fe710a77640 10 monclient: monmap: 2024-09-16T04:42:09.398 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:09.398 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.398 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:09.397321+0000 2024-09-16T04:42:09.399 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:09.397321+0000 2024-09-16T04:42:09.399 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:09.399 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.399 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:09.399 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:09.399 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:09.400 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:09.400 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:09.400 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.397+0000 7fe710a77640 1 Processor -- start 2024-09-16T04:42:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.397+0000 7fe710a77640 1 -- start start 2024-09-16T04:42:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.397+0000 7fe710a77640 10 monclient: init 2024-09-16T04:42:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe710a77640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe710a77640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-16T04:42:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe710a77640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe70c06e160 0x7fe70c0a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe710a77640 10 monclient(hunting): picked mon.noname-c con 0x7fe70c06e160 addr v2:172.21.15.114:3300/0 2024-09-16T04:42:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe710a77640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe70c0a2ee0 addr v1:172.21.15.46:6789/0 2024-09-16T04:42:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe710a77640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe70c137050 0x7fe70c13ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe710a77640 10 monclient(hunting): picked mon.noname-a con 0x7fe70c137050 addr v2:172.21.15.46:3300/0 2024-09-16T04:42:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe710a77640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe70c0a3fd0 con 0x7fe70c0a2ee0 2024-09-16T04:42:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe710a77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe710a77640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe70c0a3300 con 0x7fe70c137050 2024-09-16T04:42:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe710a77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe710a77640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fe70c06e540 con 0x7fe70c06e160 2024-09-16T04:42:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe710a77640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe710a77640 10 monclient(hunting): authenticate will time out at 927.813159s 2024-09-16T04:42:09.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe70c137050 0x7fe70c13ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe709d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe70c0a2ee0 0x7fe70c136940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51698/0 (socket says 172.21.15.46:51698) 2024-09-16T04:42:09.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe709d9b640 1 -- 172.21.15.46:0/3163814583 learned_addr learned my addr 172.21.15.46:0/3163814583 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:42:09.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70ad9d640 10 monclient(hunting): get_auth_request con 0x7fe70c137050 auth_method 0 2024-09-16T04:42:09.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:09.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70a59c640 1 --2- 172.21.15.46:0/3163814583 >> v2:172.21.15.114:3300/0 conn(0x7fe70c06e160 0x7fe70c0a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70a59c640 10 monclient(hunting): get_auth_request con 0x7fe70c06e160 auth_method 0 2024-09-16T04:42:09.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70ad9d640 10 monclient(hunting): handle_auth_done global_id 44157 payload 274 2024-09-16T04:42:09.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70ad9d640 1 -- 172.21.15.46:0/3163814583 >> v2:172.21.15.114:3300/0 conn(0x7fe70c06e160 msgr2=0x7fe70c0a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:09.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70ad9d640 1 --2- 172.21.15.46:0/3163814583 >> v2:172.21.15.114:3300/0 conn(0x7fe70c06e160 0x7fe70c0a29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70ad9d640 1 -- 172.21.15.46:0/3163814583 >> v1:172.21.15.46:6789/0 conn(0x7fe70c0a2ee0 legacy=0x7fe70c136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70ad9d640 10 monclient: _finish_hunting 0 2024-09-16T04:42:09.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70ad9d640 1 monclient: found mon.noname-a 2024-09-16T04:42:09.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:09.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:09.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70ad9d640 1 -- 172.21.15.46:0/3163814583 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe70c0a0bd0 con 0x7fe70c137050 2024-09-16T04:42:09.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70959a640 1 -- 172.21.15.46:0/3163814583 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3870932409 0 0) 0x7fe700001110 con 0x7fe70c0a2ee0 2024-09-16T04:42:09.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.398+0000 7fe70959a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:42:09.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.399+0000 7fe70ad9d640 1 --2- 172.21.15.46:0/3163814583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe70c137050 0x7fe70c13ec80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe70c068700 tx=0x7fe70402a580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0aa1f029f0f0738 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:09.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.399+0000 7fe70959a640 1 -- 172.21.15.46:0/3163814583 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe704007740 con 0x7fe70c137050 2024-09-16T04:42:09.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.399+0000 7fe70959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.399+0000 7fe70959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:09.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.399+0000 7fe70959a640 10 monclient: dump: 2024-09-16T04:42:09.424 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.424 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.424 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.424 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.425 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.425 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.425 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.425 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.425 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.425 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.399+0000 7fe70959a640 10 monclient: _finish_auth 0 2024-09-16T04:42:09.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.399+0000 7fe70959a640 10 monclient: _check_auth_tickets 2024-09-16T04:42:09.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.399+0000 7fe70959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:09.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.399+0000 7fe710a77640 5 monclient: authenticate success, global_id 44157 2024-09-16T04:42:09.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.399+0000 7fe70959a640 1 -- 172.21.15.46:0/3163814583 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7040078e0 con 0x7fe70c137050 2024-09-16T04:42:09.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.399+0000 7fe70959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:09.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.399+0000 7fe70959a640 1 -- 172.21.15.46:0/3163814583 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe704007cb0 con 0x7fe70c137050 2024-09-16T04:42:09.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.399+0000 7fe70959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.399+0000 7fe70959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:09.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.399+0000 7fe70959a640 10 monclient: dump: 2024-09-16T04:42:09.426 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.426 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.426 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.426 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.426 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.426 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.426 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.426 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.426 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.426 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.399+0000 7fe710a77640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:09.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.399+0000 7fe710a77640 4 set_mon_vals no callback set 2024-09-16T04:42:09.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.399+0000 7fe710a77640 10 monclient: shutdown 2024-09-16T04:42:09.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.400+0000 7fe710a77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:09.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.400+0000 7fe710a77640 1 -- 172.21.15.46:0/3163814583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe70c137050 msgr2=0x7fe70c13ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.400+0000 7fe710a77640 1 --2- 172.21.15.46:0/3163814583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe70c137050 0x7fe70c13ec80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe70c068700 tx=0x7fe70402a580 comp rx=0 tx=0).stop 2024-09-16T04:42:09.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.400+0000 7fe710a77640 1 -- 172.21.15.46:0/3163814583 shutdown_connections 2024-09-16T04:42:09.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.400+0000 7fe710a77640 1 --2- 172.21.15.46:0/3163814583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe70c137050 0x7fe70c13ec80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.400+0000 7fe710a77640 1 --2- 172.21.15.46:0/3163814583 >> v2:172.21.15.114:3300/0 conn(0x7fe70c06e160 0x7fe70c0a29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.400+0000 7fe710a77640 1 -- 172.21.15.46:0/3163814583 >> 172.21.15.46:0/3163814583 conn(0x7fe70c0a5c10 msgr2=0x7fe70c0a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:09.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.400+0000 7fe710a77640 1 -- 172.21.15.46:0/3163814583 shutdown_connections 2024-09-16T04:42:09.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.400+0000 7fe710a77640 1 -- 172.21.15.46:0/3163814583 wait complete. 2024-09-16T04:42:09.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.400+0000 7fe710a77640 10 monclient: build_initial_monmap 2024-09-16T04:42:09.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.400+0000 7fe710a77640 10 monclient: monmap: 2024-09-16T04:42:09.427 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:09.427 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:09.428 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:09.428 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:09.428 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:09.428 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.428 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:09.428 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:09.428 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:09.428 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.401+0000 7fe710a77640 1 Processor -- start 2024-09-16T04:42:09.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.401+0000 7fe710a77640 1 -- start start 2024-09-16T04:42:09.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.401+0000 7fe710a77640 10 monclient: init 2024-09-16T04:42:09.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.401+0000 7fe710a77640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:09.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.401+0000 7fe710a77640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:42:09.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.401+0000 7fe710a77640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe70c06e160 0x7fe70c1cba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.401+0000 7fe710a77640 10 monclient(hunting): picked mon.noname-c con 0x7fe70c06e160 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.401+0000 7fe710a77640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe70c0a2ee0 0x7fe70c135b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.401+0000 7fe710a77640 10 monclient(hunting): picked mon.noname-a con 0x7fe70c0a2ee0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.401+0000 7fe710a77640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe70c137050 0x7fe70c136040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.401+0000 7fe710a77640 10 monclient(hunting): picked mon.noname-b con 0x7fe70c137050 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.401+0000 7fe710a77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.401+0000 7fe710a77640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c0a34a0 con 0x7fe70c0a2ee0 2024-09-16T04:42:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.401+0000 7fe710a77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.401+0000 7fe710a77640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c0a5130 con 0x7fe70c137050 2024-09-16T04:42:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.401+0000 7fe710a77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.401+0000 7fe710a77640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe70c0a0300 con 0x7fe70c06e160 2024-09-16T04:42:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.401+0000 7fe710a77640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.401+0000 7fe710a77640 10 monclient(hunting): authenticate will time out at 927.816905s 2024-09-16T04:42:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.401+0000 7fe709d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe70c0a2ee0 0x7fe70c135b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.401+0000 7fe70a59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe70c06e160 0x7fe70c1cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe70a59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe70c06e160 0x7fe70c1cba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39600/0 (socket says 172.21.15.46:39600) 2024-09-16T04:42:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe709d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe70c0a2ee0 0x7fe70c135b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49620/0 (socket says 172.21.15.46:49620) 2024-09-16T04:42:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe70a59c640 1 -- 172.21.15.46:0/3810892678 learned_addr learned my addr 172.21.15.46:0/3810892678 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe70a59c640 10 monclient(hunting): get_auth_request con 0x7fe70c06e160 auth_method 0 2024-09-16T04:42:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe70a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe70a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe70a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe709d9b640 10 monclient(hunting): get_auth_request con 0x7fe70c0a2ee0 auth_method 0 2024-09-16T04:42:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe709d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe709d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe709d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe70ad9d640 1 --2- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe70c137050 0x7fe70c136040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe709d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe709d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe709d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:09.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe70ad9d640 10 monclient(hunting): get_auth_request con 0x7fe70c137050 auth_method 0 2024-09-16T04:42:09.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe70ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe70ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe70ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe70a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe70a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe70a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:09.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe70ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe70ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe70ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:09.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe709d9b640 10 monclient(hunting): handle_auth_done global_id 44163 payload 824 2024-09-16T04:42:09.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe709d9b640 1 -- 172.21.15.46:0/3810892678 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe70c06e160 msgr2=0x7fe70c1cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe709d9b640 1 --2- 172.21.15.46:0/3810892678 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe70c06e160 0x7fe70c1cba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe709d9b640 1 -- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe70c137050 msgr2=0x7fe70c136040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe709d9b640 1 --2- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe70c137050 0x7fe70c136040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe709d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:42:09.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe709d9b640 1 monclient: found mon.noname-a 2024-09-16T04:42:09.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe709d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:09.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe709d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:09.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe709d9b640 1 -- 172.21.15.46:0/3810892678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe70c127220 con 0x7fe70c0a2ee0 2024-09-16T04:42:09.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe709d9b640 1 --2- 172.21.15.46:0/3810892678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe70c0a2ee0 0x7fe70c135b00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe70000c780 tx=0x7fe70000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:09.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe6faffd640 1 -- 172.21.15.46:0/3810892678 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe700018070 con 0x7fe70c0a2ee0 2024-09-16T04:42:09.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe6faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe6faffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:09.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe6faffd640 10 monclient: dump: 2024-09-16T04:42:09.433 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.433 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.433 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.433 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.433 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.433 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.433 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.433 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.433 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.433 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe6faffd640 10 monclient: _finish_auth 0 2024-09-16T04:42:09.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe6faffd640 10 monclient: _check_auth_tickets 2024-09-16T04:42:09.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe6faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:09.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe710a77640 5 monclient: authenticate success, global_id 44163 2024-09-16T04:42:09.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe6faffd640 1 -- 172.21.15.46:0/3810892678 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7000070d0 con 0x7fe70c0a2ee0 2024-09-16T04:42:09.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe710a77640 10 monclient: _renew_subs 2024-09-16T04:42:09.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe710a77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:09.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe710a77640 1 -- 172.21.15.46:0/3810892678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe70c125d70 con 0x7fe70c0a2ee0 2024-09-16T04:42:09.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe6faffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:09.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe6faffd640 1 -- 172.21.15.46:0/3810892678 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7000074c0 con 0x7fe70c0a2ee0 2024-09-16T04:42:09.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe6faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe6faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:09.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe6faffd640 10 monclient: dump: 2024-09-16T04:42:09.434 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.434 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.434 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.434 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.434 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.434 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.435 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.435 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.435 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.435 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe710a77640 10 monclient: _renew_subs 2024-09-16T04:42:09.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe710a77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:09.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe708d99640 4 set_mon_vals no callback set 2024-09-16T04:42:09.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.402+0000 7fe710a77640 1 -- 172.21.15.46:0/3810892678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe70c0a81b0 con 0x7fe70c0a2ee0 2024-09-16T04:42:09.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.403+0000 7fe710a77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:09.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.403+0000 7fe710a77640 1 --2- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe70c123570 0x7fe70c123950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.403+0000 7fe710a77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.403+0000 7fe710a77640 1 -- 172.21.15.46:0/3810892678 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c0a3620 con 0x7fe70c123570 2024-09-16T04:42:09.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.403+0000 7fe6faffd640 1 -- 172.21.15.46:0/3810892678 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fe70001b070 con 0x7fe70c0a2ee0 2024-09-16T04:42:09.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.403+0000 7fe6faffd640 1 --2- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe6ec0383b0 0x7fe6ec03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.403+0000 7fe70a59c640 1 --2- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe70c123570 0x7fe70c123950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.403+0000 7fe6faffd640 1 -- 172.21.15.46:0/3810892678 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 4738+0+0 (secure 0 0 0) 0x7fe70004a9f0 con 0x7fe70c0a2ee0 2024-09-16T04:42:09.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.403+0000 7fe70a59c640 10 monclient: get_auth_request con 0x7fe70c123570 auth_method 0 2024-09-16T04:42:09.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.403+0000 7fe70a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.403+0000 7fe70a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.403+0000 7fe70a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.403+0000 7fe70a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.403+0000 7fe70a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.403+0000 7fe70a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:09.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.403+0000 7fe70ad9d640 1 --2- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe6ec0383b0 0x7fe6ec03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.403+0000 7fe70ad9d640 10 monclient: get_auth_request con 0x7fe6ec0383b0 auth_method 0 2024-09-16T04:42:09.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.404+0000 7fe70a59c640 10 monclient(hunting): handle_auth_done global_id 5455 payload 824 2024-09-16T04:42:09.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.404+0000 7fe70a59c640 1 -- 172.21.15.46:0/3810892678 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe70c1374a0 con 0x7fe70c123570 2024-09-16T04:42:09.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.404+0000 7fe70a59c640 1 --2- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe70c123570 0x7fe70c123950 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc010820 tx=0x7fe6fc010ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:09.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.404+0000 7fe70ad9d640 1 --2- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe6ec0383b0 0x7fe6ec03a870 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe704034000 tx=0x7fe70402f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:09.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.404+0000 7fe6faffd640 1 -- 172.21.15.46:0/3810892678 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc018070 con 0x7fe70c123570 2024-09-16T04:42:09.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.404+0000 7fe6faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.404+0000 7fe6faffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:42:09.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.404+0000 7fe6faffd640 10 monclient: dump: 2024-09-16T04:42:09.437 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.437 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.437 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.437 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.437 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.437 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.437 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.437 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.437 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.437 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.404+0000 7fe6faffd640 1 -- 172.21.15.46:0/3810892678 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe6fc00ea30 con 0x7fe70c123570 2024-09-16T04:42:09.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.404+0000 7fe6faffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:09.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.404+0000 7fe6faffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:09.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.404+0000 7fe6faffd640 1 -- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe70c123570 msgr2=0x7fe70c123950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.404+0000 7fe6faffd640 1 --2- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe70c123570 0x7fe70c123950 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc010820 tx=0x7fe6fc010ce0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.404+0000 7fe6faffd640 1 -- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe70c123570 msgr2=0x7fe70c123950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:09.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.404+0000 7fe6faffd640 1 --2- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe70c123570 0x7fe70c123950 secure :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc010820 tx=0x7fe6fc010ce0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.410+0000 7fe710a77640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:42:09.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.410+0000 7fe710a77640 1 --2- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe70c123e90 0x7fe70c11f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.411+0000 7fe710a77640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.411+0000 7fe710a77640 1 -- 172.21.15.46:0/3810892678 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c1cbfb0 con 0x7fe70c123e90 2024-09-16T04:42:09.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.411+0000 7fe70a59c640 1 --2- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe70c123e90 0x7fe70c11f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.411+0000 7fe70a59c640 10 monclient: get_auth_request con 0x7fe70c123e90 auth_method 0 2024-09-16T04:42:09.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.411+0000 7fe70a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.411+0000 7fe70a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.411+0000 7fe70a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.411+0000 7fe70a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.411+0000 7fe70a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.411+0000 7fe70a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:09.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.411+0000 7fe70a59c640 10 monclient(hunting): handle_auth_done global_id 5458 payload 824 2024-09-16T04:42:09.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.411+0000 7fe70a59c640 1 -- 172.21.15.46:0/3810892678 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe70c067e00 con 0x7fe70c123e90 2024-09-16T04:42:09.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.411+0000 7fe70a59c640 1 --2- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe70c123e90 0x7fe70c11f760 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc007fd0 tx=0x7fe6fc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:09.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe6faffd640 1 -- 172.21.15.46:0/3810892678 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc004230 con 0x7fe70c123e90 2024-09-16T04:42:09.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe6faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe6faffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:42:09.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe6faffd640 10 monclient: dump: 2024-09-16T04:42:09.439 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.439 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.439 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.440 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.440 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.440 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.440 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.440 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.440 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.440 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe6faffd640 1 -- 172.21.15.46:0/3810892678 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7fe6fc004230 con 0x7fe70c123e90 2024-09-16T04:42:09.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe6faffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:42:09.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe6faffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:09.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe6faffd640 1 -- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe70c123e90 msgr2=0x7fe70c11f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe6faffd640 1 --2- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe70c123e90 0x7fe70c11f760 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc007fd0 tx=0x7fe6fc0046c0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe6faffd640 1 -- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe70c123e90 msgr2=0x7fe70c11f760 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:09.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe6faffd640 1 --2- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe70c123e90 0x7fe70c11f760 secure :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc007fd0 tx=0x7fe6fc0046c0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.441 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:42:09.441 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:42:09.441 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:42:09.441 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:42:09.441 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 38, 2024-09-16T04:42:09.441 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 215313, 2024-09-16T04:42:09.441 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:42:09.441 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:42:09.441 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:42:09.442 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:42:09.442 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:09.442 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 2, 2024-09-16T04:42:09.442 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:09.442 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:42:09.442 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:42:09.442 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:09.442 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:09.442 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:09.442 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:09.442 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:09.442 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:09.442 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:09.442 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:09.443 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:09.443 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:09.443 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:09.443 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:09.443 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:42:09.443 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:42:09.443 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:09.443 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:09.443 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:09.443 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:09.443 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:09.443 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:09.443 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:09.443 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:09.443 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:09.444 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:09.444 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:09.444 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:09.444 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.444 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:42:09.444 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:42:09.444 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:42:09.444 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:42:09.444 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:42:09.444 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:42:09.444 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:09.444 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:09.444 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:42:09.445 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:42:09.445 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:42:09.445 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:42:09.445 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:42:09.445 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:42:09.445 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:42:09.445 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:09.445 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:42:09.445 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:09.445 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:09.445 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:09.445 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:09.445 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:09.445 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:09.446 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:09.446 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:09.446 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:09.446 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:09.446 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:09.446 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:09.446 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:42:09.446 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.446 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:42:09.446 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.446 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:42:09.446 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:42:09.446 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:09.446 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:09.446 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.447 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:09.447 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:42:09.447 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:09.447 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.447 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.447 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:09.447 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:42:09.447 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:09.447 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:09.447 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:09.447 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.447 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:42:09.448 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:42:09.448 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:09.448 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:09.448 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:09.448 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.448 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.448 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:42:09.448 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:42:09.448 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:09.448 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:09.448 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.448 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:09.449 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:42:09.449 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:09.449 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.449 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.449 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:09.449 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:42:09.449 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:09.449 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:09.449 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:09.449 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.449 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:42:09.449 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:42:09.449 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:09.449 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:09.450 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:09.450 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.450 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.450 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:42:09.450 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:42:09.450 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:09.450 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:09.450 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.450 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:09.450 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:42:09.450 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:09.450 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.450 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.450 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:09.451 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:42:09.451 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:09.451 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:09.451 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:09.451 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.451 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:42:09.451 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:42:09.451 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:09.451 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:09.451 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:09.451 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:09.451 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:09.451 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.451 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:42:09.452 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:42:09.452 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.452 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:09.452 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:09.452 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:42:09.452 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:09.452 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:09.452 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:42:09.452 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.452 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:09.452 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:09.452 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4 2024-09-16T04:42:09.452 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:09.453 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:09.453 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:42:09.453 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.453 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:09.453 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:09.453 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6 2024-09-16T04:42:09.453 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:09.453 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:09.453 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:42:09.453 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:09.453 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:09.453 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:09.453 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:42:09.454 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:09.454 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:09.454 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:09.454 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:42:09.454 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:42:09.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe710a77640 1 -- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe6ec0383b0 msgr2=0x7fe6ec03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe710a77640 1 --2- 172.21.15.46:0/3810892678 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe6ec0383b0 0x7fe6ec03a870 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe704034000 tx=0x7fe70402f000 comp rx=0 tx=0).stop 2024-09-16T04:42:09.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe710a77640 10 monclient: shutdown 2024-09-16T04:42:09.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe710a77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:09.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe710a77640 1 -- 172.21.15.46:0/3810892678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe70c0a2ee0 msgr2=0x7fe70c135b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe710a77640 1 --2- 172.21.15.46:0/3810892678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe70c0a2ee0 0x7fe70c135b00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe70000c780 tx=0x7fe70000cc40 comp rx=0 tx=0).stop 2024-09-16T04:42:09.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe70ad9d640 1 -- 172.21.15.46:0/3810892678 reap_dead start 2024-09-16T04:42:09.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe70ad9d640 1 -- 172.21.15.46:0/3810892678 reap_dead start 2024-09-16T04:42:09.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe710a77640 1 -- 172.21.15.46:0/3810892678 shutdown_connections 2024-09-16T04:42:09.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe710a77640 1 -- 172.21.15.46:0/3810892678 >> 172.21.15.46:0/3810892678 conn(0x7fe70c0a5c10 msgr2=0x7fe70c0a77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:09.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe710a77640 1 -- 172.21.15.46:0/3810892678 shutdown_connections 2024-09-16T04:42:09.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.412+0000 7fe710a77640 1 -- 172.21.15.46:0/3810892678 wait complete. 2024-09-16T04:42:09.456 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:42:09.457 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T04:42:09.457 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:42:09.457 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T04:42:09.457 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:42:09.457 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:42:09.457 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:42:09.468 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:42:09.468 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:42:09.468 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:42:09.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.607+0000 7fadf01a8640 10 monclient: get_monmap_and_config 2024-09-16T04:42:09.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.607+0000 7fadf01a8640 10 monclient: build_initial_monmap 2024-09-16T04:42:09.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.607+0000 7fadf01a8640 10 monclient: monmap: 2024-09-16T04:42:09.608 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:09.609 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.609 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:09.607625+0000 2024-09-16T04:42:09.609 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:09.607625+0000 2024-09-16T04:42:09.609 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:09.609 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.609 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:09.609 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:09.609 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:09.609 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:09.609 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:09.609 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.607+0000 7fadf01a8640 1 Processor -- start 2024-09-16T04:42:09.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.607+0000 7fadf01a8640 1 -- start start 2024-09-16T04:42:09.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.607+0000 7fadf01a8640 10 monclient: init 2024-09-16T04:42:09.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadf01a8640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:09.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadf01a8640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-16T04:42:09.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadf01a8640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fade80a3c10 0x7fade809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadf01a8640 10 monclient(hunting): picked mon.noname-c con 0x7fade80a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:42:09.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadf01a8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fade80a3ff0 0x7fade80a4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadf01a8640 10 monclient(hunting): picked mon.noname-a con 0x7fade80a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:42:09.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadf01a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fade81369d0 addr v1:172.21.15.46:6789/0 2024-09-16T04:42:09.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadf01a8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fade809ab70 con 0x7fade81369d0 2024-09-16T04:42:09.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadf01a8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadf01a8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fade80a4d60 con 0x7fade80a3ff0 2024-09-16T04:42:09.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadf01a8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadf01a8640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fade809b030 con 0x7fade80a3c10 2024-09-16T04:42:09.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadf01a8640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:09.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadf01a8640 10 monclient(hunting): authenticate will time out at 928.023473s 2024-09-16T04:42:09.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7faded743640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fade80a3ff0 0x7fade80a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadee745640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fade81369d0 0x7fade8136e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51708/0 (socket says 172.21.15.46:51708) 2024-09-16T04:42:09.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadee745640 1 -- 172.21.15.46:0/260004568 learned_addr learned my addr 172.21.15.46:0/260004568 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:42:09.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7faded743640 10 monclient(hunting): get_auth_request con 0x7fade80a3ff0 auth_method 0 2024-09-16T04:42:09.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7faded743640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7faded743640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7faded743640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadedf44640 1 --2- 172.21.15.46:0/260004568 >> v2:172.21.15.114:3300/0 conn(0x7fade80a3c10 0x7fade809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadedf44640 10 monclient(hunting): get_auth_request con 0x7fade80a3c10 auth_method 0 2024-09-16T04:42:09.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadedf44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadedf44640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadedf44640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7faded743640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7faded743640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7faded743640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadedf44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadedf44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7fadedf44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7faded743640 10 monclient(hunting): handle_auth_done global_id 44166 payload 274 2024-09-16T04:42:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7faded743640 1 -- 172.21.15.46:0/260004568 >> v2:172.21.15.114:3300/0 conn(0x7fade80a3c10 msgr2=0x7fade809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7faded743640 1 --2- 172.21.15.46:0/260004568 >> v2:172.21.15.114:3300/0 conn(0x7fade80a3c10 0x7fade809bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7faded743640 1 -- 172.21.15.46:0/260004568 >> v1:172.21.15.46:6789/0 conn(0x7fade81369d0 legacy=0x7fade8136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7faded743640 10 monclient: _finish_hunting 0 2024-09-16T04:42:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7faded743640 1 monclient: found mon.noname-a 2024-09-16T04:42:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7faded743640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7faded743640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.608+0000 7faded743640 1 -- 172.21.15.46:0/260004568 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fade809b310 con 0x7fade80a3ff0 2024-09-16T04:42:09.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.609+0000 7faded743640 1 --2- 172.21.15.46:0/260004568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fade80a3ff0 0x7fade80a4400 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fade806be80 tx=0x7fadd802a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81dda14df39d99a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:09.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.609+0000 7fadecf42640 1 -- 172.21.15.46:0/260004568 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd8037070 con 0x7fade80a3ff0 2024-09-16T04:42:09.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.609+0000 7fadecf42640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.609+0000 7fadecf42640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:09.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.609+0000 7fadecf42640 10 monclient: dump: 2024-09-16T04:42:09.621 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.621 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.621 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.621 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.621 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.621 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.622 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.622 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.622 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.622 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.609+0000 7fadecf42640 10 monclient: _finish_auth 0 2024-09-16T04:42:09.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.609+0000 7fadecf42640 10 monclient: _check_auth_tickets 2024-09-16T04:42:09.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.609+0000 7fadecf42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:09.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.609+0000 7fadecf42640 1 -- 172.21.15.46:0/260004568 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadd802add0 con 0x7fade80a3ff0 2024-09-16T04:42:09.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.609+0000 7fadf01a8640 5 monclient: authenticate success, global_id 44166 2024-09-16T04:42:09.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.609+0000 7fadf01a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:09.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.609+0000 7fadecf42640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:09.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.609+0000 7fadf01a8640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:09.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.609+0000 7fadecf42640 1 -- 172.21.15.46:0/260004568 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd8032660 con 0x7fade80a3ff0 2024-09-16T04:42:09.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.609+0000 7fadecf42640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.609+0000 7fadf01a8640 4 set_mon_vals no callback set 2024-09-16T04:42:09.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.609+0000 7fadecf42640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:09.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.609+0000 7fadecf42640 10 monclient: dump: 2024-09-16T04:42:09.623 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.623 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.623 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.623 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.624 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.624 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.624 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.624 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.624 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.624 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.610+0000 7fadf01a8640 10 monclient: shutdown 2024-09-16T04:42:09.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.610+0000 7fadf01a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:09.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.610+0000 7fadf01a8640 1 -- 172.21.15.46:0/260004568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fade80a3ff0 msgr2=0x7fade80a4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.610+0000 7fadf01a8640 1 --2- 172.21.15.46:0/260004568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fade80a3ff0 0x7fade80a4400 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fade806be80 tx=0x7fadd802a080 comp rx=0 tx=0).stop 2024-09-16T04:42:09.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.610+0000 7fadf01a8640 1 -- 172.21.15.46:0/260004568 shutdown_connections 2024-09-16T04:42:09.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.610+0000 7fadf01a8640 1 --2- 172.21.15.46:0/260004568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fade80a3ff0 0x7fade80a4400 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.610+0000 7fadf01a8640 1 --2- 172.21.15.46:0/260004568 >> v2:172.21.15.114:3300/0 conn(0x7fade80a3c10 0x7fade809bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.610+0000 7fadf01a8640 1 -- 172.21.15.46:0/260004568 >> 172.21.15.46:0/260004568 conn(0x7fade80a5d10 msgr2=0x7fade80a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:09.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.610+0000 7fadf01a8640 1 -- 172.21.15.46:0/260004568 shutdown_connections 2024-09-16T04:42:09.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.610+0000 7fadf01a8640 1 -- 172.21.15.46:0/260004568 wait complete. 2024-09-16T04:42:09.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.611+0000 7fadf01a8640 10 monclient: build_initial_monmap 2024-09-16T04:42:09.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.611+0000 7fadf01a8640 10 monclient: monmap: 2024-09-16T04:42:09.625 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:09.625 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:09.625 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:09.625 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:09.625 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:09.625 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.625 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:09.625 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:09.625 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:09.625 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.611+0000 7fadf01a8640 1 Processor -- start 2024-09-16T04:42:09.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.611+0000 7fadf01a8640 1 -- start start 2024-09-16T04:42:09.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.611+0000 7fadf01a8640 10 monclient: init 2024-09-16T04:42:09.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.611+0000 7fadf01a8640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:09.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.611+0000 7fadf01a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:42:09.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.611+0000 7fadf01a8640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fade80a3c10 0x7fade81cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.611+0000 7fadf01a8640 10 monclient(hunting): picked mon.noname-c con 0x7fade80a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:09.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.611+0000 7fadf01a8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fade80a3ff0 0x7fade81cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.611+0000 7fadf01a8640 10 monclient(hunting): picked mon.noname-a con 0x7fade80a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:09.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.611+0000 7fadf01a8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fade81369d0 0x7fade8123180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.611+0000 7fadf01a8640 10 monclient(hunting): picked mon.noname-b con 0x7fade81369d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:09.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.611+0000 7fadf01a8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.611+0000 7fadf01a8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fade806e3d0 con 0x7fade80a3ff0 2024-09-16T04:42:09.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.611+0000 7fadf01a8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.611+0000 7fadedf44640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fade80a3c10 msgr2=0x7fade81cbee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:42:09.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.611+0000 7fadf01a8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fade80a4940 con 0x7fade81369d0 2024-09-16T04:42:09.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.611+0000 7fadf01a8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:09.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.611+0000 7fadf01a8640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fade806e550 con 0x7fade80a3c10 2024-09-16T04:42:09.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.611+0000 7fadedf44640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fade80a3c10 0x7fade81cbee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:09.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadf01a8640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:09.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadf01a8640 10 monclient(hunting): authenticate will time out at 928.027017s 2024-09-16T04:42:09.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faded743640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fade80a3ff0 0x7fade81cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faded743640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fade80a3ff0 0x7fade81cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49656/0 (socket says 172.21.15.46:49656) 2024-09-16T04:42:09.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faded743640 1 -- 172.21.15.46:0/3431236104 learned_addr learned my addr 172.21.15.46:0/3431236104 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:09.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faded743640 10 monclient(hunting): get_auth_request con 0x7fade80a3ff0 auth_method 0 2024-09-16T04:42:09.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faded743640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:09.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faded743640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:09.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faded743640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:09.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faded743640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:09.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faded743640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:09.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faded743640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:09.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadee745640 1 --2- 172.21.15.46:0/3431236104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fade81369d0 0x7fade8123180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faded743640 10 monclient(hunting): handle_auth_done global_id 44172 payload 824 2024-09-16T04:42:09.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faded743640 1 -- 172.21.15.46:0/3431236104 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fade80a3c10 msgr2=0x7fade81cbee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:42:09.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faded743640 1 --2- 172.21.15.46:0/3431236104 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fade80a3c10 0x7fade81cbee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faded743640 1 -- 172.21.15.46:0/3431236104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fade81369d0 msgr2=0x7fade8123180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faded743640 1 --2- 172.21.15.46:0/3431236104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fade81369d0 0x7fade8123180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faded743640 10 monclient: _finish_hunting 0 2024-09-16T04:42:09.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faded743640 1 monclient: found mon.noname-a 2024-09-16T04:42:09.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faded743640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:09.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faded743640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:09.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faded743640 1 -- 172.21.15.46:0/3431236104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fade8125220 con 0x7fade80a3ff0 2024-09-16T04:42:09.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadee745640 10 monclient: get_auth_request con 0x7fade81369d0 auth_method 0 2024-09-16T04:42:09.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadee745640 1 --2- 172.21.15.46:0/3431236104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fade81369d0 0x7fade8123180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:42:09.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faded743640 1 --2- 172.21.15.46:0/3431236104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fade80a3ff0 0x7fade81cc420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fadd8000c00 tx=0x7fadd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:09.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadde7fc640 1 -- 172.21.15.46:0/3431236104 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd8037040 con 0x7fade80a3ff0 2024-09-16T04:42:09.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadde7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadde7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:09.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadde7fc640 10 monclient: dump: 2024-09-16T04:42:09.630 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.630 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.630 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.630 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.630 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.630 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.630 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.630 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.630 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.630 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadde7fc640 10 monclient: _finish_auth 0 2024-09-16T04:42:09.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadde7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:42:09.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadde7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:09.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadf01a8640 5 monclient: authenticate success, global_id 44172 2024-09-16T04:42:09.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadde7fc640 1 -- 172.21.15.46:0/3431236104 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadd8042070 con 0x7fade80a3ff0 2024-09-16T04:42:09.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadf01a8640 10 monclient: _renew_subs 2024-09-16T04:42:09.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadf01a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:09.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadf01a8640 1 -- 172.21.15.46:0/3431236104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fade81258e0 con 0x7fade80a3ff0 2024-09-16T04:42:09.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadde7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:09.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadde7fc640 1 -- 172.21.15.46:0/3431236104 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd803d660 con 0x7fade80a3ff0 2024-09-16T04:42:09.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadde7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:09.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadde7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:09.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadde7fc640 10 monclient: dump: 2024-09-16T04:42:09.631 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:09.631 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:09.631 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.631 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:09.632 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:09.632 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:09.632 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:09.632 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:09.632 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:09.632 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:09.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadf01a8640 10 monclient: _renew_subs 2024-09-16T04:42:09.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadf01a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:09.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7faddffff640 4 set_mon_vals no callback set 2024-09-16T04:42:09.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.612+0000 7fadf01a8640 1 -- 172.21.15.46:0/3431236104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fade8126ab0 con 0x7fade80a3ff0 2024-09-16T04:42:09.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.613+0000 7fadde7fc640 1 -- 172.21.15.46:0/3431236104 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fadd803d800 con 0x7fade80a3ff0 2024-09-16T04:42:09.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.613+0000 7fadf01a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:09.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.613+0000 7fadf01a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:09.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.613+0000 7fadf01a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:09.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.613+0000 7fadf01a8640 1 -- 172.21.15.46:0/3431236104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fade8069150 con 0x7fade80a3ff0 2024-09-16T04:42:09.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.613+0000 7fadde7fc640 1 --2- 172.21.15.46:0/3431236104 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fadcc038390 0x7fadcc03a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:09.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.614+0000 7fadde7fc640 1 -- 172.21.15.46:0/3431236104 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 4738+0+0 (secure 0 0 0) 0x7fadd806bd70 con 0x7fade80a3ff0 2024-09-16T04:42:09.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.614+0000 7fadedf44640 1 --2- 172.21.15.46:0/3431236104 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fadcc038390 0x7fadcc03a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:09.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.614+0000 7fadedf44640 10 monclient: get_auth_request con 0x7fadcc038390 auth_method 0 2024-09-16T04:42:09.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.614+0000 7fadedf44640 1 --2- 172.21.15.46:0/3431236104 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fadcc038390 0x7fadcc03a850 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fade40094f0 tx=0x7fade4009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:09.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.618+0000 7fadde7fc640 1 -- 172.21.15.46:0/3431236104 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fadd8030e20 con 0x7fade80a3ff0 2024-09-16T04:42:09.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.618+0000 7fadde7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:09.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.618+0000 7fadde7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:09.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.860+0000 7fadf01a8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:42:09.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.861+0000 7fadf01a8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:09.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.861+0000 7fadf01a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:09.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.861+0000 7fadf01a8640 1 -- 172.21.15.46:0/3431236104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fade80008d0 con 0x7fade80a3ff0 2024-09-16T04:42:09.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.861+0000 7fadde7fc640 1 -- 172.21.15.46:0/3431236104 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fade80008d0 con 0x7fade80a3ff0 2024-09-16T04:42:09.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.861+0000 7fadde7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:09.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.861+0000 7fadde7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:09.863 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:42:09.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.863+0000 7fadf01a8640 1 -- 172.21.15.46:0/3431236104 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fadcc038390 msgr2=0x7fadcc03a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.863+0000 7fadf01a8640 1 --2- 172.21.15.46:0/3431236104 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fadcc038390 0x7fadcc03a850 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fade40094f0 tx=0x7fade4009190 comp rx=0 tx=0).stop 2024-09-16T04:42:09.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.863+0000 7fadf01a8640 10 monclient: shutdown 2024-09-16T04:42:09.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.863+0000 7fadf01a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:09.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.863+0000 7fadf01a8640 1 -- 172.21.15.46:0/3431236104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fade80a3ff0 msgr2=0x7fade81cc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:09.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.863+0000 7fadf01a8640 1 --2- 172.21.15.46:0/3431236104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fade80a3ff0 0x7fade81cc420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fadd8000c00 tx=0x7fadd80046c0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.863+0000 7fadf01a8640 1 -- 172.21.15.46:0/3431236104 shutdown_connections 2024-09-16T04:42:09.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.863+0000 7fadf01a8640 1 --2- 172.21.15.46:0/3431236104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fade81369d0 0x7fade8123180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.863+0000 7fadf01a8640 1 --2- 172.21.15.46:0/3431236104 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fadcc038390 0x7fadcc03a850 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.863+0000 7fadf01a8640 1 --2- 172.21.15.46:0/3431236104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fade80a3ff0 0x7fade81cc420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.863+0000 7fadf01a8640 1 --2- 172.21.15.46:0/3431236104 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fade80a3c10 0x7fade81cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:09.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.863+0000 7fadf01a8640 1 -- 172.21.15.46:0/3431236104 >> 172.21.15.46:0/3431236104 conn(0x7fade80a5d10 msgr2=0x7fade80a2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:09.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.863+0000 7fadf01a8640 1 -- 172.21.15.46:0/3431236104 shutdown_connections 2024-09-16T04:42:09.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:09.863+0000 7fadf01a8640 1 -- 172.21.15.46:0/3431236104 wait complete. 2024-09-16T04:42:12.876 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:42:12.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.988+0000 7facb5dc4640 10 monclient: get_monmap_and_config 2024-09-16T04:42:12.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.988+0000 7facb5dc4640 10 monclient: build_initial_monmap 2024-09-16T04:42:12.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.988+0000 7facb5dc4640 10 monclient: monmap: 2024-09-16T04:42:12.990 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:12.990 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:12.990 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:12.989107+0000 2024-09-16T04:42:12.990 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:12.989107+0000 2024-09-16T04:42:12.990 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:12.990 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:12.990 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:12.990 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:12.990 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:12.990 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:12.990 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:12.990 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:12.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.989+0000 7facb5dc4640 1 Processor -- start 2024-09-16T04:42:12.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.989+0000 7facb5dc4640 1 -- start start 2024-09-16T04:42:12.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.989+0000 7facb5dc4640 10 monclient: init 2024-09-16T04:42:12.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.989+0000 7facb5dc4640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:12.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facb5dc4640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-16T04:42:12.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facb5dc4640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7facb00a3c10 0x7facb009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:12.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facb5dc4640 10 monclient(hunting): picked mon.noname-c con 0x7facb00a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:42:12.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facb5dc4640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facb00a3ff0 0x7facb00a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:12.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facb5dc4640 10 monclient(hunting): picked mon.noname-b con 0x7facb00a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:12.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facb5dc4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7facb01369d0 addr v1:172.21.15.114:6789/0 2024-09-16T04:42:12.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facb5dc4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facb009b940 con 0x7facb01369d0 2024-09-16T04:42:12.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facb5dc4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:12.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facb5dc4640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7facb009ab00 con 0x7facb00a3c10 2024-09-16T04:42:12.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facb5dc4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:12.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facb5dc4640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7facb009b030 con 0x7facb00a3ff0 2024-09-16T04:42:12.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facb5dc4640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:12.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facb5dc4640 10 monclient(hunting): authenticate will time out at 931.405087s 2024-09-16T04:42:12.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facaeffd640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facb00a3ff0 msgr2=0x7facb00a43d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:42:13.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facaeffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facb00a3ff0 0x7facb00a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:13.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facaf7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7facb00a3c10 0x7facb009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:13.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facaffff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7facb01369d0 0x7facb0136e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.46:55520/0 (socket says 172.21.15.46:55520) 2024-09-16T04:42:13.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facaffff640 1 -- 172.21.15.46:0/560590798 learned_addr learned my addr 172.21.15.46:0/560590798 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:42:13.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facaf7fe640 10 monclient(hunting): get_auth_request con 0x7facb00a3c10 auth_method 0 2024-09-16T04:42:13.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facaf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:13.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facaf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:13.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:13.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facaf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:13.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facaf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:13.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facaf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:13.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facaf7fe640 10 monclient(hunting): handle_auth_done global_id 5464 payload 274 2024-09-16T04:42:13.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facaf7fe640 1 -- 172.21.15.46:0/560590798 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facb00a3ff0 msgr2=0x7facb00a43d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:42:13.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facaf7fe640 1 --2- 172.21.15.46:0/560590798 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facb00a3ff0 0x7facb00a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:13.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facaf7fe640 1 -- 172.21.15.46:0/560590798 >> v1:172.21.15.114:6789/0 conn(0x7facb01369d0 legacy=0x7facb0136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:13.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facaf7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:42:13.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facaf7fe640 1 monclient: found mon.noname-c 2024-09-16T04:42:13.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:13.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facaf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:42:13.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.990+0000 7facaf7fe640 1 -- 172.21.15.46:0/560590798 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facb009b310 con 0x7facb00a3c10 2024-09-16T04:42:13.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.991+0000 7facaf7fe640 1 --2- 172.21.15.46:0/560590798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facb00a3c10 0x7facb009bcf0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fac980097a0 tx=0x7fac9802a080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7eba575c7c936079 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:13.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.991+0000 7facae7fc640 1 -- 172.21.15.46:0/560590798 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac98037070 con 0x7facb00a3c10 2024-09-16T04:42:13.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.991+0000 7facae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:13.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.991+0000 7facae7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:42:13.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.991+0000 7facae7fc640 10 monclient: dump: 2024-09-16T04:42:13.002 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:13.002 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:13.002 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:13.002 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:13.002 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:13.002 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:13.002 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:13.002 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:13.002 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:13.002 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:13.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.991+0000 7facae7fc640 10 monclient: _finish_auth 0 2024-09-16T04:42:13.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.991+0000 7facae7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:42:13.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.991+0000 7facae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:13.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.991+0000 7facae7fc640 1 -- 172.21.15.46:0/560590798 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac9802add0 con 0x7facb00a3c10 2024-09-16T04:42:13.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.991+0000 7facae7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:13.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.991+0000 7facae7fc640 1 -- 172.21.15.46:0/560590798 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac98032660 con 0x7facb00a3c10 2024-09-16T04:42:13.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.991+0000 7facae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:13.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.991+0000 7facae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:42:13.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.991+0000 7facae7fc640 10 monclient: dump: 2024-09-16T04:42:13.003 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:13.003 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:13.003 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:13.003 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:13.003 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:13.003 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:13.003 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:13.003 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:13.004 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:13.004 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:13.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.991+0000 7facb5dc4640 5 monclient: authenticate success, global_id 5464 2024-09-16T04:42:13.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.991+0000 7facb5dc4640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:13.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.991+0000 7facb5dc4640 4 set_mon_vals no callback set 2024-09-16T04:42:13.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.992+0000 7facb5dc4640 10 monclient: shutdown 2024-09-16T04:42:13.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.992+0000 7facb5dc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:13.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.992+0000 7facb5dc4640 1 -- 172.21.15.46:0/560590798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facb00a3c10 msgr2=0x7facb009bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:13.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.992+0000 7facb5dc4640 1 --2- 172.21.15.46:0/560590798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facb00a3c10 0x7facb009bcf0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fac980097a0 tx=0x7fac9802a080 comp rx=0 tx=0).stop 2024-09-16T04:42:13.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.992+0000 7facb5dc4640 1 -- 172.21.15.46:0/560590798 shutdown_connections 2024-09-16T04:42:13.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.992+0000 7facb5dc4640 1 --2- 172.21.15.46:0/560590798 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facb00a3ff0 0x7facb00a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:13.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.992+0000 7facb5dc4640 1 --2- 172.21.15.46:0/560590798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facb00a3c10 0x7facb009bcf0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:13.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.992+0000 7facb5dc4640 1 -- 172.21.15.46:0/560590798 >> 172.21.15.46:0/560590798 conn(0x7facb00a6340 msgr2=0x7facb006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.992+0000 7facb5dc4640 1 -- 172.21.15.46:0/560590798 shutdown_connections 2024-09-16T04:42:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.992+0000 7facb5dc4640 1 -- 172.21.15.46:0/560590798 wait complete. 2024-09-16T04:42:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.992+0000 7facb5dc4640 10 monclient: build_initial_monmap 2024-09-16T04:42:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.992+0000 7facb5dc4640 10 monclient: monmap: 2024-09-16T04:42:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:13.005 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.993+0000 7facb5dc4640 1 Processor -- start 2024-09-16T04:42:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.993+0000 7facb5dc4640 1 -- start start 2024-09-16T04:42:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.993+0000 7facb5dc4640 10 monclient: init 2024-09-16T04:42:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.993+0000 7facb5dc4640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.993+0000 7facb5dc4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:42:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.993+0000 7facb5dc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facb00a3c10 0x7facb01cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.993+0000 7facb5dc4640 10 monclient(hunting): picked mon.noname-a con 0x7facb00a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.993+0000 7facb5dc4640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facb00a3ff0 0x7facb0135cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.993+0000 7facb5dc4640 10 monclient(hunting): picked mon.noname-c con 0x7facb00a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facb5dc4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facb01369d0 0x7facb01361f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facb5dc4640 10 monclient(hunting): picked mon.noname-b con 0x7facb01369d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facb5dc4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facb5dc4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7facb00a4910 con 0x7facb00a3c10 2024-09-16T04:42:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facb5dc4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:13.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facb5dc4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7facb00a69c0 con 0x7facb01369d0 2024-09-16T04:42:13.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facb5dc4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:13.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facb5dc4640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7facb01273d0 con 0x7facb00a3ff0 2024-09-16T04:42:13.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facb5dc4640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:13.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facb5dc4640 10 monclient(hunting): authenticate will time out at 931.409030s 2024-09-16T04:42:13.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaeffd640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facb00a3ff0 msgr2=0x7facb0135cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:42:13.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaeffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facb00a3ff0 0x7facb0135cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:13.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facb00a3c10 0x7facb01cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:13.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facb01369d0 0x7facb01361f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:13.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facb00a3c10 0x7facb01cbef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49672/0 (socket says 172.21.15.46:49672) 2024-09-16T04:42:13.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaf7fe640 1 -- 172.21.15.46:0/1689544513 learned_addr learned my addr 172.21.15.46:0/1689544513 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:13.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaf7fe640 10 monclient(hunting): get_auth_request con 0x7facb00a3c10 auth_method 0 2024-09-16T04:42:13.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:13.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:13.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:13.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaffff640 10 monclient(hunting): get_auth_request con 0x7facb01369d0 auth_method 0 2024-09-16T04:42:13.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:13.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:13.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:13.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:13.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:13.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:13.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:13.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:13.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:13.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaf7fe640 10 monclient(hunting): handle_auth_done global_id 44175 payload 824 2024-09-16T04:42:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaf7fe640 1 -- 172.21.15.46:0/1689544513 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facb00a3ff0 msgr2=0x7facb0135cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:42:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaf7fe640 1 --2- 172.21.15.46:0/1689544513 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facb00a3ff0 0x7facb0135cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaf7fe640 1 -- 172.21.15.46:0/1689544513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facb01369d0 msgr2=0x7facb01361f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaf7fe640 1 --2- 172.21.15.46:0/1689544513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facb01369d0 0x7facb01361f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaf7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:42:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaf7fe640 1 monclient: found mon.noname-a 2024-09-16T04:42:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaf7fe640 1 -- 172.21.15.46:0/1689544513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facb01cc430 con 0x7facb00a3c10 2024-09-16T04:42:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaffff640 1 --2- 172.21.15.46:0/1689544513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facb01369d0 0x7facb01361f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:42:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facaf7fe640 1 --2- 172.21.15.46:0/1689544513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facb00a3c10 0x7facb01cbef0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fac9802a670 tx=0x7fac980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7faca7fff640 1 -- 172.21.15.46:0/1689544513 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac98041070 con 0x7facb00a3c10 2024-09-16T04:42:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7faca7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7faca7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7faca7fff640 10 monclient: dump: 2024-09-16T04:42:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:13.010 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7faca7fff640 10 monclient: _finish_auth 0 2024-09-16T04:42:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7faca7fff640 10 monclient: _check_auth_tickets 2024-09-16T04:42:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7faca7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7faca7fff640 1 -- 172.21.15.46:0/1689544513 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac9803c670 con 0x7facb00a3c10 2024-09-16T04:42:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facb5dc4640 5 monclient: authenticate success, global_id 44175 2024-09-16T04:42:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facb5dc4640 10 monclient: _renew_subs 2024-09-16T04:42:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facb5dc4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facb5dc4640 1 -- 172.21.15.46:0/1689544513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facb0125860 con 0x7facb00a3c10 2024-09-16T04:42:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7faca7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7faca7fff640 1 -- 172.21.15.46:0/1689544513 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac98037040 con 0x7facb00a3c10 2024-09-16T04:42:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7faca7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7faca7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7faca7fff640 10 monclient: dump: 2024-09-16T04:42:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:13.012 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:13.012 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:13.012 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:13.012 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:13.012 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:13.012 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:13.012 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:13.012 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:13.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.994+0000 7facad7fa640 4 set_mon_vals no callback set 2024-09-16T04:42:13.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.995+0000 7facb5dc4640 10 monclient: _renew_subs 2024-09-16T04:42:13.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.995+0000 7facb5dc4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:13.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.995+0000 7facb5dc4640 1 -- 172.21.15.46:0/1689544513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facb00a8a90 con 0x7facb00a3c10 2024-09-16T04:42:13.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.995+0000 7faca7fff640 1 -- 172.21.15.46:0/1689544513 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fac98044070 con 0x7facb00a3c10 2024-09-16T04:42:13.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.995+0000 7facb5dc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.995+0000 7facb5dc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.995+0000 7facb5dc4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.995+0000 7facb5dc4640 1 -- 172.21.15.46:0/1689544513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facb009bd70 con 0x7facb00a3c10 2024-09-16T04:42:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.995+0000 7faca7fff640 1 --2- 172.21.15.46:0/1689544513 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fac90038360 0x7fac9003a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.996+0000 7faca7fff640 1 -- 172.21.15.46:0/1689544513 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 4738+0+0 (secure 0 0 0) 0x7fac9806bfa0 con 0x7facb00a3c10 2024-09-16T04:42:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.996+0000 7facaeffd640 1 --2- 172.21.15.46:0/1689544513 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fac90038360 0x7fac9003a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.996+0000 7facaeffd640 10 monclient: get_auth_request con 0x7fac90038360 auth_method 0 2024-09-16T04:42:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:12.996+0000 7facaeffd640 1 --2- 172.21.15.46:0/1689544513 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fac90038360 0x7fac9003a820 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7facb01275e0 tx=0x7faca0006d50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.000+0000 7faca7fff640 1 -- 172.21.15.46:0/1689544513 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fac98044720 con 0x7facb00a3c10 2024-09-16T04:42:13.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.000+0000 7faca7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:13.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.000+0000 7faca7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:13.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.243+0000 7facb5dc4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:42:13.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.243+0000 7facb5dc4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:13.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.243+0000 7facb5dc4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:13.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.243+0000 7facb5dc4640 1 -- 172.21.15.46:0/1689544513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7facb0067e20 con 0x7facb00a3c10 2024-09-16T04:42:13.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.244+0000 7faca7fff640 1 -- 172.21.15.46:0/1689544513 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fac9803ca60 con 0x7facb00a3c10 2024-09-16T04:42:13.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.244+0000 7faca7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:13.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.244+0000 7faca7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:13.246 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:42:13.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.246+0000 7facb5dc4640 1 -- 172.21.15.46:0/1689544513 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fac90038360 msgr2=0x7fac9003a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:13.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.246+0000 7facb5dc4640 1 --2- 172.21.15.46:0/1689544513 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fac90038360 0x7fac9003a820 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7facb01275e0 tx=0x7faca0006d50 comp rx=0 tx=0).stop 2024-09-16T04:42:13.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.246+0000 7facb5dc4640 10 monclient: shutdown 2024-09-16T04:42:13.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.246+0000 7facb5dc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:13.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.246+0000 7facb5dc4640 1 -- 172.21.15.46:0/1689544513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facb00a3c10 msgr2=0x7facb01cbef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:13.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.246+0000 7facb5dc4640 1 --2- 172.21.15.46:0/1689544513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facb00a3c10 0x7facb01cbef0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fac9802a670 tx=0x7fac980046c0 comp rx=0 tx=0).stop 2024-09-16T04:42:13.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.246+0000 7facb5dc4640 1 -- 172.21.15.46:0/1689544513 shutdown_connections 2024-09-16T04:42:13.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.246+0000 7facb5dc4640 1 --2- 172.21.15.46:0/1689544513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facb01369d0 0x7facb01361f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:13.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.246+0000 7facb5dc4640 1 --2- 172.21.15.46:0/1689544513 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7facb00a3ff0 0x7facb0135cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:13.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.246+0000 7facb5dc4640 1 --2- 172.21.15.46:0/1689544513 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fac90038360 0x7fac9003a820 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:13.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.246+0000 7facb5dc4640 1 --2- 172.21.15.46:0/1689544513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facb00a3c10 0x7facb01cbef0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:13.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.246+0000 7facb5dc4640 1 -- 172.21.15.46:0/1689544513 >> 172.21.15.46:0/1689544513 conn(0x7facb00a6340 msgr2=0x7facb00a2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:13.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.246+0000 7facb5dc4640 1 -- 172.21.15.46:0/1689544513 shutdown_connections 2024-09-16T04:42:13.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:13.246+0000 7facb5dc4640 1 -- 172.21.15.46:0/1689544513 wait complete. 2024-09-16T04:42:16.258 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:42:16.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.372+0000 7f6aeb52a640 10 monclient: get_monmap_and_config 2024-09-16T04:42:16.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.372+0000 7f6aeb52a640 10 monclient: build_initial_monmap 2024-09-16T04:42:16.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.372+0000 7f6aeb52a640 10 monclient: monmap: 2024-09-16T04:42:16.374 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:16.374 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:16.374 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:16.373047+0000 2024-09-16T04:42:16.374 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:16.373047+0000 2024-09-16T04:42:16.374 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:16.374 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:16.374 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:16.374 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:16.374 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:16.374 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:16.374 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:16.374 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:16.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.373+0000 7f6aeb52a640 1 Processor -- start 2024-09-16T04:42:16.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.373+0000 7f6aeb52a640 1 -- start start 2024-09-16T04:42:16.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.373+0000 7f6aeb52a640 10 monclient: init 2024-09-16T04:42:16.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.373+0000 7f6aeb52a640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:16.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.373+0000 7f6aeb52a640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-16T04:42:16.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.373+0000 7f6aeb52a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6ae40a3c10 0x7f6ae409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:16.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.373+0000 7f6aeb52a640 10 monclient(hunting): picked mon.noname-a con 0x7f6ae40a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:42:16.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.373+0000 7f6aeb52a640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6ae40a3ff0 0x7f6ae40a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:16.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.373+0000 7f6aeb52a640 10 monclient(hunting): picked mon.noname-b con 0x7f6ae40a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:16.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.373+0000 7f6aeb52a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ae41369d0 addr v1:172.21.15.46:6789/0 2024-09-16T04:42:16.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.373+0000 7f6aeb52a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ae409b940 con 0x7f6ae41369d0 2024-09-16T04:42:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6aeb52a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6aeb52a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f6ae409ab00 con 0x7f6ae40a3c10 2024-09-16T04:42:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6aeb52a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6aeb52a640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f6ae409b030 con 0x7f6ae40a3ff0 2024-09-16T04:42:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6aeb52a640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6aeb52a640 10 monclient(hunting): authenticate will time out at 934.789019s 2024-09-16T04:42:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae8ac5640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6ae40a3ff0 msgr2=0x7f6ae40a43d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:42:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae8ac5640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6ae40a3ff0 0x7f6ae40a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae92c6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6ae40a3c10 0x7f6ae409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae92c6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6ae40a3c10 0x7f6ae409bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55902/0 (socket says 172.21.15.46:55902) 2024-09-16T04:42:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae92c6640 1 -- 172.21.15.46:0/3776099832 learned_addr learned my addr 172.21.15.46:0/3776099832 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae92c6640 10 monclient(hunting): get_auth_request con 0x7f6ae40a3c10 auth_method 0 2024-09-16T04:42:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae92c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae9ac7640 1 --1- 172.21.15.46:0/3776099832 >> v1:172.21.15.46:6789/0 conn(0x7f6ae41369d0 0x7f6ae4136e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54840/0 (socket says 172.21.15.46:54840) 2024-09-16T04:42:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae92c6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae92c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae92c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae92c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae92c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae92c6640 10 monclient(hunting): handle_auth_done global_id 44178 payload 274 2024-09-16T04:42:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae92c6640 1 -- 172.21.15.46:0/3776099832 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6ae40a3ff0 msgr2=0x7f6ae40a43d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:42:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae92c6640 1 --2- 172.21.15.46:0/3776099832 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6ae40a3ff0 0x7f6ae40a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae92c6640 1 -- 172.21.15.46:0/3776099832 >> v1:172.21.15.46:6789/0 conn(0x7f6ae41369d0 legacy=0x7f6ae4136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae92c6640 10 monclient: _finish_hunting 0 2024-09-16T04:42:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae92c6640 1 monclient: found mon.noname-a 2024-09-16T04:42:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae92c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae92c6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.374+0000 7f6ae92c6640 1 -- 172.21.15.46:0/3776099832 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae409b310 con 0x7f6ae40a3c10 2024-09-16T04:42:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6ae92c6640 1 --2- 172.21.15.46:0/3776099832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae40a3c10 0x7f6ae409bcf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0009870 tx=0x7f6ae002a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c79ad587eeb893 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6adbfff640 1 -- 172.21.15.46:0/3776099832 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0037070 con 0x7f6ae40a3c10 2024-09-16T04:42:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6adbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6adbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6adbfff640 10 monclient: dump: 2024-09-16T04:42:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:16.386 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6adbfff640 10 monclient: _finish_auth 0 2024-09-16T04:42:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6adbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:42:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6adbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6aeb52a640 5 monclient: authenticate success, global_id 44178 2024-09-16T04:42:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6aeb52a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6adbfff640 1 -- 172.21.15.46:0/3776099832 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ae0032440 con 0x7f6ae40a3c10 2024-09-16T04:42:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6adbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6aeb52a640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6aeb52a640 4 set_mon_vals no callback set 2024-09-16T04:42:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6adbfff640 1 -- 172.21.15.46:0/3776099832 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0032810 con 0x7f6ae40a3c10 2024-09-16T04:42:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6adbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6adbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6adbfff640 10 monclient: dump: 2024-09-16T04:42:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:16.387 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6aeb52a640 10 monclient: shutdown 2024-09-16T04:42:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6aeb52a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6aeb52a640 1 -- 172.21.15.46:0/3776099832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae40a3c10 msgr2=0x7f6ae409bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6aeb52a640 1 --2- 172.21.15.46:0/3776099832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae40a3c10 0x7f6ae409bcf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0009870 tx=0x7f6ae002a150 comp rx=0 tx=0).stop 2024-09-16T04:42:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6aeb52a640 1 -- 172.21.15.46:0/3776099832 shutdown_connections 2024-09-16T04:42:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6aeb52a640 1 --2- 172.21.15.46:0/3776099832 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6ae40a3ff0 0x7f6ae40a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6aeb52a640 1 --2- 172.21.15.46:0/3776099832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae40a3c10 0x7f6ae409bcf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.375+0000 7f6aeb52a640 1 -- 172.21.15.46:0/3776099832 >> 172.21.15.46:0/3776099832 conn(0x7f6ae40a6340 msgr2=0x7f6ae406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.376+0000 7f6aeb52a640 1 -- 172.21.15.46:0/3776099832 shutdown_connections 2024-09-16T04:42:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.376+0000 7f6aeb52a640 1 -- 172.21.15.46:0/3776099832 wait complete. 2024-09-16T04:42:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.376+0000 7f6aeb52a640 10 monclient: build_initial_monmap 2024-09-16T04:42:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.376+0000 7f6aeb52a640 10 monclient: monmap: 2024-09-16T04:42:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:16.389 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:16.389 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:16.389 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:16.389 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:16.389 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:16.389 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:16.389 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:16.389 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:16.389 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:16.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.376+0000 7f6aeb52a640 1 Processor -- start 2024-09-16T04:42:16.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6aeb52a640 1 -- start start 2024-09-16T04:42:16.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6aeb52a640 10 monclient: init 2024-09-16T04:42:16.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6aeb52a640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:16.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6aeb52a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:42:16.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6aeb52a640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6ae40a3c10 0x7f6ae41cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:16.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6aeb52a640 10 monclient(hunting): picked mon.noname-c con 0x7f6ae40a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:16.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6aeb52a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae40a3ff0 0x7f6ae4135cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:16.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6aeb52a640 10 monclient(hunting): picked mon.noname-a con 0x7f6ae40a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:16.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6aeb52a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ae41369d0 0x7f6ae41361f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:16.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6aeb52a640 10 monclient(hunting): picked mon.noname-b con 0x7f6ae41369d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:16.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6aeb52a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:16.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6aeb52a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae40a4910 con 0x7f6ae40a3ff0 2024-09-16T04:42:16.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6aeb52a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:16.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6aeb52a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae40a69c0 con 0x7f6ae41369d0 2024-09-16T04:42:16.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6aeb52a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6aeb52a640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f6ae41273d0 con 0x7f6ae40a3c10 2024-09-16T04:42:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6aeb52a640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6aeb52a640 10 monclient(hunting): authenticate will time out at 934.792525s 2024-09-16T04:42:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae92c6640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6ae40a3c10 msgr2=0x7f6ae41cbef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:42:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae92c6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6ae40a3c10 0x7f6ae41cbef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae8ac5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae40a3ff0 0x7f6ae4135cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae8ac5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae40a3ff0 0x7f6ae4135cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55916/0 (socket says 172.21.15.46:55916) 2024-09-16T04:42:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae8ac5640 1 -- 172.21.15.46:0/4109380420 learned_addr learned my addr 172.21.15.46:0/4109380420 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae8ac5640 10 monclient(hunting): get_auth_request con 0x7f6ae40a3ff0 auth_method 0 2024-09-16T04:42:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae8ac5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae8ac5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae8ac5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae8ac5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae8ac5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae8ac5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae8ac5640 10 monclient(hunting): handle_auth_done global_id 44181 payload 824 2024-09-16T04:42:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae8ac5640 1 -- 172.21.15.46:0/4109380420 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6ae40a3c10 msgr2=0x7f6ae41cbef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:42:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae8ac5640 1 --2- 172.21.15.46:0/4109380420 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6ae40a3c10 0x7f6ae41cbef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae8ac5640 1 -- 172.21.15.46:0/4109380420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ae41369d0 msgr2=0x7f6ae41361f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae9ac7640 1 --2- 172.21.15.46:0/4109380420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ae41369d0 0x7f6ae41361f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae8ac5640 1 --2- 172.21.15.46:0/4109380420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ae41369d0 0x7f6ae41361f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae8ac5640 10 monclient: _finish_hunting 0 2024-09-16T04:42:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae8ac5640 1 monclient: found mon.noname-a 2024-09-16T04:42:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae8ac5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae8ac5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae8ac5640 1 -- 172.21.15.46:0/4109380420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae41cc430 con 0x7f6ae40a3ff0 2024-09-16T04:42:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae9ac7640 10 monclient: get_auth_request con 0x7f6ae41369d0 auth_method 0 2024-09-16T04:42:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.377+0000 7f6ae9ac7640 1 --2- 172.21.15.46:0/4109380420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ae41369d0 0x7f6ae41361f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:42:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6ae8ac5640 1 --2- 172.21.15.46:0/4109380420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae40a3ff0 0x7f6ae4135cb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6ad400b510 tx=0x7f6ad400b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6ad9ffb640 1 -- 172.21.15.46:0/4109380420 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4003d90 con 0x7f6ae40a3ff0 2024-09-16T04:42:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6ad9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6ad9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6ad9ffb640 10 monclient: dump: 2024-09-16T04:42:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:16.394 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6ad9ffb640 10 monclient: _finish_auth 0 2024-09-16T04:42:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6ad9ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:42:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6ad9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6ad9ffb640 1 -- 172.21.15.46:0/4109380420 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ad4015070 con 0x7f6ae40a3ff0 2024-09-16T04:42:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6aeb52a640 5 monclient: authenticate success, global_id 44181 2024-09-16T04:42:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6aeb52a640 10 monclient: _renew_subs 2024-09-16T04:42:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6aeb52a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6aeb52a640 1 -- 172.21.15.46:0/4109380420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ae4125860 con 0x7f6ae40a3ff0 2024-09-16T04:42:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6ad9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6ad9ffb640 1 -- 172.21.15.46:0/4109380420 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad40052b0 con 0x7f6ae40a3ff0 2024-09-16T04:42:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6ad9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6ad9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:16.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6ad9ffb640 10 monclient: dump: 2024-09-16T04:42:16.395 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:16.395 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:16.395 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:16.395 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:16.395 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:16.395 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:16.395 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:16.395 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:16.395 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:16.395 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:16.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6aeb52a640 10 monclient: _renew_subs 2024-09-16T04:42:16.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6adb7fe640 4 set_mon_vals no callback set 2024-09-16T04:42:16.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6aeb52a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.378+0000 7f6aeb52a640 1 -- 172.21.15.46:0/4109380420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ae40a8a90 con 0x7f6ae40a3ff0 2024-09-16T04:42:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.379+0000 7f6ad9ffb640 1 -- 172.21.15.46:0/4109380420 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f6ad4005450 con 0x7f6ae40a3ff0 2024-09-16T04:42:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.379+0000 7f6ad9ffb640 1 --2- 172.21.15.46:0/4109380420 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6ac8038360 0x7f6ac803a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.379+0000 7f6ad9ffb640 1 -- 172.21.15.46:0/4109380420 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 4738+0+0 (secure 0 0 0) 0x7f6ad404b500 con 0x7f6ae40a3ff0 2024-09-16T04:42:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.379+0000 7f6aeb52a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.379+0000 7f6aeb52a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.379+0000 7f6aeb52a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.379+0000 7f6aeb52a640 1 -- 172.21.15.46:0/4109380420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ae409bd70 con 0x7f6ae40a3ff0 2024-09-16T04:42:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.379+0000 7f6ae92c6640 1 --2- 172.21.15.46:0/4109380420 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6ac8038360 0x7f6ac803a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.379+0000 7f6ae92c6640 10 monclient: get_auth_request con 0x7f6ac8038360 auth_method 0 2024-09-16T04:42:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.380+0000 7f6ae92c6640 1 --2- 172.21.15.46:0/4109380420 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6ac8038360 0x7f6ac803a820 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0034000 tx=0x7f6ae002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:16.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.384+0000 7f6ad9ffb640 1 -- 172.21.15.46:0/4109380420 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6ad40193c0 con 0x7f6ae40a3ff0 2024-09-16T04:42:16.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.384+0000 7f6ad9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:16.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.384+0000 7f6ad9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.627+0000 7f6aeb52a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:42:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.627+0000 7f6aeb52a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.627+0000 7f6aeb52a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.627+0000 7f6aeb52a640 1 -- 172.21.15.46:0/4109380420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6ae4067e20 con 0x7f6ae40a3ff0 2024-09-16T04:42:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.628+0000 7f6ad9ffb640 1 -- 172.21.15.46:0/4109380420 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f6ad400de00 con 0x7f6ae40a3ff0 2024-09-16T04:42:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.628+0000 7f6ad9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.628+0000 7f6ad9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:16.629 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:42:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.630+0000 7f6aeb52a640 1 -- 172.21.15.46:0/4109380420 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6ac8038360 msgr2=0x7f6ac803a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.630+0000 7f6aeb52a640 1 --2- 172.21.15.46:0/4109380420 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6ac8038360 0x7f6ac803a820 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0034000 tx=0x7f6ae002e000 comp rx=0 tx=0).stop 2024-09-16T04:42:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.630+0000 7f6aeb52a640 10 monclient: shutdown 2024-09-16T04:42:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.630+0000 7f6aeb52a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.630+0000 7f6aeb52a640 1 -- 172.21.15.46:0/4109380420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae40a3ff0 msgr2=0x7f6ae4135cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.630+0000 7f6aeb52a640 1 --2- 172.21.15.46:0/4109380420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae40a3ff0 0x7f6ae4135cb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6ad400b510 tx=0x7f6ad400b9d0 comp rx=0 tx=0).stop 2024-09-16T04:42:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.630+0000 7f6aeb52a640 1 -- 172.21.15.46:0/4109380420 shutdown_connections 2024-09-16T04:42:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.630+0000 7f6aeb52a640 1 --2- 172.21.15.46:0/4109380420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ae41369d0 0x7f6ae41361f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:16.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.630+0000 7f6aeb52a640 1 --2- 172.21.15.46:0/4109380420 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6ac8038360 0x7f6ac803a820 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:16.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.630+0000 7f6aeb52a640 1 --2- 172.21.15.46:0/4109380420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae40a3ff0 0x7f6ae4135cb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:16.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.630+0000 7f6aeb52a640 1 --2- 172.21.15.46:0/4109380420 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6ae40a3c10 0x7f6ae41cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:16.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.630+0000 7f6aeb52a640 1 -- 172.21.15.46:0/4109380420 >> 172.21.15.46:0/4109380420 conn(0x7f6ae40a6340 msgr2=0x7f6ae40a2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:16.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.630+0000 7f6aeb52a640 1 -- 172.21.15.46:0/4109380420 shutdown_connections 2024-09-16T04:42:16.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:16.630+0000 7f6aeb52a640 1 -- 172.21.15.46:0/4109380420 wait complete. 2024-09-16T04:42:19.643 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:42:19.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.756+0000 7fc9be378640 10 monclient: get_monmap_and_config 2024-09-16T04:42:19.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.756+0000 7fc9be378640 10 monclient: build_initial_monmap 2024-09-16T04:42:19.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.756+0000 7fc9be378640 10 monclient: monmap: 2024-09-16T04:42:19.757 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:19.757 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:19.758 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:19.756757+0000 2024-09-16T04:42:19.758 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:19.756757+0000 2024-09-16T04:42:19.758 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:19.758 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:19.758 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:19.758 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:19.758 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:19.758 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:19.758 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:19.758 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:19.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9be378640 1 Processor -- start 2024-09-16T04:42:19.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9be378640 1 -- start start 2024-09-16T04:42:19.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9be378640 10 monclient: init 2024-09-16T04:42:19.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9be378640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:19.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9be378640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-16T04:42:19.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9be378640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 0x7fc9b809e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:19.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9be378640 10 monclient(hunting): picked mon.noname-a con 0x7fc9b809f810 addr v2:172.21.15.46:3300/0 2024-09-16T04:42:19.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9be378640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc9b809fbf0 addr v1:172.21.15.114:6789/0 2024-09-16T04:42:19.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9be378640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 0x7fc9b813aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:19.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9be378640 10 monclient(hunting): picked mon.noname-c con 0x7fc9b8138610 addr v2:172.21.15.114:3300/0 2024-09-16T04:42:19.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9be378640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9b809c310 con 0x7fc9b809fbf0 2024-09-16T04:42:19.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9be378640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:19.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9be378640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc9b809d290 con 0x7fc9b809f810 2024-09-16T04:42:19.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9be378640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9be378640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fc9b809c840 con 0x7fc9b8138610 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9be378640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9be378640 10 monclient(hunting): authenticate will time out at 938.172741s 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9b7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 0x7fc9b809e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9b7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 0x7fc9b809e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55930/0 (socket says 172.21.15.46:55930) 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9b7fff640 1 -- 172.21.15.46:0/2196987540 learned_addr learned my addr 172.21.15.46:0/2196987540 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9b7fff640 10 monclient(hunting): get_auth_request con 0x7fc9b809f810 auth_method 0 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.757+0000 7fc9b7fff640 1 -- 172.21.15.46:0/2196987540 >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 msgr2=0x7fc9b809e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.758+0000 7fc9b7fff640 1 -- 172.21.15.46:0/2196987540 >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 msgr2=0x7fc9b809e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.758+0000 7fc9b7fff640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 0x7fc9b809e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.758+0000 7fc9b7fff640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 0x7fc9b809e2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.758+0000 7fc9bc915640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 0x7fc9b813aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.758+0000 7fc9bc915640 10 monclient(hunting): get_auth_request con 0x7fc9b8138610 auth_method 0 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.758+0000 7fc9bc915640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.758+0000 7fc9bc915640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.758+0000 7fc9bc915640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.758+0000 7fc9bc915640 1 -- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 msgr2=0x7fc9b813aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.758+0000 7fc9bc915640 1 -- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 msgr2=0x7fc9b813aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.758+0000 7fc9bc915640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 0x7fc9b813aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.758+0000 7fc9bc915640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 0x7fc9b813aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:19.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.959+0000 7fc9b7fff640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 0x7fc9b809e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:19.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.959+0000 7fc9b7fff640 10 monclient(hunting): get_auth_request con 0x7fc9b809f810 auth_method 0 2024-09-16T04:42:19.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.959+0000 7fc9b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:19.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.959+0000 7fc9b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:19.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.959+0000 7fc9b7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:19.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.959+0000 7fc9b7fff640 1 -- 172.21.15.46:0/2196987540 >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 msgr2=0x7fc9b809e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:42:19.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.959+0000 7fc9b7fff640 1 -- 172.21.15.46:0/2196987540 >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 msgr2=0x7fc9b809e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:42:19.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.959+0000 7fc9b7fff640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 0x7fc9b809e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:19.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.959+0000 7fc9b7fff640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 0x7fc9b809e2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:42:19.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.959+0000 7fc9bc915640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 0x7fc9b813aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:19.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.959+0000 7fc9bc915640 10 monclient(hunting): get_auth_request con 0x7fc9b8138610 auth_method 0 2024-09-16T04:42:19.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.959+0000 7fc9bc915640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:19.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.959+0000 7fc9bc915640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:19.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.959+0000 7fc9bc915640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:19.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.959+0000 7fc9bc915640 1 -- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 msgr2=0x7fc9b813aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:42:19.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.959+0000 7fc9bc915640 1 -- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 msgr2=0x7fc9b813aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:42:19.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.959+0000 7fc9bc915640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 0x7fc9b813aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:19.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:19.959+0000 7fc9bc915640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 0x7fc9b813aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:42:20.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:20.360+0000 7fc9b7fff640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 0x7fc9b809e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:20.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:20.360+0000 7fc9b7fff640 10 monclient(hunting): get_auth_request con 0x7fc9b809f810 auth_method 0 2024-09-16T04:42:20.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:20.360+0000 7fc9b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:20.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:20.360+0000 7fc9b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:20.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:20.360+0000 7fc9b7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:20.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:20.360+0000 7fc9b7fff640 1 -- 172.21.15.46:0/2196987540 >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 msgr2=0x7fc9b809e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:42:20.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:20.360+0000 7fc9b7fff640 1 -- 172.21.15.46:0/2196987540 >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 msgr2=0x7fc9b809e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:42:20.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:20.360+0000 7fc9b7fff640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 0x7fc9b809e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:20.360+0000 7fc9b7fff640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 0x7fc9b809e2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:20.360+0000 7fc9bc915640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 0x7fc9b813aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:20.360+0000 7fc9bc915640 10 monclient(hunting): get_auth_request con 0x7fc9b8138610 auth_method 0 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:20.360+0000 7fc9bc915640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:20.360+0000 7fc9bc915640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:20.360+0000 7fc9bc915640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:20.360+0000 7fc9bc915640 1 -- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 msgr2=0x7fc9b813aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:20.360+0000 7fc9bc915640 1 -- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 msgr2=0x7fc9b813aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:20.360+0000 7fc9bc915640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 0x7fc9b813aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:20.360+0000 7fc9bc915640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 0x7fc9b813aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:42:21.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:21.161+0000 7fc9b7fff640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 0x7fc9b809e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:21.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:21.161+0000 7fc9b7fff640 10 monclient(hunting): get_auth_request con 0x7fc9b809f810 auth_method 0 2024-09-16T04:42:21.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:21.161+0000 7fc9b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:21.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:21.161+0000 7fc9b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:21.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:21.161+0000 7fc9b7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:21.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:21.161+0000 7fc9b7fff640 1 -- 172.21.15.46:0/2196987540 >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 msgr2=0x7fc9b809e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:42:21.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:21.161+0000 7fc9b7fff640 1 -- 172.21.15.46:0/2196987540 >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 msgr2=0x7fc9b809e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:42:21.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:21.161+0000 7fc9b7fff640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 0x7fc9b809e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:21.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:21.161+0000 7fc9b7fff640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 0x7fc9b809e2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:42:21.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:21.161+0000 7fc9bc915640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 0x7fc9b813aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:21.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:21.161+0000 7fc9bc915640 10 monclient(hunting): get_auth_request con 0x7fc9b8138610 auth_method 0 2024-09-16T04:42:21.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:21.161+0000 7fc9bc915640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:21.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:21.161+0000 7fc9bc915640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:21.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:21.161+0000 7fc9bc915640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:21.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:21.161+0000 7fc9bc915640 1 -- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 msgr2=0x7fc9b813aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:42:21.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:21.161+0000 7fc9bc915640 1 -- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 msgr2=0x7fc9b813aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:42:21.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:21.161+0000 7fc9bc915640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 0x7fc9b813aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:21.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:21.161+0000 7fc9bc915640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 0x7fc9b813aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:42:22.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:22.758+0000 7fc9b5ffb640 10 monclient(hunting): tick 2024-09-16T04:42:22.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:22.758+0000 7fc9b5ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:42:22.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:22.758+0000 7fc9b5ffb640 1 monclient(hunting): continuing hunt 2024-09-16T04:42:22.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:22.758+0000 7fc9b5ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:42:22.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:22.758+0000 7fc9b5ffb640 1 -- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 msgr2=0x7fc9b813aa00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:42:22.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:22.758+0000 7fc9b5ffb640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.114:3300/0 conn(0x7fc9b8138610 0x7fc9b813aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:22.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:22.758+0000 7fc9b5ffb640 1 -- 172.21.15.46:0/2196987540 >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 msgr2=0x7fc9b809e2c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:42:22.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:22.758+0000 7fc9b5ffb640 1 --2- 172.21.15.46:0/2196987540 >> v2:172.21.15.46:3300/0 conn(0x7fc9b809f810 0x7fc9b809e2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:22.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:22.758+0000 7fc9b5ffb640 1 -- 172.21.15.46:0/2196987540 >> v1:172.21.15.114:6789/0 conn(0x7fc9b809fbf0 legacy=0x7fc9b809ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:22.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:22.758+0000 7fc9b5ffb640 10 monclient: _add_conns ranks=[2,0] 2024-09-16T04:42:22.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:22.758+0000 7fc9b5ffb640 1 --2- 172.21.15.46:0/2196987540 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc9a40036b0 0x7fc9a4005b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:22.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:22.758+0000 7fc9b5ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fc9a40036b0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:22.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:22.758+0000 7fc9b5ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc9a4006170 addr v1:172.21.15.46:6789/0 2024-09-16T04:42:22.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:22.758+0000 7fc9b5ffb640 1 -- 172.21.15.46:0/2196987540 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9b809d850 con 0x7fc9a4006170 2024-09-16T04:42:22.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:22.758+0000 7fc9b5ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:22.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:22.758+0000 7fc9b5ffb640 1 -- 172.21.15.46:0/2196987540 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fc9b809c840 con 0x7fc9a40036b0 2024-09-16T04:42:22.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:22.758+0000 7fc9b5ffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:22.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:22.758+0000 7fc9b7fff640 1 -- 172.21.15.46:0/2196987540 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc9a40036b0 msgr2=0x7fc9a4005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:42:22.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:22.758+0000 7fc9b7fff640 1 --2- 172.21.15.46:0/2196987540 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc9a40036b0 0x7fc9a4005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:22.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:22.959+0000 7fc9b7fff640 1 -- 172.21.15.46:0/2196987540 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc9a40036b0 msgr2=0x7fc9a4005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:42:22.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:22.959+0000 7fc9b7fff640 1 --2- 172.21.15.46:0/2196987540 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc9a40036b0 0x7fc9a4005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:42:23.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.360+0000 7fc9b7fff640 1 -- 172.21.15.46:0/2196987540 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc9a40036b0 msgr2=0x7fc9a4005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:42:23.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.360+0000 7fc9b7fff640 1 --2- 172.21.15.46:0/2196987540 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc9a40036b0 0x7fc9a4005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:42:23.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.504+0000 7fc9b6ffd640 1 -- 172.21.15.46:0/2196987540 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 437031888 0 0) 0x7fc9b809d850 con 0x7fc9a4006170 2024-09-16T04:42:23.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.504+0000 7fc9b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:42:23.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.504+0000 7fc9b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:23.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.504+0000 7fc9b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:23.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.504+0000 7fc9b6ffd640 10 monclient(hunting): my global_id is 44184 2024-09-16T04:42:23.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.504+0000 7fc9b6ffd640 1 -- 172.21.15.46:0/2196987540 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9a0003520 con 0x7fc9a4006170 2024-09-16T04:42:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.505+0000 7fc9b6ffd640 1 -- 172.21.15.46:0/2196987540 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 187596878 0 0) 0x7fc9a0003520 con 0x7fc9a4006170 2024-09-16T04:42:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.505+0000 7fc9b6ffd640 1 -- 172.21.15.46:0/2196987540 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc9a40036b0 msgr2=0x7fc9a4005b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:42:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.505+0000 7fc9b6ffd640 1 --2- 172.21.15.46:0/2196987540 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc9a40036b0 0x7fc9a4005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:42:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:42:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:42:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 1 -- 172.21.15.46:0/2196987540 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a400d1a0 con 0x7fc9a4006170 2024-09-16T04:42:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:42:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:42:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 1 -- 172.21.15.46:0/2196987540 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7fc9ac0016b0 con 0x7fc9a4006170 2024-09-16T04:42:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:42:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 10 monclient: dump: 2024-09-16T04:42:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.510 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.510 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:23.510 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:23.510 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:23.510 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:23.510 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:23.510 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:23.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:42:23.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:23.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 1 -- 172.21.15.46:0/2196987540 >> v1:172.21.15.46:6789/0 conn(0x7fc9a4006170 legacy=0x7fc9a4008610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:23.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9bc915640 1 -- 172.21.15.46:0/2196987540 reap_dead start 2024-09-16T04:42:23.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:42:23.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 1 --2- 172.21.15.46:0/2196987540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9a00039b0 0x7fc9a0003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:23.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 10 monclient(hunting): picked mon.b con 0x7fc9a00039b0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 1 --2- 172.21.15.46:0/2196987540 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc9a0004800 0x7fc9a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 10 monclient(hunting): picked mon.c con 0x7fc9a0004800 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 1 --2- 172.21.15.46:0/2196987540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9a00052e0 0x7fc9a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 10 monclient(hunting): picked mon.a con 0x7fc9a00052e0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 1 -- 172.21.15.46:0/2196987540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b809c840 con 0x7fc9a00052e0 2024-09-16T04:42:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 1 -- 172.21.15.46:0/2196987540 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a000c230 con 0x7fc9a00039b0 2024-09-16T04:42:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 1 -- 172.21.15.46:0/2196987540 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fc9a000c430 con 0x7fc9a0004800 2024-09-16T04:42:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:42:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b77fe640 1 -- 172.21.15.46:0/2196987540 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc9a0004800 msgr2=0x7fc9a0004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b77fe640 1 --2- 172.21.15.46:0/2196987540 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc9a0004800 0x7fc9a0004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b7fff640 1 --2- 172.21.15.46:0/2196987540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9a00039b0 0x7fc9a0003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b7fff640 10 monclient(hunting): get_auth_request con 0x7fc9a00039b0 auth_method 0 2024-09-16T04:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b7fff640 10 monclient(hunting): handle_auth_done global_id 44184 payload 293 2024-09-16T04:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b7fff640 1 -- 172.21.15.46:0/2196987540 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc9a0004800 msgr2=0x7fc9a0004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b7fff640 1 --2- 172.21.15.46:0/2196987540 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc9a0004800 0x7fc9a0004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b7fff640 1 -- 172.21.15.46:0/2196987540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9a00052e0 msgr2=0x7fc9a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b7fff640 1 --2- 172.21.15.46:0/2196987540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9a00052e0 0x7fc9a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:42:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b7fff640 1 monclient: found mon.b 2024-09-16T04:42:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b7fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:42:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b7fff640 1 -- 172.21.15.46:0/2196987540 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a0003520 con 0x7fc9a00039b0 2024-09-16T04:42:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b7fff640 10 monclient: _finish_auth 0 2024-09-16T04:42:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b7fff640 10 monclient: _check_auth_tickets 2024-09-16T04:42:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9be378640 5 monclient: authenticate success, global_id 44184 2024-09-16T04:42:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.506+0000 7fc9be378640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:23.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.508+0000 7fc9b7fff640 1 --2- 172.21.15.46:0/2196987540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9a00039b0 0x7fc9a0003db0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009100 tx=0x7fc9a0005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee26dbec39c91ca3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:23.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.508+0000 7fc9b6ffd640 1 -- 172.21.15.46:0/2196987540 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8031070 con 0x7fc9a00039b0 2024-09-16T04:42:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.508+0000 7fc9b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.508+0000 7fc9b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:42:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.508+0000 7fc9b6ffd640 10 monclient: dump: 2024-09-16T04:42:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:23.519 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.508+0000 7fc9b6ffd640 1 -- 172.21.15.46:0/2196987540 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9a802b8a0 con 0x7fc9a00039b0 2024-09-16T04:42:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.508+0000 7fc9b6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.508+0000 7fc9b6ffd640 1 -- 172.21.15.46:0/2196987540 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a800f040 con 0x7fc9a00039b0 2024-09-16T04:42:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.508+0000 7fc9be378640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.508+0000 7fc9be378640 4 set_mon_vals no callback set 2024-09-16T04:42:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.508+0000 7fc9b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.508+0000 7fc9b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:42:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.508+0000 7fc9b6ffd640 10 monclient: dump: 2024-09-16T04:42:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:23.521 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.509+0000 7fc9be378640 10 monclient: shutdown 2024-09-16T04:42:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.509+0000 7fc9be378640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.509+0000 7fc9be378640 1 -- 172.21.15.46:0/2196987540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9a00039b0 msgr2=0x7fc9a0003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.509+0000 7fc9be378640 1 --2- 172.21.15.46:0/2196987540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9a00039b0 0x7fc9a0003db0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009100 tx=0x7fc9a0005830 comp rx=0 tx=0).stop 2024-09-16T04:42:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.509+0000 7fc9be378640 1 -- 172.21.15.46:0/2196987540 shutdown_connections 2024-09-16T04:42:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.509+0000 7fc9be378640 1 --2- 172.21.15.46:0/2196987540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9a00052e0 0x7fc9a000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.509+0000 7fc9be378640 1 --2- 172.21.15.46:0/2196987540 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc9a0004800 0x7fc9a0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.509+0000 7fc9be378640 1 --2- 172.21.15.46:0/2196987540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9a00039b0 0x7fc9a0003db0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.509+0000 7fc9be378640 1 -- 172.21.15.46:0/2196987540 >> 172.21.15.46:0/2196987540 conn(0x7fc9b80a5960 msgr2=0x7fc9b80a1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.509+0000 7fc9be378640 1 -- 172.21.15.46:0/2196987540 shutdown_connections 2024-09-16T04:42:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.509+0000 7fc9be378640 1 -- 172.21.15.46:0/2196987540 wait complete. 2024-09-16T04:42:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.510+0000 7fc9be378640 10 monclient: build_initial_monmap 2024-09-16T04:42:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.510+0000 7fc9be378640 10 monclient: monmap: 2024-09-16T04:42:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:23.522 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.511+0000 7fc9be378640 1 Processor -- start 2024-09-16T04:42:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.511+0000 7fc9be378640 1 -- start start 2024-09-16T04:42:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.511+0000 7fc9be378640 10 monclient: init 2024-09-16T04:42:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.511+0000 7fc9be378640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.511+0000 7fc9be378640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:42:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.511+0000 7fc9be378640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc9a00039b0 0x7fc9b81cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.511+0000 7fc9be378640 10 monclient(hunting): picked mon.noname-c con 0x7fc9a00039b0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.511+0000 7fc9be378640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9a0004800 0x7fc9b81cc000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9be378640 10 monclient(hunting): picked mon.noname-b con 0x7fc9a0004800 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9be378640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9a00052e0 0x7fc9b81365b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9be378640 10 monclient(hunting): picked mon.noname-a con 0x7fc9a00052e0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9b7fff640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc9a00039b0 msgr2=0x7fc9b81cbac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:42:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9be378640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9be378640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b809e2c0 con 0x7fc9a00052e0 2024-09-16T04:42:23.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9be378640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:23.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9be378640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b809f090 con 0x7fc9a0004800 2024-09-16T04:42:23.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9be378640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:23.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9be378640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fc9b809e440 con 0x7fc9a00039b0 2024-09-16T04:42:23.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9b7fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc9a00039b0 0x7fc9b81cbac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:23.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9be378640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:23.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9be378640 10 monclient(hunting): authenticate will time out at 941.927098s 2024-09-16T04:42:23.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9bc915640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9a00052e0 0x7fc9b81365b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:23.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9b77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9a0004800 0x7fc9b81cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:23.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9bc915640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9a00052e0 0x7fc9b81365b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55988/0 (socket says 172.21.15.46:55988) 2024-09-16T04:42:23.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9bc915640 1 -- 172.21.15.46:0/3404592583 learned_addr learned my addr 172.21.15.46:0/3404592583 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:23.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9bc915640 10 monclient(hunting): get_auth_request con 0x7fc9a00052e0 auth_method 0 2024-09-16T04:42:23.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9bc915640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:23.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9bc915640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:23.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9bc915640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:23.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9b77fe640 10 monclient(hunting): get_auth_request con 0x7fc9a0004800 auth_method 0 2024-09-16T04:42:23.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:23.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:23.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:23.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9bc915640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:23.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9bc915640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:23.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9bc915640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:23.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:23.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:23.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:23.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9bc915640 10 monclient(hunting): handle_auth_done global_id 44187 payload 824 2024-09-16T04:42:23.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9bc915640 1 -- 172.21.15.46:0/3404592583 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc9a00039b0 msgr2=0x7fc9b81cbac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:42:23.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9bc915640 1 --2- 172.21.15.46:0/3404592583 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc9a00039b0 0x7fc9b81cbac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:23.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9bc915640 1 -- 172.21.15.46:0/3404592583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9a0004800 msgr2=0x7fc9b81cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:23.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9bc915640 1 --2- 172.21.15.46:0/3404592583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9a0004800 0x7fc9b81cc000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:23.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9bc915640 10 monclient: _finish_hunting 0 2024-09-16T04:42:23.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9bc915640 1 monclient: found mon.noname-a 2024-09-16T04:42:23.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9bc915640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:23.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9bc915640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:23.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9bc915640 1 -- 172.21.15.46:0/3404592583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b80a3f20 con 0x7fc9a00052e0 2024-09-16T04:42:23.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9bc915640 1 --2- 172.21.15.46:0/3404592583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9a00052e0 0x7fc9b81365b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc9b80a2ce0 tx=0x7fc9b0009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:23.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.512+0000 7fc9b4ff9640 1 -- 172.21.15.46:0/3404592583 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b001b070 con 0x7fc9a00052e0 2024-09-16T04:42:23.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.513+0000 7fc9b4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:23.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.513+0000 7fc9b4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.513+0000 7fc9b4ff9640 10 monclient: dump: 2024-09-16T04:42:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:23.528 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.513+0000 7fc9b4ff9640 10 monclient: _finish_auth 0 2024-09-16T04:42:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.513+0000 7fc9b4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:42:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.513+0000 7fc9b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.513+0000 7fc9b4ff9640 1 -- 172.21.15.46:0/3404592583 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9b00139b0 con 0x7fc9a00052e0 2024-09-16T04:42:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.513+0000 7fc9be378640 5 monclient: authenticate success, global_id 44187 2024-09-16T04:42:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.513+0000 7fc9be378640 10 monclient: _renew_subs 2024-09-16T04:42:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.513+0000 7fc9be378640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.513+0000 7fc9be378640 1 -- 172.21.15.46:0/3404592583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9b809c100 con 0x7fc9a00052e0 2024-09-16T04:42:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.513+0000 7fc9b4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.513+0000 7fc9b4ff9640 1 -- 172.21.15.46:0/3404592583 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b0013d80 con 0x7fc9a00052e0 2024-09-16T04:42:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.513+0000 7fc9b4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.513+0000 7fc9b4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.513+0000 7fc9b4ff9640 10 monclient: dump: 2024-09-16T04:42:23.529 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:23.529 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:23.529 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.529 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.529 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:23.529 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:23.529 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:23.529 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:23.529 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:23.529 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:23.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.513+0000 7fc9be378640 10 monclient: _renew_subs 2024-09-16T04:42:23.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.513+0000 7fc9be378640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:23.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.513+0000 7fc9be378640 1 -- 172.21.15.46:0/3404592583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9b81219b0 con 0x7fc9a00052e0 2024-09-16T04:42:23.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.513+0000 7fc9b67fc640 4 set_mon_vals no callback set 2024-09-16T04:42:23.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.514+0000 7fc9b4ff9640 1 -- 172.21.15.46:0/3404592583 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fc9b001e070 con 0x7fc9a00052e0 2024-09-16T04:42:23.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.514+0000 7fc9be378640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:23.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.514+0000 7fc9be378640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:23.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.514+0000 7fc9be378640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:23.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.514+0000 7fc9be378640 1 -- 172.21.15.46:0/3404592583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9b809ffd0 con 0x7fc9a00052e0 2024-09-16T04:42:23.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.514+0000 7fc9b4ff9640 1 --2- 172.21.15.46:0/3404592583 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc9a4038ce0 0x7fc9a403b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:23.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.514+0000 7fc9b4ff9640 1 -- 172.21.15.46:0/3404592583 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 4738+0+0 (secure 0 0 0) 0x7fc9b004c400 con 0x7fc9a00052e0 2024-09-16T04:42:23.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.514+0000 7fc9b7fff640 1 --2- 172.21.15.46:0/3404592583 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc9a4038ce0 0x7fc9a403b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:23.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.514+0000 7fc9b7fff640 10 monclient: get_auth_request con 0x7fc9a4038ce0 auth_method 0 2024-09-16T04:42:23.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.515+0000 7fc9b7fff640 1 --2- 172.21.15.46:0/3404592583 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc9a4038ce0 0x7fc9a403b1a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8036000 tx=0x7fc9a8010040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:23.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.519+0000 7fc9b4ff9640 1 -- 172.21.15.46:0/3404592583 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc9b0025020 con 0x7fc9a00052e0 2024-09-16T04:42:23.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.519+0000 7fc9b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:23.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.519+0000 7fc9b4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:23.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.769+0000 7fc9be378640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:42:23.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.769+0000 7fc9be378640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:23.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.769+0000 7fc9be378640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:23.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.769+0000 7fc9be378640 1 -- 172.21.15.46:0/3404592583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc9b806c020 con 0x7fc9a00052e0 2024-09-16T04:42:23.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.769+0000 7fc9b4ff9640 1 -- 172.21.15.46:0/3404592583 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fc9b0028350 con 0x7fc9a00052e0 2024-09-16T04:42:23.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.769+0000 7fc9b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:23.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.769+0000 7fc9b4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:23.771 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":42,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:42:23.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.772+0000 7fc9be378640 1 -- 172.21.15.46:0/3404592583 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc9a4038ce0 msgr2=0x7fc9a403b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:23.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.772+0000 7fc9be378640 1 --2- 172.21.15.46:0/3404592583 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc9a4038ce0 0x7fc9a403b1a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8036000 tx=0x7fc9a8010040 comp rx=0 tx=0).stop 2024-09-16T04:42:23.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.772+0000 7fc9be378640 10 monclient: shutdown 2024-09-16T04:42:23.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.772+0000 7fc9be378640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:23.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.772+0000 7fc9be378640 1 -- 172.21.15.46:0/3404592583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9a00052e0 msgr2=0x7fc9b81365b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:23.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.772+0000 7fc9be378640 1 --2- 172.21.15.46:0/3404592583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9a00052e0 0x7fc9b81365b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc9b80a2ce0 tx=0x7fc9b0009cd0 comp rx=0 tx=0).stop 2024-09-16T04:42:23.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.772+0000 7fc9be378640 1 -- 172.21.15.46:0/3404592583 shutdown_connections 2024-09-16T04:42:23.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.772+0000 7fc9be378640 1 --2- 172.21.15.46:0/3404592583 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc9a4038ce0 0x7fc9a403b1a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:23.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.772+0000 7fc9be378640 1 --2- 172.21.15.46:0/3404592583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9a00052e0 0x7fc9b81365b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:23.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.772+0000 7fc9be378640 1 --2- 172.21.15.46:0/3404592583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9a0004800 0x7fc9b81cc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:23.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.772+0000 7fc9be378640 1 --2- 172.21.15.46:0/3404592583 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc9a00039b0 0x7fc9b81cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:23.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.772+0000 7fc9be378640 1 -- 172.21.15.46:0/3404592583 >> 172.21.15.46:0/3404592583 conn(0x7fc9b80a5960 msgr2=0x7fc9b8072ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:23.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.772+0000 7fc9be378640 1 -- 172.21.15.46:0/3404592583 shutdown_connections 2024-09-16T04:42:23.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.772+0000 7fc9be378640 1 -- 172.21.15.46:0/3404592583 wait complete. 2024-09-16T04:42:23.784 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:42:23.784 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:42:23.784 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:42:23.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.885+0000 7f17f5bfd640 10 monclient: get_monmap_and_config 2024-09-16T04:42:23.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.885+0000 7f17f5bfd640 10 monclient: build_initial_monmap 2024-09-16T04:42:23.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.885+0000 7f17f5bfd640 10 monclient: monmap: 2024-09-16T04:42:23.886 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:23.886 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:23.887 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:23.885856+0000 2024-09-16T04:42:23.887 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:23.885856+0000 2024-09-16T04:42:23.887 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:23.887 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:23.887 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:23.887 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:23.887 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:23.887 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:23.887 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:23.887 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:23.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.886+0000 7f17f5bfd640 1 Processor -- start 2024-09-16T04:42:23.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.886+0000 7f17f5bfd640 1 -- start start 2024-09-16T04:42:23.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.886+0000 7f17f5bfd640 10 monclient: init 2024-09-16T04:42:23.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.886+0000 7f17f5bfd640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:23.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.886+0000 7f17f5bfd640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-16T04:42:23.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.886+0000 7f17f5bfd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f17f0131a10 0x7f17f009ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:23.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.886+0000 7f17f5bfd640 10 monclient(hunting): picked mon.noname-b con 0x7f17f0131a10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:23.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.886+0000 7f17f5bfd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f17f009e760 addr v1:172.21.15.46:6789/0 2024-09-16T04:42:23.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.886+0000 7f17f5bfd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f17f0139af0 0x7f17f013bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:23.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.886+0000 7f17f5bfd640 10 monclient(hunting): picked mon.noname-c con 0x7f17f0139af0 addr v2:172.21.15.114:3300/0 2024-09-16T04:42:23.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.886+0000 7f17f5bfd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17f00a0f40 con 0x7f17f009e760 2024-09-16T04:42:23.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.886+0000 7f17f5bfd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:23.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.886+0000 7f17f5bfd640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f17f00a1220 con 0x7f17f0139af0 2024-09-16T04:42:23.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.886+0000 7f17f5bfd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:23.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.886+0000 7f17f5bfd640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f17f009f810 con 0x7f17f0131a10 2024-09-16T04:42:23.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.886+0000 7f17f5bfd640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:23.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.886+0000 7f17f5bfd640 10 monclient(hunting): authenticate will time out at 942.301852s 2024-09-16T04:42:23.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.886+0000 7f17ef7fe640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f17f0131a10 msgr2=0x7f17f009ec70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:42:23.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.886+0000 7f17ef7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f17f0131a10 0x7f17f009ec70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:23.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.886+0000 7f17e7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f17f009e760 0x7f17f0131df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54852/0 (socket says 172.21.15.46:54852) 2024-09-16T04:42:23.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17e7fff640 1 -- 172.21.15.46:0/4199806034 learned_addr learned my addr 172.21.15.46:0/4199806034 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:42:23.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17effff640 1 --2- 172.21.15.46:0/4199806034 >> v2:172.21.15.114:3300/0 conn(0x7f17f0139af0 0x7f17f013bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:23.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17eeffd640 1 -- 172.21.15.46:0/4199806034 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2353678712 0 0) 0x7f17f00a0f40 con 0x7f17f009e760 2024-09-16T04:42:23.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17eeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:42:23.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:23.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:23.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17eeffd640 10 monclient(hunting): my global_id is 44190 2024-09-16T04:42:23.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17eeffd640 1 -- 172.21.15.46:0/4199806034 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17dc003520 con 0x7f17f009e760 2024-09-16T04:42:23.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17effff640 10 monclient(hunting): get_auth_request con 0x7f17f0139af0 auth_method 0 2024-09-16T04:42:23.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:23.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17effff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:23.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:23.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:23.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:23.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:23.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17effff640 10 monclient(hunting): handle_auth_done global_id 5473 payload 274 2024-09-16T04:42:23.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17effff640 1 -- 172.21.15.46:0/4199806034 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f17f0131a10 msgr2=0x7f17f009ec70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:42:23.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17effff640 1 --2- 172.21.15.46:0/4199806034 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f17f0131a10 0x7f17f009ec70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:23.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17eeffd640 1 -- 172.21.15.46:0/4199806034 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 29919334 0 0) 0x7f17dc003520 con 0x7f17f009e760 2024-09-16T04:42:23.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17effff640 1 -- 172.21.15.46:0/4199806034 >> v1:172.21.15.46:6789/0 conn(0x7f17f009e760 legacy=0x7f17f0131df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:23.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17effff640 10 monclient: _finish_hunting 0 2024-09-16T04:42:23.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17effff640 1 monclient: found mon.noname-c 2024-09-16T04:42:23.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:23.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17effff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:42:23.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17effff640 1 -- 172.21.15.46:0/4199806034 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f00a2290 con 0x7f17f0139af0 2024-09-16T04:42:23.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.887+0000 7f17eeffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:42:23.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.888+0000 7f17effff640 1 --2- 172.21.15.46:0/4199806034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f0139af0 0x7f17f013bee0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009d90 tx=0x7f17e802a4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6443d0f1af87a966 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:23.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.888+0000 7f17eeffd640 1 -- 172.21.15.46:0/4199806034 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d8001670 con 0x7f17f0139af0 2024-09-16T04:42:23.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.888+0000 7f17eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:23.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.888+0000 7f17eeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:42:23.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.888+0000 7f17eeffd640 10 monclient: dump: 2024-09-16T04:42:23.901 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:23.901 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:23.901 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.901 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.901 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:23.901 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:23.901 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:23.901 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:23.901 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:23.901 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:23.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.888+0000 7f17eeffd640 10 monclient: _finish_auth 0 2024-09-16T04:42:23.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.888+0000 7f17eeffd640 10 monclient: _check_auth_tickets 2024-09-16T04:42:23.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.888+0000 7f17eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:23.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.888+0000 7f17eeffd640 1 -- 172.21.15.46:0/4199806034 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17e8007540 con 0x7f17f0139af0 2024-09-16T04:42:23.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.888+0000 7f17f5bfd640 5 monclient: authenticate success, global_id 5473 2024-09-16T04:42:23.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.888+0000 7f17f5bfd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:23.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.888+0000 7f17eeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:23.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.888+0000 7f17f5bfd640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:23.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.888+0000 7f17f5bfd640 4 set_mon_vals no callback set 2024-09-16T04:42:23.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.888+0000 7f17eeffd640 1 -- 172.21.15.46:0/4199806034 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8007910 con 0x7f17f0139af0 2024-09-16T04:42:23.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.888+0000 7f17eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:23.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.888+0000 7f17eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:42:23.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.888+0000 7f17eeffd640 10 monclient: dump: 2024-09-16T04:42:23.902 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:23.903 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:23.903 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.903 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.903 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:23.903 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:23.903 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:23.903 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:23.903 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:23.903 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:23.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.889+0000 7f17f5bfd640 10 monclient: shutdown 2024-09-16T04:42:23.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.889+0000 7f17f5bfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:23.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.889+0000 7f17f5bfd640 1 -- 172.21.15.46:0/4199806034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f0139af0 msgr2=0x7f17f013bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:23.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.889+0000 7f17f5bfd640 1 --2- 172.21.15.46:0/4199806034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f0139af0 0x7f17f013bee0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009d90 tx=0x7f17e802a4e0 comp rx=0 tx=0).stop 2024-09-16T04:42:23.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.889+0000 7f17f5bfd640 1 -- 172.21.15.46:0/4199806034 shutdown_connections 2024-09-16T04:42:23.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.889+0000 7f17f5bfd640 1 --2- 172.21.15.46:0/4199806034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f0139af0 0x7f17f013bee0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:23.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.889+0000 7f17f5bfd640 1 --2- 172.21.15.46:0/4199806034 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f17f0131a10 0x7f17f009ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:23.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.889+0000 7f17f5bfd640 1 -- 172.21.15.46:0/4199806034 >> 172.21.15.46:0/4199806034 conn(0x7f17f00a5960 msgr2=0x7f17f00a1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:23.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.889+0000 7f17f5bfd640 1 -- 172.21.15.46:0/4199806034 shutdown_connections 2024-09-16T04:42:23.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.889+0000 7f17f5bfd640 1 -- 172.21.15.46:0/4199806034 wait complete. 2024-09-16T04:42:23.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.889+0000 7f17f5bfd640 10 monclient: build_initial_monmap 2024-09-16T04:42:23.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.889+0000 7f17f5bfd640 10 monclient: monmap: 2024-09-16T04:42:23.904 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:23.904 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:23.904 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:23.904 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:23.904 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:23.904 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:23.904 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:23.905 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:23.905 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:23.905 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:23.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17f5bfd640 1 Processor -- start 2024-09-16T04:42:23.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17f5bfd640 1 -- start start 2024-09-16T04:42:23.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17f5bfd640 10 monclient: init 2024-09-16T04:42:23.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17f5bfd640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:23.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17f5bfd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:42:23.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17f5bfd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f009e760 0x7f17f0135f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:23.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17f5bfd640 10 monclient(hunting): picked mon.noname-a con 0x7f17f009e760 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:23.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17f5bfd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f17f0131a10 0x7f17f0136450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:23.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17f5bfd640 10 monclient(hunting): picked mon.noname-c con 0x7f17f0131a10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:23.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17f5bfd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f0139af0 0x7f17f0136990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:23.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17f5bfd640 10 monclient(hunting): picked mon.noname-b con 0x7f17f0139af0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:23.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17f5bfd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:23.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17f5bfd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f17f009c6e0 con 0x7f17f009e760 2024-09-16T04:42:23.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17f5bfd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:23.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17f5bfd640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f17f009c560 con 0x7f17f0139af0 2024-09-16T04:42:23.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17f5bfd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:23.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17f5bfd640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f17f009c860 con 0x7f17f0131a10 2024-09-16T04:42:23.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17f5bfd640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:23.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17f5bfd640 10 monclient(hunting): authenticate will time out at 942.305588s 2024-09-16T04:42:23.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17e7fff640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f17f0131a10 msgr2=0x7f17f0136450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:42:23.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17e7fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f17f0131a10 0x7f17f0136450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:23.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17ef7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f009e760 0x7f17f0135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:23.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17ef7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f009e760 0x7f17f0135f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55996/0 (socket says 172.21.15.46:55996) 2024-09-16T04:42:23.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17ef7fe640 1 -- 172.21.15.46:0/2599311441 learned_addr learned my addr 172.21.15.46:0/2599311441 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17ef7fe640 10 monclient(hunting): get_auth_request con 0x7f17f009e760 auth_method 0 2024-09-16T04:42:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17effff640 1 --2- 172.21.15.46:0/2599311441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f0139af0 0x7f17f0136990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17effff640 10 monclient(hunting): get_auth_request con 0x7f17f0139af0 auth_method 0 2024-09-16T04:42:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17effff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17ef7fe640 10 monclient(hunting): handle_auth_done global_id 44193 payload 824 2024-09-16T04:42:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17ef7fe640 1 -- 172.21.15.46:0/2599311441 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f17f0131a10 msgr2=0x7f17f0136450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:42:23.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17ef7fe640 1 --2- 172.21.15.46:0/2599311441 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f17f0131a10 0x7f17f0136450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:23.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17ef7fe640 1 -- 172.21.15.46:0/2599311441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f0139af0 msgr2=0x7f17f0136990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:23.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17ef7fe640 1 --2- 172.21.15.46:0/2599311441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f0139af0 0x7f17f0136990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:23.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17ef7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:42:23.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17ef7fe640 1 monclient: found mon.noname-a 2024-09-16T04:42:23.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:23.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17ef7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:23.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.890+0000 7f17ef7fe640 1 -- 172.21.15.46:0/2599311441 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f0137bb0 con 0x7f17f009e760 2024-09-16T04:42:23.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17ef7fe640 1 --2- 172.21.15.46:0/2599311441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f009e760 0x7f17f0135f10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f17e000b580 tx=0x7f17e000ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:23.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17ecff9640 1 -- 172.21.15.46:0/2599311441 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e0003dc0 con 0x7f17f009e760 2024-09-16T04:42:23.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:23.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17ecff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:23.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17ecff9640 10 monclient: dump: 2024-09-16T04:42:23.909 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:23.909 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:23.909 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.909 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.909 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:23.909 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:23.909 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:23.909 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:23.909 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:23.909 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:23.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17ecff9640 10 monclient: _finish_auth 0 2024-09-16T04:42:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17ecff9640 10 monclient: _check_auth_tickets 2024-09-16T04:42:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17f5bfd640 5 monclient: authenticate success, global_id 44193 2024-09-16T04:42:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17ecff9640 1 -- 172.21.15.46:0/2599311441 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17e0015070 con 0x7f17f009e760 2024-09-16T04:42:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17f5bfd640 10 monclient: _renew_subs 2024-09-16T04:42:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17f5bfd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17f5bfd640 1 -- 172.21.15.46:0/2599311441 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f0139600 con 0x7f17f009e760 2024-09-16T04:42:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17ecff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17ecff9640 1 -- 172.21.15.46:0/2599311441 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e0005330 con 0x7f17f009e760 2024-09-16T04:42:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17ecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17ecff9640 10 monclient: dump: 2024-09-16T04:42:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:23.911 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17ee7fc640 4 set_mon_vals no callback set 2024-09-16T04:42:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17f5bfd640 10 monclient: _renew_subs 2024-09-16T04:42:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17f5bfd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.891+0000 7f17f5bfd640 1 -- 172.21.15.46:0/2599311441 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f0138710 con 0x7f17f009e760 2024-09-16T04:42:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17f5bfd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17f5bfd640 1 --2- 172.21.15.46:0/2599311441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f0136ed0 0x7f17f006ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17f5bfd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17f5bfd640 1 -- 172.21.15.46:0/2599311441 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f17f009ec70 con 0x7f17f0136ed0 2024-09-16T04:42:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17ecff9640 1 -- 172.21.15.46:0/2599311441 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f17e00054d0 con 0x7f17f009e760 2024-09-16T04:42:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17e7fff640 1 --2- 172.21.15.46:0/2599311441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f0136ed0 0x7f17f006ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17e7fff640 10 monclient: get_auth_request con 0x7f17f0136ed0 auth_method 0 2024-09-16T04:42:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17ecff9640 1 --2- 172.21.15.46:0/2599311441 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f17d0038360 0x7f17d003a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17ecff9640 1 -- 172.21.15.46:0/2599311441 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 4738+0+0 (secure 0 0 0) 0x7f17e004b5c0 con 0x7f17f009e760 2024-09-16T04:42:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17e7fff640 10 monclient(hunting): handle_auth_done global_id 44196 payload 824 2024-09-16T04:42:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17e7fff640 1 -- 172.21.15.46:0/2599311441 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f17f01372b0 con 0x7f17f0136ed0 2024-09-16T04:42:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17effff640 1 --2- 172.21.15.46:0/2599311441 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f17d0038360 0x7f17d003a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17e7fff640 1 --2- 172.21.15.46:0/2599311441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f0136ed0 0x7f17f006ed80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f17f011baf0 tx=0x7f17d800caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17effff640 10 monclient: get_auth_request con 0x7f17d0038360 auth_method 0 2024-09-16T04:42:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17ecff9640 1 -- 172.21.15.46:0/2599311441 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d8005ce0 con 0x7f17f0136ed0 2024-09-16T04:42:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17ecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.892+0000 7f17ecff9640 10 monclient: dump: 2024-09-16T04:42:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:23.914 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:23.914 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:23.914 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:23.914 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:23.914 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:23.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.893+0000 7f17ecff9640 1 -- 172.21.15.46:0/2599311441 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f17d8005ce0 con 0x7f17f0136ed0 2024-09-16T04:42:23.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.893+0000 7f17ecff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:23.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.893+0000 7f17ecff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:23.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.893+0000 7f17ecff9640 1 -- 172.21.15.46:0/2599311441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f0136ed0 msgr2=0x7f17f006ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:23.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.893+0000 7f17ecff9640 1 --2- 172.21.15.46:0/2599311441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f0136ed0 0x7f17f006ed80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f17f011baf0 tx=0x7f17d800caa0 comp rx=0 tx=0).stop 2024-09-16T04:42:23.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.893+0000 7f17ecff9640 1 -- 172.21.15.46:0/2599311441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f0136ed0 msgr2=0x7f17f006ed80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:23.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.893+0000 7f17ecff9640 1 --2- 172.21.15.46:0/2599311441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f0136ed0 0x7f17f006ed80 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f17f011baf0 tx=0x7f17d800caa0 comp rx=0 tx=0).stop 2024-09-16T04:42:23.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.893+0000 7f17effff640 1 --2- 172.21.15.46:0/2599311441 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f17d0038360 0x7f17d003a820 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f17e8034000 tx=0x7f17e802f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:23.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.899+0000 7f17f5bfd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:42:23.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.899+0000 7f17f5bfd640 1 --2- 172.21.15.46:0/2599311441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f006f2c0 0x7f17f006f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.899+0000 7f17f5bfd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.899+0000 7f17f5bfd640 1 -- 172.21.15.46:0/2599311441 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f17f0138490 con 0x7f17f006f2c0 2024-09-16T04:42:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.899+0000 7f17e7fff640 1 --2- 172.21.15.46:0/2599311441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f006f2c0 0x7f17f006f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.899+0000 7f17e7fff640 10 monclient: get_auth_request con 0x7f17f006f2c0 auth_method 0 2024-09-16T04:42:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.899+0000 7f17e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.899+0000 7f17e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.899+0000 7f17e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.899+0000 7f17e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.899+0000 7f17e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.899+0000 7f17e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.899+0000 7f17e7fff640 10 monclient(hunting): handle_auth_done global_id 44199 payload 824 2024-09-16T04:42:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.899+0000 7f17e7fff640 1 -- 172.21.15.46:0/2599311441 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f17f006c020 con 0x7f17f006f2c0 2024-09-16T04:42:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.899+0000 7f17e7fff640 1 --2- 172.21.15.46:0/2599311441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f006f2c0 0x7f17f006f6d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f17d80017e0 tx=0x7f17d800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.899+0000 7f17ecff9640 1 -- 172.21.15.46:0/2599311441 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d801cc50 con 0x7f17f006f2c0 2024-09-16T04:42:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.899+0000 7f17ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.899+0000 7f17ecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.899+0000 7f17ecff9640 10 monclient: dump: 2024-09-16T04:42:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:23.918 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.900+0000 7f17ecff9640 1 -- 172.21.15.46:0/2599311441 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4702 (secure 0 0 0) 0x7f17d8012400 con 0x7f17f006f2c0 2024-09-16T04:42:23.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.900+0000 7f17ecff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:42:23.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.900+0000 7f17ecff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:23.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.900+0000 7f17ecff9640 1 -- 172.21.15.46:0/2599311441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f006f2c0 msgr2=0x7f17f006f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:23.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.900+0000 7f17ecff9640 1 --2- 172.21.15.46:0/2599311441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f006f2c0 0x7f17f006f6d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f17d80017e0 tx=0x7f17d800f000 comp rx=0 tx=0).stop 2024-09-16T04:42:23.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.900+0000 7f17ecff9640 1 -- 172.21.15.46:0/2599311441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f006f2c0 msgr2=0x7f17f006f6d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:23.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.900+0000 7f17ecff9640 1 --2- 172.21.15.46:0/2599311441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f006f2c0 0x7f17f006f6d0 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f17d80017e0 tx=0x7f17d800f000 comp rx=0 tx=0).stop 2024-09-16T04:42:23.919 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:42:23.919 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:42:23.919 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:42:23.919 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:42:23.919 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 42, 2024-09-16T04:42:23.920 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 23646, 2024-09-16T04:42:23.920 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:42:23.920 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:42:23.920 INFO:teuthology.orchestra.run.smithi046.stdout: 1 2024-09-16T04:42:23.920 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:23.920 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:42:23.920 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:23.920 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:42:23.920 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:42:23.920 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:23.920 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:23.920 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:23.920 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:23.921 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:23.921 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:23.921 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:23.921 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:23.921 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:23.921 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:23.921 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:23.921 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:23.921 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:42:23.921 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:42:23.921 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:23.921 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:23.921 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:23.921 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:23.922 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:23.922 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:23.922 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:23.922 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:23.922 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:23.922 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:23.922 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:23.922 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:23.922 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:23.922 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:42:23.922 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:42:23.922 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:42:23.922 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:42:23.922 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:42:23.923 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:42:23.923 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:23.923 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:23.923 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:42:23.923 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:42:23.923 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:42:23.923 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:42:23.923 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:42:23.923 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:42:23.923 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:42:23.923 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:23.923 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:42:23.923 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:23.923 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:23.924 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:23.924 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:23.924 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:23.924 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:23.924 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:23.924 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:23.924 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:23.924 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:23.924 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:23.924 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:23.925 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:42:23.925 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:23.925 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:42:23.925 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:23.925 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:42:23.925 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:42:23.925 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:23.925 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:23.925 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:23.925 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:23.925 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:42:23.925 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:23.925 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:23.925 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:23.926 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:23.926 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:42:23.926 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:23.926 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:23.926 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:23.926 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:23.926 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:42:23.926 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:42:23.926 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:23.926 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:23.926 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:23.926 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:23.926 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:23.926 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:42:23.927 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:42:23.927 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:23.927 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:23.927 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:23.927 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:23.927 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:42:23.927 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:23.927 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:23.927 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:23.927 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:23.927 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:42:23.928 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:23.928 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:23.928 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:23.928 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:23.928 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:42:23.928 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:42:23.928 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:23.928 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:23.928 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:23.929 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:23.929 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:23.929 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:42:23.929 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:42:23.929 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:23.929 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:23.929 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:23.929 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:23.929 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:42:23.929 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:23.929 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:23.929 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:23.929 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:23.929 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:42:23.930 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:23.930 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:23.930 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:23.930 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:23.930 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:42:23.930 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:42:23.930 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:23.930 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:23.930 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:23.930 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:23.930 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:23.930 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:23.930 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:42:23.930 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:42:23.931 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:23.931 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:23.931 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:23.931 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:42:23.931 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:23.931 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:23.931 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:42:23.931 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:23.931 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:23.931 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:23.931 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:42:23.931 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:23.931 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:23.931 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:42:23.932 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:23.932 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:23.932 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:23.932 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:42:23.932 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:23.932 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:23.932 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:23.932 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:42:23.932 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:42:23.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.900+0000 7f17f5bfd640 1 -- 172.21.15.46:0/2599311441 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f17d0038360 msgr2=0x7f17d003a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:23.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.900+0000 7f17f5bfd640 1 --2- 172.21.15.46:0/2599311441 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f17d0038360 0x7f17d003a820 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f17e8034000 tx=0x7f17e802f000 comp rx=0 tx=0).stop 2024-09-16T04:42:23.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.900+0000 7f17f5bfd640 10 monclient: shutdown 2024-09-16T04:42:23.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.900+0000 7f17f5bfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:23.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.900+0000 7f17f5bfd640 1 -- 172.21.15.46:0/2599311441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f009e760 msgr2=0x7f17f0135f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:23.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.900+0000 7f17f5bfd640 1 --2- 172.21.15.46:0/2599311441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17f009e760 0x7f17f0135f10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f17e000b580 tx=0x7f17e000ba40 comp rx=0 tx=0).stop 2024-09-16T04:42:23.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.900+0000 7f17effff640 1 -- 172.21.15.46:0/2599311441 reap_dead start 2024-09-16T04:42:23.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.900+0000 7f17effff640 1 -- 172.21.15.46:0/2599311441 reap_dead start 2024-09-16T04:42:23.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.900+0000 7f17f5bfd640 1 -- 172.21.15.46:0/2599311441 shutdown_connections 2024-09-16T04:42:23.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.900+0000 7f17f5bfd640 1 -- 172.21.15.46:0/2599311441 >> 172.21.15.46:0/2599311441 conn(0x7f17f00a5960 msgr2=0x7f17f00a3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:23.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.900+0000 7f17f5bfd640 1 -- 172.21.15.46:0/2599311441 shutdown_connections 2024-09-16T04:42:23.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:23.900+0000 7f17f5bfd640 1 -- 172.21.15.46:0/2599311441 wait complete. 2024-09-16T04:42:23.935 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:42:23.935 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:42:23.935 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:42:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.036+0000 7f1298949640 10 monclient: get_monmap_and_config 2024-09-16T04:42:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.036+0000 7f1298949640 10 monclient: build_initial_monmap 2024-09-16T04:42:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.036+0000 7f1298949640 10 monclient: monmap: 2024-09-16T04:42:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:24.036761+0000 2024-09-16T04:42:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:24.036761+0000 2024-09-16T04:42:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:24.039 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:24.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.036+0000 7f1298949640 1 Processor -- start 2024-09-16T04:42:24.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.036+0000 7f1298949640 1 -- start start 2024-09-16T04:42:24.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.036+0000 7f1298949640 10 monclient: init 2024-09-16T04:42:24.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f1298949640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:24.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f1298949640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-16T04:42:24.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f1298949640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f12940a3c10 0x7f129409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:24.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f1298949640 10 monclient(hunting): picked mon.noname-a con 0x7f12940a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:42:24.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f1298949640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f12940a3ff0 0x7f12940a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:24.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f1298949640 10 monclient(hunting): picked mon.noname-c con 0x7f12940a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:42:24.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f1298949640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12941369d0 0x7f1294136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:24.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f1298949640 10 monclient(hunting): picked mon.noname-b con 0x7f12941369d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:24.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f1298949640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:24.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f1298949640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f129409b940 con 0x7f12940a3c10 2024-09-16T04:42:24.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f1298949640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:24.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f1298949640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f129409ab00 con 0x7f12940a3ff0 2024-09-16T04:42:24.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f1298949640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:24.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f1298949640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f129409b030 con 0x7f12941369d0 2024-09-16T04:42:24.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f1298949640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f1298949640 10 monclient(hunting): authenticate will time out at 942.452686s 2024-09-16T04:42:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f1292d9d640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12941369d0 msgr2=0x7f1294136dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:42:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f1292d9d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12941369d0 0x7f1294136dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f129259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f12940a3c10 0x7f129409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f129259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f12940a3c10 0x7f129409bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56028/0 (socket says 172.21.15.46:56028) 2024-09-16T04:42:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f129259c640 1 -- 172.21.15.46:0/3326646439 learned_addr learned my addr 172.21.15.46:0/3326646439 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f129259c640 10 monclient(hunting): get_auth_request con 0x7f12940a3c10 auth_method 0 2024-09-16T04:42:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f129259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f129259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f129259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f129259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f129259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f129259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.037+0000 7f1291d9b640 1 --2- 172.21.15.46:0/3326646439 >> v2:172.21.15.114:3300/0 conn(0x7f12940a3ff0 0x7f12940a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:24.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.038+0000 7f1291d9b640 10 monclient(hunting): get_auth_request con 0x7f12940a3ff0 auth_method 0 2024-09-16T04:42:24.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.038+0000 7f1291d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:24.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.038+0000 7f1291d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:24.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.038+0000 7f1291d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:24.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.038+0000 7f129259c640 10 monclient(hunting): handle_auth_done global_id 44202 payload 274 2024-09-16T04:42:24.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.038+0000 7f129259c640 1 -- 172.21.15.46:0/3326646439 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12941369d0 msgr2=0x7f1294136dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:42:24.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.038+0000 7f129259c640 1 --2- 172.21.15.46:0/3326646439 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12941369d0 0x7f1294136dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:24.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.038+0000 7f129259c640 1 -- 172.21.15.46:0/3326646439 >> v2:172.21.15.114:3300/0 conn(0x7f12940a3ff0 msgr2=0x7f12940a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:24.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.038+0000 7f129259c640 1 --2- 172.21.15.46:0/3326646439 >> v2:172.21.15.114:3300/0 conn(0x7f12940a3ff0 0x7f12940a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:24.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.038+0000 7f129259c640 10 monclient: _finish_hunting 0 2024-09-16T04:42:24.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.038+0000 7f129259c640 1 monclient: found mon.noname-a 2024-09-16T04:42:24.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.038+0000 7f129259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:24.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.038+0000 7f129259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:24.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.038+0000 7f129259c640 1 -- 172.21.15.46:0/3326646439 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f129409b310 con 0x7f12940a3c10 2024-09-16T04:42:24.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.038+0000 7f129259c640 1 --2- 172.21.15.46:0/3326646439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12940a3c10 0x7f129409bcf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f12940a38d0 tx=0x7f128402a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d5498620499863 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:24.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.038+0000 7f129159a640 1 -- 172.21.15.46:0/3326646439 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1284037070 con 0x7f12940a3c10 2024-09-16T04:42:24.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.038+0000 7f129159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:24.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.038+0000 7f129159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:24.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.038+0000 7f129159a640 10 monclient: dump: 2024-09-16T04:42:24.051 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:24.051 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:24.051 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:24.051 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:24.051 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:24.051 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:24.051 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:24.051 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:24.051 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:24.052 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:24.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f129159a640 10 monclient: _finish_auth 0 2024-09-16T04:42:24.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f129159a640 10 monclient: _check_auth_tickets 2024-09-16T04:42:24.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f129159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:24.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f129159a640 1 -- 172.21.15.46:0/3326646439 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f128402ae20 con 0x7f12940a3c10 2024-09-16T04:42:24.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f1298949640 5 monclient: authenticate success, global_id 44202 2024-09-16T04:42:24.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f1298949640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:24.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f129159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:24.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f129159a640 1 -- 172.21.15.46:0/3326646439 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1284032660 con 0x7f12940a3c10 2024-09-16T04:42:24.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f1298949640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:24.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f129159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:24.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f1298949640 4 set_mon_vals no callback set 2024-09-16T04:42:24.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f129159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:24.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f129159a640 10 monclient: dump: 2024-09-16T04:42:24.053 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:24.053 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:24.053 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:24.053 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:24.053 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:24.053 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:24.053 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:24.053 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:24.053 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:24.053 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:24.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f1298949640 10 monclient: shutdown 2024-09-16T04:42:24.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f1298949640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:24.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f1298949640 1 -- 172.21.15.46:0/3326646439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12940a3c10 msgr2=0x7f129409bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:24.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f1298949640 1 --2- 172.21.15.46:0/3326646439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12940a3c10 0x7f129409bcf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f12940a38d0 tx=0x7f128402a0d0 comp rx=0 tx=0).stop 2024-09-16T04:42:24.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f1298949640 1 -- 172.21.15.46:0/3326646439 shutdown_connections 2024-09-16T04:42:24.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f1298949640 1 --2- 172.21.15.46:0/3326646439 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12941369d0 0x7f1294136dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:24.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f1298949640 1 --2- 172.21.15.46:0/3326646439 >> v2:172.21.15.114:3300/0 conn(0x7f12940a3ff0 0x7f12940a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:24.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f1298949640 1 --2- 172.21.15.46:0/3326646439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12940a3c10 0x7f129409bcf0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:24.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f1298949640 1 -- 172.21.15.46:0/3326646439 >> 172.21.15.46:0/3326646439 conn(0x7f12940a6340 msgr2=0x7f129406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:24.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f1298949640 1 -- 172.21.15.46:0/3326646439 shutdown_connections 2024-09-16T04:42:24.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.039+0000 7f1298949640 1 -- 172.21.15.46:0/3326646439 wait complete. 2024-09-16T04:42:24.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.040+0000 7f1298949640 10 monclient: build_initial_monmap 2024-09-16T04:42:24.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.040+0000 7f1298949640 10 monclient: monmap: 2024-09-16T04:42:24.054 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:24.054 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:24.054 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:24.054 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:24.054 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:24.055 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:24.055 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:24.055 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:24.055 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:24.055 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:24.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.040+0000 7f1298949640 1 Processor -- start 2024-09-16T04:42:24.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.040+0000 7f1298949640 1 -- start start 2024-09-16T04:42:24.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.040+0000 7f1298949640 10 monclient: init 2024-09-16T04:42:24.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1298949640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:24.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1298949640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:42:24.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1298949640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12940a3c10 0x7f12941cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:24.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1298949640 10 monclient(hunting): picked mon.noname-c con 0x7f12940a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:24.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1298949640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12940a3ff0 0x7f1294123340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:24.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1298949640 10 monclient(hunting): picked mon.noname-a con 0x7f12940a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:24.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1298949640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12941369d0 0x7f1294123880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:24.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1298949640 10 monclient(hunting): picked mon.noname-b con 0x7f12941369d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:24.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1298949640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:24.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1298949640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f12940a4910 con 0x7f12940a3ff0 2024-09-16T04:42:24.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1298949640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:24.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1298949640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f12940a69c0 con 0x7f12941369d0 2024-09-16T04:42:24.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1298949640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:24.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1298949640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f1294127cd0 con 0x7f12940a3c10 2024-09-16T04:42:24.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1298949640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:24.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f129259c640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12940a3c10 msgr2=0x7f12941cc330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:42:24.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1298949640 10 monclient(hunting): authenticate will time out at 942.456358s 2024-09-16T04:42:24.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f129259c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12940a3c10 0x7f12941cc330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:24.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1291d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12940a3ff0 0x7f1294123340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:24.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1291d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12940a3ff0 0x7f1294123340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56034/0 (socket says 172.21.15.46:56034) 2024-09-16T04:42:24.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1291d9b640 1 -- 172.21.15.46:0/2368113933 learned_addr learned my addr 172.21.15.46:0/2368113933 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:24.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1291d9b640 10 monclient(hunting): get_auth_request con 0x7f12940a3ff0 auth_method 0 2024-09-16T04:42:24.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1291d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:24.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1291d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:24.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1291d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:24.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1292d9d640 1 --2- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12941369d0 0x7f1294123880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:24.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1291d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:24.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1291d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:24.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1291d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:24.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1292d9d640 10 monclient(hunting): get_auth_request con 0x7f12941369d0 auth_method 0 2024-09-16T04:42:24.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1292d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:24.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1292d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:24.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1292d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:24.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1292d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:24.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1292d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:24.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1292d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:24.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1291d9b640 10 monclient(hunting): handle_auth_done global_id 44205 payload 824 2024-09-16T04:42:24.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1291d9b640 1 -- 172.21.15.46:0/2368113933 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12940a3c10 msgr2=0x7f12941cc330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:42:24.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1291d9b640 1 --2- 172.21.15.46:0/2368113933 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12940a3c10 0x7f12941cc330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:24.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1291d9b640 1 -- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12941369d0 msgr2=0x7f1294123880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:24.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1291d9b640 1 --2- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12941369d0 0x7f1294123880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:24.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1291d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:42:24.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1291d9b640 1 monclient: found mon.noname-a 2024-09-16T04:42:24.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1291d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:24.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1291d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:24.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1291d9b640 1 -- 172.21.15.46:0/2368113933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1294125b40 con 0x7f12940a3ff0 2024-09-16T04:42:24.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.041+0000 7f1291d9b640 1 --2- 172.21.15.46:0/2368113933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12940a3ff0 0x7f1294123340 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f128800d830 tx=0x7f128800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:24.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1282ffd640 1 -- 172.21.15.46:0/2368113933 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f128800b6c0 con 0x7f12940a3ff0 2024-09-16T04:42:24.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1282ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:24.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1282ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:24.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1282ffd640 10 monclient: dump: 2024-09-16T04:42:24.059 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:24.059 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:24.059 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:24.059 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:24.059 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:24.059 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:24.059 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:24.059 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:24.059 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:24.059 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:24.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1282ffd640 10 monclient: _finish_auth 0 2024-09-16T04:42:24.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1282ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:42:24.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1282ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:24.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1298949640 5 monclient: authenticate success, global_id 44205 2024-09-16T04:42:24.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1282ffd640 1 -- 172.21.15.46:0/2368113933 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1288019070 con 0x7f12940a3ff0 2024-09-16T04:42:24.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1282ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:24.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1282ffd640 1 -- 172.21.15.46:0/2368113933 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f128800ba80 con 0x7f12940a3ff0 2024-09-16T04:42:24.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1298949640 10 monclient: _renew_subs 2024-09-16T04:42:24.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1298949640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:24.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1298949640 1 -- 172.21.15.46:0/2368113933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12941272d0 con 0x7f12940a3ff0 2024-09-16T04:42:24.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1282ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:24.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1282ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:24.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1282ffd640 10 monclient: dump: 2024-09-16T04:42:24.060 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:24.061 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:24.061 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:24.061 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:24.061 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:24.061 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:24.061 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:24.061 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:24.061 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:24.061 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:24.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1290d99640 4 set_mon_vals no callback set 2024-09-16T04:42:24.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1298949640 10 monclient: _renew_subs 2024-09-16T04:42:24.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1298949640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:24.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1298949640 1 -- 172.21.15.46:0/2368113933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12940a8df0 con 0x7f12940a3ff0 2024-09-16T04:42:24.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1282ffd640 1 -- 172.21.15.46:0/2368113933 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f128801e460 con 0x7f12940a3ff0 2024-09-16T04:42:24.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.042+0000 7f1298949640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:24.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.043+0000 7f1298949640 1 --2- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12941cc870 0x7f1294126c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:24.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.043+0000 7f1298949640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:24.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.043+0000 7f1298949640 1 -- 172.21.15.46:0/2368113933 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f12940a4a90 con 0x7f12941cc870 2024-09-16T04:42:24.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.043+0000 7f1282ffd640 1 --2- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1274038360 0x7f127403a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:24.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.043+0000 7f129259c640 1 --2- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12941cc870 0x7f1294126c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:24.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.043+0000 7f1282ffd640 1 -- 172.21.15.46:0/2368113933 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 4738+0+0 (secure 0 0 0) 0x7f128804bd30 con 0x7f12940a3ff0 2024-09-16T04:42:24.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.043+0000 7f129259c640 10 monclient: get_auth_request con 0x7f12941cc870 auth_method 0 2024-09-16T04:42:24.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.043+0000 7f129259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:24.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.043+0000 7f129259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:24.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.043+0000 7f129259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:24.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.043+0000 7f129259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:24.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.043+0000 7f129259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:24.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.043+0000 7f129259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:24.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.043+0000 7f1292d9d640 1 --2- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1274038360 0x7f127403a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:24.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.043+0000 7f1292d9d640 10 monclient: get_auth_request con 0x7f1274038360 auth_method 0 2024-09-16T04:42:24.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.043+0000 7f129259c640 10 monclient(hunting): handle_auth_done global_id 5485 payload 824 2024-09-16T04:42:24.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.043+0000 7f129259c640 1 -- 172.21.15.46:0/2368113933 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f12940a43d0 con 0x7f12941cc870 2024-09-16T04:42:24.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.043+0000 7f129259c640 1 --2- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12941cc870 0x7f1294126c20 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1294068240 tx=0x7f12840046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:24.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.043+0000 7f1292d9d640 1 --2- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1274038360 0x7f127403a820 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f12940a8650 tx=0x7f128c00d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:24.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.044+0000 7f1282ffd640 1 -- 172.21.15.46:0/2368113933 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1284037040 con 0x7f12941cc870 2024-09-16T04:42:24.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.044+0000 7f1282ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:24.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.044+0000 7f1282ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:42:24.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.044+0000 7f1282ffd640 10 monclient: dump: 2024-09-16T04:42:24.063 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:24.063 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:24.063 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:24.063 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:24.063 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:24.064 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:24.064 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:24.064 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:24.064 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:24.064 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:24.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.044+0000 7f1282ffd640 1 -- 172.21.15.46:0/2368113933 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1284032c90 con 0x7f12941cc870 2024-09-16T04:42:24.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.044+0000 7f1282ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:24.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.044+0000 7f1282ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:24.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.044+0000 7f1282ffd640 1 -- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12941cc870 msgr2=0x7f1294126c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:24.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.044+0000 7f1282ffd640 1 --2- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12941cc870 0x7f1294126c20 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1294068240 tx=0x7f12840046c0 comp rx=0 tx=0).stop 2024-09-16T04:42:24.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.044+0000 7f1282ffd640 1 -- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12941cc870 msgr2=0x7f1294126c20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:24.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.044+0000 7f1282ffd640 1 --2- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12941cc870 0x7f1294126c20 secure :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1294068240 tx=0x7f12840046c0 comp rx=0 tx=0).stop 2024-09-16T04:42:24.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.050+0000 7f1298949640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:42:24.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.050+0000 7f1298949640 1 --2- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1294123dc0 0x7f12941241a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:24.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.050+0000 7f1298949640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:24.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.050+0000 7f1298949640 1 -- 172.21.15.46:0/2368113933 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1294125040 con 0x7f1294123dc0 2024-09-16T04:42:24.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.050+0000 7f129259c640 1 --2- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1294123dc0 0x7f12941241a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:24.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.050+0000 7f129259c640 10 monclient: get_auth_request con 0x7f1294123dc0 auth_method 0 2024-09-16T04:42:24.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.050+0000 7f129259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:24.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.050+0000 7f129259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:24.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.050+0000 7f129259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:24.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.050+0000 7f129259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:24.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.050+0000 7f129259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:24.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.050+0000 7f129259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:24.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.051+0000 7f129259c640 10 monclient(hunting): handle_auth_done global_id 5488 payload 824 2024-09-16T04:42:24.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.051+0000 7f129259c640 1 -- 172.21.15.46:0/2368113933 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f12941246e0 con 0x7f1294123dc0 2024-09-16T04:42:24.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.051+0000 7f129259c640 1 --2- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1294123dc0 0x7f12941241a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1284000c00 tx=0x7f1284009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:24.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.051+0000 7f1282ffd640 1 -- 172.21.15.46:0/2368113933 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1284037050 con 0x7f1294123dc0 2024-09-16T04:42:24.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.051+0000 7f1282ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:24.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.051+0000 7f1282ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:42:24.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.051+0000 7f1282ffd640 10 monclient: dump: 2024-09-16T04:42:24.068 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:24.068 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:24.068 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:24.068 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:24.068 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:24.068 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:24.068 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:24.068 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:24.068 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:24.068 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:24.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.051+0000 7f1282ffd640 1 -- 172.21.15.46:0/2368113933 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4868 (secure 0 0 0) 0x7f128404c070 con 0x7f1294123dc0 2024-09-16T04:42:24.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.051+0000 7f1282ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:42:24.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.051+0000 7f1282ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:24.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.051+0000 7f1282ffd640 1 -- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1294123dc0 msgr2=0x7f12941241a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:24.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.051+0000 7f1282ffd640 1 --2- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1294123dc0 0x7f12941241a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1284000c00 tx=0x7f1284009240 comp rx=0 tx=0).stop 2024-09-16T04:42:24.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.051+0000 7f1282ffd640 1 -- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1294123dc0 msgr2=0x7f12941241a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:24.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.051+0000 7f1282ffd640 1 --2- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1294123dc0 0x7f12941241a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:24.069 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:42:24.069 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:42:24.069 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:42:24.069 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:42:24.069 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 42, 2024-09-16T04:42:24.069 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 229953, 2024-09-16T04:42:24.070 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:42:24.070 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:42:24.070 INFO:teuthology.orchestra.run.smithi046.stdout: 1 2024-09-16T04:42:24.070 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:24.070 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:42:24.070 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:24.070 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:42:24.070 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:42:24.070 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:24.070 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:24.070 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:24.070 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:24.070 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:24.071 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:24.071 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:24.071 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:24.071 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:24.071 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:24.071 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:24.071 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:24.071 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:42:24.071 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:42:24.071 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:24.071 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:24.071 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:24.071 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:24.071 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:24.072 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:24.072 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:24.072 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:24.072 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:24.072 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:24.072 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:24.072 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:24.072 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:24.072 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:42:24.072 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:42:24.072 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:42:24.072 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:42:24.072 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:42:24.073 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:42:24.073 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:24.073 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:24.073 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:42:24.073 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:42:24.073 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:42:24.073 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:42:24.073 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:42:24.073 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:42:24.073 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:42:24.073 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:24.073 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:42:24.073 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:24.074 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:24.074 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:24.074 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:24.074 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:24.074 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:24.074 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:24.074 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:24.074 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:24.074 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:24.074 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:24.074 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:24.074 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:42:24.074 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:24.074 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:42:24.075 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:24.075 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:42:24.075 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:42:24.075 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:24.075 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:24.075 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:24.075 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:24.075 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:42:24.075 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:24.075 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:24.075 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:24.075 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:24.075 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:42:24.075 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:24.076 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:24.076 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:24.076 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:24.076 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:42:24.076 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:42:24.076 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:24.076 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:24.076 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:24.076 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:24.076 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:24.076 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:42:24.076 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:42:24.076 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:24.077 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:24.077 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:24.077 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:24.077 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:42:24.077 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:24.077 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:24.077 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:24.077 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:24.077 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:42:24.077 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:24.077 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:24.077 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:24.077 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:24.077 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:42:24.078 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:42:24.078 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:24.078 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:24.078 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:24.078 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:24.078 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:24.078 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:42:24.078 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:42:24.078 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:24.078 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:24.078 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:24.078 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:24.078 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:42:24.078 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:24.079 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:24.079 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:24.079 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:24.079 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:42:24.079 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:24.079 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:24.079 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:24.079 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:24.079 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:42:24.079 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:42:24.079 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:24.079 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:24.079 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:24.079 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:24.080 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:24.080 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:24.080 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:42:24.080 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:42:24.080 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:24.080 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:24.080 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:24.080 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:42:24.080 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:24.080 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:24.080 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:42:24.080 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:24.080 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:24.080 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:24.081 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4 2024-09-16T04:42:24.081 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:24.081 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:24.081 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:42:24.081 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:24.081 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:24.081 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:24.081 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6 2024-09-16T04:42:24.081 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:24.081 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:24.081 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:42:24.081 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:24.081 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:24.081 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:24.082 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:42:24.082 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:24.082 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:24.082 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:24.082 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:42:24.082 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:42:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.052+0000 7f1298949640 1 -- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1274038360 msgr2=0x7f127403a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.052+0000 7f1298949640 1 --2- 172.21.15.46:0/2368113933 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1274038360 0x7f127403a820 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f12940a8650 tx=0x7f128c00d040 comp rx=0 tx=0).stop 2024-09-16T04:42:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.052+0000 7f1298949640 10 monclient: shutdown 2024-09-16T04:42:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.052+0000 7f1298949640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.052+0000 7f1298949640 1 -- 172.21.15.46:0/2368113933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12940a3ff0 msgr2=0x7f1294123340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.052+0000 7f1298949640 1 --2- 172.21.15.46:0/2368113933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12940a3ff0 0x7f1294123340 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f128800d830 tx=0x7f128800dcf0 comp rx=0 tx=0).stop 2024-09-16T04:42:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.052+0000 7f1292d9d640 1 -- 172.21.15.46:0/2368113933 reap_dead start 2024-09-16T04:42:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.052+0000 7f1292d9d640 1 -- 172.21.15.46:0/2368113933 reap_dead start 2024-09-16T04:42:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.052+0000 7f1298949640 1 -- 172.21.15.46:0/2368113933 shutdown_connections 2024-09-16T04:42:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.052+0000 7f1298949640 1 -- 172.21.15.46:0/2368113933 >> 172.21.15.46:0/2368113933 conn(0x7f12940a6340 msgr2=0x7f129409f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.052+0000 7f1298949640 1 -- 172.21.15.46:0/2368113933 shutdown_connections 2024-09-16T04:42:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:24.052+0000 7f1298949640 1 -- 172.21.15.46:0/2368113933 wait complete. 2024-09-16T04:42:24.084 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:42:24.084 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:42:24.510 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37325]: starting. 2024-09-16T04:42:24.511 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37325]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-37323 2024-09-16T04:42:24.511 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37325]: shutting down. 2024-09-16T04:42:26.623 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:26.623 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37360]: starting. 2024-09-16T04:42:26.624 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37360]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-37323 2024-09-16T04:42:26.624 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37360]: created object 0... 2024-09-16T04:42:26.624 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37360]: created object 25... 2024-09-16T04:42:26.624 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37360]: created object 49... 2024-09-16T04:42:26.624 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37360]: finishing. 2024-09-16T04:42:26.624 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37360]: shutting down. 2024-09-16T04:42:26.636 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:26.636 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37362]: starting. 2024-09-16T04:42:26.636 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37362]: listing objects. 2024-09-16T04:42:26.636 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37362]: listed object 0... 2024-09-16T04:42:26.636 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37362]: listed object 25... 2024-09-16T04:42:26.636 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37362]: saw 50 objects 2024-09-16T04:42:26.636 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37362]: shutting down. 2024-09-16T04:42:27.528 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:27.528 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37361]: starting. 2024-09-16T04:42:27.528 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37361]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-37323 2024-09-16T04:42:27.528 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37361]: shutting down. 2024-09-16T04:42:27.530 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:27.530 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:27.530 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:42:27.532 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:42:29.623 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37429]: starting. 2024-09-16T04:42:29.623 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37429]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-37428 2024-09-16T04:42:29.623 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37429]: created object 0... 2024-09-16T04:42:29.623 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37429]: created object 25... 2024-09-16T04:42:29.623 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37429]: created object 49... 2024-09-16T04:42:29.623 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37429]: finishing. 2024-09-16T04:42:29.623 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37429]: shutting down. 2024-09-16T04:42:30.267 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37430]: starting. 2024-09-16T04:42:30.267 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37430]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-37428 2024-09-16T04:42:30.267 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37430]: shutting down. 2024-09-16T04:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37463]: starting. 2024-09-16T04:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37463]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-37428 2024-09-16T04:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37463]: created object 0... 2024-09-16T04:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37463]: created object 25... 2024-09-16T04:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37463]: created object 49... 2024-09-16T04:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37463]: finishing. 2024-09-16T04:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37463]: shutting down. 2024-09-16T04:42:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37465]: starting. 2024-09-16T04:42:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37465]: listing objects. 2024-09-16T04:42:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37465]: listed object 0... 2024-09-16T04:42:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37465]: listed object 25... 2024-09-16T04:42:32.379 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37465]: saw 50 objects 2024-09-16T04:42:32.379 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37465]: shutting down. 2024-09-16T04:42:33.279 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:33.279 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37464]: starting. 2024-09-16T04:42:33.279 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37464]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-37428 2024-09-16T04:42:33.279 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37464]: shutting down. 2024-09-16T04:42:33.280 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:33.281 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:33.281 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:42:33.283 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:42:35.381 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37532]: starting. 2024-09-16T04:42:35.381 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37532]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-37531 2024-09-16T04:42:35.382 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37532]: created object 0... 2024-09-16T04:42:35.382 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37532]: created object 25... 2024-09-16T04:42:35.382 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37532]: created object 49... 2024-09-16T04:42:35.382 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37532]: finishing. 2024-09-16T04:42:35.382 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37532]: shutting down. 2024-09-16T04:42:36.306 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37533]: starting. 2024-09-16T04:42:36.306 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37533]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-37531 2024-09-16T04:42:36.307 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37533]: shutting down. 2024-09-16T04:42:38.405 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:38.405 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37566]: starting. 2024-09-16T04:42:38.405 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37566]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-37531 2024-09-16T04:42:38.405 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37566]: created object 0... 2024-09-16T04:42:38.405 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37566]: created object 25... 2024-09-16T04:42:38.406 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37566]: created object 49... 2024-09-16T04:42:38.406 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37566]: finishing. 2024-09-16T04:42:38.406 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37566]: shutting down. 2024-09-16T04:42:38.418 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:38.418 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37568]: starting. 2024-09-16T04:42:38.418 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37568]: listing objects. 2024-09-16T04:42:38.418 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37568]: listed object 0... 2024-09-16T04:42:38.418 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37568]: listed object 25... 2024-09-16T04:42:38.418 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37568]: saw 50 objects 2024-09-16T04:42:38.418 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37568]: shutting down. 2024-09-16T04:42:39.315 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:39.315 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37567]: starting. 2024-09-16T04:42:39.315 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37567]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-37531 2024-09-16T04:42:39.315 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37567]: shutting down. 2024-09-16T04:42:39.317 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:39.317 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:39.317 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:42:39.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:42:41.359 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37635]: starting. 2024-09-16T04:42:41.359 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37635]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-37634 2024-09-16T04:42:41.359 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37635]: created object 0... 2024-09-16T04:42:41.359 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37635]: created object 25... 2024-09-16T04:42:41.359 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37635]: created object 49... 2024-09-16T04:42:41.359 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37635]: finishing. 2024-09-16T04:42:41.359 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37635]: shutting down. 2024-09-16T04:42:42.273 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37636]: starting. 2024-09-16T04:42:42.274 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37636]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-37634 2024-09-16T04:42:42.274 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37636]: shutting down. 2024-09-16T04:42:44.085 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:42:44.085 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T04:42:44.085 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:42:44.085 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:42:44.088 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:42:44.088 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:42:44.089 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:42:44.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.199+0000 7fb1743a8640 10 monclient: get_monmap_and_config 2024-09-16T04:42:44.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.199+0000 7fb1743a8640 10 monclient: build_initial_monmap 2024-09-16T04:42:44.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.199+0000 7fb1743a8640 10 monclient: monmap: 2024-09-16T04:42:44.202 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:44.202 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:44.202 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:44.200924+0000 2024-09-16T04:42:44.202 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:44.200924+0000 2024-09-16T04:42:44.202 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:44.202 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:44.202 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:44.202 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:44.203 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:44.203 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:44.203 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:44.203 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:44.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb1743a8640 1 Processor -- start 2024-09-16T04:42:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb1743a8640 1 -- start start 2024-09-16T04:42:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb1743a8640 10 monclient: init 2024-09-16T04:42:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb1743a8640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb1743a8640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-16T04:42:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb1743a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb16c0a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:42:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb1743a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb16c0a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:42:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb1743a8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb16c13cea0 0x7fb16c13f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb1743a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb16c13cea0 addr v2:172.21.15.46:3300/0 2024-09-16T04:42:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb1743a8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb16c09b940 con 0x7fb16c0a3c10 2024-09-16T04:42:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb1743a8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb16c09ab00 con 0x7fb16c0a3ff0 2024-09-16T04:42:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb1743a8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb1743a8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb16c09b030 con 0x7fb16c13cea0 2024-09-16T04:42:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb1743a8640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb1743a8640 10 monclient(hunting): authenticate will time out at 962.616796s 2024-09-16T04:42:44.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb172144640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb16c0a3c10 0x7fb16c09bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41710/0 (socket says 172.21.15.46:41710) 2024-09-16T04:42:44.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb172144640 1 -- 172.21.15.46:0/34523335 learned_addr learned my addr 172.21.15.46:0/34523335 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:42:44.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb172945640 1 --2- 172.21.15.46:0/34523335 >> v2:172.21.15.46:3300/0 conn(0x7fb16c13cea0 0x7fb16c13f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:44.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb172945640 10 monclient(hunting): get_auth_request con 0x7fb16c13cea0 auth_method 0 2024-09-16T04:42:44.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb172945640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:44.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb172945640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:44.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.200+0000 7fb172945640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:44.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.201+0000 7fb172945640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:44.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.201+0000 7fb172945640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:44.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.201+0000 7fb172945640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:44.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.201+0000 7fb172945640 10 monclient(hunting): handle_auth_done global_id 44322 payload 274 2024-09-16T04:42:44.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.201+0000 7fb172945640 1 -- 172.21.15.46:0/34523335 >> v1:172.21.15.114:6789/0 conn(0x7fb16c0a3ff0 legacy=0x7fb16c0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:44.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.201+0000 7fb172945640 1 -- 172.21.15.46:0/34523335 >> v1:172.21.15.46:6789/0 conn(0x7fb16c0a3c10 legacy=0x7fb16c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:44.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.201+0000 7fb172945640 10 monclient: _finish_hunting 0 2024-09-16T04:42:44.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.201+0000 7fb172945640 1 monclient: found mon.noname-a 2024-09-16T04:42:44.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.201+0000 7fb172945640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:44.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.201+0000 7fb172945640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:44.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.201+0000 7fb171142640 1 -- 172.21.15.46:0/34523335 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2520043808 0 0) 0x7fb16c09b940 con 0x7fb16c0a3c10 2024-09-16T04:42:44.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.201+0000 7fb172945640 1 -- 172.21.15.46:0/34523335 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb16c09b310 con 0x7fb16c13cea0 2024-09-16T04:42:44.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.201+0000 7fb171142640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:42:44.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb172945640 1 --2- 172.21.15.46:0/34523335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c13cea0 0x7fb16c13f290 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb164009d90 tx=0x7fb16402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca97d75ac31aeff5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:44.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb171142640 1 -- 172.21.15.46:0/34523335 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb164007740 con 0x7fb16c13cea0 2024-09-16T04:42:44.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb171142640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:44.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb171142640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:44.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb171142640 10 monclient: dump: 2024-09-16T04:42:44.214 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:44.214 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:44.214 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:44.214 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:44.214 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:44.215 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb171142640 10 monclient: _finish_auth 0 2024-09-16T04:42:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb171142640 10 monclient: _check_auth_tickets 2024-09-16T04:42:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb171142640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb171142640 1 -- 172.21.15.46:0/34523335 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1640078e0 con 0x7fb16c13cea0 2024-09-16T04:42:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb1743a8640 5 monclient: authenticate success, global_id 44322 2024-09-16T04:42:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb171142640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb1743a8640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb1743a8640 4 set_mon_vals no callback set 2024-09-16T04:42:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb171142640 1 -- 172.21.15.46:0/34523335 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb164007cb0 con 0x7fb16c13cea0 2024-09-16T04:42:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb171142640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb171142640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb171142640 10 monclient: dump: 2024-09-16T04:42:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:44.216 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:44.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb1743a8640 10 monclient: shutdown 2024-09-16T04:42:44.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb1743a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:44.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb1743a8640 1 -- 172.21.15.46:0/34523335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c13cea0 msgr2=0x7fb16c13f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:44.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb1743a8640 1 --2- 172.21.15.46:0/34523335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c13cea0 0x7fb16c13f290 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb164009d90 tx=0x7fb16402a660 comp rx=0 tx=0).stop 2024-09-16T04:42:44.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb1743a8640 1 -- 172.21.15.46:0/34523335 shutdown_connections 2024-09-16T04:42:44.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb1743a8640 1 --2- 172.21.15.46:0/34523335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c13cea0 0x7fb16c13f290 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:44.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb1743a8640 1 -- 172.21.15.46:0/34523335 >> 172.21.15.46:0/34523335 conn(0x7fb16c0a6340 msgr2=0x7fb16c06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:44.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb1743a8640 1 -- 172.21.15.46:0/34523335 shutdown_connections 2024-09-16T04:42:44.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.202+0000 7fb1743a8640 1 -- 172.21.15.46:0/34523335 wait complete. 2024-09-16T04:42:44.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.203+0000 7fb1743a8640 10 monclient: build_initial_monmap 2024-09-16T04:42:44.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.203+0000 7fb1743a8640 10 monclient: monmap: 2024-09-16T04:42:44.217 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:44.217 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:44.218 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:44.218 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:44.218 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:44.218 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:44.218 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:44.218 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:44.218 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:44.218 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:44.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.203+0000 7fb1743a8640 1 Processor -- start 2024-09-16T04:42:44.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.203+0000 7fb1743a8640 1 -- start start 2024-09-16T04:42:44.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.203+0000 7fb1743a8640 10 monclient: init 2024-09-16T04:42:44.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb1743a8640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:44.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb1743a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:42:44.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb1743a8640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb16c0a3c10 0x7fb16c1cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:44.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb1743a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb16c0a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:44.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb1743a8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c0a3ff0 0x7fb16c1cc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:44.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb1743a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb16c0a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:44.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb1743a8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb16c13cea0 0x7fb16c139f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:44.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb1743a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb16c13cea0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:44.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb1743a8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:44.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb1743a8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c13f950 con 0x7fb16c0a3ff0 2024-09-16T04:42:44.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb1743a8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:44.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb1743a8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c13f7d0 con 0x7fb16c13cea0 2024-09-16T04:42:44.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb1743a8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:44.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb1743a8640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fb16c13fad0 con 0x7fb16c0a3c10 2024-09-16T04:42:44.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb1743a8640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:44.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb1743a8640 10 monclient(hunting): authenticate will time out at 962.620369s 2024-09-16T04:42:44.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb171943640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c0a3ff0 0x7fb16c1cc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:44.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb172144640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb16c0a3c10 0x7fb16c1cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:44.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb171943640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c0a3ff0 0x7fb16c1cc210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54682/0 (socket says 172.21.15.46:54682) 2024-09-16T04:42:44.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb171943640 1 -- 172.21.15.46:0/1858591628 learned_addr learned my addr 172.21.15.46:0/1858591628 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:44.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb171943640 10 monclient(hunting): get_auth_request con 0x7fb16c0a3ff0 auth_method 0 2024-09-16T04:42:44.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb171943640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:44.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb171943640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:44.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb171943640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:44.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb172144640 10 monclient(hunting): get_auth_request con 0x7fb16c0a3c10 auth_method 0 2024-09-16T04:42:44.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb172144640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:44.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb172144640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:44.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb172144640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb172945640 1 --2- 172.21.15.46:0/1858591628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb16c13cea0 0x7fb16c139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb172945640 10 monclient(hunting): get_auth_request con 0x7fb16c13cea0 auth_method 0 2024-09-16T04:42:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb172945640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb172945640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb172945640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb171943640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb171943640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb172144640 1 -- 172.21.15.46:0/1858591628 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb16c0a3c10 msgr2=0x7fb16c1cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:42:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb172144640 1 -- 172.21.15.46:0/1858591628 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb16c0a3c10 msgr2=0x7fb16c1cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:42:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb172144640 1 --2- 172.21.15.46:0/1858591628 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb16c0a3c10 0x7fb16c1cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb171943640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb172144640 1 --2- 172.21.15.46:0/1858591628 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb16c0a3c10 0x7fb16c1cba40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb172945640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb172945640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:44.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb172945640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:44.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb171943640 10 monclient(hunting): handle_auth_done global_id 44328 payload 824 2024-09-16T04:42:44.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb171943640 1 -- 172.21.15.46:0/1858591628 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb16c0a3c10 msgr2=0x7fb16c1cba40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:42:44.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb171943640 1 --2- 172.21.15.46:0/1858591628 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb16c0a3c10 0x7fb16c1cba40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:44.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb171943640 1 -- 172.21.15.46:0/1858591628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb16c13cea0 msgr2=0x7fb16c139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:44.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb171943640 1 --2- 172.21.15.46:0/1858591628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb16c13cea0 0x7fb16c139f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:44.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb171943640 10 monclient: _finish_hunting 0 2024-09-16T04:42:44.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb171943640 1 monclient: found mon.noname-a 2024-09-16T04:42:44.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb171943640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:44.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb171943640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:44.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.204+0000 7fb171943640 1 -- 172.21.15.46:0/1858591628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb16c13d280 con 0x7fb16c0a3ff0 2024-09-16T04:42:44.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb171943640 1 --2- 172.21.15.46:0/1858591628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c0a3ff0 0x7fb16c1cc210 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb16000c480 tx=0x7fb16000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:44.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb15effd640 1 -- 172.21.15.46:0/1858591628 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1600052d0 con 0x7fb16c0a3ff0 2024-09-16T04:42:44.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb15effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:44.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb15effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:44.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb15effd640 10 monclient: dump: 2024-09-16T04:42:44.224 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:44.224 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:44.225 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb15effd640 10 monclient: _finish_auth 0 2024-09-16T04:42:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb15effd640 10 monclient: _check_auth_tickets 2024-09-16T04:42:44.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb15effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:44.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb1743a8640 5 monclient: authenticate success, global_id 44328 2024-09-16T04:42:44.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb15effd640 1 -- 172.21.15.46:0/1858591628 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb160005470 con 0x7fb16c0a3ff0 2024-09-16T04:42:44.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb15effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:44.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb15effd640 1 -- 172.21.15.46:0/1858591628 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb160012870 con 0x7fb16c0a3ff0 2024-09-16T04:42:44.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb1743a8640 10 monclient: _renew_subs 2024-09-16T04:42:44.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb1743a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:44.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb1743a8640 1 -- 172.21.15.46:0/1858591628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb16c0a0400 con 0x7fb16c0a3ff0 2024-09-16T04:42:44.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb170941640 4 set_mon_vals no callback set 2024-09-16T04:42:44.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb15effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:44.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb15effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:44.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb15effd640 10 monclient: dump: 2024-09-16T04:42:44.226 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:44.226 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:44.227 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:44.227 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:44.227 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:44.227 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:44.227 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:44.227 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:44.227 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:44.227 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:44.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb1743a8640 10 monclient: _renew_subs 2024-09-16T04:42:44.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb1743a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:44.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.205+0000 7fb1743a8640 1 -- 172.21.15.46:0/1858591628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb16c11f860 con 0x7fb16c0a3ff0 2024-09-16T04:42:44.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.206+0000 7fb15effd640 1 -- 172.21.15.46:0/1858591628 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fb160012a10 con 0x7fb16c0a3ff0 2024-09-16T04:42:44.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.206+0000 7fb1743a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:44.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.206+0000 7fb1743a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:44.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.206+0000 7fb1743a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:44.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.206+0000 7fb1743a8640 1 -- 172.21.15.46:0/1858591628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb16c09bcf0 con 0x7fb16c0a3ff0 2024-09-16T04:42:44.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.206+0000 7fb15effd640 1 --2- 172.21.15.46:0/1858591628 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb1500383b0 0x7fb15003a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:44.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.206+0000 7fb15effd640 1 -- 172.21.15.46:0/1858591628 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 4738+0+0 (secure 0 0 0) 0x7fb16004b380 con 0x7fb16c0a3ff0 2024-09-16T04:42:44.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.206+0000 7fb172144640 1 --2- 172.21.15.46:0/1858591628 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb1500383b0 0x7fb15003a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:44.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.206+0000 7fb172144640 10 monclient: get_auth_request con 0x7fb1500383b0 auth_method 0 2024-09-16T04:42:44.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.207+0000 7fb172144640 1 --2- 172.21.15.46:0/1858591628 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb1500383b0 0x7fb15003a870 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb168000f60 tx=0x7fb168007b90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:44.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.211+0000 7fb15effd640 1 -- 172.21.15.46:0/1858591628 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb160003df0 con 0x7fb16c0a3ff0 2024-09-16T04:42:44.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.211+0000 7fb15effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:44.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.211+0000 7fb15effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:44.365 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:44.366 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37669]: starting. 2024-09-16T04:42:44.366 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37669]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-37634 2024-09-16T04:42:44.366 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37669]: created object 0... 2024-09-16T04:42:44.366 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37669]: created object 25... 2024-09-16T04:42:44.366 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37669]: created object 49... 2024-09-16T04:42:44.366 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37669]: finishing. 2024-09-16T04:42:44.366 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37669]: shutting down. 2024-09-16T04:42:44.382 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:44.382 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37671]: starting. 2024-09-16T04:42:44.382 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37671]: listing objects. 2024-09-16T04:42:44.382 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37671]: listed object 0... 2024-09-16T04:42:44.382 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37671]: listed object 25... 2024-09-16T04:42:44.382 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37671]: saw 50 objects 2024-09-16T04:42:44.382 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37671]: shutting down. 2024-09-16T04:42:44.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.478+0000 7fb1743a8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:42:44.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.478+0000 7fb1743a8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:44.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.478+0000 7fb1743a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:44.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:44.478+0000 7fb1743a8640 1 -- 172.21.15.46:0/1858591628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb16c067e20 con 0x7fb16c0a3ff0 2024-09-16T04:42:47.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:47.204+0000 7fb15dffb640 10 monclient: tick 2024-09-16T04:42:47.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:47.204+0000 7fb15dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:42:47.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:47.204+0000 7fb15dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:47.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:47.204+0000 7fb15dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:48.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:48.204+0000 7fb15dffb640 10 monclient: tick 2024-09-16T04:42:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:48.204+0000 7fb15dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:42:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:48.204+0000 7fb15dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:49.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:49.205+0000 7fb15dffb640 10 monclient: tick 2024-09-16T04:42:49.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:49.205+0000 7fb15dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:42:49.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:49.205+0000 7fb15dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:49.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:49.405+0000 7fb15effd640 1 -- 172.21.15.46:0/1858591628 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fb16001a520 con 0x7fb16c0a3ff0 2024-09-16T04:42:49.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:49.405+0000 7fb15effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:49.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:49.405+0000 7fb15effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:49.408 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":46,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:42:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:49.408+0000 7fb1743a8640 1 -- 172.21.15.46:0/1858591628 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb1500383b0 msgr2=0x7fb15003a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:49.408+0000 7fb1743a8640 1 --2- 172.21.15.46:0/1858591628 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb1500383b0 0x7fb15003a870 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb168000f60 tx=0x7fb168007b90 comp rx=0 tx=0).stop 2024-09-16T04:42:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:49.408+0000 7fb1743a8640 10 monclient: shutdown 2024-09-16T04:42:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:49.408+0000 7fb1743a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:49.408+0000 7fb1743a8640 1 -- 172.21.15.46:0/1858591628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c0a3ff0 msgr2=0x7fb16c1cc210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:49.408+0000 7fb1743a8640 1 --2- 172.21.15.46:0/1858591628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c0a3ff0 0x7fb16c1cc210 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb16000c480 tx=0x7fb16000c940 comp rx=0 tx=0).stop 2024-09-16T04:42:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:49.408+0000 7fb1743a8640 1 -- 172.21.15.46:0/1858591628 shutdown_connections 2024-09-16T04:42:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:49.408+0000 7fb1743a8640 1 --2- 172.21.15.46:0/1858591628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb16c13cea0 0x7fb16c139f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:49.408+0000 7fb1743a8640 1 --2- 172.21.15.46:0/1858591628 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb1500383b0 0x7fb15003a870 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:49.408+0000 7fb1743a8640 1 --2- 172.21.15.46:0/1858591628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c0a3ff0 0x7fb16c1cc210 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:49.408+0000 7fb1743a8640 1 --2- 172.21.15.46:0/1858591628 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb16c0a3c10 0x7fb16c1cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:49.408+0000 7fb1743a8640 1 -- 172.21.15.46:0/1858591628 >> 172.21.15.46:0/1858591628 conn(0x7fb16c0a6340 msgr2=0x7fb16c09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:49.408+0000 7fb1743a8640 1 -- 172.21.15.46:0/1858591628 shutdown_connections 2024-09-16T04:42:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:49.408+0000 7fb1743a8640 1 -- 172.21.15.46:0/1858591628 wait complete. 2024-09-16T04:42:49.479 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:49.479 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37670]: starting. 2024-09-16T04:42:49.479 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37670]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-37634 2024-09-16T04:42:49.479 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37670]: shutting down. 2024-09-16T04:42:49.480 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:49.480 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:49.480 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:42:49.483 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:42:51.362 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37738]: starting. 2024-09-16T04:42:51.362 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37738]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-37737 2024-09-16T04:42:51.362 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37738]: created object 0... 2024-09-16T04:42:51.362 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37738]: created object 25... 2024-09-16T04:42:51.362 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37738]: created object 49... 2024-09-16T04:42:51.362 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37738]: finishing. 2024-09-16T04:42:51.363 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37738]: shutting down. 2024-09-16T04:42:52.282 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37739]: starting. 2024-09-16T04:42:52.282 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37739]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-37737 2024-09-16T04:42:52.282 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37739]: shutting down. 2024-09-16T04:42:52.425 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:42:52.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.535+0000 7f7dfeffd640 10 monclient: get_monmap_and_config 2024-09-16T04:42:52.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.535+0000 7f7dfeffd640 10 monclient: build_initial_monmap 2024-09-16T04:42:52.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.535+0000 7f7dfeffd640 10 monclient: monmap: 2024-09-16T04:42:52.538 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:52.538 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:52.538 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:52.537066+0000 2024-09-16T04:42:52.538 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:52.537066+0000 2024-09-16T04:42:52.538 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:52.538 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:52.538 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:52.538 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:52.538 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:52.538 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:52.538 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:52.538 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:52.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.536+0000 7f7dfeffd640 1 Processor -- start 2024-09-16T04:42:52.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.536+0000 7f7dfeffd640 1 -- start start 2024-09-16T04:42:52.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.536+0000 7f7dfeffd640 10 monclient: init 2024-09-16T04:42:52.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.536+0000 7f7dfeffd640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:52.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.536+0000 7f7dfeffd640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-16T04:42:52.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.536+0000 7f7dfeffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7df8136e50 0x7f7df8137230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:52.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.536+0000 7f7dfeffd640 10 monclient(hunting): picked mon.noname-b con 0x7f7df8136e50 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:52.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.536+0000 7f7dfeffd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7df80a3500 0x7f7df80a3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:52.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.536+0000 7f7dfeffd640 10 monclient(hunting): picked mon.noname-c con 0x7f7df80a3500 addr v2:172.21.15.114:3300/0 2024-09-16T04:42:52.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.536+0000 7f7dfeffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7df80a4180 0x7f7df80a45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:52.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.536+0000 7f7dfeffd640 10 monclient(hunting): picked mon.noname-a con 0x7f7df80a4180 addr v2:172.21.15.46:3300/0 2024-09-16T04:42:52.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.536+0000 7f7dfeffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:52.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.536+0000 7f7dfeffd640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f7df80a5410 con 0x7f7df80a4180 2024-09-16T04:42:52.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.536+0000 7f7dfeffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:52.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.536+0000 7f7dfeffd640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f7df80a4e90 con 0x7f7df80a3500 2024-09-16T04:42:52.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.536+0000 7f7dfeffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:52.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.536+0000 7f7dfeffd640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f7df80a0dc0 con 0x7f7df8136e50 2024-09-16T04:42:52.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.536+0000 7f7dfeffd640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:52.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.536+0000 7f7dfeffd640 10 monclient(hunting): authenticate will time out at 970.952967s 2024-09-16T04:42:52.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfcd99640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7df8136e50 0x7f7df8137230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:52.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfd59a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7df80a4180 0x7f7df80a45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:52.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfcd99640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7df8136e50 0x7f7df8137230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:49820/0 (socket says 172.21.15.46:49820) 2024-09-16T04:42:52.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfcd99640 1 -- 172.21.15.46:0/2855340091 learned_addr learned my addr 172.21.15.46:0/2855340091 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:52.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfcd99640 10 monclient(hunting): get_auth_request con 0x7f7df8136e50 auth_method 0 2024-09-16T04:42:52.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfcd99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:52.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfcd99640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:52.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfcd99640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:52.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfd59a640 10 monclient(hunting): get_auth_request con 0x7f7df80a4180 auth_method 0 2024-09-16T04:42:52.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfd59a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:52.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfd59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:52.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:52.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfd59a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:52.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfd59a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:52.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfcd99640 1 -- 172.21.15.46:0/2855340091 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7df8136e50 msgr2=0x7f7df8137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:42:52.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfcd99640 1 -- 172.21.15.46:0/2855340091 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7df8136e50 msgr2=0x7f7df8137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:42:52.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfcd99640 1 --2- 172.21.15.46:0/2855340091 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7df8136e50 0x7f7df8137230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:52.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7deffff640 1 --2- 172.21.15.46:0/2855340091 >> v2:172.21.15.114:3300/0 conn(0x7f7df80a3500 0x7f7df80a3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:52.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfd59a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:52.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7deffff640 10 monclient(hunting): get_auth_request con 0x7f7df80a3500 auth_method 0 2024-09-16T04:42:52.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfcd99640 1 --2- 172.21.15.46:0/2855340091 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7df8136e50 0x7f7df8137230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:52.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7deffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:52.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7deffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:52.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7deffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:52.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7deffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:52.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7deffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:52.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7deffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:52.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfd59a640 10 monclient(hunting): handle_auth_done global_id 44364 payload 274 2024-09-16T04:42:52.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfd59a640 1 -- 172.21.15.46:0/2855340091 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7df8136e50 msgr2=0x7f7df8137230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:42:52.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfd59a640 1 --2- 172.21.15.46:0/2855340091 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7df8136e50 0x7f7df8137230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfd59a640 1 -- 172.21.15.46:0/2855340091 >> v2:172.21.15.114:3300/0 conn(0x7f7df80a3500 msgr2=0x7f7df80a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:52.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfd59a640 1 --2- 172.21.15.46:0/2855340091 >> v2:172.21.15.114:3300/0 conn(0x7f7df80a3500 0x7f7df80a3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfd59a640 10 monclient: _finish_hunting 0 2024-09-16T04:42:52.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfd59a640 1 monclient: found mon.noname-a 2024-09-16T04:42:52.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:52.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfd59a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:52.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.537+0000 7f7dfd59a640 1 -- 172.21.15.46:0/2855340091 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df80a10a0 con 0x7f7df80a4180 2024-09-16T04:42:52.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.538+0000 7f7dfd59a640 1 --2- 172.21.15.46:0/2855340091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df80a4180 0x7f7df80a45f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7df8074320 tx=0x7f7df002a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b327c5a924ed2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:52.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.538+0000 7f7def7fe640 1 -- 172.21.15.46:0/2855340091 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df0007740 con 0x7f7df80a4180 2024-09-16T04:42:52.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.538+0000 7f7def7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:52.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.538+0000 7f7def7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:52.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.538+0000 7f7def7fe640 10 monclient: dump: 2024-09-16T04:42:52.551 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:52.551 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:52.551 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:52.551 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:52.551 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:52.552 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:52.552 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:52.552 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:52.552 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:52.552 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:52.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.538+0000 7f7def7fe640 10 monclient: _finish_auth 0 2024-09-16T04:42:52.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.538+0000 7f7def7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:42:52.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.538+0000 7f7def7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:52.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.538+0000 7f7def7fe640 1 -- 172.21.15.46:0/2855340091 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7df00078e0 con 0x7f7df80a4180 2024-09-16T04:42:52.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.538+0000 7f7dfeffd640 5 monclient: authenticate success, global_id 44364 2024-09-16T04:42:52.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.538+0000 7f7dfeffd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:52.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.538+0000 7f7def7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:52.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.538+0000 7f7dfeffd640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:52.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.538+0000 7f7dfeffd640 4 set_mon_vals no callback set 2024-09-16T04:42:52.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.538+0000 7f7def7fe640 1 -- 172.21.15.46:0/2855340091 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df0007cb0 con 0x7f7df80a4180 2024-09-16T04:42:52.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.538+0000 7f7def7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:52.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.538+0000 7f7def7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:52.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.538+0000 7f7def7fe640 10 monclient: dump: 2024-09-16T04:42:52.553 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:52.553 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:52.553 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:52.553 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:52.553 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:52.553 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:52.553 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:52.553 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:52.553 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:52.553 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:52.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.539+0000 7f7dfeffd640 10 monclient: shutdown 2024-09-16T04:42:52.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.539+0000 7f7dfeffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:52.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.539+0000 7f7dfeffd640 1 -- 172.21.15.46:0/2855340091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df80a4180 msgr2=0x7f7df80a45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:52.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.539+0000 7f7dfeffd640 1 --2- 172.21.15.46:0/2855340091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df80a4180 0x7f7df80a45f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7df8074320 tx=0x7f7df002a4b0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.539+0000 7f7dfeffd640 1 -- 172.21.15.46:0/2855340091 shutdown_connections 2024-09-16T04:42:52.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.539+0000 7f7dfeffd640 1 --2- 172.21.15.46:0/2855340091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df80a4180 0x7f7df80a45f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.539+0000 7f7dfeffd640 1 --2- 172.21.15.46:0/2855340091 >> v2:172.21.15.114:3300/0 conn(0x7f7df80a3500 0x7f7df80a3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.539+0000 7f7dfeffd640 1 --2- 172.21.15.46:0/2855340091 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7df8136e50 0x7f7df8137230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.539+0000 7f7dfeffd640 1 -- 172.21.15.46:0/2855340091 >> 172.21.15.46:0/2855340091 conn(0x7f7df80a5f20 msgr2=0x7f7df80a6330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:52.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.539+0000 7f7dfeffd640 1 -- 172.21.15.46:0/2855340091 shutdown_connections 2024-09-16T04:42:52.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.539+0000 7f7dfeffd640 1 -- 172.21.15.46:0/2855340091 wait complete. 2024-09-16T04:42:52.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.539+0000 7f7dfeffd640 10 monclient: build_initial_monmap 2024-09-16T04:42:52.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.539+0000 7f7dfeffd640 10 monclient: monmap: 2024-09-16T04:42:52.554 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:52.555 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:52.555 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:52.555 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:52.555 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:52.555 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:52.555 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:52.555 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:52.555 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:52.555 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:52.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfeffd640 1 Processor -- start 2024-09-16T04:42:52.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfeffd640 1 -- start start 2024-09-16T04:42:52.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfeffd640 10 monclient: init 2024-09-16T04:42:52.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfeffd640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:52.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfeffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:42:52.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfeffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7df80a3500 0x7f7df81cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:52.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfeffd640 10 monclient(hunting): picked mon.noname-c con 0x7f7df80a3500 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:52.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df80a4180 0x7f7df81cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:52.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfeffd640 10 monclient(hunting): picked mon.noname-a con 0x7f7df80a4180 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:52.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df8136e50 0x7f7df81234d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:52.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfeffd640 10 monclient(hunting): picked mon.noname-b con 0x7f7df8136e50 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:52.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfeffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:52.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfeffd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7df8137770 con 0x7f7df80a4180 2024-09-16T04:42:52.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfeffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:52.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfeffd640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7df80a09b0 con 0x7f7df8136e50 2024-09-16T04:42:52.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfeffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:52.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfeffd640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f7df8127e60 con 0x7f7df80a3500 2024-09-16T04:42:52.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfeffd640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:52.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfeffd640 10 monclient(hunting): authenticate will time out at 970.956870s 2024-09-16T04:42:52.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfcd99640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7df80a3500 0x7f7df81cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:52.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfcd99640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7df80a3500 0x7f7df81cbeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:49828/0 (socket says 172.21.15.46:49828) 2024-09-16T04:42:52.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfcd99640 1 -- 172.21.15.46:0/364287385 learned_addr learned my addr 172.21.15.46:0/364287385 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:52.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfcd99640 10 monclient(hunting): get_auth_request con 0x7f7df80a3500 auth_method 0 2024-09-16T04:42:52.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfcd99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:52.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfcd99640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:52.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7dfcd99640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:52.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7deffff640 1 --2- 172.21.15.46:0/364287385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df80a4180 0x7f7df81cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:52.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7deffff640 10 monclient(hunting): get_auth_request con 0x7f7df80a4180 auth_method 0 2024-09-16T04:42:52.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7deffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:52.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7deffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:52.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.540+0000 7f7deffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:52.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7dfcd99640 1 -- 172.21.15.46:0/364287385 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7df80a3500 msgr2=0x7f7df81cbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:42:52.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7dfcd99640 1 -- 172.21.15.46:0/364287385 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7df80a3500 msgr2=0x7f7df81cbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:42:52.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7dfcd99640 1 --2- 172.21.15.46:0/364287385 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7df80a3500 0x7f7df81cbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:52.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7deffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:52.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7deffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:52.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7dfcd99640 1 --2- 172.21.15.46:0/364287385 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7df80a3500 0x7f7df81cbeb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:52.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7deffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:52.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7dfd59a640 1 --2- 172.21.15.46:0/364287385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df8136e50 0x7f7df81234d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:52.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7dfd59a640 10 monclient(hunting): get_auth_request con 0x7f7df8136e50 auth_method 0 2024-09-16T04:42:52.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7dfd59a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:52.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7dfd59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:52.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7dfd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:52.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7deffff640 10 monclient(hunting): handle_auth_done global_id 44367 payload 824 2024-09-16T04:42:52.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7deffff640 1 -- 172.21.15.46:0/364287385 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7df80a3500 msgr2=0x7f7df81cbeb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:42:52.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7deffff640 1 --2- 172.21.15.46:0/364287385 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7df80a3500 0x7f7df81cbeb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7deffff640 1 -- 172.21.15.46:0/364287385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df8136e50 msgr2=0x7f7df81234d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:52.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7deffff640 1 --2- 172.21.15.46:0/364287385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df8136e50 0x7f7df81234d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7deffff640 10 monclient: _finish_hunting 0 2024-09-16T04:42:52.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7deffff640 1 monclient: found mon.noname-a 2024-09-16T04:42:52.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7deffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:52.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7deffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:52.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7deffff640 1 -- 172.21.15.46:0/364287385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df8127180 con 0x7f7df80a4180 2024-09-16T04:42:52.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7deffff640 1 --2- 172.21.15.46:0/364287385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df80a4180 0x7f7df81cc3f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7de800e820 tx=0x7f7de800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:52.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7ded7fa640 1 -- 172.21.15.46:0/364287385 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de8017070 con 0x7f7df80a4180 2024-09-16T04:42:52.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7ded7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:52.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7ded7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:52.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7ded7fa640 10 monclient: dump: 2024-09-16T04:42:52.562 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:52.562 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:52.562 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:52.562 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:52.562 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:52.563 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:52.563 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:52.563 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:52.563 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:52.563 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:52.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7ded7fa640 10 monclient: _finish_auth 0 2024-09-16T04:42:52.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7ded7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:42:52.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7ded7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:52.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7ded7fa640 1 -- 172.21.15.46:0/364287385 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7de800c940 con 0x7f7df80a4180 2024-09-16T04:42:52.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7ded7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:52.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7ded7fa640 1 -- 172.21.15.46:0/364287385 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de800cd10 con 0x7f7df80a4180 2024-09-16T04:42:52.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7dfeffd640 5 monclient: authenticate success, global_id 44367 2024-09-16T04:42:52.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7ded7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:52.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7ded7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:52.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7ded7fa640 10 monclient: dump: 2024-09-16T04:42:52.564 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:52.564 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:52.564 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:52.564 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:52.564 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:52.564 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:52.564 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:52.564 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:52.564 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:52.564 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:52.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7dfeffd640 10 monclient: _renew_subs 2024-09-16T04:42:52.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7dfeffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:52.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7dfeffd640 1 -- 172.21.15.46:0/364287385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7df8127460 con 0x7f7df80a4180 2024-09-16T04:42:52.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7dee7fc640 4 set_mon_vals no callback set 2024-09-16T04:42:52.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7dfeffd640 10 monclient: _renew_subs 2024-09-16T04:42:52.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7dfeffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:52.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.541+0000 7f7dfeffd640 1 -- 172.21.15.46:0/364287385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7df809f2d0 con 0x7f7df80a4180 2024-09-16T04:42:52.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.542+0000 7f7dfeffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:52.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.542+0000 7f7dfeffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:52.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.542+0000 7f7dfeffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:52.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.542+0000 7f7ded7fa640 1 -- 172.21.15.46:0/364287385 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f7de801a070 con 0x7f7df80a4180 2024-09-16T04:42:52.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.542+0000 7f7dfeffd640 1 -- 172.21.15.46:0/364287385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7df80a1bf0 con 0x7f7df80a4180 2024-09-16T04:42:52.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.543+0000 7f7ded7fa640 1 --2- 172.21.15.46:0/364287385 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7de40383b0 0x7f7de403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:52.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.543+0000 7f7ded7fa640 1 -- 172.21.15.46:0/364287385 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 4346+0+0 (secure 0 0 0) 0x7f7de804b560 con 0x7f7df80a4180 2024-09-16T04:42:52.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.543+0000 7f7dfcd99640 1 --2- 172.21.15.46:0/364287385 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7de40383b0 0x7f7de403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:52.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.543+0000 7f7dfcd99640 10 monclient: get_auth_request con 0x7f7de40383b0 auth_method 0 2024-09-16T04:42:52.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.543+0000 7f7dfcd99640 1 --2- 172.21.15.46:0/364287385 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7de40383b0 0x7f7de403a870 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7df40091d0 tx=0x7f7df4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:52.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.547+0000 7f7ded7fa640 1 -- 172.21.15.46:0/364287385 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7de8011350 con 0x7f7df80a4180 2024-09-16T04:42:52.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.547+0000 7f7ded7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:52.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.547+0000 7f7ded7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:52.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.796+0000 7f7dfeffd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:42:52.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.796+0000 7f7dfeffd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:52.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.796+0000 7f7dfeffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:52.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.796+0000 7f7dfeffd640 1 -- 172.21.15.46:0/364287385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7df8067e40 con 0x7f7df80a4180 2024-09-16T04:42:52.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.796+0000 7f7ded7fa640 1 -- 172.21.15.46:0/364287385 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f7de804d6e0 con 0x7f7df80a4180 2024-09-16T04:42:52.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.796+0000 7f7ded7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:52.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.796+0000 7f7ded7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:52.799 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:42:52.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.798+0000 7f7dfeffd640 1 -- 172.21.15.46:0/364287385 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7de40383b0 msgr2=0x7f7de403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:52.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.798+0000 7f7dfeffd640 1 --2- 172.21.15.46:0/364287385 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7de40383b0 0x7f7de403a870 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7df40091d0 tx=0x7f7df4008040 comp rx=0 tx=0).stop 2024-09-16T04:42:52.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.798+0000 7f7dfeffd640 10 monclient: shutdown 2024-09-16T04:42:52.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.798+0000 7f7dfeffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:52.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.798+0000 7f7dfeffd640 1 -- 172.21.15.46:0/364287385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df80a4180 msgr2=0x7f7df81cc3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:52.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.798+0000 7f7dfeffd640 1 --2- 172.21.15.46:0/364287385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df80a4180 0x7f7df81cc3f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7de800e820 tx=0x7f7de800ece0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.798+0000 7f7dfeffd640 1 -- 172.21.15.46:0/364287385 shutdown_connections 2024-09-16T04:42:52.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.798+0000 7f7dfeffd640 1 --2- 172.21.15.46:0/364287385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df8136e50 0x7f7df81234d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.798+0000 7f7dfeffd640 1 --2- 172.21.15.46:0/364287385 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7de40383b0 0x7f7de403a870 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.798+0000 7f7dfeffd640 1 --2- 172.21.15.46:0/364287385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df80a4180 0x7f7df81cc3f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.798+0000 7f7dfeffd640 1 --2- 172.21.15.46:0/364287385 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7df80a3500 0x7f7df81cbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.798+0000 7f7dfeffd640 1 -- 172.21.15.46:0/364287385 >> 172.21.15.46:0/364287385 conn(0x7f7df80a5f20 msgr2=0x7f7df80a78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:52.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.799+0000 7f7dfeffd640 1 -- 172.21.15.46:0/364287385 shutdown_connections 2024-09-16T04:42:52.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.799+0000 7f7dfeffd640 1 -- 172.21.15.46:0/364287385 wait complete. 2024-09-16T04:42:52.811 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:42:52.811 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:42:52.812 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:42:52.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.912+0000 7fd6758f1640 10 monclient: get_monmap_and_config 2024-09-16T04:42:52.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.912+0000 7fd6758f1640 10 monclient: build_initial_monmap 2024-09-16T04:42:52.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.912+0000 7fd6758f1640 10 monclient: monmap: 2024-09-16T04:42:52.914 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:52.914 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:52.914 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:52.913529+0000 2024-09-16T04:42:52.914 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:52.913529+0000 2024-09-16T04:42:52.914 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:52.914 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:52.914 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:52.915 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:52.915 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:52.915 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:52.915 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:52.915 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:52.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.912+0000 7fd6758f1640 1 Processor -- start 2024-09-16T04:42:52.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.912+0000 7fd6758f1640 1 -- start start 2024-09-16T04:42:52.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.912+0000 7fd6758f1640 10 monclient: init 2024-09-16T04:42:52.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd6758f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:52.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd6758f1640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-16T04:42:52.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd6758f1640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6700a3c10 0x7fd67009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:52.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd6758f1640 10 monclient(hunting): picked mon.noname-b con 0x7fd6700a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:52.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd6758f1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd6700a3ff0 0x7fd6700a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:52.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd6758f1640 10 monclient(hunting): picked mon.noname-a con 0x7fd6700a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:42:52.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd6758f1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6701369d0 addr v1:172.21.15.114:6789/0 2024-09-16T04:42:52.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd6758f1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd67009ab70 con 0x7fd6701369d0 2024-09-16T04:42:52.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd6758f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:52.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd6758f1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd6700a4d60 con 0x7fd6700a3ff0 2024-09-16T04:42:52.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd6758f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:52.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd6758f1640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd67009b030 con 0x7fd6700a3c10 2024-09-16T04:42:52.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd6758f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:52.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd6758f1640 10 monclient(hunting): authenticate will time out at 971.329314s 2024-09-16T04:42:52.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66e7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd6700a3ff0 0x7fd6700a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:52.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66effd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6700a3c10 0x7fd67009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:52.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66e7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd6700a3ff0 0x7fd6700a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40084/0 (socket says 172.21.15.46:40084) 2024-09-16T04:42:52.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66effd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6700a3c10 0x7fd67009bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:49830/0 (socket says 172.21.15.46:49830) 2024-09-16T04:42:52.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66e7fc640 1 -- 172.21.15.46:0/10560090 learned_addr learned my addr 172.21.15.46:0/10560090 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:52.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66e7fc640 10 monclient(hunting): get_auth_request con 0x7fd6700a3ff0 auth_method 0 2024-09-16T04:42:52.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:52.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:52.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:52.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66effd640 10 monclient(hunting): get_auth_request con 0x7fd6700a3c10 auth_method 0 2024-09-16T04:42:52.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:52.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:52.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:52.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:52.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:52.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:52.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66effd640 1 -- 172.21.15.46:0/10560090 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6700a3c10 msgr2=0x7fd67009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:42:52.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66effd640 1 -- 172.21.15.46:0/10560090 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6700a3c10 msgr2=0x7fd67009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:42:52.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66effd640 1 --2- 172.21.15.46:0/10560090 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6700a3c10 0x7fd67009bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:52.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66effd640 1 --2- 172.21.15.46:0/10560090 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6700a3c10 0x7fd67009bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:52.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66e7fc640 10 monclient(hunting): handle_auth_done global_id 44370 payload 274 2024-09-16T04:42:52.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66e7fc640 1 -- 172.21.15.46:0/10560090 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6700a3c10 msgr2=0x7fd67009bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:42:52.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66e7fc640 1 --2- 172.21.15.46:0/10560090 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6700a3c10 0x7fd67009bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66e7fc640 1 -- 172.21.15.46:0/10560090 >> v1:172.21.15.114:6789/0 conn(0x7fd6701369d0 legacy=0x7fd670136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:52.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:42:52.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66e7fc640 1 monclient: found mon.noname-a 2024-09-16T04:42:52.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:52.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:52.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.913+0000 7fd66e7fc640 1 -- 172.21.15.46:0/10560090 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd67009b310 con 0x7fd6700a3ff0 2024-09-16T04:42:52.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.914+0000 7fd66e7fc640 1 --2- 172.21.15.46:0/10560090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6700a3ff0 0x7fd6700a43d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd664009870 tx=0x7fd66402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=676371b045ed9f2b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:52.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.914+0000 7fd66dffb640 1 -- 172.21.15.46:0/10560090 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd664037070 con 0x7fd6700a3ff0 2024-09-16T04:42:52.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.914+0000 7fd66dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:52.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.914+0000 7fd66dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:52.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.914+0000 7fd66dffb640 10 monclient: dump: 2024-09-16T04:42:52.927 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:52.927 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:52.927 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:52.928 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:52.928 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:52.928 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:52.928 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:52.928 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:52.928 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:52.928 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:52.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.914+0000 7fd66dffb640 10 monclient: _finish_auth 0 2024-09-16T04:42:52.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.914+0000 7fd66dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:42:52.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.914+0000 7fd66dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:52.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.914+0000 7fd66dffb640 1 -- 172.21.15.46:0/10560090 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd664032440 con 0x7fd6700a3ff0 2024-09-16T04:42:52.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.914+0000 7fd6758f1640 5 monclient: authenticate success, global_id 44370 2024-09-16T04:42:52.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.914+0000 7fd6758f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:52.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.914+0000 7fd66dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:52.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.914+0000 7fd6758f1640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:52.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.914+0000 7fd66dffb640 1 -- 172.21.15.46:0/10560090 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd664032810 con 0x7fd6700a3ff0 2024-09-16T04:42:52.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.914+0000 7fd6758f1640 4 set_mon_vals no callback set 2024-09-16T04:42:52.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.914+0000 7fd66dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:52.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.914+0000 7fd66dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:52.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.914+0000 7fd66dffb640 10 monclient: dump: 2024-09-16T04:42:52.929 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:52.929 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:52.929 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:52.929 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:52.929 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:52.929 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:52.929 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:52.930 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:52.930 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:52.930 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:52.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.915+0000 7fd6758f1640 10 monclient: shutdown 2024-09-16T04:42:52.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.915+0000 7fd6758f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:52.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.915+0000 7fd6758f1640 1 -- 172.21.15.46:0/10560090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6700a3ff0 msgr2=0x7fd6700a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:52.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.915+0000 7fd6758f1640 1 --2- 172.21.15.46:0/10560090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6700a3ff0 0x7fd6700a43d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd664009870 tx=0x7fd66402a180 comp rx=0 tx=0).stop 2024-09-16T04:42:52.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.915+0000 7fd6758f1640 1 -- 172.21.15.46:0/10560090 shutdown_connections 2024-09-16T04:42:52.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.915+0000 7fd6758f1640 1 --2- 172.21.15.46:0/10560090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6700a3ff0 0x7fd6700a43d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.915+0000 7fd6758f1640 1 --2- 172.21.15.46:0/10560090 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6700a3c10 0x7fd67009bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.915+0000 7fd6758f1640 1 -- 172.21.15.46:0/10560090 >> 172.21.15.46:0/10560090 conn(0x7fd6700a5d10 msgr2=0x7fd6700a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:52.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.915+0000 7fd6758f1640 1 -- 172.21.15.46:0/10560090 shutdown_connections 2024-09-16T04:42:52.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.915+0000 7fd6758f1640 1 -- 172.21.15.46:0/10560090 wait complete. 2024-09-16T04:42:52.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.915+0000 7fd6758f1640 10 monclient: build_initial_monmap 2024-09-16T04:42:52.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.915+0000 7fd6758f1640 10 monclient: monmap: 2024-09-16T04:42:52.931 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:52.931 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:52.931 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:52.931 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:52.931 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:52.931 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:52.931 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:52.931 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:52.931 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:52.931 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:52.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.916+0000 7fd6758f1640 1 Processor -- start 2024-09-16T04:42:52.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.916+0000 7fd6758f1640 1 -- start start 2024-09-16T04:42:52.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.916+0000 7fd6758f1640 10 monclient: init 2024-09-16T04:42:52.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.916+0000 7fd6758f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:52.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.916+0000 7fd6758f1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:42:52.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.916+0000 7fd6758f1640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6700a3c10 0x7fd6701cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:52.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.916+0000 7fd6758f1640 10 monclient(hunting): picked mon.noname-c con 0x7fd6700a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:52.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.916+0000 7fd6758f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6700a3ff0 0x7fd6701cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:52.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.916+0000 7fd6758f1640 10 monclient(hunting): picked mon.noname-a con 0x7fd6700a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:52.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.916+0000 7fd6758f1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6701369d0 0x7fd6701231c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:52.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.916+0000 7fd6758f1640 10 monclient(hunting): picked mon.noname-b con 0x7fd6701369d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:52.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.916+0000 7fd6758f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:52.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.916+0000 7fd6758f1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd6700a4910 con 0x7fd6700a3ff0 2024-09-16T04:42:52.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.916+0000 7fd6758f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:52.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.916+0000 7fd6758f1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd67006e3d0 con 0x7fd6701369d0 2024-09-16T04:42:52.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.916+0000 7fd6758f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.916+0000 7fd6758f1640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd6701252d0 con 0x7fd6700a3c10 2024-09-16T04:42:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.916+0000 7fd6758f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.916+0000 7fd6758f1640 10 monclient(hunting): authenticate will time out at 971.332936s 2024-09-16T04:42:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.916+0000 7fd66effd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6700a3c10 0x7fd6701cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6700a3ff0 0x7fd6701cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66effd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6700a3c10 0x7fd6701cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:49846/0 (socket says 172.21.15.46:49846) 2024-09-16T04:42:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66effd640 1 -- 172.21.15.46:0/2972094533 learned_addr learned my addr 172.21.15.46:0/2972094533 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66effd640 10 monclient(hunting): get_auth_request con 0x7fd6700a3c10 auth_method 0 2024-09-16T04:42:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6700a3ff0 0x7fd6701cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40096/0 (socket says 172.21.15.46:40096) 2024-09-16T04:42:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66e7fc640 10 monclient(hunting): get_auth_request con 0x7fd6700a3ff0 auth_method 0 2024-09-16T04:42:52.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:52.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:52.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:52.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:52.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:52.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:52.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66effd640 1 -- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6700a3c10 msgr2=0x7fd6701cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:42:52.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66effd640 1 -- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6700a3c10 msgr2=0x7fd6701cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:42:52.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66effd640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6700a3c10 0x7fd6701cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:52.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66effd640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6700a3c10 0x7fd6701cbee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:52.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66e7fc640 10 monclient(hunting): handle_auth_done global_id 44373 payload 824 2024-09-16T04:42:52.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66f7fe640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6701369d0 0x7fd6701231c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:52.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66e7fc640 1 -- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6700a3c10 msgr2=0x7fd6701cbee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:42:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66e7fc640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd6700a3c10 0x7fd6701cbee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66e7fc640 1 -- 172.21.15.46:0/2972094533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6701369d0 msgr2=0x7fd6701231c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66e7fc640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6701369d0 0x7fd6701231c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:42:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66e7fc640 1 monclient: found mon.noname-a 2024-09-16T04:42:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66e7fc640 1 -- 172.21.15.46:0/2972094533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd670126af0 con 0x7fd6700a3ff0 2024-09-16T04:42:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66f7fe640 10 monclient: get_auth_request con 0x7fd6701369d0 auth_method 0 2024-09-16T04:42:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66f7fe640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6701369d0 0x7fd6701231c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:42:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66e7fc640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6700a3ff0 0x7fd6701cc420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd664031000 tx=0x7fd6640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd657fff640 1 -- 172.21.15.46:0/2972094533 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd664041020 con 0x7fd6700a3ff0 2024-09-16T04:42:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd657fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd657fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:52.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd657fff640 10 monclient: dump: 2024-09-16T04:42:52.936 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:52.936 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:52.936 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:52.936 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:52.936 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:52.936 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:52.936 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:52.936 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:52.936 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:52.936 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:52.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd657fff640 10 monclient: _finish_auth 0 2024-09-16T04:42:52.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd657fff640 10 monclient: _check_auth_tickets 2024-09-16T04:42:52.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd657fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:52.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd657fff640 1 -- 172.21.15.46:0/2972094533 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd664030780 con 0x7fd6700a3ff0 2024-09-16T04:42:52.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd6758f1640 5 monclient: authenticate success, global_id 44373 2024-09-16T04:42:52.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd657fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:52.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd6758f1640 10 monclient: _renew_subs 2024-09-16T04:42:52.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd657fff640 1 -- 172.21.15.46:0/2972094533 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd664037040 con 0x7fd6700a3ff0 2024-09-16T04:42:52.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd6758f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:52.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd6758f1640 1 -- 172.21.15.46:0/2972094533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd670126dd0 con 0x7fd6700a3ff0 2024-09-16T04:42:52.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd657fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:52.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd657fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:52.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd657fff640 10 monclient: dump: 2024-09-16T04:42:52.937 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:52.937 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:52.937 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:52.938 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:52.938 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:52.938 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:52.938 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:52.938 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:52.938 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:52.938 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:52.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd66d7fa640 4 set_mon_vals no callback set 2024-09-16T04:42:52.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd6758f1640 10 monclient: _renew_subs 2024-09-16T04:42:52.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd6758f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:52.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.917+0000 7fd6758f1640 1 -- 172.21.15.46:0/2972094533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6701270b0 con 0x7fd6700a3ff0 2024-09-16T04:42:52.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.918+0000 7fd6758f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:52.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.918+0000 7fd6758f1640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd670123700 0x7fd670123ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:52.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.918+0000 7fd6758f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:52.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.918+0000 7fd6758f1640 1 -- 172.21.15.46:0/2972094533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd670137580 con 0x7fd670123700 2024-09-16T04:42:52.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.918+0000 7fd657fff640 1 -- 172.21.15.46:0/2972094533 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fd664044020 con 0x7fd6700a3ff0 2024-09-16T04:42:52.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.918+0000 7fd66effd640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd670123700 0x7fd670123ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:52.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.918+0000 7fd66effd640 10 monclient: get_auth_request con 0x7fd670123700 auth_method 0 2024-09-16T04:42:52.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.918+0000 7fd66effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:52.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.918+0000 7fd66effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:52.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.918+0000 7fd66effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:52.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd66effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:52.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd66effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:52.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd66effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:52.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd657fff640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd65c0383e0 0x7fd65c03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:52.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd657fff640 1 -- 172.21.15.46:0/2972094533 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 4346+0+0 (secure 0 0 0) 0x7fd66406bd10 con 0x7fd6700a3ff0 2024-09-16T04:42:52.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd66effd640 10 monclient(hunting): handle_auth_done global_id 44376 payload 824 2024-09-16T04:42:52.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd66effd640 1 -- 172.21.15.46:0/2972094533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd670069150 con 0x7fd670123700 2024-09-16T04:42:52.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd66f7fe640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd65c0383e0 0x7fd65c03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:52.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd66f7fe640 10 monclient: get_auth_request con 0x7fd65c0383e0 auth_method 0 2024-09-16T04:42:52.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd66effd640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd670123700 0x7fd670123ae0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd66000bfd0 tx=0x7fd66000f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:52.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd657fff640 1 -- 172.21.15.46:0/2972094533 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd660019020 con 0x7fd670123700 2024-09-16T04:42:52.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd657fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:52.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd657fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:52.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd657fff640 10 monclient: dump: 2024-09-16T04:42:52.940 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:52.940 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:52.940 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:52.940 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:52.941 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:52.941 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:52.941 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:52.941 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:52.941 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:52.941 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:52.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd66f7fe640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd65c0383e0 0x7fd65c03a8a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd668004540 tx=0x7fd668009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:52.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd657fff640 1 -- 172.21.15.46:0/2972094533 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd660014870 con 0x7fd670123700 2024-09-16T04:42:52.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd657fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:52.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd657fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:52.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd657fff640 1 -- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd670123700 msgr2=0x7fd670123ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:52.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd657fff640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd670123700 0x7fd670123ae0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd66000bfd0 tx=0x7fd66000f9e0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd657fff640 1 -- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd670123700 msgr2=0x7fd670123ae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:52.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.919+0000 7fd657fff640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd670123700 0x7fd670123ae0 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd66000bfd0 tx=0x7fd66000f9e0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd6758f1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:42:52.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd6758f1640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd670124020 0x7fd67011f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:52.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd6758f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:52.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd6758f1640 1 -- 172.21.15.46:0/2972094533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd670126810 con 0x7fd670124020 2024-09-16T04:42:52.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd66effd640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd670124020 0x7fd67011f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:52.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd66effd640 10 monclient: get_auth_request con 0x7fd670124020 auth_method 0 2024-09-16T04:42:52.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd66effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:52.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd66effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:52.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd66effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:52.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd66effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:52.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd66effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:52.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd66effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:52.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd66effd640 10 monclient(hunting): handle_auth_done global_id 44379 payload 824 2024-09-16T04:42:52.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd66effd640 1 -- 172.21.15.46:0/2972094533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd6701cc960 con 0x7fd670124020 2024-09-16T04:42:52.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd66effd640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd670124020 0x7fd67011f860 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd66001a000 tx=0x7fd6600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:52.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd657fff640 1 -- 172.21.15.46:0/2972094533 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd660019040 con 0x7fd670124020 2024-09-16T04:42:52.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd657fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:52.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd657fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:52.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd657fff640 10 monclient: dump: 2024-09-16T04:42:52.945 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:52.945 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:52.945 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:52.945 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:52.945 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:52.945 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:52.945 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:52.945 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:52.945 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:52.945 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:52.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd657fff640 1 -- 172.21.15.46:0/2972094533 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7fd66001e420 con 0x7fd670124020 2024-09-16T04:42:52.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd657fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:42:52.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd657fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:52.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd657fff640 1 -- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd670124020 msgr2=0x7fd67011f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:52.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd657fff640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd670124020 0x7fd67011f860 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd66001a000 tx=0x7fd6600046c0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd657fff640 1 -- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd670124020 msgr2=0x7fd67011f860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:52.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.926+0000 7fd657fff640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd670124020 0x7fd67011f860 secure :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd66001a000 tx=0x7fd6600046c0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.946 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:42:52.946 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:42:52.946 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:42:52.946 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:42:52.946 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 48, 2024-09-16T04:42:52.947 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 52674, 2024-09-16T04:42:52.947 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:42:52.947 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:42:52.947 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:42:52.947 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:42:52.947 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:52.947 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 3, 2024-09-16T04:42:52.947 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:52.947 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:42:52.947 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:42:52.947 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:52.947 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:52.947 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:52.947 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:52.948 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:52.948 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:52.948 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:52.948 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:52.948 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:52.948 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:52.948 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:52.948 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:52.948 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:42:52.948 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:42:52.948 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:52.948 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:52.948 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:52.949 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:52.949 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:52.949 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:52.949 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:52.949 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:52.949 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:52.949 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:52.949 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:52.949 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:52.949 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:52.949 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:42:52.949 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:42:52.949 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:42:52.950 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:42:52.950 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:42:52.950 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:42:52.950 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:52.950 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:52.950 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:42:52.950 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:42:52.950 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:42:52.950 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:42:52.950 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:42:52.950 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:42:52.950 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:42:52.950 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:52.950 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:42:52.951 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:52.951 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:52.951 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:52.951 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:52.951 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:52.951 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:52.951 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:52.951 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:52.951 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:52.951 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:52.951 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:52.951 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:52.951 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:42:52.952 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:52.952 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:42:52.952 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:52.952 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:42:52.952 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:42:52.952 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:52.952 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:52.952 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:52.952 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:52.952 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:42:52.952 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:52.952 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:52.952 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:52.953 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:52.953 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:42:52.953 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:52.953 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:52.953 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:52.953 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:52.953 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:42:52.953 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:42:52.953 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:52.953 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:52.953 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:52.953 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:52.953 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:52.954 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:42:52.954 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:42:52.954 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:52.954 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:52.954 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:52.954 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:52.954 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:42:52.954 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:52.954 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:52.954 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:52.954 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:52.954 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:42:52.954 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:52.954 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:52.955 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:52.955 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:52.955 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:42:52.955 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:42:52.955 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:52.955 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:52.955 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:52.955 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:52.955 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:52.955 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:42:52.955 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:42:52.955 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:52.955 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:52.955 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:52.956 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:52.956 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:42:52.956 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:52.956 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:52.956 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:52.956 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:52.956 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:42:52.956 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:52.956 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:52.956 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:52.956 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:52.956 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:42:52.956 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:42:52.957 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:52.957 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:52.957 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:52.957 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:52.957 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:52.957 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:52.957 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:42:52.957 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:42:52.957 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:52.957 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:52.957 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:52.957 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:42:52.957 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:52.957 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:52.958 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:42:52.958 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:52.958 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:52.958 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:52.958 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:42:52.958 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:52.958 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:52.958 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:42:52.958 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:52.958 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:52.958 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:52.958 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:42:52.958 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:52.959 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:52.959 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:52.959 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:42:52.959 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:42:52.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.927+0000 7fd6758f1640 1 -- 172.21.15.46:0/2972094533 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd65c0383e0 msgr2=0x7fd65c03a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:52.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.927+0000 7fd6758f1640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd65c0383e0 0x7fd65c03a8a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd668004540 tx=0x7fd668009040 comp rx=0 tx=0).stop 2024-09-16T04:42:52.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.927+0000 7fd6758f1640 10 monclient: shutdown 2024-09-16T04:42:52.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.927+0000 7fd6758f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:52.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.927+0000 7fd6758f1640 1 -- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6700a3ff0 msgr2=0x7fd6701cc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:52.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.927+0000 7fd66f7fe640 1 -- 172.21.15.46:0/2972094533 reap_dead start 2024-09-16T04:42:52.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.927+0000 7fd6758f1640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6700a3ff0 0x7fd6701cc420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd664031000 tx=0x7fd6640046c0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.927+0000 7fd6758f1640 1 -- 172.21.15.46:0/2972094533 shutdown_connections 2024-09-16T04:42:52.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.927+0000 7fd6758f1640 1 --2- 172.21.15.46:0/2972094533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6700a3ff0 0x7fd6701cc420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:52.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.927+0000 7fd6758f1640 1 -- 172.21.15.46:0/2972094533 >> 172.21.15.46:0/2972094533 conn(0x7fd6700a5d10 msgr2=0x7fd67009e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:52.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.927+0000 7fd6758f1640 1 -- 172.21.15.46:0/2972094533 shutdown_connections 2024-09-16T04:42:52.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:52.927+0000 7fd6758f1640 1 -- 172.21.15.46:0/2972094533 wait complete. 2024-09-16T04:42:52.961 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:42:52.961 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:42:52.962 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:42:53.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.062+0000 7f46d123c640 10 monclient: get_monmap_and_config 2024-09-16T04:42:53.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.062+0000 7f46d123c640 10 monclient: build_initial_monmap 2024-09-16T04:42:53.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.062+0000 7f46d123c640 10 monclient: monmap: 2024-09-16T04:42:53.064 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:53.065 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:53.065 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:53.063940+0000 2024-09-16T04:42:53.065 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:53.063940+0000 2024-09-16T04:42:53.065 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:53.065 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:53.065 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:53.065 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:53.065 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:53.065 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:53.065 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:53.065 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46d123c640 1 Processor -- start 2024-09-16T04:42:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46d123c640 1 -- start start 2024-09-16T04:42:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46d123c640 10 monclient: init 2024-09-16T04:42:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46d123c640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46d123c640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-16T04:42:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46d123c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc136e50 0x7f46cc137230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46d123c640 10 monclient(hunting): picked mon.noname-b con 0x7f46cc136e50 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46d123c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f46cc0a3500 addr v1:172.21.15.114:6789/0 2024-09-16T04:42:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46d123c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46cc0a4350 addr v1:172.21.15.46:6789/0 2024-09-16T04:42:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46d123c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46cc0a4e90 con 0x7f46cc0a4350 2024-09-16T04:42:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46d123c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46cc0a0dc0 con 0x7f46cc0a3500 2024-09-16T04:42:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46d123c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46d123c640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f46cc0a5410 con 0x7f46cc136e50 2024-09-16T04:42:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46d123c640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46d123c640 10 monclient(hunting): authenticate will time out at 971.479692s 2024-09-16T04:42:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46cad9d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc136e50 0x7f46cc137230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46cb59e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f46cc0a4350 0x7f46cc13ecd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48476/0 (socket says 172.21.15.46:48476) 2024-09-16T04:42:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46cb59e640 1 -- 172.21.15.46:0/578526335 learned_addr learned my addr 172.21.15.46:0/578526335 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:42:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46cad9d640 10 monclient(hunting): get_auth_request con 0x7f46cc136e50 auth_method 0 2024-09-16T04:42:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46cad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46cad9d640 1 -- 172.21.15.46:0/578526335 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc136e50 msgr2=0x7f46cc137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:42:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46cad9d640 1 -- 172.21.15.46:0/578526335 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc136e50 msgr2=0x7f46cc137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:42:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.063+0000 7f46cad9d640 1 --2- 172.21.15.46:0/578526335 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc136e50 0x7f46cc137230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:53.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 1 --2- 172.21.15.46:0/578526335 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc136e50 0x7f46cc137230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:53.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 1 -- 172.21.15.46:0/578526335 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4150198607 0 0) 0x7f46cc0a4e90 con 0x7f46cc0a4350 2024-09-16T04:42:53.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:42:53.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:53.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:53.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient(hunting): my global_id is 44382 2024-09-16T04:42:53.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 1 -- 172.21.15.46:0/578526335 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46b4003520 con 0x7f46cc0a4350 2024-09-16T04:42:53.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 1 -- 172.21.15.46:0/578526335 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3680183894 0 0) 0x7f46cc0a0dc0 con 0x7f46cc0a3500 2024-09-16T04:42:53.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:42:53.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:53.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:53.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient(hunting): my global_id is 5659 2024-09-16T04:42:53.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 1 -- 172.21.15.46:0/578526335 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46cc0a4e90 con 0x7f46cc0a3500 2024-09-16T04:42:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 1 -- 172.21.15.46:0/578526335 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1353930608 0 0) 0x7f46b4003520 con 0x7f46cc0a4350 2024-09-16T04:42:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 1 -- 172.21.15.46:0/578526335 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc136e50 msgr2=0x7f46cc137230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:42:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 1 --2- 172.21.15.46:0/578526335 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc136e50 0x7f46cc137230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 1 -- 172.21.15.46:0/578526335 >> v1:172.21.15.114:6789/0 conn(0x7f46cc0a3500 legacy=0x7f46cc0a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:42:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:42:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:42:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 1 -- 172.21.15.46:0/578526335 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46cc0a10a0 con 0x7f46cc0a4350 2024-09-16T04:42:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient: _finish_auth 0 2024-09-16T04:42:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:42:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46d123c640 5 monclient: authenticate success, global_id 44382 2024-09-16T04:42:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 1 -- 172.21.15.46:0/578526335 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f46c40019f0 con 0x7f46cc0a4350 2024-09-16T04:42:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46d123c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:42:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient: dump: 2024-09-16T04:42:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:53.081 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:53.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:42:53.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:53.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 1 -- 172.21.15.46:0/578526335 >> v1:172.21.15.46:6789/0 conn(0x7f46cc0a4350 legacy=0x7f46cc13ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:53.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:42:53.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 1 --2- 172.21.15.46:0/578526335 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc136e50 0x7f46b4003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:53.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient(hunting): picked mon.c con 0x7f46cc136e50 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:53.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 1 --2- 172.21.15.46:0/578526335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46b40040d0 0x7f46b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:53.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient(hunting): picked mon.a con 0x7f46b40040d0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:53.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 1 --2- 172.21.15.46:0/578526335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46b4004d90 0x7f46b4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:53.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient(hunting): picked mon.b con 0x7f46b4004d90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:53.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:53.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 1 -- 172.21.15.46:0/578526335 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc0a5410 con 0x7f46b40040d0 2024-09-16T04:42:53.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 1 -- 172.21.15.46:0/578526335 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f46b400bc70 con 0x7f46b4004d90 2024-09-16T04:42:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 1 -- 172.21.15.46:0/578526335 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f46b400be70 con 0x7f46cc136e50 2024-09-16T04:42:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46c9d9b640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:42:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46d123c640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 1 --2- 172.21.15.46:0/578526335 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc136e50 0x7f46b4003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 10 monclient(hunting): get_auth_request con 0x7f46cc136e50 auth_method 0 2024-09-16T04:42:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46ca59c640 1 --2- 172.21.15.46:0/578526335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46b40040d0 0x7f46b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46ca59c640 10 monclient(hunting): get_auth_request con 0x7f46b40040d0 auth_method 0 2024-09-16T04:42:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46ca59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46ca59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:42:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cb59e640 1 --2- 172.21.15.46:0/578526335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46b4004d90 0x7f46b4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cb59e640 10 monclient(hunting): get_auth_request con 0x7f46b4004d90 auth_method 0 2024-09-16T04:42:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cb59e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cb59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:42:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 10 monclient(hunting): handle_auth_done global_id 44382 payload 293 2024-09-16T04:42:53.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 1 -- 172.21.15.46:0/578526335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46b4004d90 msgr2=0x7f46b4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:53.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 1 --2- 172.21.15.46:0/578526335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46b4004d90 0x7f46b4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:53.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 1 -- 172.21.15.46:0/578526335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46b40040d0 msgr2=0x7f46b4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:53.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 1 --2- 172.21.15.46:0/578526335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46b40040d0 0x7f46b4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:53.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 10 monclient: _finish_hunting 0 2024-09-16T04:42:53.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 1 monclient: found mon.c 2024-09-16T04:42:53.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:53.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:53.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 1 -- 172.21.15.46:0/578526335 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46b4003520 con 0x7f46cc136e50 2024-09-16T04:42:53.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 10 monclient: _finish_auth 0 2024-09-16T04:42:53.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 10 monclient: _check_auth_tickets 2024-09-16T04:42:53.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cad9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:53.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.064+0000 7f46cb59e640 1 --2- 172.21.15.46:0/578526335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46b4004d90 0x7f46b4005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:42:53.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.065+0000 7f46cad9d640 1 --2- 172.21.15.46:0/578526335 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc136e50 0x7f46b4003b70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f46bc00b9e0 tx=0x7f46b4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d24f0e033f29d12 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:53.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.065+0000 7f46c9d9b640 1 -- 172.21.15.46:0/578526335 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc030070 con 0x7f46cc136e50 2024-09-16T04:42:53.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.065+0000 7f46c9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:53.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.065+0000 7f46c9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:53.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.065+0000 7f46c9d9b640 10 monclient: dump: 2024-09-16T04:42:53.085 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:53.085 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:53.085 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:53.085 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:53.085 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:53.085 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:53.085 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:53.085 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:53.086 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:53.086 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:53.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.065+0000 7f46d123c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:53.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.065+0000 7f46c9d9b640 1 -- 172.21.15.46:0/578526335 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46bc02adb0 con 0x7f46cc136e50 2024-09-16T04:42:53.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.065+0000 7f46c9d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:53.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.065+0000 7f46d123c640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:53.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.065+0000 7f46c9d9b640 1 -- 172.21.15.46:0/578526335 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc00b5e0 con 0x7f46cc136e50 2024-09-16T04:42:53.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.065+0000 7f46d123c640 4 set_mon_vals no callback set 2024-09-16T04:42:53.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.065+0000 7f46c9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:53.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.065+0000 7f46c9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:53.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.065+0000 7f46c9d9b640 10 monclient: dump: 2024-09-16T04:42:53.086 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:53.086 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:53.086 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:53.087 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:53.087 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:53.087 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:53.087 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:53.087 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:53.087 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:53.087 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:53.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.066+0000 7f46d123c640 10 monclient: shutdown 2024-09-16T04:42:53.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.066+0000 7f46d123c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:53.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.066+0000 7f46d123c640 1 -- 172.21.15.46:0/578526335 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc136e50 msgr2=0x7f46b4003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:53.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.066+0000 7f46d123c640 1 --2- 172.21.15.46:0/578526335 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc136e50 0x7f46b4003b70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f46bc00b9e0 tx=0x7f46b4005830 comp rx=0 tx=0).stop 2024-09-16T04:42:53.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.066+0000 7f46cb59e640 1 -- 172.21.15.46:0/578526335 reap_dead start 2024-09-16T04:42:53.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.066+0000 7f46d123c640 1 -- 172.21.15.46:0/578526335 shutdown_connections 2024-09-16T04:42:53.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.066+0000 7f46d123c640 1 -- 172.21.15.46:0/578526335 >> 172.21.15.46:0/578526335 conn(0x7f46cc0a5f20 msgr2=0x7f46cc0a6330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:53.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.066+0000 7f46d123c640 1 -- 172.21.15.46:0/578526335 shutdown_connections 2024-09-16T04:42:53.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.066+0000 7f46d123c640 1 -- 172.21.15.46:0/578526335 wait complete. 2024-09-16T04:42:53.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.067+0000 7f46d123c640 10 monclient: build_initial_monmap 2024-09-16T04:42:53.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.067+0000 7f46d123c640 10 monclient: monmap: 2024-09-16T04:42:53.088 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:53.088 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:53.088 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:53.088 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:53.088 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:53.088 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:53.088 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:53.088 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:53.088 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:53.089 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:53.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.067+0000 7f46d123c640 1 Processor -- start 2024-09-16T04:42:53.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.067+0000 7f46d123c640 1 -- start start 2024-09-16T04:42:53.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.067+0000 7f46d123c640 10 monclient: init 2024-09-16T04:42:53.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.067+0000 7f46d123c640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:53.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.067+0000 7f46d123c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:42:53.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.067+0000 7f46d123c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46cc136e50 0x7f46cc1cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:53.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.067+0000 7f46d123c640 10 monclient(hunting): picked mon.noname-b con 0x7f46cc136e50 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:53.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.067+0000 7f46d123c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc1cc7f0 0x7f46cc123070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:53.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.067+0000 7f46d123c640 10 monclient(hunting): picked mon.noname-c con 0x7f46cc1cc7f0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:53.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46d123c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46cc1ccc30 0x7f46cc1235b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:53.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46d123c640 10 monclient(hunting): picked mon.noname-a con 0x7f46cc1ccc30 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:53.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46d123c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:53.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46d123c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc0a1ce0 con 0x7f46cc1ccc30 2024-09-16T04:42:53.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46d123c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:53.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46d123c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc0a1b60 con 0x7f46cc136e50 2024-09-16T04:42:53.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46d123c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:53.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46d123c640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f46cc0a47c0 con 0x7f46cc1cc7f0 2024-09-16T04:42:53.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46d123c640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:53.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46d123c640 10 monclient(hunting): authenticate will time out at 971.484044s 2024-09-16T04:42:53.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46ca59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc1cc7f0 0x7f46cc123070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:53.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46ca59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc1cc7f0 0x7f46cc123070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:49868/0 (socket says 172.21.15.46:49868) 2024-09-16T04:42:53.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46ca59c640 1 -- 172.21.15.46:0/2576837584 learned_addr learned my addr 172.21.15.46:0/2576837584 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:53.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46ca59c640 10 monclient(hunting): get_auth_request con 0x7f46cc1cc7f0 auth_method 0 2024-09-16T04:42:53.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:53.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46ca59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:53.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:53.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cb59e640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46cc1ccc30 0x7f46cc1235b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:53.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cb59e640 10 monclient(hunting): get_auth_request con 0x7f46cc1ccc30 auth_method 0 2024-09-16T04:42:53.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:53.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cb59e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:53.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:53.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46ca59c640 1 -- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc1cc7f0 msgr2=0x7f46cc123070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:42:53.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46ca59c640 1 -- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc1cc7f0 msgr2=0x7f46cc123070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:42:53.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46ca59c640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc1cc7f0 0x7f46cc123070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:53.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46ca59c640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc1cc7f0 0x7f46cc123070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:53.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:53.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:53.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cad9d640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46cc136e50 0x7f46cc1cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:53.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:53.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cad9d640 10 monclient(hunting): get_auth_request con 0x7f46cc136e50 auth_method 0 2024-09-16T04:42:53.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:53.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:53.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:53.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:53.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:53.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:53.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cb59e640 10 monclient(hunting): handle_auth_done global_id 44385 payload 824 2024-09-16T04:42:53.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cb59e640 1 -- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc1cc7f0 msgr2=0x7f46cc123070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:42:53.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cb59e640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc1cc7f0 0x7f46cc123070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:53.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cb59e640 1 -- 172.21.15.46:0/2576837584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46cc136e50 msgr2=0x7f46cc1cc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:53.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cb59e640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46cc136e50 0x7f46cc1cc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:53.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cb59e640 10 monclient: _finish_hunting 0 2024-09-16T04:42:53.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cb59e640 1 monclient: found mon.noname-a 2024-09-16T04:42:53.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:53.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cb59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:53.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cb59e640 1 -- 172.21.15.46:0/2576837584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46cc125160 con 0x7f46cc1ccc30 2024-09-16T04:42:53.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46cb59e640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46cc1ccc30 0x7f46cc1235b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f46cc0a4730 tx=0x7f46c4007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:53.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46b37fe640 1 -- 172.21.15.46:0/2576837584 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c400eb00 con 0x7f46cc1ccc30 2024-09-16T04:42:53.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:53.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46b37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:53.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46b37fe640 10 monclient: dump: 2024-09-16T04:42:53.093 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:53.093 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:53.093 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:53.093 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:53.093 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:53.093 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:53.093 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:53.093 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:53.094 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:53.094 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:53.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46b37fe640 10 monclient: _finish_auth 0 2024-09-16T04:42:53.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46b37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:42:53.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:53.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46b37fe640 1 -- 172.21.15.46:0/2576837584 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46c400ecc0 con 0x7f46cc1ccc30 2024-09-16T04:42:53.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46d123c640 5 monclient: authenticate success, global_id 44385 2024-09-16T04:42:53.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46d123c640 10 monclient: _renew_subs 2024-09-16T04:42:53.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46d123c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:53.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46d123c640 1 -- 172.21.15.46:0/2576837584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46cc126fb0 con 0x7f46cc1ccc30 2024-09-16T04:42:53.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46b37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:53.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46b37fe640 1 -- 172.21.15.46:0/2576837584 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c40146c0 con 0x7f46cc1ccc30 2024-09-16T04:42:53.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:53.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46b37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:53.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46b37fe640 10 monclient: dump: 2024-09-16T04:42:53.095 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:53.095 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:53.095 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:53.095 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:53.095 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:53.095 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:53.095 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:53.095 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:53.095 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:53.095 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:53.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46d123c640 10 monclient: _renew_subs 2024-09-16T04:42:53.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46d123c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:53.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46d123c640 1 -- 172.21.15.46:0/2576837584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46cc0a87d0 con 0x7f46cc1ccc30 2024-09-16T04:42:53.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.068+0000 7f46c959a640 4 set_mon_vals no callback set 2024-09-16T04:42:53.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.069+0000 7f46d123c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:53.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.069+0000 7f46d123c640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 0x7f46cc123af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:53.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.069+0000 7f46d123c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:53.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.069+0000 7f46d123c640 1 -- 172.21.15.46:0/2576837584 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f46cc135870 con 0x7f46cc126950 2024-09-16T04:42:53.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.069+0000 7f46b37fe640 1 -- 172.21.15.46:0/2576837584 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f46c401b060 con 0x7f46cc1ccc30 2024-09-16T04:42:53.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.069+0000 7f46cad9d640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 0x7f46cc123af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:53.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.069+0000 7f46cad9d640 10 monclient: get_auth_request con 0x7f46cc126950 auth_method 0 2024-09-16T04:42:53.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.069+0000 7f46cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:53.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.069+0000 7f46cad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:53.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.069+0000 7f46cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:53.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.070+0000 7f46cad9d640 1 -- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 msgr2=0x7f46cc123af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:42:53.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.070+0000 7f46cad9d640 1 -- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 msgr2=0x7f46cc123af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:42:53.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.070+0000 7f46cad9d640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 0x7f46cc123af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:53.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.070+0000 7f46b37fe640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f46b80383b0 0x7f46b803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:53.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.070+0000 7f46cad9d640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 0x7f46cc123af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:53.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.070+0000 7f46b37fe640 1 -- 172.21.15.46:0/2576837584 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 4346+0+0 (secure 0 0 0) 0x7f46c404b070 con 0x7f46cc1ccc30 2024-09-16T04:42:53.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.070+0000 7f46ca59c640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f46b80383b0 0x7f46b803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:53.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.070+0000 7f46ca59c640 10 monclient: get_auth_request con 0x7f46b80383b0 auth_method 0 2024-09-16T04:42:53.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.070+0000 7f46ca59c640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f46b80383b0 0x7f46b803a870 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f46c00010e0 tx=0x7f46c0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:53.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.271+0000 7f46cad9d640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 0x7f46cc123af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:53.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.271+0000 7f46cad9d640 10 monclient: get_auth_request con 0x7f46cc126950 auth_method 0 2024-09-16T04:42:53.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.271+0000 7f46cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:53.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.271+0000 7f46cad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:53.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.271+0000 7f46cad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:53.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.271+0000 7f46cad9d640 1 -- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 msgr2=0x7f46cc123af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:42:53.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.271+0000 7f46cad9d640 1 -- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 msgr2=0x7f46cc123af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:42:53.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.271+0000 7f46cad9d640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 0x7f46cc123af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:53.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.271+0000 7f46cad9d640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 0x7f46cc123af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:42:53.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.672+0000 7f46cad9d640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 0x7f46cc123af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:53.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.672+0000 7f46cad9d640 10 monclient: get_auth_request con 0x7f46cc126950 auth_method 0 2024-09-16T04:42:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.672+0000 7f46cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.672+0000 7f46cad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.672+0000 7f46cad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.672+0000 7f46cad9d640 1 -- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 msgr2=0x7f46cc123af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:42:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.672+0000 7f46cad9d640 1 -- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 msgr2=0x7f46cc123af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:42:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.672+0000 7f46cad9d640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 0x7f46cc123af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:42:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:53.672+0000 7f46cad9d640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 0x7f46cc123af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:42:54.373 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:54.373 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37772]: starting. 2024-09-16T04:42:54.373 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37772]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-37737 2024-09-16T04:42:54.373 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37772]: created object 0... 2024-09-16T04:42:54.373 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37772]: created object 25... 2024-09-16T04:42:54.374 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37772]: created object 49... 2024-09-16T04:42:54.374 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37772]: finishing. 2024-09-16T04:42:54.374 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37772]: shutting down. 2024-09-16T04:42:54.385 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:54.385 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37774]: starting. 2024-09-16T04:42:54.385 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37774]: listing objects. 2024-09-16T04:42:54.386 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37774]: listed object 0... 2024-09-16T04:42:54.386 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37774]: listed object 25... 2024-09-16T04:42:54.386 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37774]: saw 50 objects 2024-09-16T04:42:54.386 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37774]: shutting down. 2024-09-16T04:42:54.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.473+0000 7f46cad9d640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 0x7f46cc123af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.473+0000 7f46cad9d640 10 monclient: get_auth_request con 0x7f46cc126950 auth_method 0 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.473+0000 7f46cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.473+0000 7f46cad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.473+0000 7f46cad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.473+0000 7f46cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.473+0000 7f46cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.473+0000 7f46cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.473+0000 7f46cad9d640 10 monclient(hunting): handle_auth_done global_id 54101 payload 824 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.473+0000 7f46cad9d640 1 -- 172.21.15.46:0/2576837584 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f46cc0a4ba0 con 0x7f46cc126950 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.474+0000 7f46cad9d640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 0x7f46cc123af0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f46bc006fd0 tx=0x7f46bc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.474+0000 7f46b37fe640 1 -- 172.21.15.46:0/2576837584 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc04b070 con 0x7f46cc126950 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.474+0000 7f46b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.474+0000 7f46b37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.474+0000 7f46b37fe640 10 monclient: dump: 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.474+0000 7f46b37fe640 1 -- 172.21.15.46:0/2576837584 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f46bc034b60 con 0x7f46cc126950 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.474+0000 7f46b37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.474+0000 7f46b37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:54.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.474+0000 7f46b37fe640 1 -- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 msgr2=0x7f46cc123af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:54.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.474+0000 7f46b37fe640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 0x7f46cc123af0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f46bc006fd0 tx=0x7f46bc004730 comp rx=0 tx=0).stop 2024-09-16T04:42:54.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.474+0000 7f46b37fe640 1 -- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 msgr2=0x7f46cc123af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:54.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.474+0000 7f46b37fe640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc126950 0x7f46cc123af0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:54.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.481+0000 7f46d123c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:42:54.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.481+0000 7f46d123c640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc124030 0x7f46cc11f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:54.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.481+0000 7f46d123c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.481+0000 7f46d123c640 1 -- 172.21.15.46:0/2576837584 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f46cc124410 con 0x7f46cc124030 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.481+0000 7f46cad9d640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc124030 0x7f46cc11f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.481+0000 7f46cad9d640 10 monclient: get_auth_request con 0x7f46cc124030 auth_method 0 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.481+0000 7f46cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.481+0000 7f46cad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.481+0000 7f46cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.481+0000 7f46cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.481+0000 7f46cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.481+0000 7f46cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.481+0000 7f46cad9d640 10 monclient(hunting): handle_auth_done global_id 54104 payload 824 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.481+0000 7f46cad9d640 1 -- 172.21.15.46:0/2576837584 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f46cc124cc0 con 0x7f46cc124030 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.481+0000 7f46cad9d640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc124030 0x7f46cc11f890 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f46bc006b10 tx=0x7f46bc02e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.481+0000 7f46b37fe640 1 -- 172.21.15.46:0/2576837584 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc053070 con 0x7f46cc124030 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.481+0000 7f46b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.481+0000 7f46b37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.481+0000 7f46b37fe640 10 monclient: dump: 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.482+0000 7f46b37fe640 1 -- 172.21.15.46:0/2576837584 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f46bc057450 con 0x7f46cc124030 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.482+0000 7f46b37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.482+0000 7f46b37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.482+0000 7f46b37fe640 1 -- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc124030 msgr2=0x7f46cc11f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.482+0000 7f46b37fe640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc124030 0x7f46cc11f890 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f46bc006b10 tx=0x7f46bc02e000 comp rx=0 tx=0).stop 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.482+0000 7f46b37fe640 1 -- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc124030 msgr2=0x7f46cc11f890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.482+0000 7f46b37fe640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f46cc124030 0x7f46cc11f890 secure :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f46bc006b10 tx=0x7f46bc02e000 comp rx=0 tx=0).stop 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 48, 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 10301, 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 4, 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:54.495 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.495 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:54.501 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:54.501 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:42:54.501 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:54.501 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:54.501 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.501 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:42:54.501 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:42:54.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.482+0000 7f46d123c640 1 -- 172.21.15.46:0/2576837584 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f46b80383b0 msgr2=0x7f46b803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.482+0000 7f46d123c640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f46b80383b0 0x7f46b803a870 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f46c00010e0 tx=0x7f46c0009040 comp rx=0 tx=0).stop 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.482+0000 7f46d123c640 10 monclient: shutdown 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.482+0000 7f46d123c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.482+0000 7f46d123c640 1 -- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46cc1ccc30 msgr2=0x7f46cc1235b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.482+0000 7f46d123c640 1 --2- 172.21.15.46:0/2576837584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46cc1ccc30 0x7f46cc1235b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f46cc0a4730 tx=0x7f46c4007bc0 comp rx=0 tx=0).stop 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.482+0000 7f46cb59e640 1 -- 172.21.15.46:0/2576837584 reap_dead start 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.482+0000 7f46cb59e640 1 -- 172.21.15.46:0/2576837584 reap_dead start 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.482+0000 7f46d123c640 1 -- 172.21.15.46:0/2576837584 shutdown_connections 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.482+0000 7f46d123c640 1 -- 172.21.15.46:0/2576837584 >> 172.21.15.46:0/2576837584 conn(0x7f46cc0a5f20 msgr2=0x7f46cc0a7140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.482+0000 7f46d123c640 1 -- 172.21.15.46:0/2576837584 shutdown_connections 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.482+0000 7f46d123c640 1 -- 172.21.15.46:0/2576837584 wait complete. 2024-09-16T04:42:54.504 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:42:54.504 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:42:54.504 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:42:54.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.607+0000 7f49d9fbd640 10 monclient: get_monmap_and_config 2024-09-16T04:42:54.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.607+0000 7f49d9fbd640 10 monclient: build_initial_monmap 2024-09-16T04:42:54.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.607+0000 7f49d9fbd640 10 monclient: monmap: 2024-09-16T04:42:54.622 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:54.622 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:54.622 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:54.609130+0000 2024-09-16T04:42:54.622 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:54.609130+0000 2024-09-16T04:42:54.622 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:54.622 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:54.622 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:54.623 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:54.623 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:54.623 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:54.623 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:54.623 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:54.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.608+0000 7f49d9fbd640 1 Processor -- start 2024-09-16T04:42:54.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.608+0000 7f49d9fbd640 1 -- start start 2024-09-16T04:42:54.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.608+0000 7f49d9fbd640 10 monclient: init 2024-09-16T04:42:54.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.608+0000 7f49d9fbd640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:54.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.608+0000 7f49d9fbd640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-16T04:42:54.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d9fbd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d406e160 0x7f49d40a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:54.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d9fbd640 10 monclient(hunting): picked mon.noname-b con 0x7f49d406e160 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:54.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d9fbd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f49d40a2ee0 0x7f49d4136940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d9fbd640 10 monclient(hunting): picked mon.noname-a con 0x7f49d40a2ee0 addr v2:172.21.15.46:3300/0 2024-09-16T04:42:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d9fbd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f49d4136e80 addr v1:172.21.15.114:6789/0 2024-09-16T04:42:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d9fbd640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49d40a3fd0 con 0x7f49d4136e80 2024-09-16T04:42:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d9fbd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d9fbd640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f49d40a3300 con 0x7f49d40a2ee0 2024-09-16T04:42:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d9fbd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d9fbd640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f49d406e540 con 0x7f49d406e160 2024-09-16T04:42:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d9fbd640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d9fbd640 10 monclient(hunting): authenticate will time out at 973.025149s 2024-09-16T04:42:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d37fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d406e160 0x7f49d40a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f49d40a2ee0 0x7f49d4136940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d37fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d406e160 0x7f49d40a29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:54222/0 (socket says 172.21.15.46:54222) 2024-09-16T04:42:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d37fe640 1 -- 172.21.15.46:0/1659379503 learned_addr learned my addr 172.21.15.46:0/1659379503 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d37fe640 10 monclient(hunting): get_auth_request con 0x7f49d406e160 auth_method 0 2024-09-16T04:42:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d2ffd640 10 monclient(hunting): get_auth_request con 0x7f49d40a2ee0 auth_method 0 2024-09-16T04:42:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d37fe640 10 monclient(hunting): handle_auth_done global_id 54107 payload 274 2024-09-16T04:42:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d37fe640 1 -- 172.21.15.46:0/1659379503 >> v2:172.21.15.46:3300/0 conn(0x7f49d40a2ee0 msgr2=0x7f49d4136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d37fe640 1 --2- 172.21.15.46:0/1659379503 >> v2:172.21.15.46:3300/0 conn(0x7f49d40a2ee0 0x7f49d4136940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d37fe640 1 -- 172.21.15.46:0/1659379503 >> v1:172.21.15.114:6789/0 conn(0x7f49d4136e80 legacy=0x7f49d413dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:42:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d37fe640 1 monclient: found mon.noname-b 2024-09-16T04:42:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:42:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d37fe640 1 -- 172.21.15.46:0/1659379503 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d40a0bd0 con 0x7f49d406e160 2024-09-16T04:42:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.609+0000 7f49d2ffd640 1 --2- 172.21.15.46:0/1659379503 >> v2:172.21.15.46:3300/0 conn(0x7f49d40a2ee0 0x7f49d4136940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:42:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d37fe640 1 --2- 172.21.15.46:0/1659379503 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d406e160 0x7f49d40a29a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f49c4009870 tx=0x7f49c402a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3069db1aa9b89a1e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d27fc640 1 -- 172.21.15.46:0/1659379503 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c4037070 con 0x7f49d406e160 2024-09-16T04:42:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:42:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d27fc640 10 monclient: dump: 2024-09-16T04:42:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:54.628 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d27fc640 10 monclient: _finish_auth 0 2024-09-16T04:42:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:42:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d27fc640 1 -- 172.21.15.46:0/1659379503 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49c4032440 con 0x7f49d406e160 2024-09-16T04:42:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d9fbd640 5 monclient: authenticate success, global_id 54107 2024-09-16T04:42:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d9fbd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d9fbd640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d27fc640 1 -- 172.21.15.46:0/1659379503 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c4032810 con 0x7f49d406e160 2024-09-16T04:42:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d9fbd640 4 set_mon_vals no callback set 2024-09-16T04:42:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d27fc640 10 monclient: dump: 2024-09-16T04:42:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:54.630 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:54.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d9fbd640 10 monclient: shutdown 2024-09-16T04:42:54.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d9fbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:54.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d9fbd640 1 -- 172.21.15.46:0/1659379503 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d406e160 msgr2=0x7f49d40a29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:54.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.610+0000 7f49d9fbd640 1 --2- 172.21.15.46:0/1659379503 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d406e160 0x7f49d40a29a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f49c4009870 tx=0x7f49c402a130 comp rx=0 tx=0).stop 2024-09-16T04:42:54.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.611+0000 7f49d9fbd640 1 -- 172.21.15.46:0/1659379503 shutdown_connections 2024-09-16T04:42:54.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.611+0000 7f49d9fbd640 1 --2- 172.21.15.46:0/1659379503 >> v2:172.21.15.46:3300/0 conn(0x7f49d40a2ee0 0x7f49d4136940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:54.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.611+0000 7f49d9fbd640 1 --2- 172.21.15.46:0/1659379503 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d406e160 0x7f49d40a29a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:54.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.611+0000 7f49d9fbd640 1 -- 172.21.15.46:0/1659379503 >> 172.21.15.46:0/1659379503 conn(0x7f49d40a5c10 msgr2=0x7f49d40a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:54.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.611+0000 7f49d9fbd640 1 -- 172.21.15.46:0/1659379503 shutdown_connections 2024-09-16T04:42:54.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.611+0000 7f49d9fbd640 1 -- 172.21.15.46:0/1659379503 wait complete. 2024-09-16T04:42:54.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.611+0000 7f49d9fbd640 10 monclient: build_initial_monmap 2024-09-16T04:42:54.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.611+0000 7f49d9fbd640 10 monclient: monmap: 2024-09-16T04:42:54.631 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:54.631 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:54.631 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:54.631 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:54.631 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:54.631 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:54.631 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:54.631 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:54.631 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:54.631 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:54.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d9fbd640 1 Processor -- start 2024-09-16T04:42:54.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d9fbd640 1 -- start start 2024-09-16T04:42:54.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d9fbd640 10 monclient: init 2024-09-16T04:42:54.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d9fbd640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:54.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d9fbd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:42:54.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d9fbd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d406e160 0x7f49d41cc310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:54.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d9fbd640 10 monclient(hunting): picked mon.noname-a con 0x7f49d406e160 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:54.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d9fbd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d40a2ee0 0x7f49d41cc850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:54.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d9fbd640 10 monclient(hunting): picked mon.noname-c con 0x7f49d40a2ee0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:54.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d9fbd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d4136e80 0x7f49d4123d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:54.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d9fbd640 10 monclient(hunting): picked mon.noname-b con 0x7f49d4136e80 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:54.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d9fbd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:54.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d9fbd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f49d40a34a0 con 0x7f49d406e160 2024-09-16T04:42:54.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d9fbd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:54.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d9fbd640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f49d40a5130 con 0x7f49d4136e80 2024-09-16T04:42:54.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d9fbd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:54.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d9fbd640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f49d41267b0 con 0x7f49d40a2ee0 2024-09-16T04:42:54.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d9fbd640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:54.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d9fbd640 10 monclient(hunting): authenticate will time out at 973.028787s 2024-09-16T04:42:54.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d406e160 0x7f49d41cc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:54.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d2ffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d40a2ee0 0x7f49d41cc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:54.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d406e160 0x7f49d41cc310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51496/0 (socket says 172.21.15.46:51496) 2024-09-16T04:42:54.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d37fe640 1 -- 172.21.15.46:0/3166501517 learned_addr learned my addr 172.21.15.46:0/3166501517 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:54.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d37fe640 10 monclient(hunting): get_auth_request con 0x7f49d406e160 auth_method 0 2024-09-16T04:42:54.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:54.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:54.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:54.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d2ffd640 10 monclient(hunting): get_auth_request con 0x7f49d40a2ee0 auth_method 0 2024-09-16T04:42:54.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:54.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:54.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:54.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:54.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:54.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:54.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:54.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:54.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.612+0000 7f49d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:54.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d3fff640 1 --2- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d4136e80 0x7f49d4123d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:54.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d3fff640 10 monclient(hunting): get_auth_request con 0x7f49d4136e80 auth_method 0 2024-09-16T04:42:54.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:54.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:54.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:54.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d37fe640 10 monclient(hunting): handle_auth_done global_id 44391 payload 824 2024-09-16T04:42:54.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d37fe640 1 -- 172.21.15.46:0/3166501517 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d40a2ee0 msgr2=0x7f49d41cc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:54.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d37fe640 1 --2- 172.21.15.46:0/3166501517 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d40a2ee0 0x7f49d41cc850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:54.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d37fe640 1 -- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d4136e80 msgr2=0x7f49d4123d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:54.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d37fe640 1 --2- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d4136e80 0x7f49d4123d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:54.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:42:54.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d37fe640 1 monclient: found mon.noname-a 2024-09-16T04:42:54.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:54.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:54.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d37fe640 1 -- 172.21.15.46:0/3166501517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d4125ed0 con 0x7f49d406e160 2024-09-16T04:42:54.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d2ffd640 1 --2- 172.21.15.46:0/3166501517 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f49d40a2ee0 0x7f49d41cc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:42:54.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d37fe640 1 --2- 172.21.15.46:0/3166501517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d406e160 0x7f49d41cc310 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f49c4000c00 tx=0x7f49c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:54.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49bbfff640 1 -- 172.21.15.46:0/3166501517 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c4041070 con 0x7f49d406e160 2024-09-16T04:42:54.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:54.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49bbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:54.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49bbfff640 10 monclient: dump: 2024-09-16T04:42:54.636 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:54.636 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:54.636 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.636 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.636 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:54.636 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:54.636 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:54.636 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:54.637 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:54.637 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:54.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49bbfff640 10 monclient: _finish_auth 0 2024-09-16T04:42:54.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49bbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:42:54.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:54.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d9fbd640 5 monclient: authenticate success, global_id 44391 2024-09-16T04:42:54.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49bbfff640 1 -- 172.21.15.46:0/3166501517 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49c403c870 con 0x7f49d406e160 2024-09-16T04:42:54.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49bbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:54.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d9fbd640 10 monclient: _renew_subs 2024-09-16T04:42:54.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49bbfff640 1 -- 172.21.15.46:0/3166501517 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c4037040 con 0x7f49d406e160 2024-09-16T04:42:54.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d9fbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:54.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d9fbd640 1 -- 172.21.15.46:0/3166501517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49d4127d20 con 0x7f49d406e160 2024-09-16T04:42:54.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:54.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:54.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49bbfff640 10 monclient: dump: 2024-09-16T04:42:54.638 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:54.638 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:54.638 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.638 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.638 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:54.638 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:54.638 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:54.638 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:54.638 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:54.639 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:54.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d1ffb640 4 set_mon_vals no callback set 2024-09-16T04:42:54.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d9fbd640 10 monclient: _renew_subs 2024-09-16T04:42:54.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d9fbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:54.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.613+0000 7f49d9fbd640 1 -- 172.21.15.46:0/3166501517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49d40a89b0 con 0x7f49d406e160 2024-09-16T04:42:54.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.614+0000 7f49d9fbd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:54.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.614+0000 7f49bbfff640 1 -- 172.21.15.46:0/3166501517 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f49c4044070 con 0x7f49d406e160 2024-09-16T04:42:54.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.614+0000 7f49d9fbd640 1 --2- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d41276c0 0x7f49d41242d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:54.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.614+0000 7f49d9fbd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:54.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.614+0000 7f49d9fbd640 1 -- 172.21.15.46:0/3166501517 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f49d40a3620 con 0x7f49d41276c0 2024-09-16T04:42:54.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.614+0000 7f49bbfff640 1 --2- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f49c00383b0 0x7f49c003a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:54.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.614+0000 7f49d2ffd640 1 --2- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d41276c0 0x7f49d41242d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:54.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.614+0000 7f49d2ffd640 10 monclient: get_auth_request con 0x7f49d41276c0 auth_method 0 2024-09-16T04:42:54.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.614+0000 7f49d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:54.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.614+0000 7f49d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:54.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.614+0000 7f49d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:54.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.614+0000 7f49bbfff640 1 -- 172.21.15.46:0/3166501517 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 4738+0+0 (secure 0 0 0) 0x7f49c406c160 con 0x7f49d406e160 2024-09-16T04:42:54.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.614+0000 7f49d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:54.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.614+0000 7f49d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:54.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.614+0000 7f49d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:54.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.614+0000 7f49d3fff640 1 --2- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f49c00383b0 0x7f49c003a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:54.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.614+0000 7f49d3fff640 10 monclient: get_auth_request con 0x7f49c00383b0 auth_method 0 2024-09-16T04:42:54.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.614+0000 7f49d2ffd640 10 monclient(hunting): handle_auth_done global_id 5668 payload 824 2024-09-16T04:42:54.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.614+0000 7f49d2ffd640 1 -- 172.21.15.46:0/3166501517 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f49d40a29a0 con 0x7f49d41276c0 2024-09-16T04:42:54.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.615+0000 7f49d2ffd640 1 --2- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d41276c0 0x7f49d41242d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f49d4125630 tx=0x7f49c800f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:54.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.615+0000 7f49d3fff640 1 --2- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f49c00383b0 0x7f49c003a870 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f49cc007fd0 tx=0x7f49cc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:54.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.615+0000 7f49bbfff640 1 -- 172.21.15.46:0/3166501517 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c8014440 con 0x7f49d41276c0 2024-09-16T04:42:54.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.615+0000 7f49bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:54.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.615+0000 7f49bbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:42:54.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.615+0000 7f49bbfff640 10 monclient: dump: 2024-09-16T04:42:54.642 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:54.642 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:54.642 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.642 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.642 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:54.642 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:54.642 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:54.643 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:54.643 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:54.643 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:54.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.615+0000 7f49bbfff640 1 -- 172.21.15.46:0/3166501517 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f49c8014440 con 0x7f49d41276c0 2024-09-16T04:42:54.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.615+0000 7f49bbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:54.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.615+0000 7f49bbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:54.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.615+0000 7f49bbfff640 1 -- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d41276c0 msgr2=0x7f49d41242d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:54.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.615+0000 7f49bbfff640 1 --2- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d41276c0 0x7f49d41242d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f49d4125630 tx=0x7f49c800f610 comp rx=0 tx=0).stop 2024-09-16T04:42:54.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.615+0000 7f49bbfff640 1 -- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d41276c0 msgr2=0x7f49d41242d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:54.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.615+0000 7f49bbfff640 1 --2- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d41276c0 0x7f49d41242d0 secure :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f49d4125630 tx=0x7f49c800f610 comp rx=0 tx=0).stop 2024-09-16T04:42:54.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.621+0000 7f49d9fbd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:42:54.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.621+0000 7f49d9fbd640 1 --2- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d4124810 0x7f49d4124bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:54.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.621+0000 7f49d9fbd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:54.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.621+0000 7f49d9fbd640 1 -- 172.21.15.46:0/3166501517 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f49d4125130 con 0x7f49d4124810 2024-09-16T04:42:54.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49d2ffd640 1 --2- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d4124810 0x7f49d4124bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:54.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49d2ffd640 10 monclient: get_auth_request con 0x7f49d4124810 auth_method 0 2024-09-16T04:42:54.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:54.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:54.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:54.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:54.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:54.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:54.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49d2ffd640 10 monclient(hunting): handle_auth_done global_id 5671 payload 824 2024-09-16T04:42:54.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49d2ffd640 1 -- 172.21.15.46:0/3166501517 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f49d4127aa0 con 0x7f49d4124810 2024-09-16T04:42:54.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49d2ffd640 1 --2- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d4124810 0x7f49d4124bf0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f49d4125630 tx=0x7f49c80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:54.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49bbfff640 1 -- 172.21.15.46:0/3166501517 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c8022980 con 0x7f49d4124810 2024-09-16T04:42:54.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:54.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49bbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:42:54.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49bbfff640 10 monclient: dump: 2024-09-16T04:42:54.647 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:54.647 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:54.647 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.647 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.647 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:54.647 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:54.647 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:54.647 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:54.647 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:54.647 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:54.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49bbfff640 1 -- 172.21.15.46:0/3166501517 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f49c8022980 con 0x7f49d4124810 2024-09-16T04:42:54.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49bbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:42:54.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49bbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:54.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49bbfff640 1 -- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d4124810 msgr2=0x7f49d4124bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:54.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49bbfff640 1 --2- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d4124810 0x7f49d4124bf0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f49d4125630 tx=0x7f49c80046c0 comp rx=0 tx=0).stop 2024-09-16T04:42:54.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49bbfff640 1 -- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d4124810 msgr2=0x7f49d4124bf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:54.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.622+0000 7f49bbfff640 1 --2- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49d4124810 0x7f49d4124bf0 secure :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f49d4125630 tx=0x7f49c80046c0 comp rx=0 tx=0).stop 2024-09-16T04:42:54.648 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:42:54.648 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:42:54.648 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:42:54.648 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:42:54.649 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 48, 2024-09-16T04:42:54.649 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 260525, 2024-09-16T04:42:54.649 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:42:54.649 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:42:54.649 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:42:54.649 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:42:54.649 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:54.649 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 4, 2024-09-16T04:42:54.649 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:54.649 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:42:54.649 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:42:54.649 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:54.649 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:54.650 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:54.650 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:54.650 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:54.650 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:54.650 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:54.650 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:54.650 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:54.650 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:54.650 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:54.650 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:54.650 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:42:54.650 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:42:54.650 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:54.650 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:54.650 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:54.651 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:54.651 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:54.651 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:54.651 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:54.651 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:54.651 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:54.651 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:54.651 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:54.651 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:54.651 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.651 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:42:54.651 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:42:54.651 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:42:54.651 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:42:54.652 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:42:54.652 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:42:54.652 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:54.652 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:54.652 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:42:54.652 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:42:54.652 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:42:54.652 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:42:54.652 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:42:54.652 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:42:54.652 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:42:54.652 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:54.652 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:42:54.652 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:54.653 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:54.653 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:54.653 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:54.653 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:54.653 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:54.653 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:54.653 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:54.653 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:54.653 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:54.653 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:54.653 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:54.653 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:42:54.653 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.653 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:42:54.654 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.654 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:42:54.654 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:42:54.654 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:54.654 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:54.654 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.654 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:54.654 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:42:54.654 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:54.654 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.654 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.654 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:54.654 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:42:54.654 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:54.655 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:54.655 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:54.655 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.655 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:42:54.655 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:42:54.655 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:54.655 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:54.655 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:54.655 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.655 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.655 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:42:54.655 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:42:54.655 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:54.655 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:54.655 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.656 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:54.656 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:42:54.656 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:54.656 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.656 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.656 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:54.656 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:42:54.656 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:54.656 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:54.656 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:54.656 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.656 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:42:54.656 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:42:54.656 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:54.657 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:54.657 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:54.657 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.657 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.657 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:42:54.657 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:42:54.657 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:54.657 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:54.657 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.657 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:54.657 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:42:54.657 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:54.657 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.657 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.658 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:54.658 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:42:54.658 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:54.658 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:54.658 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:54.658 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.658 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:42:54.658 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:42:54.658 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:54.658 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:54.658 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:54.658 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:54.658 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:54.658 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.659 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:42:54.659 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:42:54.659 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.659 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:54.659 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:54.659 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:42:54.659 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:54.659 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:54.659 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:42:54.659 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.659 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:54.659 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:54.659 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4 2024-09-16T04:42:54.659 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:54.659 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:54.660 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:42:54.660 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.660 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:54.660 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:54.660 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6 2024-09-16T04:42:54.660 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:54.660 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:54.660 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:42:54.660 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:54.660 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:54.660 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:54.660 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:42:54.660 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:54.660 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:54.661 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:54.661 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:42:54.661 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:42:54.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.623+0000 7f49d9fbd640 1 -- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f49c00383b0 msgr2=0x7f49c003a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:54.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.623+0000 7f49d9fbd640 1 --2- 172.21.15.46:0/3166501517 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f49c00383b0 0x7f49c003a870 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f49cc007fd0 tx=0x7f49cc009040 comp rx=0 tx=0).stop 2024-09-16T04:42:54.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.623+0000 7f49d9fbd640 10 monclient: shutdown 2024-09-16T04:42:54.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.623+0000 7f49d9fbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:54.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.623+0000 7f49d9fbd640 1 -- 172.21.15.46:0/3166501517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d406e160 msgr2=0x7f49d41cc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:54.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.623+0000 7f49d9fbd640 1 --2- 172.21.15.46:0/3166501517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d406e160 0x7f49d41cc310 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f49c4000c00 tx=0x7f49c40046c0 comp rx=0 tx=0).stop 2024-09-16T04:42:54.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.623+0000 7f49d3fff640 1 -- 172.21.15.46:0/3166501517 reap_dead start 2024-09-16T04:42:54.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.623+0000 7f49d3fff640 1 -- 172.21.15.46:0/3166501517 reap_dead start 2024-09-16T04:42:54.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.623+0000 7f49d9fbd640 1 -- 172.21.15.46:0/3166501517 shutdown_connections 2024-09-16T04:42:54.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.623+0000 7f49d9fbd640 1 -- 172.21.15.46:0/3166501517 >> 172.21.15.46:0/3166501517 conn(0x7f49d40a5c10 msgr2=0x7f49d409d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:54.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.623+0000 7f49d9fbd640 1 -- 172.21.15.46:0/3166501517 shutdown_connections 2024-09-16T04:42:54.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.623+0000 7f49d9fbd640 1 -- 172.21.15.46:0/3166501517 wait complete. 2024-09-16T04:42:54.663 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:42:54.663 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:42:54.663 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:42:54.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.762+0000 7fc5ded9d640 10 monclient: get_monmap_and_config 2024-09-16T04:42:54.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.762+0000 7fc5ded9d640 10 monclient: build_initial_monmap 2024-09-16T04:42:54.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.762+0000 7fc5ded9d640 10 monclient: monmap: 2024-09-16T04:42:54.765 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:54.765 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:54.765 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:54.764074+0000 2024-09-16T04:42:54.765 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:54.764074+0000 2024-09-16T04:42:54.765 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:54.765 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:54.765 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:54.765 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:54.765 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:54.765 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:54.765 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:54.765 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:54.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.763+0000 7fc5ded9d640 1 Processor -- start 2024-09-16T04:42:54.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.763+0000 7fc5ded9d640 1 -- start start 2024-09-16T04:42:54.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.763+0000 7fc5ded9d640 10 monclient: init 2024-09-16T04:42:54.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.763+0000 7fc5ded9d640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:54.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.763+0000 7fc5ded9d640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-16T04:42:54.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.763+0000 7fc5ded9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc5d80a3ba0 addr v1:172.21.15.114:6789/0 2024-09-16T04:42:54.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.763+0000 7fc5ded9d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc5d80a3f80 0x7fc5d80a4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:54.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.763+0000 7fc5ded9d640 10 monclient(hunting): picked mon.noname-b con 0x7fc5d80a3f80 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:54.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.763+0000 7fc5ded9d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc5d813bdb0 0x7fc5d813e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:54.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.763+0000 7fc5ded9d640 10 monclient(hunting): picked mon.noname-c con 0x7fc5d813bdb0 addr v2:172.21.15.114:3300/0 2024-09-16T04:42:54.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.763+0000 7fc5ded9d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5d809aa70 con 0x7fc5d80a3ba0 2024-09-16T04:42:54.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.763+0000 7fc5ded9d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:54.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.763+0000 7fc5ded9d640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fc5d809af50 con 0x7fc5d813bdb0 2024-09-16T04:42:54.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.763+0000 7fc5ded9d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:54.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.763+0000 7fc5ded9d640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fc5d809b510 con 0x7fc5d80a3f80 2024-09-16T04:42:54.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.763+0000 7fc5ded9d640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:54.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.763+0000 7fc5ded9d640 10 monclient(hunting): authenticate will time out at 973.179911s 2024-09-16T04:42:54.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5dd59a640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc5d80a3f80 0x7fc5d80a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:54.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5dd59a640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc5d80a3f80 0x7fc5d80a4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:54240/0 (socket says 172.21.15.46:54240) 2024-09-16T04:42:54.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5dd59a640 1 -- 172.21.15.46:0/2706339817 learned_addr learned my addr 172.21.15.46:0/2706339817 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:54.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5dd59a640 10 monclient(hunting): get_auth_request con 0x7fc5d80a3f80 auth_method 0 2024-09-16T04:42:54.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5dd59a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:54.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5dd59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:54.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:54.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5dd59a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:54.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5dd59a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:54.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5dd59a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:54.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5de59c640 1 --2- 172.21.15.46:0/2706339817 >> v2:172.21.15.114:3300/0 conn(0x7fc5d813bdb0 0x7fc5d813e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:54.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5de59c640 10 monclient(hunting): get_auth_request con 0x7fc5d813bdb0 auth_method 0 2024-09-16T04:42:54.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:54.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5de59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:54.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5de59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:54.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5dd59a640 10 monclient(hunting): handle_auth_done global_id 54113 payload 274 2024-09-16T04:42:54.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5dd59a640 1 -- 172.21.15.46:0/2706339817 >> v2:172.21.15.114:3300/0 conn(0x7fc5d813bdb0 msgr2=0x7fc5d813e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:54.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5dd59a640 1 --2- 172.21.15.46:0/2706339817 >> v2:172.21.15.114:3300/0 conn(0x7fc5d813bdb0 0x7fc5d813e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:54.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5dd59a640 1 -- 172.21.15.46:0/2706339817 >> v1:172.21.15.114:6789/0 conn(0x7fc5d80a3ba0 legacy=0x7fc5d809bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:54.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5dd59a640 10 monclient: _finish_hunting 0 2024-09-16T04:42:54.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5dd59a640 1 monclient: found mon.noname-b 2024-09-16T04:42:54.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5dd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:54.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5dd59a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:42:54.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.764+0000 7fc5dd59a640 1 -- 172.21.15.46:0/2706339817 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5d806e9a0 con 0x7fc5d80a3f80 2024-09-16T04:42:54.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5dd59a640 1 --2- 172.21.15.46:0/2706339817 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc5d80a3f80 0x7fc5d80a4360 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc5d809b320 tx=0x7fc5c802a140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9372c36ad92ea0b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:54.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5dcd99640 1 -- 172.21.15.46:0/2706339817 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8037070 con 0x7fc5d80a3f80 2024-09-16T04:42:54.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5dcd99640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:54.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5dcd99640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:42:54.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5dcd99640 10 monclient: dump: 2024-09-16T04:42:54.776 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:54.776 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:54.776 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.777 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.777 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:54.777 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:54.777 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:54.777 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:54.777 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:54.777 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:54.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5dcd99640 10 monclient: _finish_auth 0 2024-09-16T04:42:54.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5dcd99640 10 monclient: _check_auth_tickets 2024-09-16T04:42:54.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5dcd99640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:54.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5ded9d640 5 monclient: authenticate success, global_id 54113 2024-09-16T04:42:54.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5ded9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:54.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5dcd99640 1 -- 172.21.15.46:0/2706339817 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5c802ae50 con 0x7fc5d80a3f80 2024-09-16T04:42:54.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5dcd99640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:54.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5ded9d640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:54.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5ded9d640 4 set_mon_vals no callback set 2024-09-16T04:42:54.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5dcd99640 1 -- 172.21.15.46:0/2706339817 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8032660 con 0x7fc5d80a3f80 2024-09-16T04:42:54.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5dcd99640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:54.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5dcd99640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:54.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5dcd99640 10 monclient: dump: 2024-09-16T04:42:54.778 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:54.778 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:54.779 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.779 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.779 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:54.779 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:54.779 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:54.779 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:54.779 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:54.779 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:54.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5ded9d640 10 monclient: shutdown 2024-09-16T04:42:54.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5ded9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:54.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5ded9d640 1 -- 172.21.15.46:0/2706339817 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc5d80a3f80 msgr2=0x7fc5d80a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:54.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5ded9d640 1 --2- 172.21.15.46:0/2706339817 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc5d80a3f80 0x7fc5d80a4360 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc5d809b320 tx=0x7fc5c802a140 comp rx=0 tx=0).stop 2024-09-16T04:42:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5ded9d640 1 -- 172.21.15.46:0/2706339817 shutdown_connections 2024-09-16T04:42:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5ded9d640 1 --2- 172.21.15.46:0/2706339817 >> v2:172.21.15.114:3300/0 conn(0x7fc5d813bdb0 0x7fc5d813e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5ded9d640 1 --2- 172.21.15.46:0/2706339817 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc5d80a3f80 0x7fc5d80a4360 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5ded9d640 1 -- 172.21.15.46:0/2706339817 >> 172.21.15.46:0/2706339817 conn(0x7fc5d80a60c0 msgr2=0x7fc5d80a64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5ded9d640 1 -- 172.21.15.46:0/2706339817 shutdown_connections 2024-09-16T04:42:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.765+0000 7fc5ded9d640 1 -- 172.21.15.46:0/2706339817 wait complete. 2024-09-16T04:42:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.766+0000 7fc5ded9d640 10 monclient: build_initial_monmap 2024-09-16T04:42:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.766+0000 7fc5ded9d640 10 monclient: monmap: 2024-09-16T04:42:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:54.781 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:54.781 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:54.781 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:54.781 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:54.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.766+0000 7fc5ded9d640 1 Processor -- start 2024-09-16T04:42:54.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.766+0000 7fc5ded9d640 1 -- start start 2024-09-16T04:42:54.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.766+0000 7fc5ded9d640 10 monclient: init 2024-09-16T04:42:54.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ded9d640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:54.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ded9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:42:54.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ded9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5d80a3ba0 0x7fc5d8133c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:54.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ded9d640 10 monclient(hunting): picked mon.noname-b con 0x7fc5d80a3ba0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:54.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ded9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5d80a3f80 0x7fc5d81341c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:54.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ded9d640 10 monclient(hunting): picked mon.noname-a con 0x7fc5d80a3f80 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:54.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ded9d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc5d813bdb0 0x7fc5d81cb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:54.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ded9d640 10 monclient(hunting): picked mon.noname-c con 0x7fc5d813bdb0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:54.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ded9d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:54.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ded9d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d80a54d0 con 0x7fc5d80a3f80 2024-09-16T04:42:54.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ded9d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:54.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ded9d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d80a5350 con 0x7fc5d80a3ba0 2024-09-16T04:42:54.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ded9d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:54.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ded9d640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fc5d813e6e0 con 0x7fc5d813bdb0 2024-09-16T04:42:54.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ded9d640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:54.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ded9d640 10 monclient(hunting): authenticate will time out at 973.183211s 2024-09-16T04:42:54.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5de59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc5d813bdb0 0x7fc5d81cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:54.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5de59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc5d813bdb0 0x7fc5d81cb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:54244/0 (socket says 172.21.15.46:54244) 2024-09-16T04:42:54.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5de59c640 1 -- 172.21.15.46:0/3945956201 learned_addr learned my addr 172.21.15.46:0/3945956201 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:54.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5de59c640 10 monclient(hunting): get_auth_request con 0x7fc5d813bdb0 auth_method 0 2024-09-16T04:42:54.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5dd59a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5d80a3f80 0x7fc5d81341c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:54.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:54.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5de59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:54.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5de59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:54.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5dd59a640 10 monclient(hunting): get_auth_request con 0x7fc5d80a3f80 auth_method 0 2024-09-16T04:42:54.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5dd59a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:54.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5dd59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:54.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:54.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5de59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:54.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5de59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:54.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ddd9b640 1 --2- 172.21.15.46:0/3945956201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5d80a3ba0 0x7fc5d8133c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:54.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5de59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:54.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5dd59a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:54.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5dd59a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:54.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5dd59a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:54.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ddd9b640 10 monclient(hunting): get_auth_request con 0x7fc5d80a3ba0 auth_method 0 2024-09-16T04:42:54.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ddd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:54.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ddd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:54.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:54.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ddd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:54.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ddd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:54.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ddd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:54.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5de59c640 10 monclient(hunting): handle_auth_done global_id 54116 payload 824 2024-09-16T04:42:54.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5de59c640 1 -- 172.21.15.46:0/3945956201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5d80a3ba0 msgr2=0x7fc5d8133c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:54.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5de59c640 1 --2- 172.21.15.46:0/3945956201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5d80a3ba0 0x7fc5d8133c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:54.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5de59c640 1 -- 172.21.15.46:0/3945956201 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5d80a3f80 msgr2=0x7fc5d81341c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:54.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5de59c640 1 --2- 172.21.15.46:0/3945956201 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5d80a3f80 0x7fc5d81341c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:54.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5de59c640 10 monclient: _finish_hunting 0 2024-09-16T04:42:54.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5de59c640 1 monclient: found mon.noname-c 2024-09-16T04:42:54.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5de59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:54.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5de59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:42:54.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5de59c640 1 -- 172.21.15.46:0/3945956201 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5d813c340 con 0x7fc5d813bdb0 2024-09-16T04:42:54.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5dd59a640 1 --2- 172.21.15.46:0/3945956201 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5d80a3f80 0x7fc5d81341c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:42:54.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5de59c640 1 --2- 172.21.15.46:0/3945956201 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc5d813bdb0 0x7fc5d81cb990 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0007e40 tx=0x7fc5d000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:54.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ce7fc640 1 -- 172.21.15.46:0/3945956201 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d0014620 con 0x7fc5d813bdb0 2024-09-16T04:42:54.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:54.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:42:54.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ce7fc640 10 monclient: dump: 2024-09-16T04:42:54.785 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:54.785 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:54.785 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.786 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.786 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:54.786 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:54.786 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:54.786 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:54.786 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:54.786 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:54.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ce7fc640 10 monclient: _finish_auth 0 2024-09-16T04:42:54.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ce7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:42:54.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.767+0000 7fc5ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:54.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.768+0000 7fc5ce7fc640 1 -- 172.21.15.46:0/3945956201 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5d0019070 con 0x7fc5d813bdb0 2024-09-16T04:42:54.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.768+0000 7fc5ce7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:54.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.768+0000 7fc5ce7fc640 1 -- 172.21.15.46:0/3945956201 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d0014a20 con 0x7fc5d813bdb0 2024-09-16T04:42:54.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.768+0000 7fc5ded9d640 5 monclient: authenticate success, global_id 54116 2024-09-16T04:42:54.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.768+0000 7fc5ded9d640 10 monclient: _renew_subs 2024-09-16T04:42:54.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.768+0000 7fc5ded9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:54.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.768+0000 7fc5ded9d640 1 -- 172.21.15.46:0/3945956201 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5d80a2310 con 0x7fc5d813bdb0 2024-09-16T04:42:54.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.768+0000 7fc5ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:54.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.768+0000 7fc5ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:54.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.768+0000 7fc5ce7fc640 10 monclient: dump: 2024-09-16T04:42:54.787 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:54.787 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:54.787 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.787 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:54.787 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:54.787 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:54.787 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:54.788 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:54.788 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:54.788 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:54.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.768+0000 7fc5cffff640 4 set_mon_vals no callback set 2024-09-16T04:42:54.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.768+0000 7fc5ded9d640 10 monclient: _renew_subs 2024-09-16T04:42:54.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.768+0000 7fc5ded9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:54.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.768+0000 7fc5ded9d640 1 -- 172.21.15.46:0/3945956201 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5d809ff30 con 0x7fc5d813bdb0 2024-09-16T04:42:54.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.768+0000 7fc5ce7fc640 1 -- 172.21.15.46:0/3945956201 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fc5d0014bc0 con 0x7fc5d813bdb0 2024-09-16T04:42:54.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.768+0000 7fc5ded9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:54.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.768+0000 7fc5ded9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:54.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.768+0000 7fc5ded9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:54.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.768+0000 7fc5ded9d640 1 -- 172.21.15.46:0/3945956201 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5d80a4360 con 0x7fc5d813bdb0 2024-09-16T04:42:54.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.769+0000 7fc5ce7fc640 1 --2- 172.21.15.46:0/3945956201 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc5bc0383e0 0x7fc5bc03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:54.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.769+0000 7fc5ce7fc640 1 -- 172.21.15.46:0/3945956201 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(163..163 src has 1..163) ==== 4738+0+0 (secure 0 0 0) 0x7fc5d004c150 con 0x7fc5d813bdb0 2024-09-16T04:42:54.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.769+0000 7fc5ddd9b640 1 --2- 172.21.15.46:0/3945956201 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc5bc0383e0 0x7fc5bc03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:54.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.769+0000 7fc5ddd9b640 10 monclient: get_auth_request con 0x7fc5bc0383e0 auth_method 0 2024-09-16T04:42:54.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.769+0000 7fc5ddd9b640 1 --2- 172.21.15.46:0/3945956201 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc5bc0383e0 0x7fc5bc03a8a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4004f60 tx=0x7fc5d4007e30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:54.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.774+0000 7fc5ce7fc640 1 -- 172.21.15.46:0/3945956201 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc5d813c340 con 0x7fc5d813bdb0 2024-09-16T04:42:54.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.774+0000 7fc5ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:54.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:54.774+0000 7fc5ce7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:55.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:55.016+0000 7fc5ded9d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:42:55.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:55.016+0000 7fc5ded9d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:42:55.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:55.016+0000 7fc5ded9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:55.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:55.016+0000 7fc5ded9d640 1 -- 172.21.15.46:0/3945956201 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fc5d8067d60 con 0x7fc5d813bdb0 2024-09-16T04:42:55.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:55.018+0000 7fc5ce7fc640 1 -- 172.21.15.46:0/3945956201 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fc5d001c520 con 0x7fc5d813bdb0 2024-09-16T04:42:55.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:55.018+0000 7fc5ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:42:55.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:55.018+0000 7fc5ce7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:55.024 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.018+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.025 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.018+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=1,health=12,kv=11,logm=65} crc {auth=4208897817,config=555582061,health=2972789322,kv=1889325247,logm=1753838555}) 2024-09-16T04:42:55.025 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.018+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=12,kv=12,logm=63} crc {auth=910244236,config=3541143443,health=2972789322,kv=4017288519,logm=113471879}) 2024-09-16T04:42:55.025 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.018+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.025 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.018+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=1,health=12,kv=11,logm=65} crc {auth=4208897817,config=555582061,health=2972789322,kv=1889325247,logm=1753838555}) 2024-09-16T04:42:55.025 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.018+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=12,kv=11,logm=63} crc {auth=3481876681,config=3541143443,health=2972789322,kv=89513504,logm=3021508242}) 2024-09-16T04:42:55.025 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.019+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.025 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.019+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2347297209}) 2024-09-16T04:42:55.025 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.019+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3773287286}) 2024-09-16T04:42:55.026 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.019+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.026 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.019+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2347297209}) 2024-09-16T04:42:55.026 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.019+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=370622765}) 2024-09-16T04:42:55.026 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.019+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.026 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.019+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2503791280}) 2024-09-16T04:42:55.027 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.019+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2986890858}) 2024-09-16T04:42:55.027 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.019+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.027 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.019+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2503791280}) 2024-09-16T04:42:55.027 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.019+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=855143313}) 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:55.020+0000 7fc5ded9d640 1 -- 172.21.15.46:0/3945956201 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc5bc0383e0 msgr2=0x7fc5bc03a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:55.020+0000 7fc5ded9d640 1 --2- 172.21.15.46:0/3945956201 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc5bc0383e0 0x7fc5bc03a8a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4004f60 tx=0x7fc5d4007e30 comp rx=0 tx=0).stop 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:55.020+0000 7fc5ded9d640 10 monclient: shutdown 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:55.020+0000 7fc5ded9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:55.020+0000 7fc5ded9d640 1 -- 172.21.15.46:0/3945956201 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc5d813bdb0 msgr2=0x7fc5d81cb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:55.020+0000 7fc5ded9d640 1 --2- 172.21.15.46:0/3945956201 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc5d813bdb0 0x7fc5d81cb990 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0007e40 tx=0x7fc5d000e4b0 comp rx=0 tx=0).stop 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:55.020+0000 7fc5ded9d640 1 -- 172.21.15.46:0/3945956201 shutdown_connections 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:55.020+0000 7fc5ded9d640 1 --2- 172.21.15.46:0/3945956201 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc5d813bdb0 0x7fc5d81cb990 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:55.020+0000 7fc5ded9d640 1 --2- 172.21.15.46:0/3945956201 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc5bc0383e0 0x7fc5bc03a8a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:55.020+0000 7fc5ded9d640 1 --2- 172.21.15.46:0/3945956201 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5d80a3f80 0x7fc5d81341c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:55.020+0000 7fc5ded9d640 1 --2- 172.21.15.46:0/3945956201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5d80a3ba0 0x7fc5d8133c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:55.020+0000 7fc5ded9d640 1 -- 172.21.15.46:0/3945956201 >> 172.21.15.46:0/3945956201 conn(0x7fc5d80a60c0 msgr2=0x7fc5d813dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:55.020+0000 7fc5ded9d640 1 -- 172.21.15.46:0/3945956201 shutdown_connections 2024-09-16T04:42:55.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:55.020+0000 7fc5ded9d640 1 -- 172.21.15.46:0/3945956201 wait complete. 2024-09-16T04:42:55.033 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.020+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.033 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.020+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3297374524}) 2024-09-16T04:42:55.034 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.020+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4185746541}) 2024-09-16T04:42:55.034 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.020+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.034 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.020+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3297374524}) 2024-09-16T04:42:55.034 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.020+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=49125803}) 2024-09-16T04:42:55.034 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.021+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.034 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.021+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2412663613}) 2024-09-16T04:42:55.034 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.021+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3013852277}) 2024-09-16T04:42:55.034 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.021+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.034 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.021+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2412663613}) 2024-09-16T04:42:55.034 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.021+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4057802742}) 2024-09-16T04:42:55.034 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.022+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.034 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.022+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2360275022}) 2024-09-16T04:42:55.034 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.022+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1378721663}) 2024-09-16T04:42:55.034 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.022+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.035 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.022+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2360275022}) 2024-09-16T04:42:55.035 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.022+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3147492458}) 2024-09-16T04:42:55.035 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.022+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.035 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.022+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4106895874}) 2024-09-16T04:42:55.036 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.022+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=856437953}) 2024-09-16T04:42:55.036 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.022+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.036 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.022+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4106895874}) 2024-09-16T04:42:55.036 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.023+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=606561851}) 2024-09-16T04:42:55.036 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.023+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.036 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.023+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3111497918}) 2024-09-16T04:42:55.036 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.024+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1962050014}) 2024-09-16T04:42:55.036 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.024+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.037 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.024+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3111497918}) 2024-09-16T04:42:55.037 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.024+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=178063240}) 2024-09-16T04:42:55.037 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.024+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.037 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.024+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206398045}) 2024-09-16T04:42:55.037 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.024+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2965208497}) 2024-09-16T04:42:55.037 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.024+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.037 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.024+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206398045}) 2024-09-16T04:42:55.037 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.025+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1851425646}) 2024-09-16T04:42:55.037 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.025+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.037 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.025+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3108881300}) 2024-09-16T04:42:55.037 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.025+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1329497703}) 2024-09-16T04:42:55.037 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.025+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.037 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.025+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3108881300}) 2024-09-16T04:42:55.037 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.025+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3652413343}) 2024-09-16T04:42:55.038 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.026+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.038 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.026+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3796775316}) 2024-09-16T04:42:55.038 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.026+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3026270240}) 2024-09-16T04:42:55.038 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.026+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.038 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.026+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3796775316}) 2024-09-16T04:42:55.038 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.026+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=603495177}) 2024-09-16T04:42:55.038 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.027+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.038 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.027+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=86,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=1920604021,mdsmap=3140514009,mgr=1081907435,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=1762537289}) 2024-09-16T04:42:55.038 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.027+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=88,mdsmap=3,mgr=8,mgr_command_descs=1} crc {logm=1010681126,mdsmap=3956149183,mgr=1081907435,mgr_command_descs=3961477881}) 2024-09-16T04:42:55.038 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.027+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.038 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.027+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=86,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=1920604021,mdsmap=3140514009,mgr=1081907435,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=1762537289}) 2024-09-16T04:42:55.038 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.027+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=89,mdsmap=3,mgr=8} crc {logm=36573593,mdsmap=3956149183,mgr=1081907435}) 2024-09-16T04:42:55.038 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.028+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.038 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.028+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=8,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=66} crc {mgrstat=4128457423,mon_config_key=1385660123,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=2071603610,osd_snap=723727393,osdmap=1866179637}) 2024-09-16T04:42:55.039 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.028+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=8,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=67} crc {mgrstat=4128457423,mon_config_key=124642177,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=2071603610,osd_snap=723727393,osdmap=802656302}) 2024-09-16T04:42:55.039 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.028+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.039 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.028+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=8,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=66} crc {mgrstat=4128457423,mon_config_key=1385660123,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=2071603610,osd_snap=723727393,osdmap=1866179637}) 2024-09-16T04:42:55.039 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.028+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=7,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=68} crc {mgrstat=1503112857,mon_config_key=132059253,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=2071603610,osd_snap=723727393,osdmap=992165802}) 2024-09-16T04:42:55.039 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.028+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.039 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.028+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3360891762}) 2024-09-16T04:42:55.039 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.028+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3229733524}) 2024-09-16T04:42:55.039 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.029+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.039 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.029+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3360891762}) 2024-09-16T04:42:55.039 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.029+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4196977440}) 2024-09-16T04:42:55.040 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:42:55.041 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.032+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.041 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.032+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=88} crc {osdmap=840368616}) 2024-09-16T04:42:55.041 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.032+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=89} crc {osdmap=1743040990}) 2024-09-16T04:42:55.041 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.032+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:42:55.041 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.032+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=88} crc {osdmap=840368616}) 2024-09-16T04:42:55.041 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:42:55.032+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=89} crc {osdmap=3429658557}) 2024-09-16T04:42:55.288 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:55.289 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37773]: starting. 2024-09-16T04:42:55.289 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37773]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-37737 2024-09-16T04:42:55.289 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37773]: shutting down. 2024-09-16T04:42:55.290 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:55.290 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:42:55.290 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:42:55.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:42:56.041 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:42:56.042 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:42:56.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.156+0000 7f9838a0a640 10 monclient: get_monmap_and_config 2024-09-16T04:42:56.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.156+0000 7f9838a0a640 10 monclient: build_initial_monmap 2024-09-16T04:42:56.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.156+0000 7f9838a0a640 10 monclient: monmap: 2024-09-16T04:42:56.158 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:56.158 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:56.157707+0000 2024-09-16T04:42:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:56.157707+0000 2024-09-16T04:42:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:56.160 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.156+0000 7f9838a0a640 1 Processor -- start 2024-09-16T04:42:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.156+0000 7f9838a0a640 1 -- start start 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.156+0000 7f9838a0a640 10 monclient: init 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f9838a0a640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f9838a0a640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f9838a0a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f98340a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f9838a0a640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f98340a3ff0 0x7f98340a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f9838a0a640 10 monclient(hunting): picked mon.noname-b con 0x7f98340a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f9838a0a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f983413be90 addr v1:172.21.15.114:6789/0 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f9838a0a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f983409b940 con 0x7f98340a3c10 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f9838a0a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f983409ab00 con 0x7f983413be90 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f9838a0a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f9838a0a640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f983409b030 con 0x7f98340a3ff0 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f9838a0a640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f9838a0a640 10 monclient(hunting): authenticate will time out at 974.573718s 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f9831d9b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f98340a3ff0 0x7f98340a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f983259c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f98340a3c10 0x7f983409bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60294/0 (socket says 172.21.15.46:60294) 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f9831d9b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f98340a3ff0 0x7f98340a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:54260/0 (socket says 172.21.15.46:54260) 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f983259c640 1 -- 172.21.15.46:0/155799307 learned_addr learned my addr 172.21.15.46:0/155799307 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f9831d9b640 10 monclient(hunting): get_auth_request con 0x7f98340a3ff0 auth_method 0 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f9831d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f9831d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f9831d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f9831d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f9831d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.157+0000 7f9831d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.158+0000 7f9831d9b640 10 monclient(hunting): handle_auth_done global_id 54128 payload 274 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.158+0000 7f9831d9b640 1 -- 172.21.15.46:0/155799307 >> v1:172.21.15.114:6789/0 conn(0x7f983413be90 legacy=0x7f983413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.158+0000 7f983159a640 1 -- 172.21.15.46:0/155799307 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2850317999 0 0) 0x7f983409b940 con 0x7f98340a3c10 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.158+0000 7f9831d9b640 1 -- 172.21.15.46:0/155799307 >> v1:172.21.15.46:6789/0 conn(0x7f98340a3c10 legacy=0x7f983409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.158+0000 7f9831d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.158+0000 7f9831d9b640 1 monclient: found mon.noname-b 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.158+0000 7f9831d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.158+0000 7f9831d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.158+0000 7f9831d9b640 1 -- 172.21.15.46:0/155799307 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f983409b310 con 0x7f98340a3ff0 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.158+0000 7f983159a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.158+0000 7f9831d9b640 1 --2- 172.21.15.46:0/155799307 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f98340a3ff0 0x7f98340a43d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f98240097d0 tx=0x7f982402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f157e08bbb52562a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f983159a640 1 -- 172.21.15.46:0/155799307 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9824037070 con 0x7f98340a3ff0 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f983159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f983159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f983159a640 10 monclient: dump: 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f983159a640 10 monclient: _finish_auth 0 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f983159a640 10 monclient: _check_auth_tickets 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f983159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f983159a640 1 -- 172.21.15.46:0/155799307 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9824032440 con 0x7f98340a3ff0 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f9838a0a640 5 monclient: authenticate success, global_id 54128 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f9838a0a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f983159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f9838a0a640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f983159a640 1 -- 172.21.15.46:0/155799307 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9824032810 con 0x7f98340a3ff0 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f983159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f9838a0a640 4 set_mon_vals no callback set 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f983159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f983159a640 10 monclient: dump: 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f9838a0a640 10 monclient: shutdown 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f9838a0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f9838a0a640 1 -- 172.21.15.46:0/155799307 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f98340a3ff0 msgr2=0x7f98340a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f9838a0a640 1 --2- 172.21.15.46:0/155799307 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f98340a3ff0 0x7f98340a43d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f98240097d0 tx=0x7f982402a180 comp rx=0 tx=0).stop 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f9838a0a640 1 -- 172.21.15.46:0/155799307 shutdown_connections 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f9838a0a640 1 --2- 172.21.15.46:0/155799307 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f98340a3ff0 0x7f98340a43d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f9838a0a640 1 -- 172.21.15.46:0/155799307 >> 172.21.15.46:0/155799307 conn(0x7f98340a6340 msgr2=0x7f983406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f9838a0a640 1 -- 172.21.15.46:0/155799307 shutdown_connections 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.159+0000 7f9838a0a640 1 -- 172.21.15.46:0/155799307 wait complete. 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.160+0000 7f9838a0a640 10 monclient: build_initial_monmap 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.160+0000 7f9838a0a640 10 monclient: monmap: 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.160+0000 7f9838a0a640 1 Processor -- start 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.160+0000 7f9838a0a640 1 -- start start 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.160+0000 7f9838a0a640 10 monclient: init 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9838a0a640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9838a0a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9838a0a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98340a3c10 0x7f98341cbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9838a0a640 10 monclient(hunting): picked mon.noname-b con 0x7f98340a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9838a0a640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f98340a3ff0 0x7f9834137070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9838a0a640 10 monclient(hunting): picked mon.noname-c con 0x7f98340a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9838a0a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f983413be90 0x7f983409d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9838a0a640 10 monclient(hunting): picked mon.noname-a con 0x7f983413be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9838a0a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9838a0a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f98340a6b40 con 0x7f983413be90 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9838a0a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9838a0a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f98340a69c0 con 0x7f98340a3c10 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9838a0a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9838a0a640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f983413f9a0 con 0x7f98340a3ff0 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9838a0a640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9838a0a640 10 monclient(hunting): authenticate will time out at 974.577416s 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9831d9b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f98340a3ff0 0x7f9834137070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9831d9b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f98340a3ff0 0x7f9834137070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:54264/0 (socket says 172.21.15.46:54264) 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9831d9b640 1 -- 172.21.15.46:0/2163470964 learned_addr learned my addr 172.21.15.46:0/2163470964 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9831d9b640 10 monclient(hunting): get_auth_request con 0x7f98340a3ff0 auth_method 0 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9831d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9831d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9831d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9832d9d640 1 --2- 172.21.15.46:0/2163470964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f983413be90 0x7f983409d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f983259c640 1 --2- 172.21.15.46:0/2163470964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98340a3c10 0x7f98341cbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9832d9d640 10 monclient(hunting): get_auth_request con 0x7f983413be90 auth_method 0 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9832d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9832d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9832d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9831d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9831d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9831d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f983259c640 10 monclient(hunting): get_auth_request con 0x7f98340a3c10 auth_method 0 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f983259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f983259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f983259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9832d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9832d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9832d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f983259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f983259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f983259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9831d9b640 10 monclient(hunting): handle_auth_done global_id 54131 payload 824 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9831d9b640 1 -- 172.21.15.46:0/2163470964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98340a3c10 msgr2=0x7f98341cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9831d9b640 1 --2- 172.21.15.46:0/2163470964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98340a3c10 0x7f98341cbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9831d9b640 1 -- 172.21.15.46:0/2163470964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f983413be90 msgr2=0x7f983409d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9831d9b640 1 --2- 172.21.15.46:0/2163470964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f983413be90 0x7f983409d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9831d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9831d9b640 1 monclient: found mon.noname-c 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9831d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9831d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.161+0000 7f9831d9b640 1 -- 172.21.15.46:0/2163470964 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98340a0a10 con 0x7f98340a3ff0 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f9831d9b640 1 --2- 172.21.15.46:0/2163470964 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f98340a3ff0 0x7f9834137070 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f98240097d0 tx=0x7f98240046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f982affd640 1 -- 172.21.15.46:0/2163470964 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9824041070 con 0x7f98340a3ff0 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f982affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f982affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f982affd640 10 monclient: dump: 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f982affd640 10 monclient: _finish_auth 0 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f982affd640 10 monclient: _check_auth_tickets 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f982affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f982affd640 1 -- 172.21.15.46:0/2163470964 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f982403c890 con 0x7f98340a3ff0 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f982affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f982affd640 1 -- 172.21.15.46:0/2163470964 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9824037040 con 0x7f98340a3ff0 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f9838a0a640 5 monclient: authenticate success, global_id 54131 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f982affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f982affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f982affd640 10 monclient: dump: 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f9838a0a640 10 monclient: _renew_subs 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f9830d99640 4 set_mon_vals no callback set 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f9838a0a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f9838a0a640 1 -- 172.21.15.46:0/2163470964 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f983413f750 con 0x7f98340a3ff0 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f9838a0a640 10 monclient: _renew_subs 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f9838a0a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.162+0000 7f9838a0a640 1 -- 172.21.15.46:0/2163470964 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f98341209f0 con 0x7f98340a3ff0 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.163+0000 7f982affd640 1 -- 172.21.15.46:0/2163470964 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f9824044070 con 0x7f98340a3ff0 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.163+0000 7f9838a0a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.163+0000 7f9838a0a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.163+0000 7f9838a0a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.163+0000 7f9838a0a640 1 -- 172.21.15.46:0/2163470964 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98340a1a30 con 0x7f98340a3ff0 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.163+0000 7f982affd640 1 --2- 172.21.15.46:0/2163470964 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f98140383b0 0x7f981403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.163+0000 7f982affd640 1 -- 172.21.15.46:0/2163470964 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(164..164 src has 1..164) ==== 4346+0+0 (secure 0 0 0) 0x7f982406c290 con 0x7f98340a3ff0 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.163+0000 7f983259c640 1 --2- 172.21.15.46:0/2163470964 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f98140383b0 0x7f981403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.163+0000 7f983259c640 10 monclient: get_auth_request con 0x7f98140383b0 auth_method 0 2024-09-16T04:42:56.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.164+0000 7f983259c640 1 --2- 172.21.15.46:0/2163470964 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f98140383b0 0x7f981403a870 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f981c007f70 tx=0x7f981c007f00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:56.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.168+0000 7f982affd640 1 -- 172.21.15.46:0/2163470964 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f98340a1a30 con 0x7f98340a3ff0 2024-09-16T04:42:56.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.168+0000 7f982affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:56.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.168+0000 7f982affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:56.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.419+0000 7f9838a0a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:42:56.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.419+0000 7f9838a0a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:56.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.419+0000 7f9838a0a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:56.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.419+0000 7f9838a0a640 1 -- 172.21.15.46:0/2163470964 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9834069810 con 0x7f98340a3ff0 2024-09-16T04:42:56.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.420+0000 7f982affd640 1 -- 172.21.15.46:0/2163470964 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f9834069810 con 0x7f98340a3ff0 2024-09-16T04:42:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.420+0000 7f982affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.420+0000 7f982affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:56.423 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:42:56.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.422+0000 7f9838a0a640 1 -- 172.21.15.46:0/2163470964 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f98140383b0 msgr2=0x7f981403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.422+0000 7f9838a0a640 1 --2- 172.21.15.46:0/2163470964 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f98140383b0 0x7f981403a870 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f981c007f70 tx=0x7f981c007f00 comp rx=0 tx=0).stop 2024-09-16T04:42:56.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.422+0000 7f9838a0a640 10 monclient: shutdown 2024-09-16T04:42:56.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.422+0000 7f9838a0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:56.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.422+0000 7f9838a0a640 1 -- 172.21.15.46:0/2163470964 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f98340a3ff0 msgr2=0x7f9834137070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.422+0000 7f9838a0a640 1 --2- 172.21.15.46:0/2163470964 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f98340a3ff0 0x7f9834137070 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f98240097d0 tx=0x7f98240046c0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.422+0000 7f9838a0a640 1 -- 172.21.15.46:0/2163470964 shutdown_connections 2024-09-16T04:42:56.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.422+0000 7f9838a0a640 1 --2- 172.21.15.46:0/2163470964 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f98140383b0 0x7f981403a870 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.422+0000 7f9838a0a640 1 --2- 172.21.15.46:0/2163470964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f983413be90 0x7f983409d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.422+0000 7f9838a0a640 1 --2- 172.21.15.46:0/2163470964 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f98340a3ff0 0x7f9834137070 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.422+0000 7f9838a0a640 1 --2- 172.21.15.46:0/2163470964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98340a3c10 0x7f98341cbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.422+0000 7f9838a0a640 1 -- 172.21.15.46:0/2163470964 >> 172.21.15.46:0/2163470964 conn(0x7f98340a6340 msgr2=0x7f98340a1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:56.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.422+0000 7f9838a0a640 1 -- 172.21.15.46:0/2163470964 shutdown_connections 2024-09-16T04:42:56.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.422+0000 7f9838a0a640 1 -- 172.21.15.46:0/2163470964 wait complete. 2024-09-16T04:42:56.435 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:42:56.435 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:42:56.435 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:42:56.435 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:42:56.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.535+0000 7fd76f027640 10 monclient: get_monmap_and_config 2024-09-16T04:42:56.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.535+0000 7fd76f027640 10 monclient: build_initial_monmap 2024-09-16T04:42:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.535+0000 7fd76f027640 10 monclient: monmap: 2024-09-16T04:42:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:56.537101+0000 2024-09-16T04:42:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:56.537101+0000 2024-09-16T04:42:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:56.538 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.536+0000 7fd76f027640 1 Processor -- start 2024-09-16T04:42:56.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.536+0000 7fd76f027640 1 -- start start 2024-09-16T04:42:56.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.536+0000 7fd76f027640 10 monclient: init 2024-09-16T04:42:56.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.536+0000 7fd76f027640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:56.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.536+0000 7fd76f027640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-16T04:42:56.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.536+0000 7fd76f027640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd7680a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:42:56.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.536+0000 7fd76f027640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd7680a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:42:56.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.536+0000 7fd76f027640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd76813cea0 0x7fd76813f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.536+0000 7fd76f027640 10 monclient(hunting): picked mon.noname-c con 0x7fd76813cea0 addr v2:172.21.15.114:3300/0 2024-09-16T04:42:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.536+0000 7fd76f027640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd76809b940 con 0x7fd7680a3c10 2024-09-16T04:42:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.536+0000 7fd76f027640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd76809ab00 con 0x7fd7680a3ff0 2024-09-16T04:42:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.536+0000 7fd76f027640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.536+0000 7fd76f027640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fd76809b030 con 0x7fd76813cea0 2024-09-16T04:42:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76f027640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76f027640 10 monclient(hunting): authenticate will time out at 974.953012s 2024-09-16T04:42:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76cdc3640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd7680a3c10 0x7fd76809bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60306/0 (socket says 172.21.15.46:60306) 2024-09-16T04:42:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76cdc3640 1 -- 172.21.15.46:0/1583403284 learned_addr learned my addr 172.21.15.46:0/1583403284 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:42:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 1 -- 172.21.15.46:0/1583403284 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 156048603 0 0) 0x7fd76809b940 con 0x7fd7680a3c10 2024-09-16T04:42:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 10 monclient(hunting): _negotiate 2024-09-16T04:42:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 10 monclient(hunting): my global_id is 44415 2024-09-16T04:42:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 1 -- 172.21.15.46:0/1583403284 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd758003520 con 0x7fd7680a3c10 2024-09-16T04:42:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76d5c4640 1 --2- 172.21.15.46:0/1583403284 >> v2:172.21.15.114:3300/0 conn(0x7fd76813cea0 0x7fd76813f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76d5c4640 10 monclient(hunting): get_auth_request con 0x7fd76813cea0 auth_method 0 2024-09-16T04:42:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76d5c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76d5c4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76d5c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76d5c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76d5c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76d5c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 1 -- 172.21.15.46:0/1583403284 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2780351287 0 0) 0x7fd758003520 con 0x7fd7680a3c10 2024-09-16T04:42:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 1 -- 172.21.15.46:0/1583403284 >> v2:172.21.15.114:3300/0 conn(0x7fd76813cea0 msgr2=0x7fd76813f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 1 --2- 172.21.15.46:0/1583403284 >> v2:172.21.15.114:3300/0 conn(0x7fd76813cea0 0x7fd76813f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 1 -- 172.21.15.46:0/1583403284 >> v1:172.21.15.114:6789/0 conn(0x7fd7680a3ff0 legacy=0x7fd7680a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 10 monclient: _finish_hunting 0 2024-09-16T04:42:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:42:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:42:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 1 -- 172.21.15.46:0/1583403284 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd76809b310 con 0x7fd7680a3c10 2024-09-16T04:42:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 10 monclient: _finish_auth 0 2024-09-16T04:42:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 10 monclient: _check_auth_tickets 2024-09-16T04:42:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76f027640 5 monclient: authenticate success, global_id 44415 2024-09-16T04:42:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 1 -- 172.21.15.46:0/1583403284 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7fd75c001670 con 0x7fd7680a3c10 2024-09-16T04:42:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76f027640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:42:56.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 10 monclient: dump: 2024-09-16T04:42:56.554 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.554 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.554 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.554 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.554 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.554 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.554 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.554 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.554 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.554 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:42:56.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:56.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 1 -- 172.21.15.46:0/1583403284 >> v1:172.21.15.46:6789/0 conn(0x7fd7680a3c10 legacy=0x7fd76809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:42:56.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 1 --2- 172.21.15.46:0/1583403284 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7580039b0 0x7fd758003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 10 monclient(hunting): picked mon.c con 0x7fd7580039b0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:56.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 1 --2- 172.21.15.46:0/1583403284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd758004800 0x7fd758004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 10 monclient(hunting): picked mon.b con 0x7fd758004800 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:56.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 1 --2- 172.21.15.46:0/1583403284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7580051c0 0x7fd75800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 10 monclient(hunting): picked mon.a con 0x7fd7580051c0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:56.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 1 -- 172.21.15.46:0/1583403284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd76809b030 con 0x7fd7580051c0 2024-09-16T04:42:56.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 1 -- 172.21.15.46:0/1583403284 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd75800c160 con 0x7fd758004800 2024-09-16T04:42:56.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 1 -- 172.21.15.46:0/1583403284 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd75800c360 con 0x7fd7580039b0 2024-09-16T04:42:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd7677fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:42:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76f027640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76cdc3640 1 --2- 172.21.15.46:0/1583403284 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7580039b0 0x7fd758003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76d5c4640 1 --2- 172.21.15.46:0/1583403284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7580051c0 0x7fd75800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76cdc3640 10 monclient(hunting): get_auth_request con 0x7fd7580039b0 auth_method 0 2024-09-16T04:42:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76cdc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76cdc3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76cdc3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76d5c4640 10 monclient(hunting): get_auth_request con 0x7fd7580051c0 auth_method 0 2024-09-16T04:42:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76d5c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76d5c4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76d5c4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd767fff640 1 --2- 172.21.15.46:0/1583403284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd758004800 0x7fd758004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76cdc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76cdc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76cdc3640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:42:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76d5c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76d5c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.537+0000 7fd76d5c4640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:42:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd767fff640 10 monclient(hunting): get_auth_request con 0x7fd758004800 auth_method 0 2024-09-16T04:42:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd767fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd767fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd767fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:42:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd767fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd767fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd767fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:42:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd76cdc3640 10 monclient(hunting): handle_auth_done global_id 44415 payload 293 2024-09-16T04:42:56.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd76cdc3640 1 -- 172.21.15.46:0/1583403284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd758004800 msgr2=0x7fd758004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:56.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd76cdc3640 1 --2- 172.21.15.46:0/1583403284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd758004800 0x7fd758004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd76cdc3640 1 -- 172.21.15.46:0/1583403284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7580051c0 msgr2=0x7fd75800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:56.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd76cdc3640 1 --2- 172.21.15.46:0/1583403284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7580051c0 0x7fd75800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd76cdc3640 10 monclient: _finish_hunting 0 2024-09-16T04:42:56.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd76cdc3640 1 monclient: found mon.c 2024-09-16T04:42:56.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd76cdc3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:56.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd76cdc3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:42:56.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd76cdc3640 1 -- 172.21.15.46:0/1583403284 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd758003520 con 0x7fd7580039b0 2024-09-16T04:42:56.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd76cdc3640 10 monclient: _finish_auth 0 2024-09-16T04:42:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd76cdc3640 10 monclient: _check_auth_tickets 2024-09-16T04:42:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd76cdc3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd76d5c4640 1 --2- 172.21.15.46:0/1583403284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7580051c0 0x7fd75800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:42:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd76d5c4640 1 -- 172.21.15.46:0/1583403284 reap_dead start 2024-09-16T04:42:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.538+0000 7fd76cdc3640 1 --2- 172.21.15.46:0/1583403284 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7580039b0 0x7fd758003dd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd75c00c210 tx=0x7fd758005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f59da29929c95af3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd7677fe640 1 -- 172.21.15.46:0/1583403284 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c007d30 con 0x7fd7580039b0 2024-09-16T04:42:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd7677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd7677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd7677fe640 10 monclient: dump: 2024-09-16T04:42:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.560 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd76f027640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd7677fe640 1 -- 172.21.15.46:0/1583403284 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd75c005af0 con 0x7fd7580039b0 2024-09-16T04:42:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd7677fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd76f027640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd7677fe640 1 -- 172.21.15.46:0/1583403284 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c007070 con 0x7fd7580039b0 2024-09-16T04:42:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd7677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd76f027640 4 set_mon_vals no callback set 2024-09-16T04:42:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd7677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd7677fe640 10 monclient: dump: 2024-09-16T04:42:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.561 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd76f027640 10 monclient: shutdown 2024-09-16T04:42:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd76f027640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd76f027640 1 -- 172.21.15.46:0/1583403284 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7580039b0 msgr2=0x7fd758003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd76f027640 1 --2- 172.21.15.46:0/1583403284 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7580039b0 0x7fd758003dd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd75c00c210 tx=0x7fd758005830 comp rx=0 tx=0).stop 2024-09-16T04:42:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd76f027640 1 -- 172.21.15.46:0/1583403284 shutdown_connections 2024-09-16T04:42:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd76f027640 1 --2- 172.21.15.46:0/1583403284 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7580039b0 0x7fd758003dd0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd76f027640 1 -- 172.21.15.46:0/1583403284 >> 172.21.15.46:0/1583403284 conn(0x7fd7680a6340 msgr2=0x7fd76806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd76f027640 1 -- 172.21.15.46:0/1583403284 shutdown_connections 2024-09-16T04:42:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.539+0000 7fd76f027640 1 -- 172.21.15.46:0/1583403284 wait complete. 2024-09-16T04:42:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.540+0000 7fd76f027640 10 monclient: build_initial_monmap 2024-09-16T04:42:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.540+0000 7fd76f027640 10 monclient: monmap: 2024-09-16T04:42:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:56.563 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.540+0000 7fd76f027640 1 Processor -- start 2024-09-16T04:42:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.540+0000 7fd76f027640 1 -- start start 2024-09-16T04:42:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.540+0000 7fd76f027640 10 monclient: init 2024-09-16T04:42:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76f027640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76f027640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:42:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76f027640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7580039b0 0x7fd76813b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76f027640 10 monclient(hunting): picked mon.noname-c con 0x7fd7580039b0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76f027640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd76811f860 0x7fd76811fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76f027640 10 monclient(hunting): picked mon.noname-a con 0x7fd76811f860 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76f027640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd768120180 0x7fd7681d2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76f027640 10 monclient(hunting): picked mon.noname-b con 0x7fd768120180 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76f027640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76f027640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd76809be70 con 0x7fd76811f860 2024-09-16T04:42:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76f027640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76f027640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd76809bcf0 con 0x7fd768120180 2024-09-16T04:42:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76f027640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76f027640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd76809bff0 con 0x7fd7580039b0 2024-09-16T04:42:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76f027640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76f027640 10 monclient(hunting): authenticate will time out at 974.957339s 2024-09-16T04:42:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd767fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd76811f860 0x7fd76811fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76cdc3640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7580039b0 0x7fd76813b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd767fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd76811f860 0x7fd76811fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51520/0 (socket says 172.21.15.46:51520) 2024-09-16T04:42:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd767fff640 1 -- 172.21.15.46:0/557991763 learned_addr learned my addr 172.21.15.46:0/557991763 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd767fff640 10 monclient(hunting): get_auth_request con 0x7fd76811f860 auth_method 0 2024-09-16T04:42:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd767fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd767fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd767fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76cdc3640 10 monclient(hunting): get_auth_request con 0x7fd7580039b0 auth_method 0 2024-09-16T04:42:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76cdc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76cdc3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76cdc3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76d5c4640 1 --2- 172.21.15.46:0/557991763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd768120180 0x7fd7681d2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd767fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd767fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd767fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76d5c4640 10 monclient(hunting): get_auth_request con 0x7fd768120180 auth_method 0 2024-09-16T04:42:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76d5c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76d5c4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76d5c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76cdc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76cdc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76cdc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76d5c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76d5c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76d5c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd767fff640 10 monclient(hunting): handle_auth_done global_id 44418 payload 824 2024-09-16T04:42:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd767fff640 1 -- 172.21.15.46:0/557991763 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7580039b0 msgr2=0x7fd76813b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd767fff640 1 --2- 172.21.15.46:0/557991763 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7580039b0 0x7fd76813b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd767fff640 1 -- 172.21.15.46:0/557991763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd768120180 msgr2=0x7fd7681d2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd767fff640 1 --2- 172.21.15.46:0/557991763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd768120180 0x7fd7681d2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd767fff640 10 monclient: _finish_hunting 0 2024-09-16T04:42:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd767fff640 1 monclient: found mon.noname-a 2024-09-16T04:42:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd767fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd767fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd767fff640 1 -- 172.21.15.46:0/557991763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd768122930 con 0x7fd76811f860 2024-09-16T04:42:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd76cdc3640 1 --2- 172.21.15.46:0/557991763 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7580039b0 0x7fd76813b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:42:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.541+0000 7fd767fff640 1 --2- 172.21.15.46:0/557991763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd76811f860 0x7fd76811fc40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd75400f5f0 tx=0x7fd75400fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd7657fa640 1 -- 172.21.15.46:0/557991763 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7540052e0 con 0x7fd76811f860 2024-09-16T04:42:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd7657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd7657fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd7657fa640 10 monclient: dump: 2024-09-16T04:42:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.568 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd7657fa640 10 monclient: _finish_auth 0 2024-09-16T04:42:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd7657fa640 10 monclient: _check_auth_tickets 2024-09-16T04:42:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd7657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd76f027640 5 monclient: authenticate success, global_id 44418 2024-09-16T04:42:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd7657fa640 1 -- 172.21.15.46:0/557991763 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd754018070 con 0x7fd76811f860 2024-09-16T04:42:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd7657fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd76f027640 10 monclient: _renew_subs 2024-09-16T04:42:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd7657fa640 1 -- 172.21.15.46:0/557991763 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd754003810 con 0x7fd76811f860 2024-09-16T04:42:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd76f027640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd76f027640 1 -- 172.21.15.46:0/557991763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd768121910 con 0x7fd76811f860 2024-09-16T04:42:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd7657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd7657fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd7657fa640 10 monclient: dump: 2024-09-16T04:42:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.569 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd766ffd640 4 set_mon_vals no callback set 2024-09-16T04:42:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd76f027640 10 monclient: _renew_subs 2024-09-16T04:42:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd76f027640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd76f027640 1 -- 172.21.15.46:0/557991763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd768120600 con 0x7fd76811f860 2024-09-16T04:42:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd76f027640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd76f027640 1 --2- 172.21.15.46:0/557991763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd76809f120 0x7fd76809f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd76f027640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd7657fa640 1 -- 172.21.15.46:0/557991763 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fd7540039b0 con 0x7fd76811f860 2024-09-16T04:42:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.542+0000 7fd76f027640 1 -- 172.21.15.46:0/557991763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd7680a0b30 con 0x7fd76809f120 2024-09-16T04:42:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.543+0000 7fd76cdc3640 1 --2- 172.21.15.46:0/557991763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd76809f120 0x7fd76809f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.543+0000 7fd76cdc3640 10 monclient: get_auth_request con 0x7fd76809f120 auth_method 0 2024-09-16T04:42:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.543+0000 7fd76cdc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.543+0000 7fd76cdc3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.543+0000 7fd76cdc3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.543+0000 7fd76cdc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.543+0000 7fd76cdc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.543+0000 7fd76cdc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.543+0000 7fd7657fa640 1 --2- 172.21.15.46:0/557991763 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd74c0383e0 0x7fd74c03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.543+0000 7fd7657fa640 1 -- 172.21.15.46:0/557991763 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 4738+0+0 (secure 0 0 0) 0x7fd75404b560 con 0x7fd76811f860 2024-09-16T04:42:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.543+0000 7fd76cdc3640 10 monclient(hunting): handle_auth_done global_id 44421 payload 824 2024-09-16T04:42:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.543+0000 7fd76cdc3640 1 -- 172.21.15.46:0/557991763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7680a43d0 con 0x7fd76809f120 2024-09-16T04:42:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.543+0000 7fd76d5c4640 1 --2- 172.21.15.46:0/557991763 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd74c0383e0 0x7fd74c03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.543+0000 7fd76d5c4640 10 monclient: get_auth_request con 0x7fd74c0383e0 auth_method 0 2024-09-16T04:42:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.543+0000 7fd76cdc3640 1 --2- 172.21.15.46:0/557991763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd76809f120 0x7fd76809f500 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd75c001eb0 tx=0x7fd75c001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.543+0000 7fd7657fa640 1 -- 172.21.15.46:0/557991763 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c043070 con 0x7fd76809f120 2024-09-16T04:42:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.543+0000 7fd7657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.543+0000 7fd7657fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.543+0000 7fd7657fa640 10 monclient: dump: 2024-09-16T04:42:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.572 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.544+0000 7fd76d5c4640 1 --2- 172.21.15.46:0/557991763 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd74c0383e0 0x7fd74c03a8a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd768067ce0 tx=0x7fd76000b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:56.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.544+0000 7fd7657fa640 1 -- 172.21.15.46:0/557991763 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd75c038040 con 0x7fd76809f120 2024-09-16T04:42:56.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.544+0000 7fd7657fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:56.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.544+0000 7fd7657fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:56.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.544+0000 7fd7657fa640 1 -- 172.21.15.46:0/557991763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd76809f120 msgr2=0x7fd76809f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.544+0000 7fd7657fa640 1 --2- 172.21.15.46:0/557991763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd76809f120 0x7fd76809f500 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd75c001eb0 tx=0x7fd75c001a40 comp rx=0 tx=0).stop 2024-09-16T04:42:56.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.544+0000 7fd7657fa640 1 -- 172.21.15.46:0/557991763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd76809f120 msgr2=0x7fd76809f500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:56.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.544+0000 7fd7657fa640 1 --2- 172.21.15.46:0/557991763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd76809f120 0x7fd76809f500 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd75c001eb0 tx=0x7fd75c001a40 comp rx=0 tx=0).stop 2024-09-16T04:42:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.550+0000 7fd76f027640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:42:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.550+0000 7fd76f027640 1 --2- 172.21.15.46:0/557991763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd76809fa40 0x7fd7681d65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.550+0000 7fd76f027640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.550+0000 7fd76f027640 1 -- 172.21.15.46:0/557991763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd768120b90 con 0x7fd76809fa40 2024-09-16T04:42:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.550+0000 7fd76cdc3640 1 --2- 172.21.15.46:0/557991763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd76809fa40 0x7fd7681d65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.550+0000 7fd76cdc3640 10 monclient: get_auth_request con 0x7fd76809fa40 auth_method 0 2024-09-16T04:42:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.550+0000 7fd76cdc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.550+0000 7fd76cdc3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.550+0000 7fd76cdc3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.550+0000 7fd76cdc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.550+0000 7fd76cdc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.550+0000 7fd76cdc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.550+0000 7fd76cdc3640 10 monclient(hunting): handle_auth_done global_id 44424 payload 824 2024-09-16T04:42:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.550+0000 7fd76cdc3640 1 -- 172.21.15.46:0/557991763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd7680008d0 con 0x7fd76809fa40 2024-09-16T04:42:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.550+0000 7fd76cdc3640 1 --2- 172.21.15.46:0/557991763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd76809fa40 0x7fd7681d65c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd768120ad0 tx=0x7fd75c02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd7657fa640 1 -- 172.21.15.46:0/557991763 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c007dd0 con 0x7fd76809fa40 2024-09-16T04:42:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd7657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd7657fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd7657fa640 10 monclient: dump: 2024-09-16T04:42:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.577 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd7657fa640 1 -- 172.21.15.46:0/557991763 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7fd75c007dd0 con 0x7fd76809fa40 2024-09-16T04:42:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd7657fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:42:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd7657fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd7657fa640 1 -- 172.21.15.46:0/557991763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd76809fa40 msgr2=0x7fd7681d65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd7657fa640 1 --2- 172.21.15.46:0/557991763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd76809fa40 0x7fd7681d65c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd768120ad0 tx=0x7fd75c02e000 comp rx=0 tx=0).stop 2024-09-16T04:42:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd7657fa640 1 -- 172.21.15.46:0/557991763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd76809fa40 msgr2=0x7fd7681d65c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd7657fa640 1 --2- 172.21.15.46:0/557991763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd76809fa40 0x7fd7681d65c0 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd768120ad0 tx=0x7fd75c02e000 comp rx=0 tx=0).stop 2024-09-16T04:42:56.578 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:42:56.578 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:42:56.578 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:42:56.578 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:42:56.578 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 48, 2024-09-16T04:42:56.578 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 56299, 2024-09-16T04:42:56.578 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:42:56.578 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:42:56.578 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:42:56.578 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:42:56.579 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:56.579 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 6, 2024-09-16T04:42:56.579 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:56.579 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:42:56.579 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:42:56.579 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:56.579 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:56.579 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:56.579 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:56.579 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:56.579 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:56.579 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:56.579 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:56.579 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:56.580 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:56.580 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:56.580 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:56.580 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:42:56.580 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:42:56.580 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:56.580 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:56.580 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:56.580 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:56.580 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:56.580 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:56.580 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:56.580 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:56.580 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:56.581 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:56.581 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:56.581 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:56.581 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.581 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:42:56.581 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:42:56.581 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:42:56.581 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:42:56.581 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:42:56.581 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:42:56.581 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:56.581 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:56.581 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:42:56.581 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:42:56.582 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:42:56.582 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:42:56.582 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:42:56.582 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:42:56.582 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:42:56.582 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:56.582 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:42:56.582 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:56.582 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:56.582 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:56.582 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:56.582 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:56.582 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:56.583 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:56.583 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:56.583 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:56.583 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:56.583 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:56.583 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:56.583 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:42:56.583 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.583 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:42:56.583 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.583 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:42:56.583 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:42:56.583 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:56.583 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:56.584 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.584 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:56.584 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:42:56.584 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:56.584 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.584 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.584 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:56.584 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:42:56.584 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:56.584 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:56.585 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:56.585 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.585 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:42:56.585 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:42:56.585 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:56.585 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:56.585 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:56.585 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.585 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.585 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:42:56.585 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:42:56.585 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:56.585 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:56.585 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.586 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:56.586 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:42:56.586 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:56.586 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.586 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.586 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:56.586 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:42:56.586 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:56.586 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:56.586 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:56.586 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.586 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:42:56.586 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:42:56.586 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:56.587 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:56.588 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:56.588 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.588 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.588 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:42:56.588 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:42:56.588 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:56.588 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:56.588 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.588 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:56.588 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:42:56.588 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:56.588 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.588 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.588 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:56.589 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:42:56.589 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:56.589 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:56.589 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:56.589 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.589 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:42:56.589 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:42:56.589 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:56.589 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:56.589 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:56.589 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:56.589 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:56.589 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.589 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:42:56.590 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:42:56.590 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.590 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:56.590 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:56.590 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:42:56.590 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:56.590 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:56.590 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:42:56.590 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.590 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:56.590 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:56.590 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:42:56.590 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:56.590 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:56.591 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:42:56.591 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.591 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:56.591 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:56.591 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:42:56.591 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:56.591 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:56.591 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.591 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:42:56.591 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:42:56.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd76f027640 1 -- 172.21.15.46:0/557991763 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd74c0383e0 msgr2=0x7fd74c03a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd76f027640 1 --2- 172.21.15.46:0/557991763 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd74c0383e0 0x7fd74c03a8a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd768067ce0 tx=0x7fd76000b040 comp rx=0 tx=0).stop 2024-09-16T04:42:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd76f027640 10 monclient: shutdown 2024-09-16T04:42:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd76f027640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd76f027640 1 -- 172.21.15.46:0/557991763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd76811f860 msgr2=0x7fd76811fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd76f027640 1 --2- 172.21.15.46:0/557991763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd76811f860 0x7fd76811fc40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd75400f5f0 tx=0x7fd75400fab0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd76d5c4640 1 -- 172.21.15.46:0/557991763 reap_dead start 2024-09-16T04:42:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd76f027640 1 -- 172.21.15.46:0/557991763 shutdown_connections 2024-09-16T04:42:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd76f027640 1 --2- 172.21.15.46:0/557991763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd76811f860 0x7fd76811fc40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd76f027640 1 -- 172.21.15.46:0/557991763 >> 172.21.15.46:0/557991763 conn(0x7fd7680a6340 msgr2=0x7fd76813c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd76f027640 1 -- 172.21.15.46:0/557991763 shutdown_connections 2024-09-16T04:42:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.551+0000 7fd76f027640 1 -- 172.21.15.46:0/557991763 wait complete. 2024-09-16T04:42:56.594 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:42:56.594 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:42:56.594 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:42:56.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.693+0000 7f0c24142640 10 monclient: get_monmap_and_config 2024-09-16T04:42:56.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.693+0000 7f0c24142640 10 monclient: build_initial_monmap 2024-09-16T04:42:56.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.693+0000 7f0c24142640 10 monclient: monmap: 2024-09-16T04:42:56.696 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:56.696 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.696 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:56.695218+0000 2024-09-16T04:42:56.697 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:56.695218+0000 2024-09-16T04:42:56.697 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:56.697 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.697 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:56.697 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:56.697 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:56.697 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:56.697 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:56.697 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.694+0000 7f0c24142640 1 Processor -- start 2024-09-16T04:42:56.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.694+0000 7f0c24142640 1 -- start start 2024-09-16T04:42:56.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.694+0000 7f0c24142640 10 monclient: init 2024-09-16T04:42:56.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c24142640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:56.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c24142640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-16T04:42:56.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c24142640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0c1c0a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:42:56.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c24142640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c0a3ff0 0x7f0c1c0a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c24142640 10 monclient(hunting): picked mon.noname-b con 0x7f0c1c0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:56.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c24142640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0c1c13be90 0x7f0c1c13e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c24142640 10 monclient(hunting): picked mon.noname-c con 0x7f0c1c13be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:42:56.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c24142640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c1c09b940 con 0x7f0c1c0a3c10 2024-09-16T04:42:56.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c24142640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c24142640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f0c1c09ab00 con 0x7f0c1c13be90 2024-09-16T04:42:56.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c24142640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c24142640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f0c1c09b030 con 0x7f0c1c0a3ff0 2024-09-16T04:42:56.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c24142640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:56.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c24142640 10 monclient(hunting): authenticate will time out at 975.111179s 2024-09-16T04:42:56.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c216dd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c0a3ff0 0x7f0c1c0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c216dd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c0a3ff0 0x7f0c1c0a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:54292/0 (socket says 172.21.15.46:54292) 2024-09-16T04:42:56.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c216dd640 1 -- 172.21.15.46:0/505759109 learned_addr learned my addr 172.21.15.46:0/505759109 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:56.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c216dd640 10 monclient(hunting): get_auth_request con 0x7f0c1c0a3ff0 auth_method 0 2024-09-16T04:42:56.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c216dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c216dd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c216dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c216dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c216dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c216dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c226df640 1 --2- 172.21.15.46:0/505759109 >> v2:172.21.15.114:3300/0 conn(0x7f0c1c13be90 0x7f0c1c13e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c226df640 10 monclient(hunting): get_auth_request con 0x7f0c1c13be90 auth_method 0 2024-09-16T04:42:56.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c226df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c226df640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c226df640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c216dd640 10 monclient(hunting): handle_auth_done global_id 54137 payload 274 2024-09-16T04:42:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c216dd640 1 -- 172.21.15.46:0/505759109 >> v2:172.21.15.114:3300/0 conn(0x7f0c1c13be90 msgr2=0x7f0c1c13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c216dd640 1 --2- 172.21.15.46:0/505759109 >> v2:172.21.15.114:3300/0 conn(0x7f0c1c13be90 0x7f0c1c13e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c216dd640 1 -- 172.21.15.46:0/505759109 >> v1:172.21.15.114:6789/0 conn(0x7f0c1c0a3c10 legacy=0x7f0c1c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c216dd640 10 monclient: _finish_hunting 0 2024-09-16T04:42:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c216dd640 1 monclient: found mon.noname-b 2024-09-16T04:42:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c216dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c216dd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:42:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.695+0000 7f0c216dd640 1 -- 172.21.15.46:0/505759109 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c1c09b310 con 0x7f0c1c0a3ff0 2024-09-16T04:42:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c216dd640 1 --2- 172.21.15.46:0/505759109 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c0a3ff0 0x7f0c1c0a43d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0c10009870 tx=0x7f0c1002a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae682697b5dd6e61 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:56.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c20edc640 1 -- 172.21.15.46:0/505759109 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c10037070 con 0x7f0c1c0a3ff0 2024-09-16T04:42:56.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c20edc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c20edc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:42:56.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c20edc640 10 monclient: dump: 2024-09-16T04:42:56.713 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.713 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.713 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.713 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.713 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.713 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.713 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.713 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.713 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.714 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c20edc640 10 monclient: _finish_auth 0 2024-09-16T04:42:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c20edc640 10 monclient: _check_auth_tickets 2024-09-16T04:42:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c20edc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c20edc640 1 -- 172.21.15.46:0/505759109 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c10032440 con 0x7f0c1c0a3ff0 2024-09-16T04:42:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c24142640 5 monclient: authenticate success, global_id 54137 2024-09-16T04:42:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c24142640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c20edc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c24142640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c20edc640 1 -- 172.21.15.46:0/505759109 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c10032810 con 0x7f0c1c0a3ff0 2024-09-16T04:42:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c20edc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c24142640 4 set_mon_vals no callback set 2024-09-16T04:42:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c20edc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c20edc640 10 monclient: dump: 2024-09-16T04:42:56.715 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.715 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.715 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.715 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.715 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.715 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.715 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.715 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.715 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.715 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c24142640 10 monclient: shutdown 2024-09-16T04:42:56.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c24142640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:56.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c24142640 1 -- 172.21.15.46:0/505759109 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c0a3ff0 msgr2=0x7f0c1c0a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.696+0000 7f0c24142640 1 --2- 172.21.15.46:0/505759109 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c0a3ff0 0x7f0c1c0a43d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0c10009870 tx=0x7f0c1002a1c0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.697+0000 7f0c24142640 1 -- 172.21.15.46:0/505759109 shutdown_connections 2024-09-16T04:42:56.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.697+0000 7f0c24142640 1 --2- 172.21.15.46:0/505759109 >> v2:172.21.15.114:3300/0 conn(0x7f0c1c13be90 0x7f0c1c13e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.697+0000 7f0c24142640 1 --2- 172.21.15.46:0/505759109 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c0a3ff0 0x7f0c1c0a43d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.697+0000 7f0c24142640 1 -- 172.21.15.46:0/505759109 >> 172.21.15.46:0/505759109 conn(0x7f0c1c0a6340 msgr2=0x7f0c1c06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:56.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.697+0000 7f0c24142640 1 -- 172.21.15.46:0/505759109 shutdown_connections 2024-09-16T04:42:56.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.697+0000 7f0c24142640 1 -- 172.21.15.46:0/505759109 wait complete. 2024-09-16T04:42:56.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.697+0000 7f0c24142640 10 monclient: build_initial_monmap 2024-09-16T04:42:56.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.697+0000 7f0c24142640 10 monclient: monmap: 2024-09-16T04:42:56.716 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:56.716 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:56.717 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:56.717 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:56.717 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:56.717 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.717 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:56.717 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:56.717 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:56.717 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c24142640 1 Processor -- start 2024-09-16T04:42:56.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c24142640 1 -- start start 2024-09-16T04:42:56.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c24142640 10 monclient: init 2024-09-16T04:42:56.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c24142640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:56.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c24142640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:42:56.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c24142640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c1c0a3c10 0x7f0c1c133df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c24142640 10 monclient(hunting): picked mon.noname-b con 0x7f0c1c0a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:56.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c24142640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c0a3ff0 0x7f0c1c134330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c24142640 10 monclient(hunting): picked mon.noname-c con 0x7f0c1c0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:56.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c24142640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c1c13be90 0x7f0c1c1cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c24142640 10 monclient(hunting): picked mon.noname-a con 0x7f0c1c13be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:56.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c24142640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c24142640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c0a6b40 con 0x7f0c1c13be90 2024-09-16T04:42:56.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c24142640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c24142640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c0a69c0 con 0x7f0c1c0a3c10 2024-09-16T04:42:56.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c24142640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c24142640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f0c1c13e7c0 con 0x7f0c1c0a3ff0 2024-09-16T04:42:56.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c24142640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:56.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c24142640 10 monclient(hunting): authenticate will time out at 975.114532s 2024-09-16T04:42:56.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c216dd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c0a3ff0 0x7f0c1c134330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c226df640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c1c13be90 0x7f0c1c1cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c226df640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c1c13be90 0x7f0c1c1cbbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51550/0 (socket says 172.21.15.46:51550) 2024-09-16T04:42:56.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c216dd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c0a3ff0 0x7f0c1c134330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:54294/0 (socket says 172.21.15.46:54294) 2024-09-16T04:42:56.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c226df640 1 -- 172.21.15.46:0/4193959776 learned_addr learned my addr 172.21.15.46:0/4193959776 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:56.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c226df640 10 monclient(hunting): get_auth_request con 0x7f0c1c13be90 auth_method 0 2024-09-16T04:42:56.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c226df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c226df640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c226df640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c216dd640 10 monclient(hunting): get_auth_request con 0x7f0c1c0a3ff0 auth_method 0 2024-09-16T04:42:56.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c216dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c216dd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c216dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c21ede640 1 --2- 172.21.15.46:0/4193959776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c1c0a3c10 0x7f0c1c133df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c21ede640 10 monclient(hunting): get_auth_request con 0x7f0c1c0a3c10 auth_method 0 2024-09-16T04:42:56.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c21ede640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c21ede640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c21ede640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c226df640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c226df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c226df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c216dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c216dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c216dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c21ede640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c21ede640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c21ede640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c226df640 10 monclient(hunting): handle_auth_done global_id 44427 payload 824 2024-09-16T04:42:56.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c226df640 1 -- 172.21.15.46:0/4193959776 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c0a3ff0 msgr2=0x7f0c1c134330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c226df640 1 --2- 172.21.15.46:0/4193959776 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c0a3ff0 0x7f0c1c134330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c226df640 1 -- 172.21.15.46:0/4193959776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c1c0a3c10 msgr2=0x7f0c1c133df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c226df640 1 --2- 172.21.15.46:0/4193959776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c1c0a3c10 0x7f0c1c133df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c226df640 10 monclient: _finish_hunting 0 2024-09-16T04:42:56.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c226df640 1 monclient: found mon.noname-a 2024-09-16T04:42:56.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c226df640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:56.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.698+0000 7f0c226df640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:56.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c226df640 1 -- 172.21.15.46:0/4193959776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c1c13a7f0 con 0x7f0c1c13be90 2024-09-16T04:42:56.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c216dd640 1 --2- 172.21.15.46:0/4193959776 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c0a3ff0 0x7f0c1c134330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:42:56.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c226df640 1 --2- 172.21.15.46:0/4193959776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c1c13be90 0x7f0c1c1cbbe0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0c14007d40 tx=0x7f0c1400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:56.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c0e7fc640 1 -- 172.21.15.46:0/4193959776 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c14014420 con 0x7f0c1c13be90 2024-09-16T04:42:56.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c0e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:56.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c0e7fc640 10 monclient: dump: 2024-09-16T04:42:56.722 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.722 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.722 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.723 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.723 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.723 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.723 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.723 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.723 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.723 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c0e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:42:56.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c0e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:42:56.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:56.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c0e7fc640 1 -- 172.21.15.46:0/4193959776 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c14019070 con 0x7f0c1c13be90 2024-09-16T04:42:56.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c24142640 5 monclient: authenticate success, global_id 44427 2024-09-16T04:42:56.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c24142640 10 monclient: _renew_subs 2024-09-16T04:42:56.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c24142640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:56.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c24142640 1 -- 172.21.15.46:0/4193959776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c1c139080 con 0x7f0c1c13be90 2024-09-16T04:42:56.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c0e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:56.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c0e7fc640 1 -- 172.21.15.46:0/4193959776 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c14014820 con 0x7f0c1c13be90 2024-09-16T04:42:56.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c0e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:56.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c0e7fc640 10 monclient: dump: 2024-09-16T04:42:56.724 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.724 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.724 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.724 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.724 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.724 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.725 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.725 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.725 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.725 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c0ffff640 4 set_mon_vals no callback set 2024-09-16T04:42:56.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c24142640 10 monclient: _renew_subs 2024-09-16T04:42:56.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c24142640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:56.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.699+0000 7f0c24142640 1 -- 172.21.15.46:0/4193959776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c1c09f910 con 0x7f0c1c13be90 2024-09-16T04:42:56.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c24142640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:56.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c24142640 1 --2- 172.21.15.46:0/4193959776 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c1cc120 0x7f0c1c1cc500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c24142640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c24142640 1 -- 172.21.15.46:0/4193959776 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f0c1c1cca40 con 0x7f0c1c1cc120 2024-09-16T04:42:56.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c0e7fc640 1 -- 172.21.15.46:0/4193959776 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f0c140149c0 con 0x7f0c1c13be90 2024-09-16T04:42:56.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c21ede640 1 --2- 172.21.15.46:0/4193959776 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c1cc120 0x7f0c1c1cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c21ede640 10 monclient: get_auth_request con 0x7f0c1c1cc120 auth_method 0 2024-09-16T04:42:56.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c21ede640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c21ede640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c21ede640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c21ede640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c21ede640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c21ede640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c0e7fc640 1 --2- 172.21.15.46:0/4193959776 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0c000383b0 0x7f0c0003a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c21ede640 10 monclient(hunting): handle_auth_done global_id 54143 payload 824 2024-09-16T04:42:56.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c0e7fc640 1 -- 172.21.15.46:0/4193959776 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 4738+0+0 (secure 0 0 0) 0x7f0c1404beb0 con 0x7f0c1c13be90 2024-09-16T04:42:56.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c21ede640 1 -- 172.21.15.46:0/4193959776 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c1c0a5730 con 0x7f0c1c1cc120 2024-09-16T04:42:56.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c21ede640 1 --2- 172.21.15.46:0/4193959776 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c1cc120 0x7f0c1c1cc500 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0c1800f540 tx=0x7f0c1800fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c216dd640 1 --2- 172.21.15.46:0/4193959776 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0c000383b0 0x7f0c0003a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c216dd640 10 monclient: get_auth_request con 0x7f0c000383b0 auth_method 0 2024-09-16T04:42:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c0e7fc640 1 -- 172.21.15.46:0/4193959776 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c180052e0 con 0x7f0c1c1cc120 2024-09-16T04:42:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c0e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.700+0000 7f0c0e7fc640 10 monclient: dump: 2024-09-16T04:42:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.728 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.701+0000 7f0c0e7fc640 1 -- 172.21.15.46:0/4193959776 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0c18018070 con 0x7f0c1c1cc120 2024-09-16T04:42:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.701+0000 7f0c0e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.701+0000 7f0c0e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.701+0000 7f0c0e7fc640 1 -- 172.21.15.46:0/4193959776 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c1cc120 msgr2=0x7f0c1c1cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.701+0000 7f0c0e7fc640 1 --2- 172.21.15.46:0/4193959776 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c1cc120 0x7f0c1c1cc500 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0c1800f540 tx=0x7f0c1800fa00 comp rx=0 tx=0).stop 2024-09-16T04:42:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.701+0000 7f0c0e7fc640 1 -- 172.21.15.46:0/4193959776 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c1cc120 msgr2=0x7f0c1c1cc500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.701+0000 7f0c0e7fc640 1 --2- 172.21.15.46:0/4193959776 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c1cc120 0x7f0c1c1cc500 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0c1800f540 tx=0x7f0c1800fa00 comp rx=0 tx=0).stop 2024-09-16T04:42:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.701+0000 7f0c216dd640 1 --2- 172.21.15.46:0/4193959776 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0c000383b0 0x7f0c0003a870 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0c10034000 tx=0x7f0c1002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:56.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c24142640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:42:56.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c24142640 1 --2- 172.21.15.46:0/4193959776 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c0a05e0 0x7f0c1c0a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c24142640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c24142640 1 -- 172.21.15.46:0/4193959776 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f0c1c0a43d0 con 0x7f0c1c0a05e0 2024-09-16T04:42:56.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c21ede640 1 --2- 172.21.15.46:0/4193959776 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c0a05e0 0x7f0c1c0a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c21ede640 10 monclient: get_auth_request con 0x7f0c1c0a05e0 auth_method 0 2024-09-16T04:42:56.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c21ede640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c21ede640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c21ede640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c21ede640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c21ede640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c21ede640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c21ede640 10 monclient(hunting): handle_auth_done global_id 54146 payload 824 2024-09-16T04:42:56.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c21ede640 1 -- 172.21.15.46:0/4193959776 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0c1c13c270 con 0x7f0c1c0a05e0 2024-09-16T04:42:56.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c21ede640 1 --2- 172.21.15.46:0/4193959776 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c0a05e0 0x7f0c1c0a09c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0c18017000 tx=0x7f0c18001080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:56.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c0e7fc640 1 -- 172.21.15.46:0/4193959776 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c180038c0 con 0x7f0c1c0a05e0 2024-09-16T04:42:56.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c0e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:42:56.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c0e7fc640 10 monclient: dump: 2024-09-16T04:42:56.732 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.732 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.732 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.732 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.732 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.733 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.733 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.733 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.733 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.733 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c0e7fc640 1 -- 172.21.15.46:0/4193959776 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f0c180038c0 con 0x7f0c1c0a05e0 2024-09-16T04:42:56.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c0e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:42:56.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c0e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:56.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c0e7fc640 1 -- 172.21.15.46:0/4193959776 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c0a05e0 msgr2=0x7f0c1c0a09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c0e7fc640 1 --2- 172.21.15.46:0/4193959776 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c0a05e0 0x7f0c1c0a09c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0c18017000 tx=0x7f0c18001080 comp rx=0 tx=0).stop 2024-09-16T04:42:56.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c0e7fc640 1 -- 172.21.15.46:0/4193959776 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c0a05e0 msgr2=0x7f0c1c0a09c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:56.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.707+0000 7f0c0e7fc640 1 --2- 172.21.15.46:0/4193959776 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0c1c0a05e0 0x7f0c1c0a09c0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0c18017000 tx=0x7f0c18001080 comp rx=0 tx=0).stop 2024-09-16T04:42:56.734 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:42:56.734 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:42:56.734 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:42:56.734 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:42:56.734 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 48, 2024-09-16T04:42:56.734 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 12527, 2024-09-16T04:42:56.734 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:42:56.734 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:42:56.734 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:42:56.735 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:42:56.735 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:56.735 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 6, 2024-09-16T04:42:56.735 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:56.735 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:42:56.735 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:42:56.735 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:56.735 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:56.735 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:56.735 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:56.735 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:56.736 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:56.736 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:56.736 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:56.736 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:56.736 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:56.736 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:56.736 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:56.736 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:42:56.736 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:42:56.736 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:56.736 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:56.737 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:56.737 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:56.737 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:56.737 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:56.737 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:56.737 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:56.737 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:56.737 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:56.737 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:56.738 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:56.738 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.738 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:42:56.738 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:42:56.738 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:42:56.738 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:42:56.738 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:42:56.738 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:42:56.738 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:56.738 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:56.738 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:42:56.739 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:42:56.739 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:42:56.739 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:42:56.739 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:42:56.739 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:42:56.739 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:42:56.739 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:56.739 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:42:56.739 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:56.739 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:56.739 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:56.740 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:56.740 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:56.740 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:56.740 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:56.740 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:56.740 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:56.740 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:56.740 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:56.740 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:56.741 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:42:56.741 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.741 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:42:56.741 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.741 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:42:56.741 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:42:56.741 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:56.741 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:56.741 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.741 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:56.741 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:42:56.742 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:56.742 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.742 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.742 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:56.742 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:42:56.742 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:56.742 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:56.742 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:56.742 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.742 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:42:56.743 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:42:56.743 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:56.743 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:56.743 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:56.743 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.743 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.743 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:42:56.743 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:42:56.743 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:56.743 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:56.743 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.744 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:56.744 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:42:56.744 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:56.744 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.744 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.744 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:56.744 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:42:56.744 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:56.744 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:56.744 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:56.744 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.745 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:42:56.745 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:42:56.745 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:56.745 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:56.745 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:56.745 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.745 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.746 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:42:56.746 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:42:56.746 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:56.746 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:56.746 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.746 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:56.746 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:42:56.747 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:56.747 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.747 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.747 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:56.747 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:42:56.747 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:56.747 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:56.749 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.749 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:42:56.749 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:42:56.749 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.749 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:56.749 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:56.749 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:42:56.749 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:56.749 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:56.749 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:42:56.749 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.750 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:56.750 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:56.750 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:42:56.750 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:56.750 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:56.750 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.750 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:42:56.750 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:42:56.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.708+0000 7f0c24142640 1 -- 172.21.15.46:0/4193959776 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0c000383b0 msgr2=0x7f0c0003a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.708+0000 7f0c24142640 1 --2- 172.21.15.46:0/4193959776 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0c000383b0 0x7f0c0003a870 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0c10034000 tx=0x7f0c1002e000 comp rx=0 tx=0).stop 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.708+0000 7f0c24142640 10 monclient: shutdown 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.708+0000 7f0c24142640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.708+0000 7f0c24142640 1 -- 172.21.15.46:0/4193959776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c1c13be90 msgr2=0x7f0c1c1cbbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.708+0000 7f0c24142640 1 --2- 172.21.15.46:0/4193959776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c1c13be90 0x7f0c1c1cbbe0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0c14007d40 tx=0x7f0c1400e4b0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.708+0000 7f0c226df640 1 -- 172.21.15.46:0/4193959776 reap_dead start 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.708+0000 7f0c226df640 1 -- 172.21.15.46:0/4193959776 reap_dead start 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.708+0000 7f0c24142640 1 -- 172.21.15.46:0/4193959776 shutdown_connections 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.708+0000 7f0c24142640 1 -- 172.21.15.46:0/4193959776 >> 172.21.15.46:0/4193959776 conn(0x7f0c1c0a6340 msgr2=0x7f0c1c13dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.708+0000 7f0c24142640 1 -- 172.21.15.46:0/4193959776 shutdown_connections 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.708+0000 7f0c24142640 1 -- 172.21.15.46:0/4193959776 wait complete. 2024-09-16T04:42:56.753 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:42:56.753 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:42:56.753 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:42:56.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.853+0000 7f9a9348d640 10 monclient: get_monmap_and_config 2024-09-16T04:42:56.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.853+0000 7f9a9348d640 10 monclient: build_initial_monmap 2024-09-16T04:42:56.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.853+0000 7f9a9348d640 10 monclient: monmap: 2024-09-16T04:42:56.857 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:56.857 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.857 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:56.855116+0000 2024-09-16T04:42:56.857 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:56.855116+0000 2024-09-16T04:42:56.857 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:56.858 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.858 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:56.858 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:56.858 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:56.858 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:56.859 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:56.859 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.854+0000 7f9a9348d640 1 Processor -- start 2024-09-16T04:42:56.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.854+0000 7f9a9348d640 1 -- start start 2024-09-16T04:42:56.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.854+0000 7f9a9348d640 10 monclient: init 2024-09-16T04:42:56.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.854+0000 7f9a9348d640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:56.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.854+0000 7f9a9348d640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-16T04:42:56.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.854+0000 7f9a9348d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9a8c06e160 0x7f9a8c0a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.854+0000 7f9a9348d640 10 monclient(hunting): picked mon.noname-a con 0x7f9a8c06e160 addr v2:172.21.15.46:3300/0 2024-09-16T04:42:56.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.854+0000 7f9a9348d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a8c0a2ee0 addr v1:172.21.15.46:6789/0 2024-09-16T04:42:56.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.854+0000 7f9a9348d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9a8c137050 0x7f9a8c13ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.854+0000 7f9a9348d640 10 monclient(hunting): picked mon.noname-b con 0x7f9a8c137050 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.854+0000 7f9a9348d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a8c0a3fd0 con 0x7f9a8c0a2ee0 2024-09-16T04:42:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.854+0000 7f9a9348d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.854+0000 7f9a9348d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f9a8c0a3300 con 0x7f9a8c06e160 2024-09-16T04:42:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.854+0000 7f9a9348d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.854+0000 7f9a9348d640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f9a8c06e540 con 0x7f9a8c137050 2024-09-16T04:42:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.854+0000 7f9a9348d640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:56.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.854+0000 7f9a9348d640 10 monclient(hunting): authenticate will time out at 975.270944s 2024-09-16T04:42:56.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91229640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9a8c06e160 0x7f9a8c0a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91229640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9a8c06e160 0x7f9a8c0a29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51566/0 (socket says 172.21.15.46:51566) 2024-09-16T04:42:56.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a90a28640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9a8c0a2ee0 0x7f9a8c136940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60314/0 (socket says 172.21.15.46:60314) 2024-09-16T04:42:56.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91a2a640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9a8c137050 0x7f9a8c13ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91229640 1 -- 172.21.15.46:0/1860711797 learned_addr learned my addr 172.21.15.46:0/1860711797 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91229640 10 monclient(hunting): get_auth_request con 0x7f9a8c06e160 auth_method 0 2024-09-16T04:42:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91229640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91229640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91229640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91a2a640 10 monclient(hunting): get_auth_request con 0x7f9a8c137050 auth_method 0 2024-09-16T04:42:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91a2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91a2a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91a2a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91229640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91229640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91229640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91a2a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91a2a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91a2a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91229640 10 monclient(hunting): handle_auth_done global_id 44430 payload 274 2024-09-16T04:42:56.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91229640 1 -- 172.21.15.46:0/1860711797 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9a8c137050 msgr2=0x7f9a8c13ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:42:56.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91229640 1 --2- 172.21.15.46:0/1860711797 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9a8c137050 0x7f9a8c13ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91229640 1 -- 172.21.15.46:0/1860711797 >> v1:172.21.15.46:6789/0 conn(0x7f9a8c0a2ee0 legacy=0x7f9a8c136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91229640 10 monclient: _finish_hunting 0 2024-09-16T04:42:56.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91229640 1 monclient: found mon.noname-a 2024-09-16T04:42:56.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a7ffff640 1 -- 172.21.15.46:0/1860711797 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2633492451 0 0) 0x7f9a88001110 con 0x7f9a8c0a2ee0 2024-09-16T04:42:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91229640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91229640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91229640 1 -- 172.21.15.46:0/1860711797 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8c0a0bd0 con 0x7f9a8c06e160 2024-09-16T04:42:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a91a2a640 1 --2- 172.21.15.46:0/1860711797 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9a8c137050 0x7f9a8c13ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:42:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.855+0000 7f9a7ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:42:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a91229640 1 --2- 172.21.15.46:0/1860711797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c06e160 0x7f9a8c0a29a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9a800098a0 tx=0x7f9a8002a160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93de958624dd2df4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a7ffff640 1 -- 172.21.15.46:0/1860711797 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a80037070 con 0x7f9a8c06e160 2024-09-16T04:42:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a7ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a7ffff640 10 monclient: dump: 2024-09-16T04:42:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.881 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.881 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.881 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.881 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.881 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.881 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.881 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.882 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a7ffff640 10 monclient: _finish_auth 0 2024-09-16T04:42:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a7ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:42:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a9348d640 5 monclient: authenticate success, global_id 44430 2024-09-16T04:42:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a7ffff640 1 -- 172.21.15.46:0/1860711797 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a80032440 con 0x7f9a8c06e160 2024-09-16T04:42:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a9348d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:42:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a7ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a9348d640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a9348d640 4 set_mon_vals no callback set 2024-09-16T04:42:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a7ffff640 1 -- 172.21.15.46:0/1860711797 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a80032810 con 0x7f9a8c06e160 2024-09-16T04:42:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a7ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a7ffff640 10 monclient: dump: 2024-09-16T04:42:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.883 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a9348d640 10 monclient: shutdown 2024-09-16T04:42:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a9348d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a9348d640 1 -- 172.21.15.46:0/1860711797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c06e160 msgr2=0x7f9a8c0a29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a9348d640 1 --2- 172.21.15.46:0/1860711797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c06e160 0x7f9a8c0a29a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9a800098a0 tx=0x7f9a8002a160 comp rx=0 tx=0).stop 2024-09-16T04:42:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a9348d640 1 -- 172.21.15.46:0/1860711797 shutdown_connections 2024-09-16T04:42:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a9348d640 1 --2- 172.21.15.46:0/1860711797 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9a8c137050 0x7f9a8c13ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a9348d640 1 --2- 172.21.15.46:0/1860711797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c06e160 0x7f9a8c0a29a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.856+0000 7f9a9348d640 1 -- 172.21.15.46:0/1860711797 >> 172.21.15.46:0/1860711797 conn(0x7f9a8c0a5c10 msgr2=0x7f9a8c0a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.857+0000 7f9a9348d640 1 -- 172.21.15.46:0/1860711797 shutdown_connections 2024-09-16T04:42:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.857+0000 7f9a9348d640 1 -- 172.21.15.46:0/1860711797 wait complete. 2024-09-16T04:42:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.857+0000 7f9a9348d640 10 monclient: build_initial_monmap 2024-09-16T04:42:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.857+0000 7f9a9348d640 10 monclient: monmap: 2024-09-16T04:42:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:56.885 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:56.885 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:56.885 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:56.885 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.885 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:56.885 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:56.885 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:56.885 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.857+0000 7f9a9348d640 1 Processor -- start 2024-09-16T04:42:56.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a9348d640 1 -- start start 2024-09-16T04:42:56.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a9348d640 10 monclient: init 2024-09-16T04:42:56.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a9348d640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:56.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a9348d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:42:56.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a9348d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a8c06e160 0x7f9a8c135df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a9348d640 10 monclient(hunting): picked mon.noname-b con 0x7f9a8c06e160 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:56.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a9348d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c0a2ee0 0x7f9a8c136330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a9348d640 10 monclient(hunting): picked mon.noname-a con 0x7f9a8c0a2ee0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:56.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a9348d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9a8c137050 0x7f9a8c1238a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a9348d640 10 monclient(hunting): picked mon.noname-c con 0x7f9a8c137050 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:56.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a9348d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a9348d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c0a34a0 con 0x7f9a8c0a2ee0 2024-09-16T04:42:56.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a9348d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a9348d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c0a5130 con 0x7f9a8c06e160 2024-09-16T04:42:56.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a9348d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a9348d640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f9a8c1277f0 con 0x7f9a8c137050 2024-09-16T04:42:56.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a9348d640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a9348d640 10 monclient(hunting): authenticate will time out at 975.274509s 2024-09-16T04:42:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a90a28640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c0a2ee0 0x7f9a8c136330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a91a2a640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9a8c137050 0x7f9a8c1238a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a90a28640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c0a2ee0 0x7f9a8c136330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51582/0 (socket says 172.21.15.46:51582) 2024-09-16T04:42:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a90a28640 1 -- 172.21.15.46:0/2804205087 learned_addr learned my addr 172.21.15.46:0/2804205087 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a91a2a640 1 --2- 172.21.15.46:0/2804205087 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9a8c137050 0x7f9a8c1238a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:54334/0 (socket says 172.21.15.46:54334) 2024-09-16T04:42:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a90a28640 10 monclient(hunting): get_auth_request con 0x7f9a8c0a2ee0 auth_method 0 2024-09-16T04:42:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a90a28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a90a28640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a90a28640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a91a2a640 10 monclient(hunting): get_auth_request con 0x7f9a8c137050 auth_method 0 2024-09-16T04:42:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a91a2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a91a2a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a91a2a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a90a28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a90a28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a90a28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a91a2a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a91a2a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a91a2a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a91229640 1 --2- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a8c06e160 0x7f9a8c135df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a90a28640 10 monclient(hunting): handle_auth_done global_id 44436 payload 824 2024-09-16T04:42:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a90a28640 1 -- 172.21.15.46:0/2804205087 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9a8c137050 msgr2=0x7f9a8c1238a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a90a28640 1 --2- 172.21.15.46:0/2804205087 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9a8c137050 0x7f9a8c1238a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a90a28640 1 -- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a8c06e160 msgr2=0x7f9a8c135df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a90a28640 1 --2- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a8c06e160 0x7f9a8c135df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a90a28640 10 monclient: _finish_hunting 0 2024-09-16T04:42:56.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a90a28640 1 monclient: found mon.noname-a 2024-09-16T04:42:56.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a90a28640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:56.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a90a28640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:56.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a90a28640 1 -- 172.21.15.46:0/2804205087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8c0a8440 con 0x7f9a8c0a2ee0 2024-09-16T04:42:56.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a91229640 10 monclient: get_auth_request con 0x7f9a8c06e160 auth_method 0 2024-09-16T04:42:56.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.858+0000 7f9a91229640 1 --2- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a8c06e160 0x7f9a8c135df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:42:56.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a90a28640 1 --2- 172.21.15.46:0/2804205087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c0a2ee0 0x7f9a8c136330 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800c7b0 tx=0x7f9a8800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:56.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a7dffb640 1 -- 172.21.15.46:0/2804205087 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88018070 con 0x7f9a8c0a2ee0 2024-09-16T04:42:56.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a7dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:56.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a7dffb640 10 monclient: dump: 2024-09-16T04:42:56.889 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.889 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.890 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.890 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.890 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.890 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.890 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.890 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.890 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.890 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a7dffb640 10 monclient: _finish_auth 0 2024-09-16T04:42:56.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a7dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:42:56.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:56.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a7dffb640 1 -- 172.21.15.46:0/2804205087 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a880072d0 con 0x7f9a8c0a2ee0 2024-09-16T04:42:56.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a9348d640 5 monclient: authenticate success, global_id 44436 2024-09-16T04:42:56.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a7dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:56.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a7dffb640 1 -- 172.21.15.46:0/2804205087 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a880047e0 con 0x7f9a8c0a2ee0 2024-09-16T04:42:56.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a9348d640 10 monclient: _renew_subs 2024-09-16T04:42:56.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a9348d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:56.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a9348d640 1 -- 172.21.15.46:0/2804205087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a8c136960 con 0x7f9a8c0a2ee0 2024-09-16T04:42:56.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a7dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:56.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a7dffb640 10 monclient: dump: 2024-09-16T04:42:56.891 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.891 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.891 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.891 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.891 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.891 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.892 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.892 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.892 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.892 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a7f7fe640 4 set_mon_vals no callback set 2024-09-16T04:42:56.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a9348d640 10 monclient: _renew_subs 2024-09-16T04:42:56.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a9348d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:56.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.859+0000 7f9a9348d640 1 -- 172.21.15.46:0/2804205087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a8c127510 con 0x7f9a8c0a2ee0 2024-09-16T04:42:56.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.860+0000 7f9a7dffb640 1 -- 172.21.15.46:0/2804205087 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f9a8801b070 con 0x7f9a8c0a2ee0 2024-09-16T04:42:56.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.860+0000 7f9a9348d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:56.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.860+0000 7f9a9348d640 1 --2- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a8c1cbe90 0x7f9a8c123de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.860+0000 7f9a9348d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.860+0000 7f9a9348d640 1 -- 172.21.15.46:0/2804205087 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c0a3620 con 0x7f9a8c1cbe90 2024-09-16T04:42:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.860+0000 7f9a7dffb640 1 --2- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9a700383e0 0x7f9a7003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.860+0000 7f9a7dffb640 1 -- 172.21.15.46:0/2804205087 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 4738+0+0 (secure 0 0 0) 0x7f9a8804aa70 con 0x7f9a8c0a2ee0 2024-09-16T04:42:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.860+0000 7f9a91229640 1 --2- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a8c1cbe90 0x7f9a8c123de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.860+0000 7f9a91229640 10 monclient: get_auth_request con 0x7f9a8c1cbe90 auth_method 0 2024-09-16T04:42:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.860+0000 7f9a91229640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.860+0000 7f9a91229640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.860+0000 7f9a91229640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.860+0000 7f9a91a2a640 1 --2- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9a700383e0 0x7f9a7003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.860+0000 7f9a91229640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.860+0000 7f9a91229640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.860+0000 7f9a91229640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.860+0000 7f9a91a2a640 10 monclient: get_auth_request con 0x7f9a700383e0 auth_method 0 2024-09-16T04:42:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.860+0000 7f9a91229640 10 monclient(hunting): handle_auth_done global_id 5710 payload 824 2024-09-16T04:42:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.860+0000 7f9a91229640 1 -- 172.21.15.46:0/2804205087 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9a8c127030 con 0x7f9a8c1cbe90 2024-09-16T04:42:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.861+0000 7f9a91229640 1 --2- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a8c1cbe90 0x7f9a8c123de0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c09fec0 tx=0x7f9a800046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.861+0000 7f9a91a2a640 1 --2- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9a700383e0 0x7f9a7003a8a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c0a8210 tx=0x7f9a8400b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.861+0000 7f9a7dffb640 1 -- 172.21.15.46:0/2804205087 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a80041070 con 0x7f9a8c1cbe90 2024-09-16T04:42:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.861+0000 7f9a7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.861+0000 7f9a7dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:42:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.861+0000 7f9a7dffb640 10 monclient: dump: 2024-09-16T04:42:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.895 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.895 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.895 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.895 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.861+0000 7f9a7dffb640 1 -- 172.21.15.46:0/2804205087 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9a80009490 con 0x7f9a8c1cbe90 2024-09-16T04:42:56.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.861+0000 7f9a7dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:56.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.861+0000 7f9a7dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:56.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.861+0000 7f9a7dffb640 1 -- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a8c1cbe90 msgr2=0x7f9a8c123de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.861+0000 7f9a7dffb640 1 --2- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a8c1cbe90 0x7f9a8c123de0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c09fec0 tx=0x7f9a800046c0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.861+0000 7f9a7dffb640 1 -- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a8c1cbe90 msgr2=0x7f9a8c123de0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:56.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.861+0000 7f9a7dffb640 1 --2- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a8c1cbe90 0x7f9a8c123de0 secure :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c09fec0 tx=0x7f9a800046c0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.867+0000 7f9a9348d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:42:56.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.867+0000 7f9a9348d640 1 --2- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a8c124320 0x7f9a8c11f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:56.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.867+0000 7f9a9348d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:56.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.867+0000 7f9a9348d640 1 -- 172.21.15.46:0/2804205087 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c124730 con 0x7f9a8c124320 2024-09-16T04:42:56.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.868+0000 7f9a91229640 1 --2- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a8c124320 0x7f9a8c11f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:56.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.868+0000 7f9a91229640 10 monclient: get_auth_request con 0x7f9a8c124320 auth_method 0 2024-09-16T04:42:56.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.868+0000 7f9a91229640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:56.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.868+0000 7f9a91229640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:56.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.868+0000 7f9a91229640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:56.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.868+0000 7f9a91229640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:56.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.868+0000 7f9a91229640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:56.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.868+0000 7f9a91229640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:56.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.868+0000 7f9a91229640 10 monclient(hunting): handle_auth_done global_id 5713 payload 824 2024-09-16T04:42:56.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.868+0000 7f9a91229640 1 -- 172.21.15.46:0/2804205087 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9a8c1374c0 con 0x7f9a8c124320 2024-09-16T04:42:56.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.868+0000 7f9a91229640 1 --2- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a8c124320 0x7f9a8c11f760 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9a80000c00 tx=0x7f9a80008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:56.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.868+0000 7f9a7dffb640 1 -- 172.21.15.46:0/2804205087 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a80037040 con 0x7f9a8c124320 2024-09-16T04:42:56.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.868+0000 7f9a7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:56.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.868+0000 7f9a7dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:42:56.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.868+0000 7f9a7dffb640 10 monclient: dump: 2024-09-16T04:42:56.897 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:56.897 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:56.897 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.897 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:56.897 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:56.897 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:56.897 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:56.897 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:56.897 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:56.897 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:56.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.869+0000 7f9a7dffb640 1 -- 172.21.15.46:0/2804205087 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f9a8003cd70 con 0x7f9a8c124320 2024-09-16T04:42:56.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.869+0000 7f9a7dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:42:56.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.869+0000 7f9a7dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:56.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.869+0000 7f9a7dffb640 1 -- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a8c124320 msgr2=0x7f9a8c11f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.869+0000 7f9a7dffb640 1 --2- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a8c124320 0x7f9a8c11f760 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9a80000c00 tx=0x7f9a80008030 comp rx=0 tx=0).stop 2024-09-16T04:42:56.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.869+0000 7f9a7dffb640 1 -- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a8c124320 msgr2=0x7f9a8c11f760 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:42:56.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.869+0000 7f9a7dffb640 1 --2- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a8c124320 0x7f9a8c11f760 secure :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9a80000c00 tx=0x7f9a80008030 comp rx=0 tx=0).stop 2024-09-16T04:42:56.898 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:42:56.898 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:42:56.898 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:42:56.899 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:42:56.899 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 48, 2024-09-16T04:42:56.899 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 262771, 2024-09-16T04:42:56.899 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:42:56.899 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:42:56.899 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:42:56.899 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:42:56.899 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:56.899 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 6, 2024-09-16T04:42:56.899 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:56.899 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:42:56.899 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:42:56.899 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:56.900 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:56.900 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:56.900 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:56.900 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:56.900 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:56.900 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:56.900 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:56.900 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:56.900 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:56.900 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:56.900 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:56.900 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:42:56.900 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:42:56.900 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:56.901 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:56.901 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:56.901 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:56.901 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:56.901 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:56.901 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:56.901 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:56.901 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:56.901 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:56.901 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:56.901 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:56.901 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.901 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:42:56.902 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:42:56.902 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:42:56.902 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:42:56.902 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:42:56.902 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:42:56.902 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:56.902 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:42:56.902 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:42:56.902 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:42:56.902 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:42:56.902 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:42:56.902 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:42:56.902 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:42:56.902 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:42:56.903 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:42:56.903 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:42:56.903 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:42:56.903 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:42:56.903 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:42:56.903 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:42:56.903 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:42:56.903 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:42:56.903 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:42:56.903 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:42:56.903 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:42:56.903 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:42:56.903 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:42:56.903 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:56.904 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:42:56.904 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.904 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:42:56.904 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.904 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:42:56.904 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:42:56.904 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:56.904 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:56.904 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.904 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:56.904 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:42:56.904 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:56.904 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.905 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.905 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:56.905 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:42:56.905 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:56.905 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:56.905 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:56.905 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.905 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:42:56.905 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:42:56.905 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:56.905 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:56.905 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:56.905 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.906 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.906 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:42:56.906 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:42:56.906 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:56.906 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:56.906 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.906 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:56.906 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:42:56.906 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:56.906 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.906 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.906 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:56.906 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:42:56.906 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:56.907 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:56.907 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:56.907 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.907 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:42:56.907 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:42:56.907 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:56.907 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:56.907 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:56.907 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.907 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.907 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:42:56.907 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:42:56.907 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:42:56.908 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:42:56.908 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.908 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:42:56.908 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:42:56.908 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:56.908 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.908 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.908 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:42:56.908 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:42:56.908 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:42:56.908 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:56.908 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:56.908 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.909 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:42:56.909 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:42:56.909 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:42:56.909 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:42:56.909 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:42:56.909 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:56.909 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:56.909 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.909 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:42:56.909 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:42:56.909 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.909 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:56.909 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:56.909 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:42:56.910 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:56.910 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:56.910 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:42:56.910 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.910 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:56.910 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:56.910 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4 2024-09-16T04:42:56.910 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:56.910 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:56.910 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:42:56.910 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.910 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:56.910 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:56.911 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 7 2024-09-16T04:42:56.911 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:56.911 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:42:56.911 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:42:56.911 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:42:56.911 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:42:56.911 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:42:56.911 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:42:56.911 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:42:56.911 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:42:56.911 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:42:56.911 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:42:56.911 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:42:56.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.869+0000 7f9a9348d640 1 -- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9a700383e0 msgr2=0x7f9a7003a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.869+0000 7f9a9348d640 1 --2- 172.21.15.46:0/2804205087 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9a700383e0 0x7f9a7003a8a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c0a8210 tx=0x7f9a8400b040 comp rx=0 tx=0).stop 2024-09-16T04:42:56.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.869+0000 7f9a9348d640 10 monclient: shutdown 2024-09-16T04:42:56.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.869+0000 7f9a9348d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:56.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.869+0000 7f9a9348d640 1 -- 172.21.15.46:0/2804205087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c0a2ee0 msgr2=0x7f9a8c136330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:56.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.869+0000 7f9a9348d640 1 --2- 172.21.15.46:0/2804205087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c0a2ee0 0x7f9a8c136330 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800c7b0 tx=0x7f9a8800cbb0 comp rx=0 tx=0).stop 2024-09-16T04:42:56.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.869+0000 7f9a91a2a640 1 -- 172.21.15.46:0/2804205087 reap_dead start 2024-09-16T04:42:56.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.869+0000 7f9a91a2a640 1 -- 172.21.15.46:0/2804205087 reap_dead start 2024-09-16T04:42:56.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.869+0000 7f9a9348d640 1 -- 172.21.15.46:0/2804205087 shutdown_connections 2024-09-16T04:42:56.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.869+0000 7f9a9348d640 1 -- 172.21.15.46:0/2804205087 >> 172.21.15.46:0/2804205087 conn(0x7f9a8c0a5c10 msgr2=0x7f9a8c0a8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:56.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.869+0000 7f9a9348d640 1 -- 172.21.15.46:0/2804205087 shutdown_connections 2024-09-16T04:42:56.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:56.869+0000 7f9a9348d640 1 -- 172.21.15.46:0/2804205087 wait complete. 2024-09-16T04:42:56.914 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:42:56.914 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T04:42:56.914 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:42:56.914 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T04:42:56.914 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:42:56.915 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:42:56.915 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:42:56.924 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:42:56.924 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:42:56.924 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:42:57.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.062+0000 7f60bf326640 10 monclient: get_monmap_and_config 2024-09-16T04:42:57.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.062+0000 7f60bf326640 10 monclient: build_initial_monmap 2024-09-16T04:42:57.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.062+0000 7f60bf326640 10 monclient: monmap: 2024-09-16T04:42:57.064 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:57.064 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:57.064 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:42:57.063579+0000 2024-09-16T04:42:57.065 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:42:57.063579+0000 2024-09-16T04:42:57.065 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:57.065 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:57.065 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:42:57.065 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:42:57.065 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:42:57.065 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:42:57.065 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:42:57.065 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:57.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.062+0000 7f60bf326640 1 Processor -- start 2024-09-16T04:42:57.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.062+0000 7f60bf326640 1 -- start start 2024-09-16T04:42:57.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.062+0000 7f60bf326640 10 monclient: init 2024-09-16T04:42:57.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bf326640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:57.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bf326640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-16T04:42:57.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bf326640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f60b80a3c10 0x7f60b809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:57.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bf326640 10 monclient(hunting): picked mon.noname-b con 0x7f60b80a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:57.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bf326640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60b80a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:42:57.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bf326640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f60b813be90 0x7f60b813e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bf326640 10 monclient(hunting): picked mon.noname-a con 0x7f60b813be90 addr v2:172.21.15.46:3300/0 2024-09-16T04:42:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bf326640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60b809ab70 con 0x7f60b80a3ff0 2024-09-16T04:42:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bf326640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bf326640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f60b80a4d60 con 0x7f60b813be90 2024-09-16T04:42:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bf326640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bf326640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f60b809b030 con 0x7f60b80a3c10 2024-09-16T04:42:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bf326640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bf326640 10 monclient(hunting): authenticate will time out at 975.479413s 2024-09-16T04:42:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bd0c2640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f60b80a3c10 msgr2=0x7f60b809bcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:42:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bd0c2640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f60b80a3c10 0x7f60b809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bd8c3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f60b813be90 0x7f60b813e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bd8c3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f60b813be90 0x7f60b813e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51592/0 (socket says 172.21.15.46:51592) 2024-09-16T04:42:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bc8c1640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f60b80a3ff0 0x7f60b80a43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60330/0 (socket says 172.21.15.46:60330) 2024-09-16T04:42:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bd8c3640 1 -- 172.21.15.46:0/477099230 learned_addr learned my addr 172.21.15.46:0/477099230 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:57.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bd8c3640 10 monclient(hunting): get_auth_request con 0x7f60b813be90 auth_method 0 2024-09-16T04:42:57.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bd8c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:57.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bd8c3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:57.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bd8c3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:57.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bd8c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:57.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bd8c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:57.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bd8c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:57.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bd8c3640 10 monclient(hunting): handle_auth_done global_id 44439 payload 274 2024-09-16T04:42:57.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bd8c3640 1 -- 172.21.15.46:0/477099230 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f60b80a3c10 msgr2=0x7f60b809bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:42:57.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bd8c3640 1 --2- 172.21.15.46:0/477099230 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f60b80a3c10 0x7f60b809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:57.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bd8c3640 1 -- 172.21.15.46:0/477099230 >> v1:172.21.15.46:6789/0 conn(0x7f60b80a3ff0 legacy=0x7f60b80a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:57.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bd8c3640 10 monclient: _finish_hunting 0 2024-09-16T04:42:57.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bd8c3640 1 monclient: found mon.noname-a 2024-09-16T04:42:57.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bd8c3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:57.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bd8c3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:57.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60affff640 1 -- 172.21.15.46:0/477099230 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3717062966 0 0) 0x7f60b809ab70 con 0x7f60b80a3ff0 2024-09-16T04:42:57.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60bd8c3640 1 -- 172.21.15.46:0/477099230 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b809b310 con 0x7f60b813be90 2024-09-16T04:42:57.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.063+0000 7f60affff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:42:57.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.064+0000 7f60bd8c3640 1 --2- 172.21.15.46:0/477099230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b813be90 0x7f60b813e280 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f60b0009e50 tx=0x7f60b002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e5702150b6fc3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:57.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.064+0000 7f60affff640 1 -- 172.21.15.46:0/477099230 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b0036070 con 0x7f60b813be90 2024-09-16T04:42:57.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.064+0000 7f60affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:57.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.064+0000 7f60affff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:57.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.064+0000 7f60affff640 10 monclient: dump: 2024-09-16T04:42:57.078 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:57.079 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:57.079 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:57.079 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:57.079 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:57.079 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:57.079 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:57.079 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:57.079 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:57.079 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:57.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.064+0000 7f60affff640 10 monclient: _finish_auth 0 2024-09-16T04:42:57.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.064+0000 7f60affff640 10 monclient: _check_auth_tickets 2024-09-16T04:42:57.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.064+0000 7f60affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:57.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.064+0000 7f60bf326640 5 monclient: authenticate success, global_id 44439 2024-09-16T04:42:57.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.064+0000 7f60affff640 1 -- 172.21.15.46:0/477099230 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60b0031600 con 0x7f60b813be90 2024-09-16T04:42:57.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.064+0000 7f60affff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:57.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.064+0000 7f60bf326640 10 monclient: get_monmap_and_config success 2024-09-16T04:42:57.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.064+0000 7f60affff640 1 -- 172.21.15.46:0/477099230 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b00319d0 con 0x7f60b813be90 2024-09-16T04:42:57.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.064+0000 7f60bf326640 4 set_mon_vals no callback set 2024-09-16T04:42:57.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.064+0000 7f60affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:57.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.064+0000 7f60affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:57.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.064+0000 7f60affff640 10 monclient: dump: 2024-09-16T04:42:57.080 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:57.080 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:57.080 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:57.080 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:57.081 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:57.081 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:57.081 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:57.081 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:57.081 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:57.081 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:57.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.065+0000 7f60bf326640 10 monclient: shutdown 2024-09-16T04:42:57.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.065+0000 7f60bf326640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:57.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.065+0000 7f60bf326640 1 -- 172.21.15.46:0/477099230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b813be90 msgr2=0x7f60b813e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:57.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.065+0000 7f60bf326640 1 --2- 172.21.15.46:0/477099230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b813be90 0x7f60b813e280 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f60b0009e50 tx=0x7f60b002a610 comp rx=0 tx=0).stop 2024-09-16T04:42:57.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.065+0000 7f60bf326640 1 -- 172.21.15.46:0/477099230 shutdown_connections 2024-09-16T04:42:57.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.065+0000 7f60bf326640 1 --2- 172.21.15.46:0/477099230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b813be90 0x7f60b813e280 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:57.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.065+0000 7f60bf326640 1 --2- 172.21.15.46:0/477099230 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f60b80a3c10 0x7f60b809bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:57.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.065+0000 7f60bf326640 1 -- 172.21.15.46:0/477099230 >> 172.21.15.46:0/477099230 conn(0x7f60b80a5d10 msgr2=0x7f60b80a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:57.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.065+0000 7f60bf326640 1 -- 172.21.15.46:0/477099230 shutdown_connections 2024-09-16T04:42:57.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.065+0000 7f60bf326640 1 -- 172.21.15.46:0/477099230 wait complete. 2024-09-16T04:42:57.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.065+0000 7f60bf326640 10 monclient: build_initial_monmap 2024-09-16T04:42:57.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.065+0000 7f60bf326640 10 monclient: monmap: 2024-09-16T04:42:57.082 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:42:57.082 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:42:57.082 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:42:57.082 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:42:57.082 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:42:57.082 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:57.082 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:42:57.082 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:42:57.082 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:42:57.083 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:57.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bf326640 1 Processor -- start 2024-09-16T04:42:57.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bf326640 1 -- start start 2024-09-16T04:42:57.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bf326640 10 monclient: init 2024-09-16T04:42:57.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bf326640 10 monclient: _reopen_session rank -1 2024-09-16T04:42:57.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bf326640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:42:57.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bf326640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f60b80a3c10 0x7f60b81cba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:57.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bf326640 10 monclient(hunting): picked mon.noname-c con 0x7f60b80a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:42:57.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bf326640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b80a3ff0 0x7f60b8136e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:57.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bf326640 10 monclient(hunting): picked mon.noname-a con 0x7f60b80a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:42:57.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bf326640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60b813be90 0x7f60b8139ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:57.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bf326640 10 monclient(hunting): picked mon.noname-b con 0x7f60b813be90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:42:57.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bf326640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:57.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bf326640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f60b806e550 con 0x7f60b80a3ff0 2024-09-16T04:42:57.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bf326640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:57.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bf326640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f60b806e3d0 con 0x7f60b813be90 2024-09-16T04:42:57.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bf326640 10 monclient(hunting): start opening mon connection 2024-09-16T04:42:57.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bf326640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f60b813e7c0 con 0x7f60b80a3c10 2024-09-16T04:42:57.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bf326640 10 monclient(hunting): _renew_subs 2024-09-16T04:42:57.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bf326640 10 monclient(hunting): authenticate will time out at 975.482752s 2024-09-16T04:42:57.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bd0c2640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f60b80a3c10 msgr2=0x7f60b81cba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:42:57.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bd0c2640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f60b80a3c10 0x7f60b81cba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:42:57.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bc8c1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b80a3ff0 0x7f60b8136e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:57.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bc8c1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b80a3ff0 0x7f60b8136e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51594/0 (socket says 172.21.15.46:51594) 2024-09-16T04:42:57.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bc8c1640 1 -- 172.21.15.46:0/1915430049 learned_addr learned my addr 172.21.15.46:0/1915430049 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:42:57.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bc8c1640 10 monclient(hunting): get_auth_request con 0x7f60b80a3ff0 auth_method 0 2024-09-16T04:42:57.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bc8c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:57.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bc8c1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:57.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bc8c1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:57.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bc8c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:42:57.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bc8c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:42:57.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.066+0000 7f60bc8c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:42:57.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bd8c3640 1 --2- 172.21.15.46:0/1915430049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60b813be90 0x7f60b8139ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:57.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bd8c3640 10 monclient(hunting): get_auth_request con 0x7f60b813be90 auth_method 0 2024-09-16T04:42:57.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bd8c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:42:57.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bd8c3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:42:57.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bd8c3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:42:57.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bc8c1640 10 monclient(hunting): handle_auth_done global_id 44445 payload 824 2024-09-16T04:42:57.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bc8c1640 1 -- 172.21.15.46:0/1915430049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f60b80a3c10 msgr2=0x7f60b81cba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:42:57.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bc8c1640 1 --2- 172.21.15.46:0/1915430049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f60b80a3c10 0x7f60b81cba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:57.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bc8c1640 1 -- 172.21.15.46:0/1915430049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60b813be90 msgr2=0x7f60b8139ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:57.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bc8c1640 1 --2- 172.21.15.46:0/1915430049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60b813be90 0x7f60b8139ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:57.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bc8c1640 10 monclient: _finish_hunting 0 2024-09-16T04:42:57.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bc8c1640 1 monclient: found mon.noname-a 2024-09-16T04:42:57.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bc8c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:42:57.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bc8c1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:42:57.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bc8c1640 1 -- 172.21.15.46:0/1915430049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b80a0400 con 0x7f60b80a3ff0 2024-09-16T04:42:57.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bc8c1640 1 --2- 172.21.15.46:0/1915430049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b80a3ff0 0x7f60b8136e90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f60a800c4d0 tx=0x7f60a800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:57.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60adffb640 1 -- 172.21.15.46:0/1915430049 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a8018070 con 0x7f60b80a3ff0 2024-09-16T04:42:57.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:57.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60adffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:42:57.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60adffb640 10 monclient: dump: 2024-09-16T04:42:57.087 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:57.087 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:57.087 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:57.087 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:57.087 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:57.087 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:57.087 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:57.087 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:57.087 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:57.087 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:57.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60adffb640 10 monclient: _finish_auth 0 2024-09-16T04:42:57.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60adffb640 10 monclient: _check_auth_tickets 2024-09-16T04:42:57.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:42:57.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60adffb640 1 -- 172.21.15.46:0/1915430049 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60a8007500 con 0x7f60b80a3ff0 2024-09-16T04:42:57.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bf326640 5 monclient: authenticate success, global_id 44445 2024-09-16T04:42:57.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bf326640 10 monclient: _renew_subs 2024-09-16T04:42:57.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bf326640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:57.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bf326640 1 -- 172.21.15.46:0/1915430049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60b81cc900 con 0x7f60b80a3ff0 2024-09-16T04:42:57.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60adffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:42:57.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60adffb640 1 -- 172.21.15.46:0/1915430049 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a8004a20 con 0x7f60b80a3ff0 2024-09-16T04:42:57.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:42:57.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60adffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:42:57.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60adffb640 10 monclient: dump: 2024-09-16T04:42:57.088 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:42:57.088 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:42:57.088 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:57.089 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:42:57.089 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:42:57.089 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:42:57.089 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:42:57.089 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:42:57.089 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:42:57.089 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:42:57.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bf326640 10 monclient: _renew_subs 2024-09-16T04:42:57.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60aeffd640 4 set_mon_vals no callback set 2024-09-16T04:42:57.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bf326640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:57.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.067+0000 7f60bf326640 1 -- 172.21.15.46:0/1915430049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60b8121840 con 0x7f60b80a3ff0 2024-09-16T04:42:57.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.068+0000 7f60adffb640 1 -- 172.21.15.46:0/1915430049 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f60a801b070 con 0x7f60b80a3ff0 2024-09-16T04:42:57.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.068+0000 7f60bf326640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:57.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.068+0000 7f60bf326640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:57.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.068+0000 7f60bf326640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:57.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.068+0000 7f60bf326640 1 -- 172.21.15.46:0/1915430049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60b806d910 con 0x7f60b80a3ff0 2024-09-16T04:42:57.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.068+0000 7f60adffb640 1 --2- 172.21.15.46:0/1915430049 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f609c038360 0x7f609c03a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:42:57.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.068+0000 7f60adffb640 1 -- 172.21.15.46:0/1915430049 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 4738+0+0 (secure 0 0 0) 0x7f60a804ab80 con 0x7f60b80a3ff0 2024-09-16T04:42:57.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.069+0000 7f60bd0c2640 1 --2- 172.21.15.46:0/1915430049 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f609c038360 0x7f609c03a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:42:57.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.069+0000 7f60bd0c2640 10 monclient: get_auth_request con 0x7f609c038360 auth_method 0 2024-09-16T04:42:57.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.069+0000 7f60bd0c2640 1 --2- 172.21.15.46:0/1915430049 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f609c038360 0x7f609c03a820 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f60b4003980 tx=0x7f60b4006d30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:42:57.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.073+0000 7f60adffb640 1 -- 172.21.15.46:0/1915430049 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f60b80a0400 con 0x7f60b80a3ff0 2024-09-16T04:42:57.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.073+0000 7f60adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:42:57.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.073+0000 7f60adffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.316+0000 7f60bf326640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.316+0000 7f60bf326640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.316+0000 7f60bf326640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.316+0000 7f60bf326640 1 -- 172.21.15.46:0/1915430049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f60b809bcf0 con 0x7f60b80a3ff0 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.317+0000 7f60adffb640 1 -- 172.21.15.46:0/1915430049 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f60a804ae20 con 0x7f60b80a3ff0 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.317+0000 7f60adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.317+0000 7f60adffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.318+0000 7f60bf326640 1 -- 172.21.15.46:0/1915430049 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f609c038360 msgr2=0x7f609c03a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.318+0000 7f60bf326640 1 --2- 172.21.15.46:0/1915430049 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f609c038360 0x7f609c03a820 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f60b4003980 tx=0x7f60b4006d30 comp rx=0 tx=0).stop 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.318+0000 7f60bf326640 10 monclient: shutdown 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.318+0000 7f60bf326640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.318+0000 7f60bf326640 1 -- 172.21.15.46:0/1915430049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b80a3ff0 msgr2=0x7f60b8136e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.318+0000 7f60bf326640 1 --2- 172.21.15.46:0/1915430049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b80a3ff0 0x7f60b8136e90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f60a800c4d0 tx=0x7f60a800c990 comp rx=0 tx=0).stop 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.319+0000 7f60bf326640 1 -- 172.21.15.46:0/1915430049 shutdown_connections 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.319+0000 7f60bf326640 1 --2- 172.21.15.46:0/1915430049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60b813be90 0x7f60b8139ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.319+0000 7f60bf326640 1 --2- 172.21.15.46:0/1915430049 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f609c038360 0x7f609c03a820 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:57.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.319+0000 7f60bf326640 1 --2- 172.21.15.46:0/1915430049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b80a3ff0 0x7f60b8136e90 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:57.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.319+0000 7f60bf326640 1 --2- 172.21.15.46:0/1915430049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f60b80a3c10 0x7f60b81cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:42:57.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.319+0000 7f60bf326640 1 -- 172.21.15.46:0/1915430049 >> 172.21.15.46:0/1915430049 conn(0x7f60b80a5d10 msgr2=0x7f60b809efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:42:57.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.319+0000 7f60bf326640 1 -- 172.21.15.46:0/1915430049 shutdown_connections 2024-09-16T04:42:57.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:42:57.319+0000 7f60bf326640 1 -- 172.21.15.46:0/1915430049 wait complete. 2024-09-16T04:43:00.332 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:43:00.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.443+0000 7f313cdc4640 10 monclient: get_monmap_and_config 2024-09-16T04:43:00.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.443+0000 7f313cdc4640 10 monclient: build_initial_monmap 2024-09-16T04:43:00.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.443+0000 7f313cdc4640 10 monclient: monmap: 2024-09-16T04:43:00.445 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:00.445 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:00.445 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:00.444875+0000 2024-09-16T04:43:00.446 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:00.444875+0000 2024-09-16T04:43:00.446 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:00.446 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:00.446 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:00.446 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:00.446 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:00.446 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:00.446 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:00.446 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:00.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.443+0000 7f313cdc4640 1 Processor -- start 2024-09-16T04:43:00.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f313cdc4640 1 -- start start 2024-09-16T04:43:00.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f313cdc4640 10 monclient: init 2024-09-16T04:43:00.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f313cdc4640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:00.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f313cdc4640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-16T04:43:00.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f313cdc4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31380a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:43:00.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f313cdc4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f31380a3ff0 0x7f31380a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:00.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f313cdc4640 10 monclient(hunting): picked mon.noname-a con 0x7f31380a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:43:00.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f313cdc4640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f313813be90 0x7f313813e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:00.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f313cdc4640 10 monclient(hunting): picked mon.noname-c con 0x7f313813be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:43:00.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f313cdc4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f313809b940 con 0x7f31380a3c10 2024-09-16T04:43:00.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f313cdc4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:00.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f313cdc4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f313809ab00 con 0x7f31380a3ff0 2024-09-16T04:43:00.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f313cdc4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:00.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f313cdc4640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f313809b030 con 0x7f313813be90 2024-09-16T04:43:00.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f313cdc4640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:00.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f313cdc4640 10 monclient(hunting): authenticate will time out at 978.860770s 2024-09-16T04:43:00.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f3135d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f31380a3ff0 0x7f31380a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:00.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f3135d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f31380a3ff0 0x7f31380a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51598/0 (socket says 172.21.15.46:51598) 2024-09-16T04:43:00.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f3135d9b640 1 -- 172.21.15.46:0/4088244522 learned_addr learned my addr 172.21.15.46:0/4088244522 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:00.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f3135d9b640 10 monclient(hunting): get_auth_request con 0x7f31380a3ff0 auth_method 0 2024-09-16T04:43:00.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f313659c640 1 --1- 172.21.15.46:0/4088244522 >> v1:172.21.15.46:6789/0 conn(0x7f31380a3c10 0x7f313809bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60340/0 (socket says 172.21.15.46:60340) 2024-09-16T04:43:00.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f3135d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:00.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f3135d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:00.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.444+0000 7f3135d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:00.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.445+0000 7f3136d9d640 1 --2- 172.21.15.46:0/4088244522 >> v2:172.21.15.114:3300/0 conn(0x7f313813be90 0x7f313813e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:00.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.445+0000 7f3135d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:00.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.445+0000 7f3135d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:00.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.445+0000 7f3135d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:00.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.445+0000 7f3136d9d640 10 monclient(hunting): get_auth_request con 0x7f313813be90 auth_method 0 2024-09-16T04:43:00.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.445+0000 7f3136d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:00.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.445+0000 7f3136d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:00.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.445+0000 7f3136d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:00.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.445+0000 7f3136d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.445+0000 7f3136d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.445+0000 7f3136d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.445+0000 7f3135d9b640 10 monclient(hunting): handle_auth_done global_id 44448 payload 274 2024-09-16T04:43:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.445+0000 7f3135d9b640 1 -- 172.21.15.46:0/4088244522 >> v2:172.21.15.114:3300/0 conn(0x7f313813be90 msgr2=0x7f313813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:43:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.445+0000 7f3135d9b640 1 --2- 172.21.15.46:0/4088244522 >> v2:172.21.15.114:3300/0 conn(0x7f313813be90 0x7f313813e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.445+0000 7f3135d9b640 1 -- 172.21.15.46:0/4088244522 >> v1:172.21.15.46:6789/0 conn(0x7f31380a3c10 legacy=0x7f313809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.445+0000 7f3135d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:43:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.445+0000 7f3135d9b640 1 monclient: found mon.noname-a 2024-09-16T04:43:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.445+0000 7f3135d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.445+0000 7f3135d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:43:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.445+0000 7f3135d9b640 1 -- 172.21.15.46:0/4088244522 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f313809b310 con 0x7f31380a3ff0 2024-09-16T04:43:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f3135d9b640 1 --2- 172.21.15.46:0/4088244522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31380a3ff0 0x7f31380a43d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f313806ee70 tx=0x7f312c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5fa68f63e2ebff server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313559a640 1 -- 172.21.15.46:0/4088244522 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312c037070 con 0x7f31380a3ff0 2024-09-16T04:43:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:43:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313559a640 10 monclient: dump: 2024-09-16T04:43:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:00.457 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313559a640 10 monclient: _finish_auth 0 2024-09-16T04:43:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313559a640 10 monclient: _check_auth_tickets 2024-09-16T04:43:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313559a640 1 -- 172.21.15.46:0/4088244522 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f312c032440 con 0x7f31380a3ff0 2024-09-16T04:43:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313cdc4640 5 monclient: authenticate success, global_id 44448 2024-09-16T04:43:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313cdc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313559a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313cdc4640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313559a640 1 -- 172.21.15.46:0/4088244522 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312c032810 con 0x7f31380a3ff0 2024-09-16T04:43:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313cdc4640 4 set_mon_vals no callback set 2024-09-16T04:43:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313559a640 10 monclient: dump: 2024-09-16T04:43:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:00.459 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:00.459 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:00.459 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:00.459 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:00.459 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:00.460 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:00.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313cdc4640 10 monclient: shutdown 2024-09-16T04:43:00.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313cdc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:00.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313cdc4640 1 -- 172.21.15.46:0/4088244522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31380a3ff0 msgr2=0x7f31380a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:00.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313cdc4640 1 --2- 172.21.15.46:0/4088244522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31380a3ff0 0x7f31380a43d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f313806ee70 tx=0x7f312c02a1e0 comp rx=0 tx=0).stop 2024-09-16T04:43:00.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313cdc4640 1 -- 172.21.15.46:0/4088244522 shutdown_connections 2024-09-16T04:43:00.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313cdc4640 1 --2- 172.21.15.46:0/4088244522 >> v2:172.21.15.114:3300/0 conn(0x7f313813be90 0x7f313813e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:00.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313cdc4640 1 --2- 172.21.15.46:0/4088244522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31380a3ff0 0x7f31380a43d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:00.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313cdc4640 1 -- 172.21.15.46:0/4088244522 >> 172.21.15.46:0/4088244522 conn(0x7f31380a6340 msgr2=0x7f313806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:00.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313cdc4640 1 -- 172.21.15.46:0/4088244522 shutdown_connections 2024-09-16T04:43:00.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.446+0000 7f313cdc4640 1 -- 172.21.15.46:0/4088244522 wait complete. 2024-09-16T04:43:00.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.447+0000 7f313cdc4640 10 monclient: build_initial_monmap 2024-09-16T04:43:00.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.447+0000 7f313cdc4640 10 monclient: monmap: 2024-09-16T04:43:00.461 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:00.461 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:00.461 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:00.461 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:00.461 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:00.461 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:00.461 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:00.461 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:00.461 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:00.461 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:00.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.447+0000 7f313cdc4640 1 Processor -- start 2024-09-16T04:43:00.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313cdc4640 1 -- start start 2024-09-16T04:43:00.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313cdc4640 10 monclient: init 2024-09-16T04:43:00.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313cdc4640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:00.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313cdc4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:43:00.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313cdc4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31380a3c10 0x7f31381cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:00.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313cdc4640 10 monclient(hunting): picked mon.noname-b con 0x7f31380a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:00.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313cdc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31380a3ff0 0x7f31381cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:00.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313cdc4640 10 monclient(hunting): picked mon.noname-a con 0x7f31380a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:00.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313cdc4640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f313813be90 0x7f31381cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:00.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313cdc4640 10 monclient(hunting): picked mon.noname-c con 0x7f313813be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:00.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313cdc4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:00.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313cdc4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f31380a6b40 con 0x7f31380a3ff0 2024-09-16T04:43:00.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313cdc4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:00.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313cdc4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f31380a69c0 con 0x7f31380a3c10 2024-09-16T04:43:00.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313cdc4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:00.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313cdc4640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f313813e7c0 con 0x7f313813be90 2024-09-16T04:43:00.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313cdc4640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:00.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313cdc4640 10 monclient(hunting): authenticate will time out at 978.864494s 2024-09-16T04:43:00.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3136d9d640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f313813be90 msgr2=0x7f31381cc3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:43:00.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3136d9d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f313813be90 0x7f31381cc3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:00.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3135d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31380a3ff0 0x7f31381cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:00.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3135d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31380a3ff0 0x7f31381cbe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51612/0 (socket says 172.21.15.46:51612) 2024-09-16T04:43:00.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3135d9b640 1 -- 172.21.15.46:0/2604998672 learned_addr learned my addr 172.21.15.46:0/2604998672 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:00.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3135d9b640 10 monclient(hunting): get_auth_request con 0x7f31380a3ff0 auth_method 0 2024-09-16T04:43:00.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3135d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3135d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3135d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3135d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3135d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3135d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313659c640 1 --2- 172.21.15.46:0/2604998672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31380a3c10 0x7f31381cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313659c640 10 monclient(hunting): get_auth_request con 0x7f31380a3c10 auth_method 0 2024-09-16T04:43:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3135d9b640 10 monclient(hunting): handle_auth_done global_id 44454 payload 824 2024-09-16T04:43:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3135d9b640 1 -- 172.21.15.46:0/2604998672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f313813be90 msgr2=0x7f31381cc3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:43:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3135d9b640 1 --2- 172.21.15.46:0/2604998672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f313813be90 0x7f31381cc3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3135d9b640 1 -- 172.21.15.46:0/2604998672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31380a3c10 msgr2=0x7f31381cb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3135d9b640 1 --2- 172.21.15.46:0/2604998672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31380a3c10 0x7f31381cb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3135d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:43:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3135d9b640 1 monclient: found mon.noname-a 2024-09-16T04:43:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3135d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3135d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:43:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f3135d9b640 1 -- 172.21.15.46:0/2604998672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f313813a3f0 con 0x7f31380a3ff0 2024-09-16T04:43:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.448+0000 7f313659c640 1 --2- 172.21.15.46:0/2604998672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31380a3c10 0x7f31381cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:43:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f3135d9b640 1 --2- 172.21.15.46:0/2604998672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31380a3ff0 0x7f31381cbe80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f312c02a220 tx=0x7f312c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f3126ffd640 1 -- 172.21.15.46:0/2604998672 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312c037040 con 0x7f31380a3ff0 2024-09-16T04:43:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f3126ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f3126ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:43:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f3126ffd640 10 monclient: dump: 2024-09-16T04:43:00.466 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:00.466 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:00.466 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:00.466 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:00.466 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:00.466 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:00.466 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:00.466 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:00.466 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:00.466 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:00.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f3126ffd640 10 monclient: _finish_auth 0 2024-09-16T04:43:00.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f3126ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:43:00.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f3126ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:00.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f3126ffd640 1 -- 172.21.15.46:0/2604998672 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f312c042070 con 0x7f31380a3ff0 2024-09-16T04:43:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f3126ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f3126ffd640 1 -- 172.21.15.46:0/2604998672 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312c030880 con 0x7f31380a3ff0 2024-09-16T04:43:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f3126ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f3126ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f3126ffd640 10 monclient: dump: 2024-09-16T04:43:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:00.468 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f313cdc4640 5 monclient: authenticate success, global_id 44454 2024-09-16T04:43:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f313cdc4640 10 monclient: _renew_subs 2024-09-16T04:43:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f3134d99640 4 set_mon_vals no callback set 2024-09-16T04:43:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f313cdc4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f313cdc4640 1 -- 172.21.15.46:0/2604998672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f313813c850 con 0x7f31380a3ff0 2024-09-16T04:43:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f313cdc4640 10 monclient: _renew_subs 2024-09-16T04:43:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f313cdc4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.449+0000 7f313cdc4640 1 -- 172.21.15.46:0/2604998672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f313813a6d0 con 0x7f31380a3ff0 2024-09-16T04:43:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.450+0000 7f3126ffd640 1 -- 172.21.15.46:0/2604998672 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f312c030a20 con 0x7f31380a3ff0 2024-09-16T04:43:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.450+0000 7f313cdc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.450+0000 7f313cdc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.450+0000 7f313cdc4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.450+0000 7f313cdc4640 1 -- 172.21.15.46:0/2604998672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31380a43d0 con 0x7f31380a3ff0 2024-09-16T04:43:00.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.450+0000 7f3126ffd640 1 --2- 172.21.15.46:0/2604998672 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3118038360 0x7f311803a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:00.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.450+0000 7f3126ffd640 1 -- 172.21.15.46:0/2604998672 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 4738+0+0 (secure 0 0 0) 0x7f312c06bfc0 con 0x7f31380a3ff0 2024-09-16T04:43:00.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.450+0000 7f313659c640 1 --2- 172.21.15.46:0/2604998672 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3118038360 0x7f311803a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:00.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.450+0000 7f313659c640 10 monclient: get_auth_request con 0x7f3118038360 auth_method 0 2024-09-16T04:43:00.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.451+0000 7f313659c640 1 --2- 172.21.15.46:0/2604998672 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3118038360 0x7f311803a820 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3128007fd0 tx=0x7f3128007d60 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:00.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.455+0000 7f3126ffd640 1 -- 172.21.15.46:0/2604998672 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f312c03da70 con 0x7f31380a3ff0 2024-09-16T04:43:00.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.455+0000 7f3126ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:00.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.455+0000 7f3126ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:00.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.702+0000 7f313cdc4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:43:00.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.702+0000 7f313cdc4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:00.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.702+0000 7f313cdc4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:00.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.702+0000 7f313cdc4640 1 -- 172.21.15.46:0/2604998672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f31380a1580 con 0x7f31380a3ff0 2024-09-16T04:43:00.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.702+0000 7f3126ffd640 1 -- 172.21.15.46:0/2604998672 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f312c030dc0 con 0x7f31380a3ff0 2024-09-16T04:43:00.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.702+0000 7f3126ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:00.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.702+0000 7f3126ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:00.705 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:43:00.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.704+0000 7f313cdc4640 1 -- 172.21.15.46:0/2604998672 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3118038360 msgr2=0x7f311803a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:00.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.704+0000 7f313cdc4640 1 --2- 172.21.15.46:0/2604998672 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3118038360 0x7f311803a820 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3128007fd0 tx=0x7f3128007d60 comp rx=0 tx=0).stop 2024-09-16T04:43:00.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.704+0000 7f313cdc4640 10 monclient: shutdown 2024-09-16T04:43:00.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.704+0000 7f313cdc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:00.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.704+0000 7f313cdc4640 1 -- 172.21.15.46:0/2604998672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31380a3ff0 msgr2=0x7f31381cbe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:00.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.704+0000 7f313cdc4640 1 --2- 172.21.15.46:0/2604998672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31380a3ff0 0x7f31381cbe80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f312c02a220 tx=0x7f312c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:43:00.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.704+0000 7f313cdc4640 1 -- 172.21.15.46:0/2604998672 shutdown_connections 2024-09-16T04:43:00.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.704+0000 7f313cdc4640 1 --2- 172.21.15.46:0/2604998672 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f313813be90 0x7f31381cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:00.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.704+0000 7f313cdc4640 1 --2- 172.21.15.46:0/2604998672 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3118038360 0x7f311803a820 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:00.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.704+0000 7f313cdc4640 1 --2- 172.21.15.46:0/2604998672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31380a3ff0 0x7f31381cbe80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:00.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.704+0000 7f313cdc4640 1 --2- 172.21.15.46:0/2604998672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31380a3c10 0x7f31381cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:00.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.704+0000 7f313cdc4640 1 -- 172.21.15.46:0/2604998672 >> 172.21.15.46:0/2604998672 conn(0x7f31380a6340 msgr2=0x7f31380a7110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:00.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.705+0000 7f313cdc4640 1 -- 172.21.15.46:0/2604998672 shutdown_connections 2024-09-16T04:43:00.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:00.705+0000 7f313cdc4640 1 -- 172.21.15.46:0/2604998672 wait complete. 2024-09-16T04:43:03.719 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:43:03.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.829+0000 7f14b0fd6640 10 monclient: get_monmap_and_config 2024-09-16T04:43:03.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.829+0000 7f14b0fd6640 10 monclient: build_initial_monmap 2024-09-16T04:43:03.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.829+0000 7f14b0fd6640 10 monclient: monmap: 2024-09-16T04:43:03.832 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:03.832 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:03.832 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:03.831368+0000 2024-09-16T04:43:03.832 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:03.831368+0000 2024-09-16T04:43:03.832 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:03.832 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:03.832 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:03.832 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:03.833 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:03.833 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:03.833 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:03.833 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.830+0000 7f14b0fd6640 1 Processor -- start 2024-09-16T04:43:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.830+0000 7f14b0fd6640 1 -- start start 2024-09-16T04:43:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.830+0000 7f14b0fd6640 10 monclient: init 2024-09-16T04:43:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14b0fd6640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14b0fd6640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-16T04:43:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14b0fd6640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f14ac0a3c10 0x7f14ac09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14b0fd6640 10 monclient(hunting): picked mon.noname-c con 0x7f14ac0a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:43:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14b0fd6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f14ac0a3ff0 0x7f14ac0a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14b0fd6640 10 monclient(hunting): picked mon.noname-a con 0x7f14ac0a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:43:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14b0fd6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f14ac1369d0 addr v1:172.21.15.114:6789/0 2024-09-16T04:43:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14b0fd6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14ac09b940 con 0x7f14ac1369d0 2024-09-16T04:43:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14b0fd6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14b0fd6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f14ac09ab00 con 0x7f14ac0a3ff0 2024-09-16T04:43:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14b0fd6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14b0fd6640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f14ac09b030 con 0x7f14ac0a3c10 2024-09-16T04:43:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14b0fd6640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14b0fd6640 10 monclient(hunting): authenticate will time out at 982.247265s 2024-09-16T04:43:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14a9d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f14ac0a3ff0 0x7f14ac0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14a9d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f14ac0a3ff0 0x7f14ac0a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51616/0 (socket says 172.21.15.46:51616) 2024-09-16T04:43:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14a9d9b640 1 -- 172.21.15.46:0/2127217209 learned_addr learned my addr 172.21.15.46:0/2127217209 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14a9d9b640 10 monclient(hunting): get_auth_request con 0x7f14ac0a3ff0 auth_method 0 2024-09-16T04:43:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14aa59c640 1 --2- 172.21.15.46:0/2127217209 >> v2:172.21.15.114:3300/0 conn(0x7f14ac0a3c10 0x7f14ac09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14aa59c640 10 monclient(hunting): get_auth_request con 0x7f14ac0a3c10 auth_method 0 2024-09-16T04:43:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14a9d9b640 10 monclient(hunting): handle_auth_done global_id 44457 payload 274 2024-09-16T04:43:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14a9d9b640 1 -- 172.21.15.46:0/2127217209 >> v2:172.21.15.114:3300/0 conn(0x7f14ac0a3c10 msgr2=0x7f14ac09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:43:03.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14a9d9b640 1 --2- 172.21.15.46:0/2127217209 >> v2:172.21.15.114:3300/0 conn(0x7f14ac0a3c10 0x7f14ac09bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:03.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14a9d9b640 1 -- 172.21.15.46:0/2127217209 >> v1:172.21.15.114:6789/0 conn(0x7f14ac1369d0 legacy=0x7f14ac136e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:43:03.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14a9d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:43:03.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14a9d9b640 1 monclient: found mon.noname-a 2024-09-16T04:43:03.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:03.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14a9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:43:03.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.831+0000 7f14a9d9b640 1 -- 172.21.15.46:0/2127217209 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14ac09b310 con 0x7f14ac0a3ff0 2024-09-16T04:43:03.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.832+0000 7f14a9d9b640 1 --2- 172.21.15.46:0/2127217209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14ac0a3ff0 0x7f14ac0a43d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f14ac068240 tx=0x7f14a002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e8d58a2893fcf2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:03.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.832+0000 7f14a959a640 1 -- 172.21.15.46:0/2127217209 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a0037070 con 0x7f14ac0a3ff0 2024-09-16T04:43:03.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.832+0000 7f14a959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:03.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.832+0000 7f14a959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:43:03.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.832+0000 7f14a959a640 10 monclient: dump: 2024-09-16T04:43:03.844 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:03.845 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:03.845 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:03.845 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:03.845 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:03.845 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:03.845 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:03.845 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:03.845 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:03.845 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:03.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.832+0000 7f14a959a640 10 monclient: _finish_auth 0 2024-09-16T04:43:03.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.832+0000 7f14a959a640 10 monclient: _check_auth_tickets 2024-09-16T04:43:03.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.832+0000 7f14a959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:03.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.832+0000 7f14a959a640 1 -- 172.21.15.46:0/2127217209 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14a002ae20 con 0x7f14ac0a3ff0 2024-09-16T04:43:03.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.832+0000 7f14a959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.832+0000 7f14a959a640 1 -- 172.21.15.46:0/2127217209 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a0032660 con 0x7f14ac0a3ff0 2024-09-16T04:43:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.832+0000 7f14b0fd6640 5 monclient: authenticate success, global_id 44457 2024-09-16T04:43:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.832+0000 7f14b0fd6640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.832+0000 7f14b0fd6640 4 set_mon_vals no callback set 2024-09-16T04:43:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.832+0000 7f14a959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.832+0000 7f14a959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.832+0000 7f14a959a640 10 monclient: dump: 2024-09-16T04:43:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:03.847 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:03.847 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:03.847 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:03.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.833+0000 7f14b0fd6640 10 monclient: shutdown 2024-09-16T04:43:03.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.833+0000 7f14b0fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:03.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.833+0000 7f14b0fd6640 1 -- 172.21.15.46:0/2127217209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14ac0a3ff0 msgr2=0x7f14ac0a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:03.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.833+0000 7f14b0fd6640 1 --2- 172.21.15.46:0/2127217209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14ac0a3ff0 0x7f14ac0a43d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f14ac068240 tx=0x7f14a002a110 comp rx=0 tx=0).stop 2024-09-16T04:43:03.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.833+0000 7f14b0fd6640 1 -- 172.21.15.46:0/2127217209 shutdown_connections 2024-09-16T04:43:03.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.833+0000 7f14b0fd6640 1 --2- 172.21.15.46:0/2127217209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14ac0a3ff0 0x7f14ac0a43d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:03.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.833+0000 7f14b0fd6640 1 --2- 172.21.15.46:0/2127217209 >> v2:172.21.15.114:3300/0 conn(0x7f14ac0a3c10 0x7f14ac09bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:03.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.833+0000 7f14b0fd6640 1 -- 172.21.15.46:0/2127217209 >> 172.21.15.46:0/2127217209 conn(0x7f14ac0a6340 msgr2=0x7f14ac06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:03.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.833+0000 7f14b0fd6640 1 -- 172.21.15.46:0/2127217209 shutdown_connections 2024-09-16T04:43:03.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.833+0000 7f14b0fd6640 1 -- 172.21.15.46:0/2127217209 wait complete. 2024-09-16T04:43:03.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.834+0000 7f14b0fd6640 10 monclient: build_initial_monmap 2024-09-16T04:43:03.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.834+0000 7f14b0fd6640 10 monclient: monmap: 2024-09-16T04:43:03.848 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:03.848 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:03.848 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:03.848 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:03.848 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:03.848 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:03.848 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:03.848 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:03.848 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:03.848 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:03.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.834+0000 7f14b0fd6640 1 Processor -- start 2024-09-16T04:43:03.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.834+0000 7f14b0fd6640 1 -- start start 2024-09-16T04:43:03.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.834+0000 7f14b0fd6640 10 monclient: init 2024-09-16T04:43:03.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14b0fd6640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:03.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14b0fd6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:43:03.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14b0fd6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f14ac0a3c10 0x7f14ac1cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:03.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14b0fd6640 10 monclient(hunting): picked mon.noname-c con 0x7f14ac0a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:03.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14b0fd6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14ac0a3ff0 0x7f14ac1233a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:03.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14b0fd6640 10 monclient(hunting): picked mon.noname-b con 0x7f14ac0a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:03.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14b0fd6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14ac1369d0 0x7f14ac1238e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:03.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14b0fd6640 10 monclient(hunting): picked mon.noname-a con 0x7f14ac1369d0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:03.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14b0fd6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:03.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14b0fd6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f14ac0a4910 con 0x7f14ac1369d0 2024-09-16T04:43:03.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14b0fd6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:03.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14b0fd6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f14ac0a69c0 con 0x7f14ac0a3ff0 2024-09-16T04:43:03.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14b0fd6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:03.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14b0fd6640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f14ac0a2bf0 con 0x7f14ac0a3c10 2024-09-16T04:43:03.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14b0fd6640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:03.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aa59c640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f14ac0a3c10 msgr2=0x7f14ac1cc330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:43:03.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14b0fd6640 10 monclient(hunting): authenticate will time out at 982.251126s 2024-09-16T04:43:03.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aa59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f14ac0a3c10 0x7f14ac1cc330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:03.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14ac1369d0 0x7f14ac1238e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:03.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14ac1369d0 0x7f14ac1238e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51626/0 (socket says 172.21.15.46:51626) 2024-09-16T04:43:03.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aad9d640 1 -- 172.21.15.46:0/62730397 learned_addr learned my addr 172.21.15.46:0/62730397 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:03.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aad9d640 10 monclient(hunting): get_auth_request con 0x7f14ac1369d0 auth_method 0 2024-09-16T04:43:03.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:03.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:03.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:03.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:03.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:03.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:03.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14a9d9b640 1 --2- 172.21.15.46:0/62730397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14ac0a3ff0 0x7f14ac1233a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:03.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14a9d9b640 10 monclient(hunting): get_auth_request con 0x7f14ac0a3ff0 auth_method 0 2024-09-16T04:43:03.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:03.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:03.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:03.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:03.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:03.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:03.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aad9d640 10 monclient(hunting): handle_auth_done global_id 44460 payload 824 2024-09-16T04:43:03.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aad9d640 1 -- 172.21.15.46:0/62730397 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f14ac0a3c10 msgr2=0x7f14ac1cc330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:43:03.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aad9d640 1 --2- 172.21.15.46:0/62730397 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f14ac0a3c10 0x7f14ac1cc330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aad9d640 1 -- 172.21.15.46:0/62730397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14ac0a3ff0 msgr2=0x7f14ac1233a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aad9d640 1 --2- 172.21.15.46:0/62730397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14ac0a3ff0 0x7f14ac1233a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aad9d640 10 monclient: _finish_hunting 0 2024-09-16T04:43:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aad9d640 1 monclient: found mon.noname-a 2024-09-16T04:43:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:43:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aad9d640 1 -- 172.21.15.46:0/62730397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14ac127d10 con 0x7f14ac1369d0 2024-09-16T04:43:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14aad9d640 1 --2- 172.21.15.46:0/62730397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14ac1369d0 0x7f14ac1238e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f14a400c780 tx=0x7f14a400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f149affd640 1 -- 172.21.15.46:0/62730397 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a4004b10 con 0x7f14ac1369d0 2024-09-16T04:43:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f149affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f149affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:43:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f149affd640 10 monclient: dump: 2024-09-16T04:43:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:03.853 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f149affd640 10 monclient: _finish_auth 0 2024-09-16T04:43:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f149affd640 10 monclient: _check_auth_tickets 2024-09-16T04:43:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f149affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f149affd640 1 -- 172.21.15.46:0/62730397 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14a4004cb0 con 0x7f14ac1369d0 2024-09-16T04:43:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14b0fd6640 5 monclient: authenticate success, global_id 44460 2024-09-16T04:43:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14b0fd6640 10 monclient: _renew_subs 2024-09-16T04:43:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14b0fd6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14b0fd6640 1 -- 172.21.15.46:0/62730397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14ac125ba0 con 0x7f14ac1369d0 2024-09-16T04:43:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f149affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f149affd640 1 -- 172.21.15.46:0/62730397 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a4004e60 con 0x7f14ac1369d0 2024-09-16T04:43:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.836+0000 7f149affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.836+0000 7f149affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.836+0000 7f149affd640 10 monclient: dump: 2024-09-16T04:43:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:03.855 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:03.855 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:03.855 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:03.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.836+0000 7f14b0fd6640 10 monclient: _renew_subs 2024-09-16T04:43:03.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.835+0000 7f14a8d99640 4 set_mon_vals no callback set 2024-09-16T04:43:03.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.836+0000 7f14b0fd6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:03.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.836+0000 7f14b0fd6640 1 -- 172.21.15.46:0/62730397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14ac0a88f0 con 0x7f14ac1369d0 2024-09-16T04:43:03.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.836+0000 7f149affd640 1 -- 172.21.15.46:0/62730397 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f14a400aa90 con 0x7f14ac1369d0 2024-09-16T04:43:03.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.836+0000 7f14b0fd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:03.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.836+0000 7f14b0fd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:03.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.836+0000 7f14b0fd6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:03.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.836+0000 7f14b0fd6640 1 -- 172.21.15.46:0/62730397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14ac0a86f0 con 0x7f14ac1369d0 2024-09-16T04:43:03.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.837+0000 7f149affd640 1 --2- 172.21.15.46:0/62730397 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f148c038360 0x7f148c03a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:03.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.837+0000 7f149affd640 1 -- 172.21.15.46:0/62730397 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 4738+0+0 (secure 0 0 0) 0x7f14a404bb50 con 0x7f14ac1369d0 2024-09-16T04:43:03.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.837+0000 7f14aa59c640 1 --2- 172.21.15.46:0/62730397 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f148c038360 0x7f148c03a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:03.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.837+0000 7f14aa59c640 10 monclient: get_auth_request con 0x7f148c038360 auth_method 0 2024-09-16T04:43:03.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.837+0000 7f14aa59c640 1 --2- 172.21.15.46:0/62730397 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f148c038360 0x7f148c03a820 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f149c006fd0 tx=0x7f149c0092e0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:03.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.841+0000 7f149affd640 1 -- 172.21.15.46:0/62730397 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f14ac0a86f0 con 0x7f14ac1369d0 2024-09-16T04:43:03.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.841+0000 7f149affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:03.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:03.841+0000 7f149affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:04.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:04.082+0000 7f14b0fd6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:43:04.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:04.082+0000 7f14b0fd6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:04.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:04.082+0000 7f14b0fd6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:04.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:04.082+0000 7f14b0fd6640 1 -- 172.21.15.46:0/62730397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f14ac09bcf0 con 0x7f14ac1369d0 2024-09-16T04:43:04.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:04.083+0000 7f149affd640 1 -- 172.21.15.46:0/62730397 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f14ac127d10 con 0x7f14ac1369d0 2024-09-16T04:43:04.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:04.083+0000 7f149affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:04.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:04.083+0000 7f149affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:04.086 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:43:04.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:04.085+0000 7f14b0fd6640 1 -- 172.21.15.46:0/62730397 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f148c038360 msgr2=0x7f148c03a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:04.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:04.085+0000 7f14b0fd6640 1 --2- 172.21.15.46:0/62730397 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f148c038360 0x7f148c03a820 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f149c006fd0 tx=0x7f149c0092e0 comp rx=0 tx=0).stop 2024-09-16T04:43:04.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:04.085+0000 7f14b0fd6640 10 monclient: shutdown 2024-09-16T04:43:04.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:04.085+0000 7f14b0fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:04.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:04.085+0000 7f14b0fd6640 1 -- 172.21.15.46:0/62730397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14ac1369d0 msgr2=0x7f14ac1238e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:04.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:04.085+0000 7f14b0fd6640 1 --2- 172.21.15.46:0/62730397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14ac1369d0 0x7f14ac1238e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f14a400c780 tx=0x7f14a400cc40 comp rx=0 tx=0).stop 2024-09-16T04:43:04.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:04.085+0000 7f14b0fd6640 1 -- 172.21.15.46:0/62730397 shutdown_connections 2024-09-16T04:43:04.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:04.085+0000 7f14b0fd6640 1 --2- 172.21.15.46:0/62730397 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f148c038360 0x7f148c03a820 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:04.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:04.085+0000 7f14b0fd6640 1 --2- 172.21.15.46:0/62730397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14ac1369d0 0x7f14ac1238e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:04.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:04.085+0000 7f14b0fd6640 1 --2- 172.21.15.46:0/62730397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14ac0a3ff0 0x7f14ac1233a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:04.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:04.085+0000 7f14b0fd6640 1 --2- 172.21.15.46:0/62730397 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f14ac0a3c10 0x7f14ac1cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:04.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:04.085+0000 7f14b0fd6640 1 -- 172.21.15.46:0/62730397 >> 172.21.15.46:0/62730397 conn(0x7f14ac0a6340 msgr2=0x7f14ac0a7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:04.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:04.085+0000 7f14b0fd6640 1 -- 172.21.15.46:0/62730397 shutdown_connections 2024-09-16T04:43:04.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:04.085+0000 7f14b0fd6640 1 -- 172.21.15.46:0/62730397 wait complete. 2024-09-16T04:43:07.099 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:43:07.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.210+0000 7f05a420e640 10 monclient: get_monmap_and_config 2024-09-16T04:43:07.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.210+0000 7f05a420e640 10 monclient: build_initial_monmap 2024-09-16T04:43:07.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.210+0000 7f05a420e640 10 monclient: monmap: 2024-09-16T04:43:07.213 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:07.213 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:07.213 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:07.212298+0000 2024-09-16T04:43:07.213 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:07.212298+0000 2024-09-16T04:43:07.214 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:07.214 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:07.214 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:07.214 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:07.214 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:07.214 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:07.214 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:07.214 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:07.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.211+0000 7f05a420e640 1 Processor -- start 2024-09-16T04:43:07.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.211+0000 7f05a420e640 1 -- start start 2024-09-16T04:43:07.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.211+0000 7f05a420e640 10 monclient: init 2024-09-16T04:43:07.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a420e640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:07.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a420e640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-16T04:43:07.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a420e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 0x7f059c09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:07.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a420e640 10 monclient(hunting): picked mon.noname-c con 0x7f059c0a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:43:07.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a420e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f059c0a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:43:07.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a420e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 0x7f059c13e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:07.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a420e640 10 monclient(hunting): picked mon.noname-a con 0x7f059c13be90 addr v2:172.21.15.46:3300/0 2024-09-16T04:43:07.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a420e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f059c09ab70 con 0x7f059c0a3ff0 2024-09-16T04:43:07.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a420e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:07.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a420e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f059c0a4d60 con 0x7f059c13be90 2024-09-16T04:43:07.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a420e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:07.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a420e640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f059c09b030 con 0x7f059c0a3c10 2024-09-16T04:43:07.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a420e640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:07.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a420e640 10 monclient(hunting): authenticate will time out at 985.628293s 2024-09-16T04:43:07.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a27ab640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 0x7f059c13e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:07.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a27ab640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 0x7f059c13e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40000/0 (socket says 172.21.15.46:40000) 2024-09-16T04:43:07.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a27ab640 1 -- 172.21.15.46:0/3908488691 learned_addr learned my addr 172.21.15.46:0/3908488691 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:07.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a27ab640 10 monclient(hunting): get_auth_request con 0x7f059c13be90 auth_method 0 2024-09-16T04:43:07.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a27ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:07.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a27ab640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:07.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a27ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:07.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a27ab640 1 -- 172.21.15.46:0/3908488691 >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 msgr2=0x7f059c13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:43:07.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a27ab640 1 -- 172.21.15.46:0/3908488691 >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 msgr2=0x7f059c13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:07.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a27ab640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 0x7f059c13e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:07.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 0x7f059c09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:07.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a1faa640 10 monclient(hunting): get_auth_request con 0x7f059c0a3c10 auth_method 0 2024-09-16T04:43:07.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a1faa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:07.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a1faa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:07.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a1faa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:07.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a27ab640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 0x7f059c13e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:07.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a1faa640 1 -- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 msgr2=0x7f059c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:43:07.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a1faa640 1 -- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 msgr2=0x7f059c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:07.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 0x7f059c09bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:07.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.212+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 0x7f059c09bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:07.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.413+0000 7f05a27ab640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 0x7f059c13e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:07.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.413+0000 7f05a27ab640 10 monclient(hunting): get_auth_request con 0x7f059c13be90 auth_method 0 2024-09-16T04:43:07.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.413+0000 7f05a27ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:07.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.413+0000 7f05a27ab640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:07.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.413+0000 7f05a27ab640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:07.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.413+0000 7f05a27ab640 1 -- 172.21.15.46:0/3908488691 >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 msgr2=0x7f059c13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:43:07.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.413+0000 7f05a27ab640 1 -- 172.21.15.46:0/3908488691 >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 msgr2=0x7f059c13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:07.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.413+0000 7f05a27ab640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 0x7f059c13e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:07.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.413+0000 7f05a27ab640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 0x7f059c13e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:43:07.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.413+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 0x7f059c09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:07.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.413+0000 7f05a1faa640 10 monclient(hunting): get_auth_request con 0x7f059c0a3c10 auth_method 0 2024-09-16T04:43:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.413+0000 7f05a1faa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.413+0000 7f05a1faa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.413+0000 7f05a1faa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.413+0000 7f05a1faa640 1 -- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 msgr2=0x7f059c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:43:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.413+0000 7f05a1faa640 1 -- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 msgr2=0x7f059c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.413+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 0x7f059c09bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.413+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 0x7f059c09bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:43:07.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.814+0000 7f05a27ab640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 0x7f059c13e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.814+0000 7f05a27ab640 10 monclient(hunting): get_auth_request con 0x7f059c13be90 auth_method 0 2024-09-16T04:43:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.814+0000 7f05a27ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.814+0000 7f05a27ab640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.814+0000 7f05a27ab640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.814+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 0x7f059c09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.814+0000 7f05a27ab640 1 -- 172.21.15.46:0/3908488691 >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 msgr2=0x7f059c13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:43:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.814+0000 7f05a27ab640 1 -- 172.21.15.46:0/3908488691 >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 msgr2=0x7f059c13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.814+0000 7f05a27ab640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 0x7f059c13e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.814+0000 7f05a1faa640 10 monclient(hunting): get_auth_request con 0x7f059c0a3c10 auth_method 0 2024-09-16T04:43:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.814+0000 7f05a1faa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.814+0000 7f05a1faa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.814+0000 7f05a1faa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.814+0000 7f05a27ab640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 0x7f059c13e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:43:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.814+0000 7f05a1faa640 1 -- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 msgr2=0x7f059c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:43:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.814+0000 7f05a1faa640 1 -- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 msgr2=0x7f059c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.814+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 0x7f059c09bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:07.814+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 0x7f059c09bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:43:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:08.616+0000 7f05a27ab640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 0x7f059c13e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:08.616+0000 7f05a27ab640 10 monclient(hunting): get_auth_request con 0x7f059c13be90 auth_method 0 2024-09-16T04:43:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:08.616+0000 7f05a27ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:08.616+0000 7f05a27ab640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:08.616+0000 7f05a27ab640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:08.616+0000 7f05a27ab640 1 -- 172.21.15.46:0/3908488691 >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 msgr2=0x7f059c13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:43:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:08.616+0000 7f05a27ab640 1 -- 172.21.15.46:0/3908488691 >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 msgr2=0x7f059c13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:08.616+0000 7f05a27ab640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 0x7f059c13e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:08.616+0000 7f05a27ab640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 0x7f059c13e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:43:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:08.616+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 0x7f059c09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:08.616+0000 7f05a1faa640 10 monclient(hunting): get_auth_request con 0x7f059c0a3c10 auth_method 0 2024-09-16T04:43:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:08.616+0000 7f05a1faa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:08.616+0000 7f05a1faa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:08.616+0000 7f05a1faa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:08.616+0000 7f05a1faa640 1 -- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 msgr2=0x7f059c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:43:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:08.616+0000 7f05a1faa640 1 -- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 msgr2=0x7f059c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:08.616+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 0x7f059c09bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:08.616+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 0x7f059c09bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:43:10.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.213+0000 7f05937fe640 10 monclient(hunting): tick 2024-09-16T04:43:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.213+0000 7f05937fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:43:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.213+0000 7f05937fe640 1 monclient(hunting): continuing hunt 2024-09-16T04:43:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.213+0000 7f05937fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:43:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.213+0000 7f05937fe640 1 -- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 msgr2=0x7f059c09bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.213+0000 7f05937fe640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.114:3300/0 conn(0x7f059c0a3c10 0x7f059c09bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.213+0000 7f05937fe640 1 -- 172.21.15.46:0/3908488691 >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 msgr2=0x7f059c13e280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.213+0000 7f05937fe640 1 --2- 172.21.15.46:0/3908488691 >> v2:172.21.15.46:3300/0 conn(0x7f059c13be90 0x7f059c13e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.213+0000 7f05937fe640 1 -- 172.21.15.46:0/3908488691 >> v1:172.21.15.114:6789/0 conn(0x7f059c0a3ff0 legacy=0x7f059c0a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.213+0000 7f05937fe640 10 monclient: _add_conns ranks=[2,0] 2024-09-16T04:43:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.213+0000 7f05937fe640 1 --2- 172.21.15.46:0/3908488691 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0580003690 0x7f0580005ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.213+0000 7f05937fe640 10 monclient(hunting): picked mon.noname-b con 0x7f0580003690 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.213+0000 7f05937fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f05800061f0 addr v1:172.21.15.46:6789/0 2024-09-16T04:43:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.213+0000 7f05937fe640 1 -- 172.21.15.46:0/3908488691 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f059c09b310 con 0x7f05800061f0 2024-09-16T04:43:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.213+0000 7f05937fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.213+0000 7f05937fe640 1 -- 172.21.15.46:0/3908488691 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f059c0a4d60 con 0x7f0580003690 2024-09-16T04:43:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.213+0000 7f05937fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.213+0000 7f05a1faa640 1 -- 172.21.15.46:0/3908488691 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0580003690 msgr2=0x7f0580005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:43:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.213+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3908488691 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0580003690 0x7f0580005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:10.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.414+0000 7f05a1faa640 1 -- 172.21.15.46:0/3908488691 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0580003690 msgr2=0x7f0580005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:43:10.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.414+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3908488691 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0580003690 0x7f0580005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:43:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.815+0000 7f05a1faa640 1 -- 172.21.15.46:0/3908488691 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0580003690 msgr2=0x7f0580005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:43:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:10.815+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3908488691 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0580003690 0x7f0580005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:43:11.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.354+0000 7f05a0fa8640 1 -- 172.21.15.46:0/3908488691 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2071638961 0 0) 0x7f059c09b310 con 0x7f05800061f0 2024-09-16T04:43:11.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.354+0000 7f05a0fa8640 10 monclient(hunting): _negotiate 2024-09-16T04:43:11.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.354+0000 7f05a0fa8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:11.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.354+0000 7f05a0fa8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:11.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.354+0000 7f05a0fa8640 10 monclient(hunting): my global_id is 44463 2024-09-16T04:43:11.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.354+0000 7f05a0fa8640 1 -- 172.21.15.46:0/3908488691 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0588003520 con 0x7f05800061f0 2024-09-16T04:43:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.354+0000 7f05a0fa8640 1 -- 172.21.15.46:0/3908488691 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4240373557 0 0) 0x7f0588003520 con 0x7f05800061f0 2024-09-16T04:43:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.354+0000 7f05a0fa8640 1 -- 172.21.15.46:0/3908488691 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0580003690 msgr2=0x7f0580005ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.354+0000 7f05a0fa8640 1 --2- 172.21.15.46:0/3908488691 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0580003690 0x7f0580005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 10 monclient: _finish_hunting 0 2024-09-16T04:43:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:43:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:43:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 1 -- 172.21.15.46:0/3908488691 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058000b050 con 0x7f05800061f0 2024-09-16T04:43:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 10 monclient: _finish_auth 0 2024-09-16T04:43:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 10 monclient: _check_auth_tickets 2024-09-16T04:43:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 1 -- 172.21.15.46:0/3908488691 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f058c0016b0 con 0x7f05800061f0 2024-09-16T04:43:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:43:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 10 monclient: dump: 2024-09-16T04:43:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:11.361 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:43:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 1 -- 172.21.15.46:0/3908488691 >> v1:172.21.15.46:6789/0 conn(0x7f05800061f0 legacy=0x7f05800086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:43:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a27ab640 1 -- 172.21.15.46:0/3908488691 reap_dead start 2024-09-16T04:43:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 1 --2- 172.21.15.46:0/3908488691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05880039b0 0x7f0588003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 10 monclient(hunting): picked mon.b con 0x7f05880039b0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 1 --2- 172.21.15.46:0/3908488691 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0588004800 0x7f0588004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 10 monclient(hunting): picked mon.c con 0x7f0588004800 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 1 --2- 172.21.15.46:0/3908488691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05880052e0 0x7f058800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 10 monclient(hunting): picked mon.a con 0x7f05880052e0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 1 -- 172.21.15.46:0/3908488691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f059c0a4d60 con 0x7f05880052e0 2024-09-16T04:43:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 1 -- 172.21.15.46:0/3908488691 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f058800c1e0 con 0x7f05880039b0 2024-09-16T04:43:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 1 -- 172.21.15.46:0/3908488691 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f058800c3e0 con 0x7f0588004800 2024-09-16T04:43:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a0fa8640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:43:11.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a17a9640 1 -- 172.21.15.46:0/3908488691 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0588004800 msgr2=0x7f0588004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:43:11.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a17a9640 1 --2- 172.21.15.46:0/3908488691 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0588004800 0x7f0588004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:11.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3908488691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05880039b0 0x7f0588003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:11.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a1faa640 10 monclient(hunting): get_auth_request con 0x7f05880039b0 auth_method 0 2024-09-16T04:43:11.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a1faa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:11.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a1faa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:11.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a1faa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:11.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a1faa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:11.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a1faa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:11.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a1faa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:43:11.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a1faa640 10 monclient(hunting): handle_auth_done global_id 44463 payload 293 2024-09-16T04:43:11.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a1faa640 1 -- 172.21.15.46:0/3908488691 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0588004800 msgr2=0x7f0588004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:11.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3908488691 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0588004800 0x7f0588004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a1faa640 1 -- 172.21.15.46:0/3908488691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05880052e0 msgr2=0x7f058800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:43:11.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3908488691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05880052e0 0x7f058800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a1faa640 10 monclient: _finish_hunting 0 2024-09-16T04:43:11.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a1faa640 1 monclient: found mon.b 2024-09-16T04:43:11.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a1faa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:11.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a1faa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:43:11.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a1faa640 1 -- 172.21.15.46:0/3908488691 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058000b050 con 0x7f05880039b0 2024-09-16T04:43:11.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a1faa640 10 monclient: _finish_auth 0 2024-09-16T04:43:11.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a1faa640 10 monclient: _check_auth_tickets 2024-09-16T04:43:11.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.355+0000 7f05a1faa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:11.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.356+0000 7f05a420e640 5 monclient: authenticate success, global_id 44463 2024-09-16T04:43:11.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.356+0000 7f05a420e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:11.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.359+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3908488691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05880039b0 0x7f0588003db0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0598009610 tx=0x7f0588005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e45c72bcd1735c5d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:11.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.361+0000 7f05a0fa8640 1 -- 172.21.15.46:0/3908488691 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0598031070 con 0x7f05880039b0 2024-09-16T04:43:11.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.361+0000 7f05a0fa8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:11.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.361+0000 7f05a0fa8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:43:11.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.361+0000 7f05a0fa8640 10 monclient: dump: 2024-09-16T04:43:11.367 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:11.367 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:11.367 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.367 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.367 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:11.367 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.367 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:11.367 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:11.368 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:11.368 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.362+0000 7f05a0fa8640 1 -- 172.21.15.46:0/3908488691 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f059802bc70 con 0x7f05880039b0 2024-09-16T04:43:11.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.362+0000 7f05a0fa8640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:11.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.362+0000 7f05a0fa8640 1 -- 172.21.15.46:0/3908488691 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0598009830 con 0x7f05880039b0 2024-09-16T04:43:11.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.362+0000 7f05a0fa8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.362+0000 7f05a0fa8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:43:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.362+0000 7f05a0fa8640 10 monclient: dump: 2024-09-16T04:43:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:11.369 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.362+0000 7f05a420e640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.362+0000 7f05a420e640 4 set_mon_vals no callback set 2024-09-16T04:43:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.363+0000 7f05a420e640 10 monclient: shutdown 2024-09-16T04:43:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.363+0000 7f05a420e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.363+0000 7f05a420e640 1 -- 172.21.15.46:0/3908488691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05880039b0 msgr2=0x7f0588003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.363+0000 7f05a420e640 1 --2- 172.21.15.46:0/3908488691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05880039b0 0x7f0588003db0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0598009610 tx=0x7f0588005830 comp rx=0 tx=0).stop 2024-09-16T04:43:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.363+0000 7f05a420e640 1 -- 172.21.15.46:0/3908488691 shutdown_connections 2024-09-16T04:43:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.363+0000 7f05a420e640 1 --2- 172.21.15.46:0/3908488691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05880052e0 0x7f058800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.363+0000 7f05a420e640 1 --2- 172.21.15.46:0/3908488691 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0588004800 0x7f0588004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.363+0000 7f05a420e640 1 --2- 172.21.15.46:0/3908488691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05880039b0 0x7f0588003db0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.363+0000 7f05a420e640 1 -- 172.21.15.46:0/3908488691 >> 172.21.15.46:0/3908488691 conn(0x7f059c0a5d10 msgr2=0x7f059c0a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.363+0000 7f05a420e640 1 -- 172.21.15.46:0/3908488691 shutdown_connections 2024-09-16T04:43:11.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.363+0000 7f05a420e640 1 -- 172.21.15.46:0/3908488691 wait complete. 2024-09-16T04:43:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.365+0000 7f05a420e640 10 monclient: build_initial_monmap 2024-09-16T04:43:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.365+0000 7f05a420e640 10 monclient: monmap: 2024-09-16T04:43:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:11.374 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.366+0000 7f05a420e640 1 Processor -- start 2024-09-16T04:43:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.366+0000 7f05a420e640 1 -- start start 2024-09-16T04:43:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.366+0000 7f05a420e640 10 monclient: init 2024-09-16T04:43:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a420e640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a420e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:43:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a420e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05880039b0 0x7f059c136e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a420e640 10 monclient(hunting): picked mon.noname-b con 0x7f05880039b0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a420e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0588004800 0x7f059c137350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a420e640 10 monclient(hunting): picked mon.noname-c con 0x7f0588004800 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a420e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05880052e0 0x7f059c11f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a17a9640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0588004800 msgr2=0x7f059c137350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:43:11.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a17a9640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0588004800 0x7f059c137350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:11.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a420e640 10 monclient(hunting): picked mon.noname-a con 0x7f05880052e0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:11.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a420e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a420e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f059c0a4580 con 0x7f05880052e0 2024-09-16T04:43:11.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a420e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a420e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f059c0a4400 con 0x7f05880039b0 2024-09-16T04:43:11.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a420e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a420e640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f059c0a4700 con 0x7f0588004800 2024-09-16T04:43:11.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a420e640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:11.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a420e640 10 monclient(hunting): authenticate will time out at 989.783778s 2024-09-16T04:43:11.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a1faa640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05880039b0 0x7f059c136e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:11.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a1faa640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05880039b0 0x7f059c136e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.46:43866/0 (socket says 172.21.15.46:43866) 2024-09-16T04:43:11.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a1faa640 1 -- 172.21.15.46:0/3221946809 learned_addr learned my addr 172.21.15.46:0/3221946809 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a1faa640 10 monclient(hunting): get_auth_request con 0x7f05880039b0 auth_method 0 2024-09-16T04:43:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.367+0000 7f05a1faa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.368+0000 7f05a1faa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.368+0000 7f05a1faa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.368+0000 7f05a1faa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.368+0000 7f05a1faa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.368+0000 7f05a1faa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.368+0000 7f05a1faa640 10 monclient(hunting): handle_auth_done global_id 5731 payload 824 2024-09-16T04:43:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.368+0000 7f05a1faa640 1 -- 172.21.15.46:0/3221946809 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0588004800 msgr2=0x7f059c137350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:43:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.368+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3221946809 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0588004800 0x7f059c137350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.368+0000 7f05a1faa640 1 -- 172.21.15.46:0/3221946809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05880052e0 msgr2=0x7f059c11f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.368+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3221946809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05880052e0 0x7f059c11f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.368+0000 7f05a1faa640 10 monclient: _finish_hunting 0 2024-09-16T04:43:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.368+0000 7f05a1faa640 1 monclient: found mon.noname-b 2024-09-16T04:43:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.368+0000 7f05a1faa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.368+0000 7f05a1faa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3300/0 2024-09-16T04:43:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.368+0000 7f05a1faa640 1 -- 172.21.15.46:0/3221946809 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f059c122030 con 0x7f05880039b0 2024-09-16T04:43:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.369+0000 7f05a1faa640 1 --2- 172.21.15.46:0/3221946809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05880039b0 0x7f059c136e10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0598009d20 tx=0x7f0598004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.369+0000 7f05927fc640 1 -- 172.21.15.46:0/3221946809 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0598044020 con 0x7f05880039b0 2024-09-16T04:43:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.369+0000 7f05927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.369+0000 7f05927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:43:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.369+0000 7f05927fc640 10 monclient: dump: 2024-09-16T04:43:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:11.378 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.369+0000 7f05927fc640 10 monclient: _finish_auth 0 2024-09-16T04:43:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.369+0000 7f05927fc640 10 monclient: _check_auth_tickets 2024-09-16T04:43:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.369+0000 7f05927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.369+0000 7f05927fc640 1 -- 172.21.15.46:0/3221946809 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0598040670 con 0x7f05880039b0 2024-09-16T04:43:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.369+0000 7f05927fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.369+0000 7f0593fff640 4 set_mon_vals no callback set 2024-09-16T04:43:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.370+0000 7f05927fc640 1 -- 172.21.15.46:0/3221946809 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0598009830 con 0x7f05880039b0 2024-09-16T04:43:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.370+0000 7f05927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.370+0000 7f05927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:43:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.370+0000 7f05927fc640 10 monclient: dump: 2024-09-16T04:43:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:11.379 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.370+0000 7f05a420e640 5 monclient: authenticate success, global_id 5731 2024-09-16T04:43:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.370+0000 7f05a420e640 10 monclient: _renew_subs 2024-09-16T04:43:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.370+0000 7f05a420e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:43:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.370+0000 7f05a420e640 1 -- 172.21.15.46:0/3221946809 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f059c0a3360 con 0x7f05880039b0 2024-09-16T04:43:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.371+0000 7f05a420e640 10 monclient: _renew_subs 2024-09-16T04:43:11.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.371+0000 7f05a420e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:43:11.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.371+0000 7f05a420e640 1 -- 172.21.15.46:0/3221946809 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f059c1234e0 con 0x7f05880039b0 2024-09-16T04:43:11.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.372+0000 7f05927fc640 1 -- 172.21.15.46:0/3221946809 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f0598047020 con 0x7f05880039b0 2024-09-16T04:43:11.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.372+0000 7f05927fc640 1 --2- 172.21.15.46:0/3221946809 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0580038d20 0x7f058003b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.372+0000 7f05a420e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:11.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.372+0000 7f05a420e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:11.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.372+0000 7f05a420e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:43:11.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.372+0000 7f05927fc640 1 -- 172.21.15.46:0/3221946809 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 4738+0+0 (secure 0 0 0) 0x7f059806d4d0 con 0x7f05880039b0 2024-09-16T04:43:11.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.372+0000 7f05a420e640 1 -- 172.21.15.46:0/3221946809 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f059c0a75c0 con 0x7f05880039b0 2024-09-16T04:43:11.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.372+0000 7f05a17a9640 1 --2- 172.21.15.46:0/3221946809 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0580038d20 0x7f058003b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:11.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.372+0000 7f05a17a9640 10 monclient: get_auth_request con 0x7f0580038d20 auth_method 0 2024-09-16T04:43:11.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.373+0000 7f05a17a9640 1 --2- 172.21.15.46:0/3221946809 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0580038d20 0x7f058003b1e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f059c122e50 tx=0x7f058c0015c0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:11.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.378+0000 7f05927fc640 1 -- 172.21.15.46:0/3221946809 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f059804e030 con 0x7f05880039b0 2024-09-16T04:43:11.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.378+0000 7f05927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:11.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.378+0000 7f05927fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:11.443 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37841]: starting. 2024-09-16T04:43:11.444 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37841]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-37840 2024-09-16T04:43:11.444 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37841]: created object 0... 2024-09-16T04:43:11.444 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37841]: created object 25... 2024-09-16T04:43:11.444 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37841]: created object 49... 2024-09-16T04:43:11.444 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37841]: finishing. 2024-09-16T04:43:11.444 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37841]: shutting down. 2024-09-16T04:43:11.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.631+0000 7f05a420e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:43:11.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.631+0000 7f05a420e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:11.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.631+0000 7f05a420e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:43:11.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.631+0000 7f05a420e640 1 -- 172.21.15.46:0/3221946809 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f059c0a01b0 con 0x7f05880039b0 2024-09-16T04:43:11.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.632+0000 7f05927fc640 1 -- 172.21.15.46:0/3221946809 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f059802be20 con 0x7f05880039b0 2024-09-16T04:43:11.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.632+0000 7f05927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:11.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.632+0000 7f05927fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:11.634 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":52,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:43:11.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.634+0000 7f05a420e640 1 -- 172.21.15.46:0/3221946809 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0580038d20 msgr2=0x7f058003b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:11.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.634+0000 7f05a420e640 1 --2- 172.21.15.46:0/3221946809 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0580038d20 0x7f058003b1e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f059c122e50 tx=0x7f058c0015c0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.634+0000 7f05a420e640 10 monclient: shutdown 2024-09-16T04:43:11.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.634+0000 7f05a420e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:11.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.634+0000 7f05a420e640 1 -- 172.21.15.46:0/3221946809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05880039b0 msgr2=0x7f059c136e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:11.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.634+0000 7f05a420e640 1 --2- 172.21.15.46:0/3221946809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05880039b0 0x7f059c136e10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0598009d20 tx=0x7f0598004a40 comp rx=0 tx=0).stop 2024-09-16T04:43:11.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.634+0000 7f05a420e640 1 -- 172.21.15.46:0/3221946809 shutdown_connections 2024-09-16T04:43:11.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.634+0000 7f05a420e640 1 --2- 172.21.15.46:0/3221946809 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0580038d20 0x7f058003b1e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.634+0000 7f05a420e640 1 --2- 172.21.15.46:0/3221946809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05880052e0 0x7f059c11f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.634+0000 7f05a420e640 1 --2- 172.21.15.46:0/3221946809 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0588004800 0x7f059c137350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.634+0000 7f05a420e640 1 --2- 172.21.15.46:0/3221946809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05880039b0 0x7f059c136e10 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.634+0000 7f05a420e640 1 -- 172.21.15.46:0/3221946809 >> 172.21.15.46:0/3221946809 conn(0x7f059c0a5d10 msgr2=0x7f059c0a2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:11.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.634+0000 7f05a420e640 1 -- 172.21.15.46:0/3221946809 shutdown_connections 2024-09-16T04:43:11.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.634+0000 7f05a420e640 1 -- 172.21.15.46:0/3221946809 wait complete. 2024-09-16T04:43:11.647 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:43:11.648 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:43:11.648 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:43:11.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.747+0000 7ffb7c2d0640 10 monclient: get_monmap_and_config 2024-09-16T04:43:11.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.747+0000 7ffb7c2d0640 10 monclient: build_initial_monmap 2024-09-16T04:43:11.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.747+0000 7ffb7c2d0640 10 monclient: monmap: 2024-09-16T04:43:11.750 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:11.750 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:11.750 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:11.749286+0000 2024-09-16T04:43:11.750 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:11.749286+0000 2024-09-16T04:43:11.750 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:11.750 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.750 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:11.750 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:11.751 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:11.751 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:11.751 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:11.751 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.748+0000 7ffb7c2d0640 1 Processor -- start 2024-09-16T04:43:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.748+0000 7ffb7c2d0640 1 -- start start 2024-09-16T04:43:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.748+0000 7ffb7c2d0640 10 monclient: init 2024-09-16T04:43:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.748+0000 7ffb7c2d0640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.748+0000 7ffb7c2d0640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-16T04:43:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7c2d0640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ffb740a3ba0 0x7ffb7409bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7c2d0640 10 monclient(hunting): picked mon.noname-c con 0x7ffb740a3ba0 addr v2:172.21.15.114:3300/0 2024-09-16T04:43:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7c2d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffb7409c1c0 addr v1:172.21.15.46:6789/0 2024-09-16T04:43:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7c2d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb740a4690 addr v1:172.21.15.114:6789/0 2024-09-16T04:43:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7c2d0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb7409afc0 con 0x7ffb7409c1c0 2024-09-16T04:43:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7c2d0640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb7409b580 con 0x7ffb740a4690 2024-09-16T04:43:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7c2d0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7c2d0640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ffb7409b860 con 0x7ffb740a3ba0 2024-09-16T04:43:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7c2d0640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7c2d0640 10 monclient(hunting): authenticate will time out at 990.165084s 2024-09-16T04:43:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7986b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ffb7409c1c0 0x7ffb740a3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56626/0 (socket says 172.21.15.46:56626) 2024-09-16T04:43:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7986b640 1 -- 172.21.15.46:0/1066032425 learned_addr learned my addr 172.21.15.46:0/1066032425 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:43:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7a06c640 1 --2- 172.21.15.46:0/1066032425 >> v2:172.21.15.114:3300/0 conn(0x7ffb740a3ba0 0x7ffb7409bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 1 -- 172.21.15.46:0/1066032425 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3070630717 0 0) 0x7ffb7409afc0 con 0x7ffb7409c1c0 2024-09-16T04:43:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 10 monclient(hunting): _negotiate 2024-09-16T04:43:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 10 monclient(hunting): my global_id is 44466 2024-09-16T04:43:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 1 -- 172.21.15.46:0/1066032425 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb60003520 con 0x7ffb7409c1c0 2024-09-16T04:43:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7a06c640 10 monclient(hunting): get_auth_request con 0x7ffb740a3ba0 auth_method 0 2024-09-16T04:43:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7a06c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7a06c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7a06c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7a06c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7a06c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7a06c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 1 -- 172.21.15.46:0/1066032425 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2399058711 0 0) 0x7ffb60003520 con 0x7ffb7409c1c0 2024-09-16T04:43:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 1 -- 172.21.15.46:0/1066032425 >> v2:172.21.15.114:3300/0 conn(0x7ffb740a3ba0 msgr2=0x7ffb7409bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:43:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 1 --2- 172.21.15.46:0/1066032425 >> v2:172.21.15.114:3300/0 conn(0x7ffb740a3ba0 0x7ffb7409bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 1 -- 172.21.15.46:0/1066032425 >> v1:172.21.15.114:6789/0 conn(0x7ffb740a4690 legacy=0x7ffb7413df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 10 monclient: _finish_hunting 0 2024-09-16T04:43:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:43:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:11.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:43:11.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 1 -- 172.21.15.46:0/1066032425 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb740a53c0 con 0x7ffb7409c1c0 2024-09-16T04:43:11.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 10 monclient: _finish_auth 0 2024-09-16T04:43:11.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 10 monclient: _check_auth_tickets 2024-09-16T04:43:11.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:11.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 1 -- 172.21.15.46:0/1066032425 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7ffb68001670 con 0x7ffb7409c1c0 2024-09-16T04:43:11.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:11.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:43:11.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 10 monclient: dump: 2024-09-16T04:43:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:11.771 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:43:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 1 -- 172.21.15.46:0/1066032425 >> v1:172.21.15.46:6789/0 conn(0x7ffb7409c1c0 legacy=0x7ffb740a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:43:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 1 --2- 172.21.15.46:0/1066032425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb600039b0 0x7ffb60003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 10 monclient(hunting): picked mon.a con 0x7ffb600039b0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 1 --2- 172.21.15.46:0/1066032425 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb60004800 0x7ffb60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 10 monclient(hunting): picked mon.b con 0x7ffb60004800 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 1 --2- 172.21.15.46:0/1066032425 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ffb600051c0 0x7ffb6000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 10 monclient(hunting): picked mon.c con 0x7ffb600051c0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 1 -- 172.21.15.46:0/1066032425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7409b860 con 0x7ffb600039b0 2024-09-16T04:43:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 1 -- 172.21.15.46:0/1066032425 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6000c160 con 0x7ffb60004800 2024-09-16T04:43:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 1 -- 172.21.15.46:0/1066032425 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7ffb6000c360 con 0x7ffb600051c0 2024-09-16T04:43:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7906a640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:43:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7a86d640 1 -- 172.21.15.46:0/1066032425 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ffb600051c0 msgr2=0x7ffb6000b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:43:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7a86d640 1 --2- 172.21.15.46:0/1066032425 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ffb600051c0 0x7ffb6000b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7a06c640 1 --2- 172.21.15.46:0/1066032425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb600039b0 0x7ffb60003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7a06c640 10 monclient(hunting): get_auth_request con 0x7ffb600039b0 auth_method 0 2024-09-16T04:43:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7a06c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7a06c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:11.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7a06c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:11.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7986b640 1 --2- 172.21.15.46:0/1066032425 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb60004800 0x7ffb60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:11.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.749+0000 7ffb7986b640 10 monclient(hunting): get_auth_request con 0x7ffb60004800 auth_method 0 2024-09-16T04:43:11.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7986b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:11.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7986b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:11.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7986b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:11.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7a06c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:11.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7a06c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:11.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7a06c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:43:11.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7986b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:11.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7986b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:11.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7986b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:43:11.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7a06c640 10 monclient(hunting): handle_auth_done global_id 44466 payload 293 2024-09-16T04:43:11.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7a06c640 1 -- 172.21.15.46:0/1066032425 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ffb600051c0 msgr2=0x7ffb6000b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:11.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7a06c640 1 --2- 172.21.15.46:0/1066032425 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ffb600051c0 0x7ffb6000b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7a06c640 1 -- 172.21.15.46:0/1066032425 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb60004800 msgr2=0x7ffb60004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:43:11.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7a06c640 1 --2- 172.21.15.46:0/1066032425 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb60004800 0x7ffb60004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7a06c640 10 monclient: _finish_hunting 0 2024-09-16T04:43:11.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7a06c640 1 monclient: found mon.a 2024-09-16T04:43:11.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7a86d640 1 -- 172.21.15.46:0/1066032425 reap_dead start 2024-09-16T04:43:11.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7a06c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:11.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7a06c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:11.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7a06c640 1 -- 172.21.15.46:0/1066032425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb60003520 con 0x7ffb600039b0 2024-09-16T04:43:11.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7a06c640 10 monclient: _finish_auth 0 2024-09-16T04:43:11.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7a06c640 10 monclient: _check_auth_tickets 2024-09-16T04:43:11.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7a06c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:11.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7c2d0640 5 monclient: authenticate success, global_id 44466 2024-09-16T04:43:11.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7c2d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:11.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.750+0000 7ffb7a06c640 1 --2- 172.21.15.46:0/1066032425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb600039b0 0x7ffb60003dd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009ac0 tx=0x7ffb60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50fbac6ab1fbaff3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:11.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7906a640 1 -- 172.21.15.46:0/1066032425 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb70036070 con 0x7ffb600039b0 2024-09-16T04:43:11.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7906a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:11.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7906a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:11.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7906a640 10 monclient: dump: 2024-09-16T04:43:11.776 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:11.776 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:11.776 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.776 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.776 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:11.776 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.776 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:11.777 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:11.777 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:11.777 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7906a640 1 -- 172.21.15.46:0/1066032425 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb70031400 con 0x7ffb600039b0 2024-09-16T04:43:11.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7c2d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:11.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7906a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:11.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7c2d0640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:11.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7906a640 1 -- 172.21.15.46:0/1066032425 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb70009690 con 0x7ffb600039b0 2024-09-16T04:43:11.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7906a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:11.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7c2d0640 4 set_mon_vals no callback set 2024-09-16T04:43:11.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7906a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:11.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7906a640 10 monclient: dump: 2024-09-16T04:43:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:11.778 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7c2d0640 10 monclient: shutdown 2024-09-16T04:43:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7c2d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7c2d0640 1 -- 172.21.15.46:0/1066032425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb600039b0 msgr2=0x7ffb60003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7c2d0640 1 --2- 172.21.15.46:0/1066032425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb600039b0 0x7ffb60003dd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009ac0 tx=0x7ffb60005830 comp rx=0 tx=0).stop 2024-09-16T04:43:11.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7c2d0640 1 -- 172.21.15.46:0/1066032425 shutdown_connections 2024-09-16T04:43:11.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7c2d0640 1 --2- 172.21.15.46:0/1066032425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb600039b0 0x7ffb60003dd0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7c2d0640 1 -- 172.21.15.46:0/1066032425 >> 172.21.15.46:0/1066032425 conn(0x7ffb740a60c0 msgr2=0x7ffb740a64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:11.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7c2d0640 1 -- 172.21.15.46:0/1066032425 shutdown_connections 2024-09-16T04:43:11.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.751+0000 7ffb7c2d0640 1 -- 172.21.15.46:0/1066032425 wait complete. 2024-09-16T04:43:11.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.752+0000 7ffb7c2d0640 10 monclient: build_initial_monmap 2024-09-16T04:43:11.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.752+0000 7ffb7c2d0640 10 monclient: monmap: 2024-09-16T04:43:11.779 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:11.779 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:11.779 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:11.779 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:11.779 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:11.780 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.752+0000 7ffb7c2d0640 1 Processor -- start 2024-09-16T04:43:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.752+0000 7ffb7c2d0640 1 -- start start 2024-09-16T04:43:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.752+0000 7ffb7c2d0640 10 monclient: init 2024-09-16T04:43:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.752+0000 7ffb7c2d0640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.752+0000 7ffb7c2d0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:43:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.752+0000 7ffb7c2d0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb600039b0 0x7ffb741cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.752+0000 7ffb7c2d0640 10 monclient(hunting): picked mon.noname-a con 0x7ffb600039b0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7c2d0640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ffb741cc490 0x7ffb74123ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7c2d0640 10 monclient(hunting): picked mon.noname-c con 0x7ffb741cc490 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7c2d0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb741cc8d0 0x7ffb74124530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7c2d0640 10 monclient(hunting): picked mon.noname-b con 0x7ffb741cc8d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7c2d0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7c2d0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7409c010 con 0x7ffb600039b0 2024-09-16T04:43:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7c2d0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7c2d0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7409be90 con 0x7ffb741cc8d0 2024-09-16T04:43:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7c2d0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7c2d0640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7ffb7409b2a0 con 0x7ffb741cc490 2024-09-16T04:43:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7c2d0640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7c2d0640 10 monclient(hunting): authenticate will time out at 990.169058s 2024-09-16T04:43:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7986b640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ffb741cc490 msgr2=0x7ffb74123ff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:43:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7986b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ffb741cc490 0x7ffb74123ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a06c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb600039b0 0x7ffb741cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a06c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb600039b0 0x7ffb741cbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40038/0 (socket says 172.21.15.46:40038) 2024-09-16T04:43:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a06c640 1 -- 172.21.15.46:0/920939058 learned_addr learned my addr 172.21.15.46:0/920939058 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a06c640 10 monclient(hunting): get_auth_request con 0x7ffb600039b0 auth_method 0 2024-09-16T04:43:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a06c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a06c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a06c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a06c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a06c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a06c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a86d640 1 --2- 172.21.15.46:0/920939058 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb741cc8d0 0x7ffb74124530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a86d640 10 monclient(hunting): get_auth_request con 0x7ffb741cc8d0 auth_method 0 2024-09-16T04:43:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a86d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a86d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a86d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a06c640 10 monclient(hunting): handle_auth_done global_id 44469 payload 824 2024-09-16T04:43:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a06c640 1 -- 172.21.15.46:0/920939058 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ffb741cc490 msgr2=0x7ffb74123ff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:43:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a06c640 1 --2- 172.21.15.46:0/920939058 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ffb741cc490 0x7ffb74123ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a06c640 1 -- 172.21.15.46:0/920939058 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb741cc8d0 msgr2=0x7ffb74124530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a06c640 1 --2- 172.21.15.46:0/920939058 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb741cc8d0 0x7ffb74124530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a06c640 10 monclient: _finish_hunting 0 2024-09-16T04:43:11.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a06c640 1 monclient: found mon.noname-a 2024-09-16T04:43:11.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a06c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:11.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a06c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:43:11.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a06c640 1 -- 172.21.15.46:0/920939058 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb741267f0 con 0x7ffb600039b0 2024-09-16T04:43:11.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a86d640 1 --2- 172.21.15.46:0/920939058 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb741cc8d0 0x7ffb74124530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:43:11.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7a06c640 1 --2- 172.21.15.46:0/920939058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb600039b0 0x7ffb741cbf50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ffb70034040 tx=0x7ffb70004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:11.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb66ffd640 1 -- 172.21.15.46:0/920939058 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb70036050 con 0x7ffb600039b0 2024-09-16T04:43:11.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:11.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb66ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:43:11.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb66ffd640 10 monclient: dump: 2024-09-16T04:43:11.785 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:11.785 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:11.785 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.785 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.785 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:11.785 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.785 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:11.785 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:11.785 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:11.785 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb66ffd640 10 monclient: _finish_auth 0 2024-09-16T04:43:11.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb66ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:43:11.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:11.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7c2d0640 5 monclient: authenticate success, global_id 44469 2024-09-16T04:43:11.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb66ffd640 1 -- 172.21.15.46:0/920939058 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb70042070 con 0x7ffb600039b0 2024-09-16T04:43:11.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7c2d0640 10 monclient: _renew_subs 2024-09-16T04:43:11.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7c2d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:11.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7c2d0640 1 -- 172.21.15.46:0/920939058 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb741279c0 con 0x7ffb600039b0 2024-09-16T04:43:11.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb66ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:11.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb66ffd640 1 -- 172.21.15.46:0/920939058 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb700317f0 con 0x7ffb600039b0 2024-09-16T04:43:11.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:11.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:11.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb66ffd640 10 monclient: dump: 2024-09-16T04:43:11.787 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:11.787 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:11.787 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.787 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.787 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:11.787 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:11.788 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7c2d0640 10 monclient: _renew_subs 2024-09-16T04:43:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb78869640 4 set_mon_vals no callback set 2024-09-16T04:43:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7c2d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.753+0000 7ffb7c2d0640 1 -- 172.21.15.46:0/920939058 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb74127f80 con 0x7ffb600039b0 2024-09-16T04:43:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.754+0000 7ffb7c2d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.754+0000 7ffb7c2d0640 1 --2- 172.21.15.46:0/920939058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb74124a70 0x7ffb74124e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.754+0000 7ffb7c2d0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.754+0000 7ffb7c2d0640 1 -- 172.21.15.46:0/920939058 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffb740a4130 con 0x7ffb74124a70 2024-09-16T04:43:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.754+0000 7ffb66ffd640 1 -- 172.21.15.46:0/920939058 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7ffb70031990 con 0x7ffb600039b0 2024-09-16T04:43:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.754+0000 7ffb7986b640 1 --2- 172.21.15.46:0/920939058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb74124a70 0x7ffb74124e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.754+0000 7ffb7986b640 10 monclient: get_auth_request con 0x7ffb74124a70 auth_method 0 2024-09-16T04:43:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.754+0000 7ffb7986b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.754+0000 7ffb7986b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.754+0000 7ffb7986b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.754+0000 7ffb7986b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.754+0000 7ffb7986b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.754+0000 7ffb7986b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.754+0000 7ffb66ffd640 1 --2- 172.21.15.46:0/920939058 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ffb58038360 0x7ffb5803a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.754+0000 7ffb7986b640 10 monclient(hunting): handle_auth_done global_id 44472 payload 824 2024-09-16T04:43:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.754+0000 7ffb66ffd640 1 -- 172.21.15.46:0/920939058 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 4738+0+0 (secure 0 0 0) 0x7ffb7006c110 con 0x7ffb600039b0 2024-09-16T04:43:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.754+0000 7ffb7986b640 1 -- 172.21.15.46:0/920939058 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffb74125390 con 0x7ffb74124a70 2024-09-16T04:43:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.755+0000 7ffb7a86d640 1 --2- 172.21.15.46:0/920939058 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ffb58038360 0x7ffb5803a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.755+0000 7ffb7a86d640 10 monclient: get_auth_request con 0x7ffb58038360 auth_method 0 2024-09-16T04:43:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.755+0000 7ffb7986b640 1 --2- 172.21.15.46:0/920939058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb74124a70 0x7ffb74124e50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ffb74127bd0 tx=0x7ffb68007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.755+0000 7ffb66ffd640 1 -- 172.21.15.46:0/920939058 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb6800f850 con 0x7ffb74124a70 2024-09-16T04:43:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.755+0000 7ffb66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.755+0000 7ffb66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.755+0000 7ffb66ffd640 10 monclient: dump: 2024-09-16T04:43:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:11.792 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.755+0000 7ffb66ffd640 1 -- 172.21.15.46:0/920939058 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ffb68018070 con 0x7ffb74124a70 2024-09-16T04:43:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.755+0000 7ffb66ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.755+0000 7ffb66ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.755+0000 7ffb66ffd640 1 -- 172.21.15.46:0/920939058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb74124a70 msgr2=0x7ffb74124e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.755+0000 7ffb66ffd640 1 --2- 172.21.15.46:0/920939058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb74124a70 0x7ffb74124e50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ffb74127bd0 tx=0x7ffb68007720 comp rx=0 tx=0).stop 2024-09-16T04:43:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.755+0000 7ffb66ffd640 1 -- 172.21.15.46:0/920939058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb74124a70 msgr2=0x7ffb74124e50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:43:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.755+0000 7ffb66ffd640 1 --2- 172.21.15.46:0/920939058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb74124a70 0x7ffb74124e50 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.755+0000 7ffb7a86d640 1 --2- 172.21.15.46:0/920939058 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ffb58038360 0x7ffb5803a820 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ffb74068440 tx=0x7ffb6c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.761+0000 7ffb7c2d0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:43:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.761+0000 7ffb7c2d0640 1 --2- 172.21.15.46:0/920939058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb7411f7f0 0x7ffb7411fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.761+0000 7ffb7c2d0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.761+0000 7ffb7c2d0640 1 -- 172.21.15.46:0/920939058 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffb741277a0 con 0x7ffb7411f7f0 2024-09-16T04:43:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.761+0000 7ffb7986b640 1 --2- 172.21.15.46:0/920939058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb7411f7f0 0x7ffb7411fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.761+0000 7ffb7986b640 10 monclient: get_auth_request con 0x7ffb7411f7f0 auth_method 0 2024-09-16T04:43:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.761+0000 7ffb7986b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.761+0000 7ffb7986b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.761+0000 7ffb7986b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.761+0000 7ffb7986b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.761+0000 7ffb7986b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.761+0000 7ffb7986b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.762+0000 7ffb7986b640 10 monclient(hunting): handle_auth_done global_id 44475 payload 824 2024-09-16T04:43:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.762+0000 7ffb7986b640 1 -- 172.21.15.46:0/920939058 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ffb741260d0 con 0x7ffb7411f7f0 2024-09-16T04:43:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.762+0000 7ffb7986b640 1 --2- 172.21.15.46:0/920939058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb7411f7f0 0x7ffb7411fbd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ffb68001200 tx=0x7ffb68001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.762+0000 7ffb66ffd640 1 -- 172.21.15.46:0/920939058 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb68013600 con 0x7ffb7411f7f0 2024-09-16T04:43:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.762+0000 7ffb66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.762+0000 7ffb66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.762+0000 7ffb66ffd640 10 monclient: dump: 2024-09-16T04:43:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:11.795 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.762+0000 7ffb66ffd640 1 -- 172.21.15.46:0/920939058 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4702 (secure 0 0 0) 0x7ffb68013600 con 0x7ffb7411f7f0 2024-09-16T04:43:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.762+0000 7ffb66ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:43:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.762+0000 7ffb66ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.762+0000 7ffb66ffd640 1 -- 172.21.15.46:0/920939058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb7411f7f0 msgr2=0x7ffb7411fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.762+0000 7ffb66ffd640 1 --2- 172.21.15.46:0/920939058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb7411f7f0 0x7ffb7411fbd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ffb68001200 tx=0x7ffb68001a40 comp rx=0 tx=0).stop 2024-09-16T04:43:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.762+0000 7ffb66ffd640 1 -- 172.21.15.46:0/920939058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb7411f7f0 msgr2=0x7ffb7411fbd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:43:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.762+0000 7ffb66ffd640 1 --2- 172.21.15.46:0/920939058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb7411f7f0 0x7ffb7411fbd0 secure :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ffb68001200 tx=0x7ffb68001a40 comp rx=0 tx=0).stop 2024-09-16T04:43:11.795 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:43:11.796 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:43:11.796 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:43:11.796 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:43:11.796 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 52, 2024-09-16T04:43:11.796 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 71510, 2024-09-16T04:43:11.796 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:43:11.796 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:43:11.796 INFO:teuthology.orchestra.run.smithi046.stdout: 1 2024-09-16T04:43:11.796 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:11.796 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:43:11.796 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:43:11.796 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:43:11.797 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:43:11.797 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:11.797 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:11.797 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:11.797 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:11.797 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:11.797 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:11.797 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:11.797 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:11.797 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:11.797 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:11.797 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:11.797 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:11.797 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:43:11.798 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:43:11.798 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:11.798 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:11.798 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:11.798 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:11.798 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:11.798 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:11.798 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:11.798 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:11.798 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:11.798 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:11.798 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:11.798 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:11.799 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.799 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:43:11.799 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:43:11.799 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:43:11.799 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:43:11.799 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:43:11.799 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:43:11.799 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:43:11.799 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:43:11.799 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:43:11.799 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:43:11.800 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:43:11.800 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:43:11.800 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:43:11.800 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:43:11.800 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:43:11.800 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:43:11.800 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:43:11.800 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:11.800 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:11.800 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:11.800 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:11.800 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:11.800 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:11.801 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:11.801 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:11.801 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:11.801 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:11.801 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:11.801 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:11.801 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:43:11.801 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.801 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:43:11.801 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.801 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:43:11.801 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:43:11.801 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:11.802 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:11.802 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.802 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:11.802 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:43:11.802 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:11.802 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.802 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.802 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:11.802 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:43:11.802 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:11.802 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:11.802 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:11.802 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.802 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:43:11.803 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:43:11.803 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:11.803 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:11.803 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:11.803 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.803 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.803 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:43:11.803 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:43:11.803 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:11.803 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:11.803 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.803 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:11.803 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:43:11.804 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:11.804 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.804 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.804 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:11.804 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:43:11.804 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:11.804 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:11.804 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:11.804 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.804 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:43:11.804 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:43:11.804 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:11.804 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:11.804 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:11.805 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.805 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.805 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:43:11.805 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:43:11.805 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:11.805 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:11.805 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.805 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:11.805 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:43:11.805 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:11.805 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.805 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.805 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:11.806 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:43:11.806 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:11.806 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:11.806 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:11.806 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.806 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:43:11.806 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:43:11.806 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:11.806 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:11.806 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:11.806 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:11.806 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:11.806 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.806 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:43:11.807 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:43:11.807 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.807 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:11.807 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:11.807 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:43:11.807 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:11.807 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:11.807 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:43:11.807 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.807 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:11.807 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:11.807 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:43:11.807 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:11.807 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:11.808 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:43:11.808 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.808 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:11.808 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:11.808 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:43:11.808 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:11.808 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:11.808 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.808 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:43:11.808 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:43:11.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.762+0000 7ffb7c2d0640 1 -- 172.21.15.46:0/920939058 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ffb58038360 msgr2=0x7ffb5803a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:11.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.762+0000 7ffb7c2d0640 1 --2- 172.21.15.46:0/920939058 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ffb58038360 0x7ffb5803a820 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ffb74068440 tx=0x7ffb6c009040 comp rx=0 tx=0).stop 2024-09-16T04:43:11.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.762+0000 7ffb7c2d0640 10 monclient: shutdown 2024-09-16T04:43:11.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.762+0000 7ffb7c2d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:11.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.762+0000 7ffb7c2d0640 1 -- 172.21.15.46:0/920939058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb600039b0 msgr2=0x7ffb741cbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:11.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.762+0000 7ffb7c2d0640 1 --2- 172.21.15.46:0/920939058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb600039b0 0x7ffb741cbf50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ffb70034040 tx=0x7ffb70004730 comp rx=0 tx=0).stop 2024-09-16T04:43:11.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.762+0000 7ffb7a86d640 1 -- 172.21.15.46:0/920939058 reap_dead start 2024-09-16T04:43:11.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.763+0000 7ffb7a86d640 1 -- 172.21.15.46:0/920939058 reap_dead start 2024-09-16T04:43:11.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.763+0000 7ffb7c2d0640 1 -- 172.21.15.46:0/920939058 shutdown_connections 2024-09-16T04:43:11.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.763+0000 7ffb7c2d0640 1 -- 172.21.15.46:0/920939058 >> 172.21.15.46:0/920939058 conn(0x7ffb740a60c0 msgr2=0x7ffb740a8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:11.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.763+0000 7ffb7c2d0640 1 -- 172.21.15.46:0/920939058 shutdown_connections 2024-09-16T04:43:11.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.763+0000 7ffb7c2d0640 1 -- 172.21.15.46:0/920939058 wait complete. 2024-09-16T04:43:11.811 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:43:11.811 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:43:11.812 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:43:11.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.912+0000 7f4729ab7640 10 monclient: get_monmap_and_config 2024-09-16T04:43:11.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.912+0000 7f4729ab7640 10 monclient: build_initial_monmap 2024-09-16T04:43:11.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.912+0000 7f4729ab7640 10 monclient: monmap: 2024-09-16T04:43:11.914 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:11.914 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:11.914 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:11.913639+0000 2024-09-16T04:43:11.914 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:11.913639+0000 2024-09-16T04:43:11.914 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:11.915 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.915 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:11.915 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:11.915 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:11.915 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:11.915 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:11.915 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.912+0000 7f4729ab7640 1 Processor -- start 2024-09-16T04:43:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.912+0000 7f4729ab7640 1 -- start start 2024-09-16T04:43:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.912+0000 7f4729ab7640 10 monclient: init 2024-09-16T04:43:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f4729ab7640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f4729ab7640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-16T04:43:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f4729ab7640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f472406e160 0x7f47240a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f4729ab7640 10 monclient(hunting): picked mon.noname-c con 0x7f472406e160 addr v2:172.21.15.114:3300/0 2024-09-16T04:43:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f4729ab7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f47240a2ee0 addr v1:172.21.15.114:6789/0 2024-09-16T04:43:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f4729ab7640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4724137050 0x7f472413ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f4729ab7640 10 monclient(hunting): picked mon.noname-b con 0x7f4724137050 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f4729ab7640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47240a3fd0 con 0x7f47240a2ee0 2024-09-16T04:43:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f4729ab7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f4729ab7640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f47240a3300 con 0x7f472406e160 2024-09-16T04:43:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f4729ab7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f4729ab7640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f472406e540 con 0x7f4724137050 2024-09-16T04:43:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f4729ab7640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f4729ab7640 10 monclient(hunting): authenticate will time out at 990.329450s 2024-09-16T04:43:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f4723fff640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4724137050 msgr2=0x7f472413ec80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:43:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f4723fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4724137050 0x7f472413ec80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f47237fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f472406e160 0x7f47240a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f47237fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f472406e160 0x7f47240a29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.46:43904/0 (socket says 172.21.15.46:43904) 2024-09-16T04:43:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f47237fe640 1 -- 172.21.15.46:0/708462009 learned_addr learned my addr 172.21.15.46:0/708462009 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f47237fe640 10 monclient(hunting): get_auth_request con 0x7f472406e160 auth_method 0 2024-09-16T04:43:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f47237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f47237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f47237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f47237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f47237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.913+0000 7f47237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.914+0000 7f47237fe640 10 monclient(hunting): handle_auth_done global_id 5740 payload 274 2024-09-16T04:43:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.914+0000 7f47237fe640 1 -- 172.21.15.46:0/708462009 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4724137050 msgr2=0x7f472413ec80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.914+0000 7f47237fe640 1 --2- 172.21.15.46:0/708462009 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4724137050 0x7f472413ec80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.914+0000 7f47237fe640 1 -- 172.21.15.46:0/708462009 >> v1:172.21.15.114:6789/0 conn(0x7f47240a2ee0 legacy=0x7f4724136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.914+0000 7f47237fe640 10 monclient: _finish_hunting 0 2024-09-16T04:43:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.914+0000 7f47237fe640 1 monclient: found mon.noname-c 2024-09-16T04:43:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.914+0000 7f47237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.914+0000 7f47237fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:43:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.914+0000 7f47237fe640 1 -- 172.21.15.46:0/708462009 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47240a0bd0 con 0x7f472406e160 2024-09-16T04:43:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.914+0000 7f47227fc640 1 -- 172.21.15.46:0/708462009 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 250580085 0 0) 0x7f4718001110 con 0x7f47240a2ee0 2024-09-16T04:43:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.914+0000 7f47227fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:43:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.914+0000 7f47237fe640 1 --2- 172.21.15.46:0/708462009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f472406e160 0x7f47240a29a0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f47140098a0 tx=0x7f471402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30d7e79899ecb0fb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f47227fc640 1 -- 172.21.15.46:0/708462009 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4714037070 con 0x7f472406e160 2024-09-16T04:43:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f47227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f47227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:43:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f47227fc640 10 monclient: dump: 2024-09-16T04:43:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:11.931 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f47227fc640 10 monclient: _finish_auth 0 2024-09-16T04:43:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f47227fc640 10 monclient: _check_auth_tickets 2024-09-16T04:43:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f47227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f4729ab7640 5 monclient: authenticate success, global_id 5740 2024-09-16T04:43:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f47227fc640 1 -- 172.21.15.46:0/708462009 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4714032440 con 0x7f472406e160 2024-09-16T04:43:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f4729ab7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f47227fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:11.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f4729ab7640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:11.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f4729ab7640 4 set_mon_vals no callback set 2024-09-16T04:43:11.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f47227fc640 1 -- 172.21.15.46:0/708462009 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4714032810 con 0x7f472406e160 2024-09-16T04:43:11.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f47227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:11.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f47227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:43:11.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f47227fc640 10 monclient: dump: 2024-09-16T04:43:11.932 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:11.932 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:11.932 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.933 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.933 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:11.933 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.933 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:11.933 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:11.933 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:11.933 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f4729ab7640 10 monclient: shutdown 2024-09-16T04:43:11.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f4729ab7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:11.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f4729ab7640 1 -- 172.21.15.46:0/708462009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f472406e160 msgr2=0x7f47240a29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:11.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f4729ab7640 1 --2- 172.21.15.46:0/708462009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f472406e160 0x7f47240a29a0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f47140098a0 tx=0x7f471402a180 comp rx=0 tx=0).stop 2024-09-16T04:43:11.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f4729ab7640 1 -- 172.21.15.46:0/708462009 shutdown_connections 2024-09-16T04:43:11.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f4729ab7640 1 --2- 172.21.15.46:0/708462009 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4724137050 0x7f472413ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f4729ab7640 1 --2- 172.21.15.46:0/708462009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f472406e160 0x7f47240a29a0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f4729ab7640 1 -- 172.21.15.46:0/708462009 >> 172.21.15.46:0/708462009 conn(0x7f47240a5c10 msgr2=0x7f47240a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:11.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f4729ab7640 1 -- 172.21.15.46:0/708462009 shutdown_connections 2024-09-16T04:43:11.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.915+0000 7f4729ab7640 1 -- 172.21.15.46:0/708462009 wait complete. 2024-09-16T04:43:11.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.916+0000 7f4729ab7640 10 monclient: build_initial_monmap 2024-09-16T04:43:11.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.916+0000 7f4729ab7640 10 monclient: monmap: 2024-09-16T04:43:11.934 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:11.934 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:11.935 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:11.935 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:11.935 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:11.935 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.935 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:11.935 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:11.935 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:11.935 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.916+0000 7f4729ab7640 1 Processor -- start 2024-09-16T04:43:11.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.916+0000 7f4729ab7640 1 -- start start 2024-09-16T04:43:11.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.916+0000 7f4729ab7640 10 monclient: init 2024-09-16T04:43:11.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4729ab7640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:11.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4729ab7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:43:11.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4729ab7640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f472406e160 0x7f47241cbd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4729ab7640 10 monclient(hunting): picked mon.noname-c con 0x7f472406e160 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:11.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4729ab7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47240a2ee0 0x7f4724135f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4729ab7640 10 monclient(hunting): picked mon.noname-b con 0x7f47240a2ee0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:11.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4729ab7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4724137050 0x7f4724136460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f47237fe640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f472406e160 msgr2=0x7f47241cbd40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:43:11.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4729ab7640 10 monclient(hunting): picked mon.noname-a con 0x7f4724137050 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:11.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4729ab7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4729ab7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f47240a34a0 con 0x7f4724137050 2024-09-16T04:43:11.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4729ab7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4729ab7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f47240a5130 con 0x7f47240a2ee0 2024-09-16T04:43:11.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4729ab7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f47237fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f472406e160 0x7f47241cbd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:11.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4729ab7640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f47240a0300 con 0x7f472406e160 2024-09-16T04:43:11.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4729ab7640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:11.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4729ab7640 10 monclient(hunting): authenticate will time out at 990.333279s 2024-09-16T04:43:11.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4723fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4724137050 0x7f4724136460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:11.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4723fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4724137050 0x7f4724136460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40070/0 (socket says 172.21.15.46:40070) 2024-09-16T04:43:11.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4723fff640 1 -- 172.21.15.46:0/133519552 learned_addr learned my addr 172.21.15.46:0/133519552 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:11.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4723fff640 10 monclient(hunting): get_auth_request con 0x7f4724137050 auth_method 0 2024-09-16T04:43:11.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4723fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:11.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4723fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:11.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4723fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:11.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4722ffd640 1 --2- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47240a2ee0 0x7f4724135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:11.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4722ffd640 10 monclient(hunting): get_auth_request con 0x7f47240a2ee0 auth_method 0 2024-09-16T04:43:11.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4722ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:11.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4722ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:11.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4722ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:11.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4723fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:11.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4723fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:11.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4723fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:11.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4722ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:11.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4722ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:11.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4722ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:11.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4723fff640 10 monclient(hunting): handle_auth_done global_id 44478 payload 824 2024-09-16T04:43:11.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4723fff640 1 -- 172.21.15.46:0/133519552 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f472406e160 msgr2=0x7f47241cbd40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:43:11.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4723fff640 1 --2- 172.21.15.46:0/133519552 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f472406e160 0x7f47241cbd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4723fff640 1 -- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47240a2ee0 msgr2=0x7f4724135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:11.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4723fff640 1 --2- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47240a2ee0 0x7f4724135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4723fff640 10 monclient: _finish_hunting 0 2024-09-16T04:43:11.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4723fff640 1 monclient: found mon.noname-a 2024-09-16T04:43:11.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4723fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:11.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4723fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:43:11.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4723fff640 1 -- 172.21.15.46:0/133519552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47241262b0 con 0x7f4724137050 2024-09-16T04:43:11.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.917+0000 7f4723fff640 1 --2- 172.21.15.46:0/133519552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4724137050 0x7f4724136460 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f471c00be60 tx=0x7f471c00bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:11.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.918+0000 7f470bfff640 1 -- 172.21.15.46:0/133519552 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f471c016430 con 0x7f4724137050 2024-09-16T04:43:11.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.918+0000 7f470bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:11.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.918+0000 7f470bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:43:11.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.918+0000 7f470bfff640 10 monclient: dump: 2024-09-16T04:43:11.940 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:11.940 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:11.940 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.940 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.940 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:11.940 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.940 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:11.940 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:11.940 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:11.940 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.918+0000 7f470bfff640 10 monclient: _finish_auth 0 2024-09-16T04:43:11.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.918+0000 7f470bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:43:11.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.918+0000 7f470bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:11.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.918+0000 7f470bfff640 1 -- 172.21.15.46:0/133519552 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f471c010030 con 0x7f4724137050 2024-09-16T04:43:11.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.918+0000 7f4729ab7640 5 monclient: authenticate success, global_id 44478 2024-09-16T04:43:11.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.918+0000 7f470bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:11.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.918+0000 7f470bfff640 1 -- 172.21.15.46:0/133519552 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f471c016810 con 0x7f4724137050 2024-09-16T04:43:11.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.918+0000 7f4729ab7640 10 monclient: _renew_subs 2024-09-16T04:43:11.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.918+0000 7f4729ab7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:11.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.918+0000 7f4729ab7640 1 -- 172.21.15.46:0/133519552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47241369d0 con 0x7f4724137050 2024-09-16T04:43:11.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.918+0000 7f470bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:11.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.918+0000 7f470bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:11.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.918+0000 7f470bfff640 10 monclient: dump: 2024-09-16T04:43:11.941 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:11.941 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:11.941 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.942 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.942 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:11.942 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.942 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:11.942 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:11.942 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:11.942 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.918+0000 7f4721ffb640 4 set_mon_vals no callback set 2024-09-16T04:43:11.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.918+0000 7f4729ab7640 10 monclient: _renew_subs 2024-09-16T04:43:11.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.918+0000 7f4729ab7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:11.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.918+0000 7f4729ab7640 1 -- 172.21.15.46:0/133519552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4724127640 con 0x7f4724137050 2024-09-16T04:43:11.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f4729ab7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:11.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f470bfff640 1 -- 172.21.15.46:0/133519552 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f471c0169b0 con 0x7f4724137050 2024-09-16T04:43:11.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f4729ab7640 1 --2- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4724123990 0x7f4724123d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f4729ab7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f4729ab7640 1 -- 172.21.15.46:0/133519552 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f47240a3620 con 0x7f4724123990 2024-09-16T04:43:11.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f47237fe640 1 --2- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4724123990 0x7f4724123d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:11.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f470bfff640 1 --2- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4710038390 0x7f471003a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f47237fe640 10 monclient: get_auth_request con 0x7f4724123990 auth_method 0 2024-09-16T04:43:11.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f47237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:11.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f47237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:11.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f47237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:11.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f470bfff640 1 -- 172.21.15.46:0/133519552 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 4738+0+0 (secure 0 0 0) 0x7f471c04be50 con 0x7f4724137050 2024-09-16T04:43:11.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f47237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:11.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f47237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:11.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f47237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:11.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f4722ffd640 1 --2- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4710038390 0x7f471003a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:11.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f4722ffd640 10 monclient: get_auth_request con 0x7f4710038390 auth_method 0 2024-09-16T04:43:11.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f47237fe640 10 monclient(hunting): handle_auth_done global_id 5749 payload 824 2024-09-16T04:43:11.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f47237fe640 1 -- 172.21.15.46:0/133519552 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f472409efb0 con 0x7f4724123990 2024-09-16T04:43:11.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f47237fe640 1 --2- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4724123990 0x7f4724123d70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f472409fec0 tx=0x7f47140046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:11.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f4722ffd640 1 --2- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4710038390 0x7f471003a850 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f47240a7920 tx=0x7f4718009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:11.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f470bfff640 1 -- 172.21.15.46:0/133519552 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4714037040 con 0x7f4724123990 2024-09-16T04:43:11.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f470bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:11.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f470bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:43:11.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.919+0000 7f470bfff640 10 monclient: dump: 2024-09-16T04:43:11.944 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:11.944 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:11.944 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.944 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.944 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:11.944 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.945 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:11.945 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:11.945 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:11.945 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.920+0000 7f470bfff640 1 -- 172.21.15.46:0/133519552 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4714009490 con 0x7f4724123990 2024-09-16T04:43:11.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.920+0000 7f470bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:11.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.920+0000 7f470bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:11.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.920+0000 7f470bfff640 1 -- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4724123990 msgr2=0x7f4724123d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:11.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.920+0000 7f470bfff640 1 --2- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4724123990 0x7f4724123d70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f472409fec0 tx=0x7f47140046c0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.920+0000 7f470bfff640 1 -- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4724123990 msgr2=0x7f4724123d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:43:11.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.920+0000 7f470bfff640 1 --2- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4724123990 0x7f4724123d70 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:11.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.926+0000 7f4729ab7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:43:11.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.926+0000 7f4729ab7640 1 --2- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47241242b0 0x7f472411f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:11.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.926+0000 7f4729ab7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:11.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.926+0000 7f4729ab7640 1 -- 172.21.15.46:0/133519552 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f47240a4e20 con 0x7f47241242b0 2024-09-16T04:43:11.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.926+0000 7f47237fe640 1 --2- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47241242b0 0x7f472411f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:11.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.926+0000 7f47237fe640 10 monclient: get_auth_request con 0x7f47241242b0 auth_method 0 2024-09-16T04:43:11.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.926+0000 7f47237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:11.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.926+0000 7f47237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:11.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.926+0000 7f47237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:11.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.926+0000 7f47237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:11.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.926+0000 7f47237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:11.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.926+0000 7f47237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:11.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.926+0000 7f47237fe640 10 monclient(hunting): handle_auth_done global_id 5752 payload 824 2024-09-16T04:43:11.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.926+0000 7f47237fe640 1 -- 172.21.15.46:0/133519552 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4724067e00 con 0x7f47241242b0 2024-09-16T04:43:11.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f47237fe640 1 --2- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47241242b0 0x7f472411f760 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f471402a1c0 tx=0x7f4714009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:11.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f470bfff640 1 -- 172.21.15.46:0/133519552 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4714037040 con 0x7f47241242b0 2024-09-16T04:43:11.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f470bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:11.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f470bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:43:11.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f470bfff640 10 monclient: dump: 2024-09-16T04:43:11.948 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:11.948 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:11.948 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.949 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:11.949 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:11.949 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:11.949 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:11.949 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:11.949 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:11.949 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:11.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f470bfff640 1 -- 172.21.15.46:0/133519552 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4868 (secure 0 0 0) 0x7f4714046c00 con 0x7f47241242b0 2024-09-16T04:43:11.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f470bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:43:11.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f470bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:11.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f470bfff640 1 -- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47241242b0 msgr2=0x7f472411f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:11.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f470bfff640 1 --2- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47241242b0 0x7f472411f760 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f471402a1c0 tx=0x7f4714009240 comp rx=0 tx=0).stop 2024-09-16T04:43:11.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f470bfff640 1 -- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47241242b0 msgr2=0x7f472411f760 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:43:11.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f470bfff640 1 --2- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47241242b0 0x7f472411f760 secure :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f471402a1c0 tx=0x7f4714009240 comp rx=0 tx=0).stop 2024-09-16T04:43:11.950 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:43:11.950 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:43:11.950 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:43:11.950 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:43:11.950 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 52, 2024-09-16T04:43:11.950 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 277829, 2024-09-16T04:43:11.950 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:43:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:43:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: 1 2024-09-16T04:43:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:43:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:43:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:43:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:43:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:43:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:43:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:43:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:43:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:43:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:43:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:43:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:43:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:43:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:43:11.954 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:43:11.954 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:43:11.954 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:43:11.954 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:43:11.954 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:43:11.954 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:43:11.954 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:43:11.954 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:43:11.954 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:43:11.954 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:11.954 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:11.954 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:11.954 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:43:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:43:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:43:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:43:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:43:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:43:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:43:11.957 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:43:11.957 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:11.957 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:11.957 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:11.957 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.957 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.957 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:43:11.958 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:43:11.958 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:11.958 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:11.958 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.958 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:11.958 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:43:11.958 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:11.958 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.958 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.958 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:11.958 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:43:11.958 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:11.958 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:11.959 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:11.959 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.959 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:43:11.959 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:43:11.959 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:11.959 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:11.959 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:11.959 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.960 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.960 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:43:11.960 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:43:11.960 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:11.960 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:11.960 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.960 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:11.960 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:43:11.960 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:11.960 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.960 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.961 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:11.961 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:43:11.961 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:11.961 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:11.961 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:11.961 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.961 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:43:11.961 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:43:11.961 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:11.961 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:11.961 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:11.961 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:11.962 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:11.962 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.962 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:43:11.962 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:43:11.962 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.962 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:11.962 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:11.962 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:43:11.962 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:11.962 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:11.963 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:43:11.963 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.963 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:11.963 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:11.963 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4 2024-09-16T04:43:11.963 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:11.963 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:11.963 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:43:11.963 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.963 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:11.963 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:11.963 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6 2024-09-16T04:43:11.963 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:11.964 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:11.964 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:43:11.964 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:11.964 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:11.964 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:11.964 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:43:11.964 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:11.964 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:11.964 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:11.964 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:43:11.964 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:43:11.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f4729ab7640 1 -- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4710038390 msgr2=0x7f471003a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:11.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f4729ab7640 1 --2- 172.21.15.46:0/133519552 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4710038390 0x7f471003a850 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f47240a7920 tx=0x7f4718009040 comp rx=0 tx=0).stop 2024-09-16T04:43:11.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f4729ab7640 10 monclient: shutdown 2024-09-16T04:43:11.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f4729ab7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:11.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f4729ab7640 1 -- 172.21.15.46:0/133519552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4724137050 msgr2=0x7f4724136460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:11.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f4729ab7640 1 --2- 172.21.15.46:0/133519552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4724137050 0x7f4724136460 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f471c00be60 tx=0x7f471c00bf50 comp rx=0 tx=0).stop 2024-09-16T04:43:11.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f4723fff640 1 -- 172.21.15.46:0/133519552 reap_dead start 2024-09-16T04:43:11.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f4723fff640 1 -- 172.21.15.46:0/133519552 reap_dead start 2024-09-16T04:43:11.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f4729ab7640 1 -- 172.21.15.46:0/133519552 shutdown_connections 2024-09-16T04:43:11.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f4729ab7640 1 -- 172.21.15.46:0/133519552 >> 172.21.15.46:0/133519552 conn(0x7f47240a5c10 msgr2=0x7f472409d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:11.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f4729ab7640 1 -- 172.21.15.46:0/133519552 shutdown_connections 2024-09-16T04:43:11.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:11.927+0000 7f4729ab7640 1 -- 172.21.15.46:0/133519552 wait complete. 2024-09-16T04:43:11.968 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:43:11.968 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:43:12.362 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37842]: starting. 2024-09-16T04:43:12.362 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37842]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-37840 2024-09-16T04:43:12.362 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37842]: shutting down. 2024-09-16T04:43:14.455 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:14.455 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37875]: starting. 2024-09-16T04:43:14.455 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37875]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-37840 2024-09-16T04:43:14.455 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37875]: created object 0... 2024-09-16T04:43:14.455 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37875]: created object 25... 2024-09-16T04:43:14.455 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37875]: created object 49... 2024-09-16T04:43:14.455 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37875]: finishing. 2024-09-16T04:43:14.456 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37875]: shutting down. 2024-09-16T04:43:14.468 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:14.469 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37877]: starting. 2024-09-16T04:43:14.469 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37877]: listing objects. 2024-09-16T04:43:14.469 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37877]: listed object 0... 2024-09-16T04:43:14.469 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37877]: listed object 25... 2024-09-16T04:43:14.469 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37877]: saw 50 objects 2024-09-16T04:43:14.469 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37877]: shutting down. 2024-09-16T04:43:15.379 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:15.379 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37876]: starting. 2024-09-16T04:43:15.379 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37876]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-37840 2024-09-16T04:43:15.379 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37876]: shutting down. 2024-09-16T04:43:15.380 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:15.381 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:15.381 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:43:15.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:43:17.477 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37944]: starting. 2024-09-16T04:43:17.477 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37944]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-37943 2024-09-16T04:43:17.477 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37944]: created object 0... 2024-09-16T04:43:17.477 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37944]: created object 25... 2024-09-16T04:43:17.477 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37944]: created object 49... 2024-09-16T04:43:17.478 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37944]: finishing. 2024-09-16T04:43:17.478 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[37944]: shutting down. 2024-09-16T04:43:18.397 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37945]: starting. 2024-09-16T04:43:18.398 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37945]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-37943 2024-09-16T04:43:18.398 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[37945]: shutting down. 2024-09-16T04:43:20.362 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:20.362 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37978]: starting. 2024-09-16T04:43:20.362 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37978]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-37943 2024-09-16T04:43:20.362 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37978]: created object 0... 2024-09-16T04:43:20.362 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37978]: created object 25... 2024-09-16T04:43:20.362 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37978]: created object 49... 2024-09-16T04:43:20.363 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37978]: finishing. 2024-09-16T04:43:20.363 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[37978]: shutting down. 2024-09-16T04:43:20.374 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:20.374 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37980]: starting. 2024-09-16T04:43:20.374 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37980]: listing objects. 2024-09-16T04:43:20.374 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37980]: listed object 0... 2024-09-16T04:43:20.374 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37980]: listed object 25... 2024-09-16T04:43:20.375 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37980]: saw 50 objects 2024-09-16T04:43:20.375 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[37980]: shutting down. 2024-09-16T04:43:21.277 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:21.277 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37979]: starting. 2024-09-16T04:43:21.277 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37979]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-37943 2024-09-16T04:43:21.277 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[37979]: shutting down. 2024-09-16T04:43:21.278 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:21.279 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:21.279 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:43:21.281 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:43:23.375 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38047]: starting. 2024-09-16T04:43:23.375 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38047]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-38046 2024-09-16T04:43:23.375 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38047]: created object 0... 2024-09-16T04:43:23.375 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38047]: created object 25... 2024-09-16T04:43:23.375 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38047]: created object 49... 2024-09-16T04:43:23.375 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38047]: finishing. 2024-09-16T04:43:23.375 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38047]: shutting down. 2024-09-16T04:43:24.296 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38048]: starting. 2024-09-16T04:43:24.296 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38048]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-38046 2024-09-16T04:43:24.297 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38048]: shutting down. 2024-09-16T04:43:26.383 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:26.384 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38081]: starting. 2024-09-16T04:43:26.384 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38081]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-38046 2024-09-16T04:43:26.384 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38081]: created object 0... 2024-09-16T04:43:26.384 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38081]: created object 25... 2024-09-16T04:43:26.384 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38081]: created object 49... 2024-09-16T04:43:26.384 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38081]: finishing. 2024-09-16T04:43:26.384 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38081]: shutting down. 2024-09-16T04:43:26.397 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:26.397 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38083]: starting. 2024-09-16T04:43:26.397 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38083]: listing objects. 2024-09-16T04:43:26.397 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38083]: listed object 0... 2024-09-16T04:43:26.397 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38083]: listed object 25... 2024-09-16T04:43:26.397 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38083]: saw 50 objects 2024-09-16T04:43:26.397 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38083]: shutting down. 2024-09-16T04:43:27.304 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:27.305 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38082]: starting. 2024-09-16T04:43:27.305 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38082]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-38046 2024-09-16T04:43:27.305 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38082]: shutting down. 2024-09-16T04:43:27.306 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:27.306 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:27.306 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:43:27.309 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:43:29.409 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38150]: starting. 2024-09-16T04:43:29.409 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38150]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-38149 2024-09-16T04:43:29.409 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38150]: created object 0... 2024-09-16T04:43:29.409 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38150]: created object 25... 2024-09-16T04:43:29.409 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38150]: created object 49... 2024-09-16T04:43:29.409 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38150]: finishing. 2024-09-16T04:43:29.409 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38150]: shutting down. 2024-09-16T04:43:30.276 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38151]: starting. 2024-09-16T04:43:30.276 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38151]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-38149 2024-09-16T04:43:30.276 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38151]: shutting down. 2024-09-16T04:43:31.969 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:43:31.969 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T04:43:31.970 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:43:31.970 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:43:31.973 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:43:31.974 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:43:31.974 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:43:32.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.086+0000 7fe401f6d640 10 monclient: get_monmap_and_config 2024-09-16T04:43:32.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.086+0000 7fe401f6d640 10 monclient: build_initial_monmap 2024-09-16T04:43:32.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.086+0000 7fe401f6d640 10 monclient: monmap: 2024-09-16T04:43:32.087 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:32.087 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:32.087 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:32.086728+0000 2024-09-16T04:43:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:32.086728+0000 2024-09-16T04:43:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:32.088 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:32.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe401f6d640 1 Processor -- start 2024-09-16T04:43:32.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe401f6d640 1 -- start start 2024-09-16T04:43:32.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe401f6d640 10 monclient: init 2024-09-16T04:43:32.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe401f6d640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:32.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe401f6d640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-16T04:43:32.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe401f6d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3fc0a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:43:32.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe401f6d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe3fc0a3ff0 0x7fe3fc0a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:32.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe401f6d640 10 monclient(hunting): picked mon.noname-b con 0x7fe3fc0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:32.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe401f6d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe3fc13be90 0x7fe3fc13e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:32.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe401f6d640 10 monclient(hunting): picked mon.noname-c con 0x7fe3fc13be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:43:32.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe401f6d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3fc09b940 con 0x7fe3fc0a3c10 2024-09-16T04:43:32.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe401f6d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:32.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe401f6d640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fe3fc09ab00 con 0x7fe3fc13be90 2024-09-16T04:43:32.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe401f6d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:32.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe401f6d640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe3fc09b030 con 0x7fe3fc0a3ff0 2024-09-16T04:43:32.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe401f6d640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:32.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe401f6d640 10 monclient(hunting): authenticate will time out at 1010.502630s 2024-09-16T04:43:32.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe3f2ffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe3fc0a3ff0 0x7fe3fc0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:32.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe3fb7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe3fc0a3c10 0x7fe3fc09bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55234/0 (socket says 172.21.15.46:55234) 2024-09-16T04:43:32.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe3fb7fe640 1 -- 172.21.15.46:0/2815202043 learned_addr learned my addr 172.21.15.46:0/2815202043 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:43:32.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe3f2ffd640 10 monclient(hunting): get_auth_request con 0x7fe3fc0a3ff0 auth_method 0 2024-09-16T04:43:32.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe3f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:32.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe3f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:32.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe3f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:32.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe3f2ffd640 1 -- 172.21.15.46:0/2815202043 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe3fc0a3ff0 msgr2=0x7fe3fc0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:43:32.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe3f2ffd640 1 -- 172.21.15.46:0/2815202043 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe3fc0a3ff0 msgr2=0x7fe3fc0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:32.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe3f2ffd640 1 --2- 172.21.15.46:0/2815202043 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe3fc0a3ff0 0x7fe3fc0a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:32.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe3f2ffd640 1 --2- 172.21.15.46:0/2815202043 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe3fc0a3ff0 0x7fe3fc0a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:32.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.087+0000 7fe3fbfff640 1 --2- 172.21.15.46:0/2815202043 >> v2:172.21.15.114:3300/0 conn(0x7fe3fc13be90 0x7fe3fc13e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:32.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3fbfff640 10 monclient(hunting): get_auth_request con 0x7fe3fc13be90 auth_method 0 2024-09-16T04:43:32.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:32.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3faffd640 1 -- 172.21.15.46:0/2815202043 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1074728508 0 0) 0x7fe3fc09b940 con 0x7fe3fc0a3c10 2024-09-16T04:43:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3faffd640 10 monclient(hunting): _negotiate 2024-09-16T04:43:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3faffd640 10 monclient(hunting): my global_id is 44601 2024-09-16T04:43:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3faffd640 1 -- 172.21.15.46:0/2815202043 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3e8003520 con 0x7fe3fc0a3c10 2024-09-16T04:43:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3fbfff640 10 monclient(hunting): handle_auth_done global_id 5869 payload 274 2024-09-16T04:43:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3faffd640 1 -- 172.21.15.46:0/2815202043 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2004006261 0 0) 0x7fe3e8003520 con 0x7fe3fc0a3c10 2024-09-16T04:43:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3fbfff640 1 -- 172.21.15.46:0/2815202043 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe3fc0a3ff0 msgr2=0x7fe3fc0a43d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3fbfff640 1 --2- 172.21.15.46:0/2815202043 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe3fc0a3ff0 0x7fe3fc0a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:32.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3fbfff640 1 -- 172.21.15.46:0/2815202043 >> v1:172.21.15.46:6789/0 conn(0x7fe3fc0a3c10 legacy=0x7fe3fc09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:32.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3fbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:43:32.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3fbfff640 1 monclient: found mon.noname-c 2024-09-16T04:43:32.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:32.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3fbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:43:32.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3fbfff640 1 -- 172.21.15.46:0/2815202043 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3fc09b310 con 0x7fe3fc13be90 2024-09-16T04:43:32.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.088+0000 7fe3faffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:43:32.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe3fbfff640 1 --2- 172.21.15.46:0/2815202043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3fc13be90 0x7fe3fc13e280 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009d90 tx=0x7fe3f402a560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=124bf2b2a725d384 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:32.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe3faffd640 1 -- 172.21.15.46:0/2815202043 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec001670 con 0x7fe3fc13be90 2024-09-16T04:43:32.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe3faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:32.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe3faffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:43:32.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe3faffd640 10 monclient: dump: 2024-09-16T04:43:32.100 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:32.100 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:32.101 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:32.101 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:32.101 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:32.101 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:32.101 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:32.101 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:32.101 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:32.101 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:32.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe3faffd640 10 monclient: _finish_auth 0 2024-09-16T04:43:32.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe3faffd640 10 monclient: _check_auth_tickets 2024-09-16T04:43:32.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe3faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:32.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe401f6d640 5 monclient: authenticate success, global_id 5869 2024-09-16T04:43:32.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe401f6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:32.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe3faffd640 1 -- 172.21.15.46:0/2815202043 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3f4007540 con 0x7fe3fc13be90 2024-09-16T04:43:32.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe3faffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:32.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe401f6d640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:32.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe401f6d640 4 set_mon_vals no callback set 2024-09-16T04:43:32.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe3faffd640 1 -- 172.21.15.46:0/2815202043 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f40078f0 con 0x7fe3fc13be90 2024-09-16T04:43:32.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe3faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:32.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe3faffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:43:32.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe3faffd640 10 monclient: dump: 2024-09-16T04:43:32.102 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:32.102 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:32.102 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:32.102 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:32.102 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:32.102 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:32.102 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:32.103 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:32.103 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:32.103 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:32.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe401f6d640 10 monclient: shutdown 2024-09-16T04:43:32.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe401f6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:32.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe401f6d640 1 -- 172.21.15.46:0/2815202043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3fc13be90 msgr2=0x7fe3fc13e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:32.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe401f6d640 1 --2- 172.21.15.46:0/2815202043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3fc13be90 0x7fe3fc13e280 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009d90 tx=0x7fe3f402a560 comp rx=0 tx=0).stop 2024-09-16T04:43:32.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe401f6d640 1 -- 172.21.15.46:0/2815202043 shutdown_connections 2024-09-16T04:43:32.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe401f6d640 1 --2- 172.21.15.46:0/2815202043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3fc13be90 0x7fe3fc13e280 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:32.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe401f6d640 1 --2- 172.21.15.46:0/2815202043 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe3fc0a3ff0 0x7fe3fc0a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:32.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe401f6d640 1 -- 172.21.15.46:0/2815202043 >> 172.21.15.46:0/2815202043 conn(0x7fe3fc0a6340 msgr2=0x7fe3fc06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:32.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe401f6d640 1 -- 172.21.15.46:0/2815202043 shutdown_connections 2024-09-16T04:43:32.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.089+0000 7fe401f6d640 1 -- 172.21.15.46:0/2815202043 wait complete. 2024-09-16T04:43:32.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.090+0000 7fe401f6d640 10 monclient: build_initial_monmap 2024-09-16T04:43:32.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.090+0000 7fe401f6d640 10 monclient: monmap: 2024-09-16T04:43:32.104 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:32.104 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:32.104 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:32.104 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:32.104 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:32.104 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:32.104 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:32.104 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:32.104 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:32.104 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:32.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.090+0000 7fe401f6d640 1 Processor -- start 2024-09-16T04:43:32.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe401f6d640 1 -- start start 2024-09-16T04:43:32.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe401f6d640 10 monclient: init 2024-09-16T04:43:32.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe401f6d640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:32.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe401f6d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:43:32.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe401f6d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3fc0a3c10 0x7fe3fc1cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:32.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe401f6d640 10 monclient(hunting): picked mon.noname-a con 0x7fe3fc0a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:32.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe401f6d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe3fc0a3ff0 0x7fe3fc136d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:32.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe401f6d640 10 monclient(hunting): picked mon.noname-c con 0x7fe3fc0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:32.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe401f6d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3fc13be90 0x7fe3fc09e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:32.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe401f6d640 10 monclient(hunting): picked mon.noname-b con 0x7fe3fc13be90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:32.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe401f6d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:32.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe401f6d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc0a6b40 con 0x7fe3fc0a3c10 2024-09-16T04:43:32.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe401f6d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:32.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe401f6d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc0a69c0 con 0x7fe3fc13be90 2024-09-16T04:43:32.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe401f6d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:32.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe401f6d640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe3fc13e7c0 con 0x7fe3fc0a3ff0 2024-09-16T04:43:32.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe401f6d640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:32.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe401f6d640 10 monclient(hunting): authenticate will time out at 1010.506535s 2024-09-16T04:43:32.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3f2ffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe3fc0a3ff0 0x7fe3fc136d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:32.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3fb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3fc0a3c10 0x7fe3fc1cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:32.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3fb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3fc0a3c10 0x7fe3fc1cb740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44338/0 (socket says 172.21.15.46:44338) 2024-09-16T04:43:32.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3f2ffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe3fc0a3ff0 0x7fe3fc136d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:34442/0 (socket says 172.21.15.46:34442) 2024-09-16T04:43:32.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3fb7fe640 1 -- 172.21.15.46:0/2168102499 learned_addr learned my addr 172.21.15.46:0/2168102499 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:32.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3fb7fe640 10 monclient(hunting): get_auth_request con 0x7fe3fc0a3c10 auth_method 0 2024-09-16T04:43:32.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:32.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:32.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:32.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3f2ffd640 10 monclient(hunting): get_auth_request con 0x7fe3fc0a3ff0 auth_method 0 2024-09-16T04:43:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3fbfff640 1 --2- 172.21.15.46:0/2168102499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3fc13be90 0x7fe3fc09e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3f2ffd640 1 -- 172.21.15.46:0/2168102499 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe3fc0a3ff0 msgr2=0x7fe3fc136d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:43:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3f2ffd640 1 -- 172.21.15.46:0/2168102499 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe3fc0a3ff0 msgr2=0x7fe3fc136d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:43:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3f2ffd640 1 --2- 172.21.15.46:0/2168102499 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe3fc0a3ff0 0x7fe3fc136d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3fbfff640 10 monclient(hunting): get_auth_request con 0x7fe3fc13be90 auth_method 0 2024-09-16T04:43:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3f2ffd640 1 --2- 172.21.15.46:0/2168102499 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe3fc0a3ff0 0x7fe3fc136d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.091+0000 7fe3fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3fb7fe640 10 monclient(hunting): handle_auth_done global_id 44604 payload 824 2024-09-16T04:43:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3fb7fe640 1 -- 172.21.15.46:0/2168102499 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe3fc0a3ff0 msgr2=0x7fe3fc136d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:43:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3fb7fe640 1 --2- 172.21.15.46:0/2168102499 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe3fc0a3ff0 0x7fe3fc136d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3fb7fe640 1 -- 172.21.15.46:0/2168102499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3fc13be90 msgr2=0x7fe3fc09e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3fb7fe640 1 --2- 172.21.15.46:0/2168102499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3fc13be90 0x7fe3fc09e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3fb7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:43:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3fb7fe640 1 monclient: found mon.noname-a 2024-09-16T04:43:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3fb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:43:32.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3fb7fe640 1 -- 172.21.15.46:0/2168102499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3fc1216b0 con 0x7fe3fc0a3c10 2024-09-16T04:43:32.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3fb7fe640 1 --2- 172.21.15.46:0/2168102499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3fc0a3c10 0x7fe3fc1cb740 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec00c780 tx=0x7fe3ec00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:32.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3f8ff9640 1 -- 172.21.15.46:0/2168102499 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec017070 con 0x7fe3fc0a3c10 2024-09-16T04:43:32.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:32.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3f8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:43:32.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3f8ff9640 10 monclient: dump: 2024-09-16T04:43:32.109 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:32.109 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:32.109 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:32.109 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:32.109 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:32.109 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:32.109 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:32.110 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:32.110 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:32.110 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:32.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3f8ff9640 10 monclient: _finish_auth 0 2024-09-16T04:43:32.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3f8ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:43:32.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:32.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3f8ff9640 1 -- 172.21.15.46:0/2168102499 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3ec005ad0 con 0x7fe3fc0a3c10 2024-09-16T04:43:32.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3f8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:32.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3f8ff9640 1 -- 172.21.15.46:0/2168102499 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec007070 con 0x7fe3fc0a3c10 2024-09-16T04:43:32.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe401f6d640 5 monclient: authenticate success, global_id 44604 2024-09-16T04:43:32.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe401f6d640 10 monclient: _renew_subs 2024-09-16T04:43:32.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe401f6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:32.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe401f6d640 1 -- 172.21.15.46:0/2168102499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3fc13c390 con 0x7fe3fc0a3c10 2024-09-16T04:43:32.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3fa7fc640 4 set_mon_vals no callback set 2024-09-16T04:43:32.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3f8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe3f8ff9640 10 monclient: dump: 2024-09-16T04:43:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:32.111 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe401f6d640 10 monclient: _renew_subs 2024-09-16T04:43:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe401f6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.092+0000 7fe401f6d640 1 -- 172.21.15.46:0/2168102499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3fc0a0400 con 0x7fe3fc0a3c10 2024-09-16T04:43:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.093+0000 7fe3f8ff9640 1 -- 172.21.15.46:0/2168102499 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fe3ec01a070 con 0x7fe3fc0a3c10 2024-09-16T04:43:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.093+0000 7fe401f6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.093+0000 7fe401f6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.093+0000 7fe401f6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.093+0000 7fe401f6d640 1 -- 172.21.15.46:0/2168102499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3fc0a43d0 con 0x7fe3fc0a3c10 2024-09-16T04:43:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.093+0000 7fe3f8ff9640 1 --2- 172.21.15.46:0/2168102499 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe3dc0383b0 0x7fe3dc03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.093+0000 7fe3f8ff9640 1 -- 172.21.15.46:0/2168102499 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(186..186 src has 1..186) ==== 4738+0+0 (secure 0 0 0) 0x7fe3ec04adf0 con 0x7fe3fc0a3c10 2024-09-16T04:43:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.093+0000 7fe3f2ffd640 1 --2- 172.21.15.46:0/2168102499 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe3dc0383b0 0x7fe3dc03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.093+0000 7fe3f2ffd640 10 monclient: get_auth_request con 0x7fe3dc0383b0 auth_method 0 2024-09-16T04:43:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.094+0000 7fe3f2ffd640 1 --2- 172.21.15.46:0/2168102499 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe3dc0383b0 0x7fe3dc03a870 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc120860 tx=0x7fe3e4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:32.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.098+0000 7fe3f8ff9640 1 -- 172.21.15.46:0/2168102499 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe3ec01a360 con 0x7fe3fc0a3c10 2024-09-16T04:43:32.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.098+0000 7fe3f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:32.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.098+0000 7fe3f8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:32.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.354+0000 7fe401f6d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:43:32.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.354+0000 7fe401f6d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:32.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.354+0000 7fe401f6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:32.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.354+0000 7fe401f6d640 1 -- 172.21.15.46:0/2168102499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe3fc11ff20 con 0x7fe3fc0a3c10 2024-09-16T04:43:32.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.354+0000 7fe3f8ff9640 1 -- 172.21.15.46:0/2168102499 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fe3ec01bd70 con 0x7fe3fc0a3c10 2024-09-16T04:43:32.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.355+0000 7fe3f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:32.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.355+0000 7fe3f8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:32.356 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":52,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:43:32.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.357+0000 7fe401f6d640 1 -- 172.21.15.46:0/2168102499 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe3dc0383b0 msgr2=0x7fe3dc03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:32.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.357+0000 7fe401f6d640 1 --2- 172.21.15.46:0/2168102499 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe3dc0383b0 0x7fe3dc03a870 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc120860 tx=0x7fe3e4008040 comp rx=0 tx=0).stop 2024-09-16T04:43:32.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.357+0000 7fe401f6d640 10 monclient: shutdown 2024-09-16T04:43:32.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.357+0000 7fe401f6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:32.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.357+0000 7fe401f6d640 1 -- 172.21.15.46:0/2168102499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3fc0a3c10 msgr2=0x7fe3fc1cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:32.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.357+0000 7fe401f6d640 1 --2- 172.21.15.46:0/2168102499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3fc0a3c10 0x7fe3fc1cb740 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec00c780 tx=0x7fe3ec00cc40 comp rx=0 tx=0).stop 2024-09-16T04:43:32.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.357+0000 7fe401f6d640 1 -- 172.21.15.46:0/2168102499 shutdown_connections 2024-09-16T04:43:32.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.357+0000 7fe401f6d640 1 --2- 172.21.15.46:0/2168102499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3fc13be90 0x7fe3fc09e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:32.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.357+0000 7fe401f6d640 1 --2- 172.21.15.46:0/2168102499 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe3fc0a3ff0 0x7fe3fc136d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:32.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.357+0000 7fe401f6d640 1 --2- 172.21.15.46:0/2168102499 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe3dc0383b0 0x7fe3dc03a870 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:32.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.357+0000 7fe401f6d640 1 --2- 172.21.15.46:0/2168102499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3fc0a3c10 0x7fe3fc1cb740 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:32.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.357+0000 7fe401f6d640 1 -- 172.21.15.46:0/2168102499 >> 172.21.15.46:0/2168102499 conn(0x7fe3fc0a6340 msgr2=0x7fe3fc13d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:32.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.357+0000 7fe401f6d640 1 -- 172.21.15.46:0/2168102499 shutdown_connections 2024-09-16T04:43:32.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:32.357+0000 7fe401f6d640 1 -- 172.21.15.46:0/2168102499 wait complete. 2024-09-16T04:43:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38184]: starting. 2024-09-16T04:43:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38184]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-38149 2024-09-16T04:43:32.366 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38184]: created object 0... 2024-09-16T04:43:32.366 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38184]: created object 25... 2024-09-16T04:43:32.366 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38184]: created object 49... 2024-09-16T04:43:32.366 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38184]: finishing. 2024-09-16T04:43:32.366 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38184]: shutting down. 2024-09-16T04:43:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38186]: starting. 2024-09-16T04:43:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38186]: listing objects. 2024-09-16T04:43:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38186]: listed object 0... 2024-09-16T04:43:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38186]: listed object 25... 2024-09-16T04:43:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38186]: saw 50 objects 2024-09-16T04:43:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38186]: shutting down. 2024-09-16T04:43:35.371 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:43:35.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.484+0000 7f91a4b33640 10 monclient: get_monmap_and_config 2024-09-16T04:43:35.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.484+0000 7f91a4b33640 10 monclient: build_initial_monmap 2024-09-16T04:43:35.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.484+0000 7f91a4b33640 10 monclient: monmap: 2024-09-16T04:43:35.485 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:35.485 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:35.486 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:35.484906+0000 2024-09-16T04:43:35.486 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:35.484906+0000 2024-09-16T04:43:35.486 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:35.486 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:35.486 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:35.486 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:35.486 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:35.486 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:35.486 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:35.486 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:35.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.485+0000 7f91a4b33640 1 Processor -- start 2024-09-16T04:43:35.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.485+0000 7f91a4b33640 1 -- start start 2024-09-16T04:43:35.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.485+0000 7f91a4b33640 10 monclient: init 2024-09-16T04:43:35.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.485+0000 7f91a4b33640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:35.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.485+0000 7f91a4b33640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-16T04:43:35.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.485+0000 7f91a4b33640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 0x7f91a009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:35.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.485+0000 7f91a4b33640 10 monclient(hunting): picked mon.noname-c con 0x7f91a00a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:43:35.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.485+0000 7f91a4b33640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f91a00a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:43:35.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.485+0000 7f91a4b33640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 0x7f91a013e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:35.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.485+0000 7f91a4b33640 10 monclient(hunting): picked mon.noname-a con 0x7f91a013be90 addr v2:172.21.15.46:3300/0 2024-09-16T04:43:35.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.485+0000 7f91a4b33640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91a009b940 con 0x7f91a00a3ff0 2024-09-16T04:43:35.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.485+0000 7f91a4b33640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:35.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.485+0000 7f91a4b33640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f91a009ab00 con 0x7f91a013be90 2024-09-16T04:43:35.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.485+0000 7f91a4b33640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:35.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.485+0000 7f91a4b33640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f91a009b030 con 0x7f91a00a3c10 2024-09-16T04:43:35.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.485+0000 7f91a4b33640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:35.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.485+0000 7f91a4b33640 10 monclient(hunting): authenticate will time out at 1013.900945s 2024-09-16T04:43:35.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.486+0000 7f919ed9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 0x7f91a013e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:35.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.486+0000 7f919ed9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 0x7f91a013e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53354/0 (socket says 172.21.15.46:53354) 2024-09-16T04:43:35.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.486+0000 7f919ed9d640 1 -- 172.21.15.46:0/2112162247 learned_addr learned my addr 172.21.15.46:0/2112162247 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:35.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.486+0000 7f919ed9d640 10 monclient(hunting): get_auth_request con 0x7f91a013be90 auth_method 0 2024-09-16T04:43:35.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.486+0000 7f919ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:35.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.486+0000 7f919ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:35.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.486+0000 7f919ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:35.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.486+0000 7f919e59c640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 0x7f91a009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:35.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.486+0000 7f919e59c640 10 monclient(hunting): get_auth_request con 0x7f91a00a3c10 auth_method 0 2024-09-16T04:43:35.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.486+0000 7f919e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:35.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.486+0000 7f919e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:35.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.486+0000 7f919e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:35.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.486+0000 7f919ed9d640 1 -- 172.21.15.46:0/2112162247 >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 msgr2=0x7f91a013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:43:35.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.486+0000 7f919ed9d640 1 -- 172.21.15.46:0/2112162247 >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 msgr2=0x7f91a013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:35.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.486+0000 7f919ed9d640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 0x7f91a013e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:35.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.486+0000 7f919ed9d640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 0x7f91a013e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:35.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.486+0000 7f919e59c640 1 -- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 msgr2=0x7f91a009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:43:35.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.486+0000 7f919e59c640 1 -- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 msgr2=0x7f91a009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:35.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.486+0000 7f919e59c640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 0x7f91a009bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:35.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.486+0000 7f919e59c640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 0x7f91a009bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:35.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.687+0000 7f919ed9d640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 0x7f91a013e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:35.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.687+0000 7f919ed9d640 10 monclient(hunting): get_auth_request con 0x7f91a013be90 auth_method 0 2024-09-16T04:43:35.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.687+0000 7f919ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:35.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.687+0000 7f919ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:35.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.687+0000 7f919ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:35.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.687+0000 7f919ed9d640 1 -- 172.21.15.46:0/2112162247 >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 msgr2=0x7f91a013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:43:35.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.687+0000 7f919ed9d640 1 -- 172.21.15.46:0/2112162247 >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 msgr2=0x7f91a013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:35.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.687+0000 7f919ed9d640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 0x7f91a013e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:35.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.687+0000 7f919ed9d640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 0x7f91a013e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:43:35.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.687+0000 7f919e59c640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 0x7f91a009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:35.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.687+0000 7f919e59c640 10 monclient(hunting): get_auth_request con 0x7f91a00a3c10 auth_method 0 2024-09-16T04:43:35.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.687+0000 7f919e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:35.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.687+0000 7f919e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:35.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.687+0000 7f919e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:35.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.687+0000 7f919e59c640 1 -- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 msgr2=0x7f91a009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:43:35.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.687+0000 7f919e59c640 1 -- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 msgr2=0x7f91a009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:35.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.687+0000 7f919e59c640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 0x7f91a009bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:35.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:35.687+0000 7f919e59c640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 0x7f91a009bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:43:36.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.088+0000 7f919ed9d640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 0x7f91a013e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:36.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.088+0000 7f919ed9d640 10 monclient(hunting): get_auth_request con 0x7f91a013be90 auth_method 0 2024-09-16T04:43:36.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.088+0000 7f919ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:36.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.088+0000 7f919ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:36.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.088+0000 7f919ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:36.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.088+0000 7f919ed9d640 1 -- 172.21.15.46:0/2112162247 >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 msgr2=0x7f91a013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:43:36.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.088+0000 7f919ed9d640 1 -- 172.21.15.46:0/2112162247 >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 msgr2=0x7f91a013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:36.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.088+0000 7f919ed9d640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 0x7f91a013e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:36.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.088+0000 7f919e59c640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 0x7f91a009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:36.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.088+0000 7f919ed9d640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 0x7f91a013e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:43:36.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.088+0000 7f919e59c640 10 monclient(hunting): get_auth_request con 0x7f91a00a3c10 auth_method 0 2024-09-16T04:43:36.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.088+0000 7f919e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:36.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.088+0000 7f919e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:36.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.088+0000 7f919e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:36.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.088+0000 7f919e59c640 1 -- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 msgr2=0x7f91a009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:43:36.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.088+0000 7f919e59c640 1 -- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 msgr2=0x7f91a009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:36.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.088+0000 7f919e59c640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 0x7f91a009bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:36.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.088+0000 7f919e59c640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 0x7f91a009bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:43:36.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.889+0000 7f919ed9d640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 0x7f91a013e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:36.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.889+0000 7f919ed9d640 10 monclient(hunting): get_auth_request con 0x7f91a013be90 auth_method 0 2024-09-16T04:43:36.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.889+0000 7f919ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:36.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.889+0000 7f919ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:36.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.889+0000 7f919ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:36.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.889+0000 7f919ed9d640 1 -- 172.21.15.46:0/2112162247 >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 msgr2=0x7f91a013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:43:36.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.889+0000 7f919ed9d640 1 -- 172.21.15.46:0/2112162247 >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 msgr2=0x7f91a013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:36.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.889+0000 7f919ed9d640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 0x7f91a013e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:36.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.889+0000 7f919ed9d640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 0x7f91a013e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:43:36.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.889+0000 7f919e59c640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 0x7f91a009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:36.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.889+0000 7f919e59c640 10 monclient(hunting): get_auth_request con 0x7f91a00a3c10 auth_method 0 2024-09-16T04:43:36.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.889+0000 7f919e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:36.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.889+0000 7f919e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:36.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.889+0000 7f919e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:36.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.889+0000 7f919e59c640 1 -- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 msgr2=0x7f91a009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:43:36.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.889+0000 7f919e59c640 1 -- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 msgr2=0x7f91a009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:36.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.889+0000 7f919e59c640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 0x7f91a009bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:36.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:36.889+0000 7f919e59c640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 0x7f91a009bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:43:37.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 1 -- 172.21.15.46:0/2112162247 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 548525126 0 0) 0x7f91a009b940 con 0x7f91a00a3ff0 2024-09-16T04:43:37.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 10 monclient(hunting): _negotiate 2024-09-16T04:43:37.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:37.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:37.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 10 monclient(hunting): my global_id is 5875 2024-09-16T04:43:37.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 1 -- 172.21.15.46:0/2112162247 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f918c003520 con 0x7f91a00a3ff0 2024-09-16T04:43:37.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 1 -- 172.21.15.46:0/2112162247 <== mon.1 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1072096320 0 0) 0x7f918c003520 con 0x7f91a00a3ff0 2024-09-16T04:43:37.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 1 -- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 msgr2=0x7f91a009bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:37.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.114:3300/0 conn(0x7f91a00a3c10 0x7f91a009bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:37.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 1 -- 172.21.15.46:0/2112162247 >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 msgr2=0x7f91a013e280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:37.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 1 --2- 172.21.15.46:0/2112162247 >> v2:172.21.15.46:3300/0 conn(0x7f91a013be90 0x7f91a013e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:37.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 10 monclient: _finish_hunting 0 2024-09-16T04:43:37.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 1 monclient: found mon.noname-c-legacy 2024-09-16T04:43:37.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:37.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.114:6789/0 2024-09-16T04:43:37.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 1 -- 172.21.15.46:0/2112162247 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a009b310 con 0x7f91a00a3ff0 2024-09-16T04:43:37.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 10 monclient: _finish_auth 0 2024-09-16T04:43:37.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 10 monclient: _check_auth_tickets 2024-09-16T04:43:37.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:37.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 1 -- 172.21.15.46:0/2112162247 <== mon.1 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f9188001670 con 0x7f91a00a3ff0 2024-09-16T04:43:37.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f91a4b33640 5 monclient: authenticate success, global_id 5875 2024-09-16T04:43:37.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f91a4b33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T04:43:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 10 monclient: dump: 2024-09-16T04:43:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:37.696 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T04:43:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.688+0000 7f919dd9b640 1 -- 172.21.15.46:0/2112162247 >> v1:172.21.15.114:6789/0 conn(0x7f91a00a3ff0 legacy=0x7f91a00a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919dd9b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:43:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919dd9b640 1 --2- 172.21.15.46:0/2112162247 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f918c003a00 0x7f918c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919dd9b640 10 monclient(hunting): picked mon.c con 0x7f918c003a00 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919dd9b640 1 --2- 172.21.15.46:0/2112162247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f918c004800 0x7f918c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919dd9b640 10 monclient(hunting): picked mon.a con 0x7f918c004800 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919dd9b640 1 --2- 172.21.15.46:0/2112162247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f918c0052e0 0x7f918c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919dd9b640 10 monclient(hunting): picked mon.b con 0x7f918c0052e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919dd9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919dd9b640 1 -- 172.21.15.46:0/2112162247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f91a009ab00 con 0x7f918c004800 2024-09-16T04:43:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919dd9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919dd9b640 1 -- 172.21.15.46:0/2112162247 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f91a009b030 con 0x7f918c0052e0 2024-09-16T04:43:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919dd9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919dd9b640 1 -- 172.21.15.46:0/2112162247 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f918c00c230 con 0x7f918c003a00 2024-09-16T04:43:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919dd9b640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919dd9b640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:43:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f91a4b33640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919e59c640 1 --2- 172.21.15.46:0/2112162247 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f918c003a00 0x7f918c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919e59c640 10 monclient(hunting): get_auth_request con 0x7f918c003a00 auth_method 0 2024-09-16T04:43:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919ed9d640 1 --2- 172.21.15.46:0/2112162247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f918c0052e0 0x7f918c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919ed9d640 10 monclient(hunting): get_auth_request con 0x7f918c0052e0 auth_method 0 2024-09-16T04:43:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:43:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919e59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:43:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919ed9d640 10 monclient(hunting): handle_auth_done global_id 5875 payload 293 2024-09-16T04:43:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919ed9d640 1 -- 172.21.15.46:0/2112162247 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f918c003a00 msgr2=0x7f918c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:43:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919ed9d640 1 --2- 172.21.15.46:0/2112162247 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f918c003a00 0x7f918c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919ed9d640 1 -- 172.21.15.46:0/2112162247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f918c004800 msgr2=0x7f918c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:43:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919ed9d640 1 --2- 172.21.15.46:0/2112162247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f918c004800 0x7f918c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:37.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919ed9d640 10 monclient: _finish_hunting 0 2024-09-16T04:43:37.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919ed9d640 1 monclient: found mon.b 2024-09-16T04:43:37.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:37.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919ed9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:43:37.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919ed9d640 1 -- 172.21.15.46:0/2112162247 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f918c003520 con 0x7f918c0052e0 2024-09-16T04:43:37.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919ed9d640 10 monclient: _finish_auth 0 2024-09-16T04:43:37.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919ed9d640 10 monclient: _check_auth_tickets 2024-09-16T04:43:37.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.689+0000 7f919ed9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:37.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.690+0000 7f919ed9d640 1 -- 172.21.15.46:0/2112162247 reap_dead start 2024-09-16T04:43:37.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.690+0000 7f919ed9d640 1 --2- 172.21.15.46:0/2112162247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f918c0052e0 0x7f918c00b9c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f9198009be0 tx=0x7f918c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7a204c15d46dbed server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:37.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.690+0000 7f919dd9b640 1 -- 172.21.15.46:0/2112162247 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9198031070 con 0x7f918c0052e0 2024-09-16T04:43:37.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.690+0000 7f919dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:37.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.690+0000 7f919dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:43:37.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.690+0000 7f919dd9b640 10 monclient: dump: 2024-09-16T04:43:37.700 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:37.700 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:37.700 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:37.700 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:37.700 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:37.700 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:37.700 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:37.700 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:37.700 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:37.700 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:37.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.691+0000 7f91a4b33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:37.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.691+0000 7f919dd9b640 1 -- 172.21.15.46:0/2112162247 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f919802b8a0 con 0x7f918c0052e0 2024-09-16T04:43:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.691+0000 7f919dd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.691+0000 7f91a4b33640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.691+0000 7f919dd9b640 1 -- 172.21.15.46:0/2112162247 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91980097e0 con 0x7f918c0052e0 2024-09-16T04:43:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.691+0000 7f919dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.691+0000 7f91a4b33640 4 set_mon_vals no callback set 2024-09-16T04:43:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.691+0000 7f919dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:43:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.691+0000 7f919dd9b640 10 monclient: dump: 2024-09-16T04:43:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:37.702 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.691+0000 7f91a4b33640 10 monclient: shutdown 2024-09-16T04:43:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.691+0000 7f91a4b33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.691+0000 7f91a4b33640 1 -- 172.21.15.46:0/2112162247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f918c0052e0 msgr2=0x7f918c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.691+0000 7f91a4b33640 1 --2- 172.21.15.46:0/2112162247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f918c0052e0 0x7f918c00b9c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f9198009be0 tx=0x7f918c005830 comp rx=0 tx=0).stop 2024-09-16T04:43:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.692+0000 7f91a4b33640 1 -- 172.21.15.46:0/2112162247 shutdown_connections 2024-09-16T04:43:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.692+0000 7f91a4b33640 1 --2- 172.21.15.46:0/2112162247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f918c0052e0 0x7f918c00b9c0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.692+0000 7f91a4b33640 1 -- 172.21.15.46:0/2112162247 >> 172.21.15.46:0/2112162247 conn(0x7f91a00a6340 msgr2=0x7f91a006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.692+0000 7f91a4b33640 1 -- 172.21.15.46:0/2112162247 shutdown_connections 2024-09-16T04:43:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.692+0000 7f91a4b33640 1 -- 172.21.15.46:0/2112162247 wait complete. 2024-09-16T04:43:37.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.693+0000 7f91a4b33640 10 monclient: build_initial_monmap 2024-09-16T04:43:37.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.693+0000 7f91a4b33640 10 monclient: monmap: 2024-09-16T04:43:37.706 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:37.707 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.693+0000 7f91a4b33640 1 Processor -- start 2024-09-16T04:43:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.693+0000 7f91a4b33640 1 -- start start 2024-09-16T04:43:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.693+0000 7f91a4b33640 10 monclient: init 2024-09-16T04:43:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f91a4b33640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f91a4b33640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:43:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f91a4b33640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f918c0052e0 0x7f91a01333a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f91a4b33640 10 monclient(hunting): picked mon.noname-c con 0x7f918c0052e0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f91a4b33640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a009d110 0x7f91a009d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f91a4b33640 10 monclient(hunting): picked mon.noname-b con 0x7f91a009d110 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f91a4b33640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91a009da90 0x7f91a01d2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f91a4b33640 10 monclient(hunting): picked mon.noname-a con 0x7f91a009da90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f91a4b33640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f91a4b33640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f91a009be70 con 0x7f91a009da90 2024-09-16T04:43:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f91a4b33640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f91a4b33640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f91a009bcf0 con 0x7f91a009d110 2024-09-16T04:43:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f91a4b33640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f91a4b33640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f91a009bff0 con 0x7f918c0052e0 2024-09-16T04:43:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f91a4b33640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f91a4b33640 10 monclient(hunting): authenticate will time out at 1016.109365s 2024-09-16T04:43:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f919e59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f918c0052e0 0x7f91a01333a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f919ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91a009da90 0x7f91a01d2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f919e59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f918c0052e0 0x7f91a01333a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:42656/0 (socket says 172.21.15.46:42656) 2024-09-16T04:43:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f9197fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a009d110 0x7f91a009d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f919e59c640 1 -- 172.21.15.46:0/1800752070 learned_addr learned my addr 172.21.15.46:0/1800752070 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f919e59c640 10 monclient(hunting): get_auth_request con 0x7f918c0052e0 auth_method 0 2024-09-16T04:43:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f919e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f919e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f919e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f9197fff640 10 monclient(hunting): get_auth_request con 0x7f91a009d110 auth_method 0 2024-09-16T04:43:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f9197fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f9197fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f9197fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f919ed9d640 10 monclient(hunting): get_auth_request con 0x7f91a009da90 auth_method 0 2024-09-16T04:43:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f919ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f919ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f919ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f919e59c640 1 -- 172.21.15.46:0/1800752070 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f918c0052e0 msgr2=0x7f91a01333a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:43:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f919e59c640 1 -- 172.21.15.46:0/1800752070 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f918c0052e0 msgr2=0x7f91a01333a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:43:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f919e59c640 1 --2- 172.21.15.46:0/1800752070 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f918c0052e0 0x7f91a01333a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f9197fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f9197fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f9197fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f919ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f919ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f919e59c640 1 --2- 172.21.15.46:0/1800752070 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f918c0052e0 0x7f91a01333a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.694+0000 7f919ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f919ed9d640 10 monclient(hunting): handle_auth_done global_id 44607 payload 824 2024-09-16T04:43:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f919ed9d640 1 -- 172.21.15.46:0/1800752070 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f918c0052e0 msgr2=0x7f91a01333a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:43:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f919ed9d640 1 --2- 172.21.15.46:0/1800752070 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f918c0052e0 0x7f91a01333a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f919ed9d640 1 -- 172.21.15.46:0/1800752070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a009d110 msgr2=0x7f91a009d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f919ed9d640 1 --2- 172.21.15.46:0/1800752070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a009d110 0x7f91a009d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f919ed9d640 10 monclient: _finish_hunting 0 2024-09-16T04:43:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f919ed9d640 1 monclient: found mon.noname-a 2024-09-16T04:43:37.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f919ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:37.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f919ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:43:37.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f919ed9d640 1 -- 172.21.15.46:0/1800752070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a009fca0 con 0x7f91a009da90 2024-09-16T04:43:37.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f9197fff640 1 --2- 172.21.15.46:0/1800752070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a009d110 0x7f91a009d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:43:37.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f919ed9d640 1 --2- 172.21.15.46:0/1800752070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91a009da90 0x7f91a01d2060 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f91a009af30 tx=0x7f9198004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:37.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f9196ffd640 1 -- 172.21.15.46:0/1800752070 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91980097e0 con 0x7f91a009da90 2024-09-16T04:43:37.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f9196ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:37.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f9196ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:43:37.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f9196ffd640 10 monclient: dump: 2024-09-16T04:43:37.712 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:37.712 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:37.712 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:37.713 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:37.713 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:37.713 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:37.713 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:37.713 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:37.713 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:37.713 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:37.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f9196ffd640 10 monclient: _finish_auth 0 2024-09-16T04:43:37.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f9196ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:43:37.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f9196ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:37.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f91a4b33640 5 monclient: authenticate success, global_id 44607 2024-09-16T04:43:37.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f9196ffd640 1 -- 172.21.15.46:0/1800752070 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9198045070 con 0x7f91a009da90 2024-09-16T04:43:37.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f91a4b33640 10 monclient: _renew_subs 2024-09-16T04:43:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f91a4b33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f91a4b33640 1 -- 172.21.15.46:0/1800752070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91a011fab0 con 0x7f91a009da90 2024-09-16T04:43:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f9196ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f9196ffd640 1 -- 172.21.15.46:0/1800752070 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91980130b0 con 0x7f91a009da90 2024-09-16T04:43:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f9196ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f9196ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f9196ffd640 10 monclient: dump: 2024-09-16T04:43:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:37.715 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f919d59a640 4 set_mon_vals no callback set 2024-09-16T04:43:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f91a4b33640 10 monclient: _renew_subs 2024-09-16T04:43:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f91a4b33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.695+0000 7f91a4b33640 1 -- 172.21.15.46:0/1800752070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91a009df10 con 0x7f91a009da90 2024-09-16T04:43:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.696+0000 7f9196ffd640 1 -- 172.21.15.46:0/1800752070 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f919804a460 con 0x7f91a009da90 2024-09-16T04:43:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.696+0000 7f91a4b33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.696+0000 7f91a4b33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.696+0000 7f91a4b33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.697+0000 7f9196ffd640 1 --2- 172.21.15.46:0/1800752070 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f91800383a0 0x7f918003a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.696+0000 7f91a4b33640 1 -- 172.21.15.46:0/1800752070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91a00a43d0 con 0x7f91a009da90 2024-09-16T04:43:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.697+0000 7f9196ffd640 1 -- 172.21.15.46:0/1800752070 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(187..187 src has 1..187) ==== 4738+0+0 (secure 0 0 0) 0x7f919806c820 con 0x7f91a009da90 2024-09-16T04:43:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.697+0000 7f919e59c640 1 --2- 172.21.15.46:0/1800752070 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f91800383a0 0x7f918003a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.697+0000 7f919e59c640 10 monclient: get_auth_request con 0x7f91800383a0 auth_method 0 2024-09-16T04:43:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.697+0000 7f919e59c640 1 --2- 172.21.15.46:0/1800752070 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f91800383a0 0x7f918003a860 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9190004040 tx=0x7f9190008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.702+0000 7f9196ffd640 1 -- 172.21.15.46:0/1800752070 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f919804c060 con 0x7f91a009da90 2024-09-16T04:43:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.702+0000 7f9196ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.702+0000 7f9196ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:37.753 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:37.753 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38185]: starting. 2024-09-16T04:43:37.753 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38185]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-38149 2024-09-16T04:43:37.753 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38185]: shutting down. 2024-09-16T04:43:37.754 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:37.754 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:37.754 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:43:37.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:43:37.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.961+0000 7f91a4b33640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:43:37.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.961+0000 7f91a4b33640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:37.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.961+0000 7f91a4b33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:37.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.961+0000 7f91a4b33640 1 -- 172.21.15.46:0/1800752070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f91a01327b0 con 0x7f91a009da90 2024-09-16T04:43:37.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.961+0000 7f9196ffd640 1 -- 172.21.15.46:0/1800752070 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f919803e440 con 0x7f91a009da90 2024-09-16T04:43:37.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.961+0000 7f9196ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:37.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.961+0000 7f9196ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:37.963 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":56,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:43:37.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.963+0000 7f91a4b33640 1 -- 172.21.15.46:0/1800752070 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f91800383a0 msgr2=0x7f918003a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:37.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.963+0000 7f91a4b33640 1 --2- 172.21.15.46:0/1800752070 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f91800383a0 0x7f918003a860 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9190004040 tx=0x7f9190008040 comp rx=0 tx=0).stop 2024-09-16T04:43:37.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.963+0000 7f91a4b33640 10 monclient: shutdown 2024-09-16T04:43:37.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.963+0000 7f91a4b33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:37.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.963+0000 7f91a4b33640 1 -- 172.21.15.46:0/1800752070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91a009da90 msgr2=0x7f91a01d2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:37.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.963+0000 7f91a4b33640 1 --2- 172.21.15.46:0/1800752070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91a009da90 0x7f91a01d2060 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f91a009af30 tx=0x7f9198004ab0 comp rx=0 tx=0).stop 2024-09-16T04:43:37.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.963+0000 7f91a4b33640 1 -- 172.21.15.46:0/1800752070 shutdown_connections 2024-09-16T04:43:37.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.963+0000 7f91a4b33640 1 --2- 172.21.15.46:0/1800752070 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f91800383a0 0x7f918003a860 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:37.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.963+0000 7f91a4b33640 1 --2- 172.21.15.46:0/1800752070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91a009da90 0x7f91a01d2060 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:37.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.963+0000 7f91a4b33640 1 --2- 172.21.15.46:0/1800752070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a009d110 0x7f91a009d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:37.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.963+0000 7f91a4b33640 1 --2- 172.21.15.46:0/1800752070 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f918c0052e0 0x7f91a01333a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:37.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.963+0000 7f91a4b33640 1 -- 172.21.15.46:0/1800752070 >> 172.21.15.46:0/1800752070 conn(0x7f91a00a6340 msgr2=0x7f91a013b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:37.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.963+0000 7f91a4b33640 1 -- 172.21.15.46:0/1800752070 shutdown_connections 2024-09-16T04:43:37.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:37.963+0000 7f91a4b33640 1 -- 172.21.15.46:0/1800752070 wait complete. 2024-09-16T04:43:39.851 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38253]: starting. 2024-09-16T04:43:39.851 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38253]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-38252 2024-09-16T04:43:39.851 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38253]: created object 0... 2024-09-16T04:43:39.851 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38253]: created object 25... 2024-09-16T04:43:39.851 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38253]: created object 49... 2024-09-16T04:43:39.851 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38253]: finishing. 2024-09-16T04:43:39.851 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38253]: shutting down. 2024-09-16T04:43:40.279 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38254]: starting. 2024-09-16T04:43:40.279 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38254]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-38252 2024-09-16T04:43:40.279 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38254]: shutting down. 2024-09-16T04:43:40.976 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:43:41.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.087+0000 7fd0bf282640 10 monclient: get_monmap_and_config 2024-09-16T04:43:41.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.087+0000 7fd0bf282640 10 monclient: build_initial_monmap 2024-09-16T04:43:41.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.087+0000 7fd0bf282640 10 monclient: monmap: 2024-09-16T04:43:41.175 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:41.175 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:41.175 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:41.088344+0000 2024-09-16T04:43:41.175 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:41.088344+0000 2024-09-16T04:43:41.175 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:41.175 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.175 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:41.175 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:41.175 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:41.176 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:41.176 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:41.176 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.088+0000 7fd0bf282640 1 Processor -- start 2024-09-16T04:43:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.088+0000 7fd0bf282640 1 -- start start 2024-09-16T04:43:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.088+0000 7fd0bf282640 10 monclient: init 2024-09-16T04:43:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bf282640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bf282640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-16T04:43:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bf282640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0b80a3ba0 0x7fd0b809bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bf282640 10 monclient(hunting): picked mon.noname-b con 0x7fd0b80a3ba0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bf282640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd0b809c1c0 0x7fd0b80a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bf282640 10 monclient(hunting): picked mon.noname-c con 0x7fd0b809c1c0 addr v2:172.21.15.114:3300/0 2024-09-16T04:43:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bf282640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd0b80a44c0 addr v1:172.21.15.46:6789/0 2024-09-16T04:43:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bf282640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0b809afc0 con 0x7fd0b80a44c0 2024-09-16T04:43:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bf282640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bf282640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fd0b809b860 con 0x7fd0b809c1c0 2024-09-16T04:43:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bf282640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bf282640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd0b809b2a0 con 0x7fd0b80a3ba0 2024-09-16T04:43:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bf282640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bf282640 10 monclient(hunting): authenticate will time out at 1019.504291s 2024-09-16T04:43:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bd01e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0b80a3ba0 0x7fd0b809bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bd81f640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd0b80a44c0 0x7fd0b8135950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37060/0 (socket says 172.21.15.46:37060) 2024-09-16T04:43:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bd81f640 1 -- 172.21.15.46:0/262457285 learned_addr learned my addr 172.21.15.46:0/262457285 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:43:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bd01e640 10 monclient(hunting): get_auth_request con 0x7fd0b80a3ba0 auth_method 0 2024-09-16T04:43:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bd01e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bd01e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bd01e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bd01e640 1 -- 172.21.15.46:0/262457285 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0b80a3ba0 msgr2=0x7fd0b809bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:43:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bd01e640 1 -- 172.21.15.46:0/262457285 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0b80a3ba0 msgr2=0x7fd0b809bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bd01e640 1 --2- 172.21.15.46:0/262457285 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0b80a3ba0 0x7fd0b809bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bd01e640 1 --2- 172.21.15.46:0/262457285 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0b80a3ba0 0x7fd0b809bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bc81d640 1 --2- 172.21.15.46:0/262457285 >> v2:172.21.15.114:3300/0 conn(0x7fd0b809c1c0 0x7fd0b80a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bc81d640 10 monclient(hunting): get_auth_request con 0x7fd0b809c1c0 auth_method 0 2024-09-16T04:43:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bc81d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bc81d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bc81d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0affff640 1 -- 172.21.15.46:0/262457285 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4205514789 0 0) 0x7fd0b809afc0 con 0x7fd0b80a44c0 2024-09-16T04:43:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0affff640 10 monclient(hunting): _negotiate 2024-09-16T04:43:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0affff640 10 monclient(hunting): my global_id is 44637 2024-09-16T04:43:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0affff640 1 -- 172.21.15.46:0/262457285 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0a4003520 con 0x7fd0b80a44c0 2024-09-16T04:43:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bc81d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bc81d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0bc81d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0affff640 1 -- 172.21.15.46:0/262457285 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4251966209 0 0) 0x7fd0a4003520 con 0x7fd0b80a44c0 2024-09-16T04:43:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0affff640 1 -- 172.21.15.46:0/262457285 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0b80a3ba0 msgr2=0x7fd0b809bc80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.089+0000 7fd0affff640 1 --2- 172.21.15.46:0/262457285 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0b80a3ba0 0x7fd0b809bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 1 -- 172.21.15.46:0/262457285 >> v2:172.21.15.114:3300/0 conn(0x7fd0b809c1c0 msgr2=0x7fd0b80a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:43:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 1 --2- 172.21.15.46:0/262457285 >> v2:172.21.15.114:3300/0 conn(0x7fd0b809c1c0 0x7fd0b80a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 10 monclient: _finish_hunting 0 2024-09-16T04:43:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:43:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:43:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 1 -- 172.21.15.46:0/262457285 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b809b580 con 0x7fd0b80a44c0 2024-09-16T04:43:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 10 monclient: _finish_auth 0 2024-09-16T04:43:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 10 monclient: _check_auth_tickets 2024-09-16T04:43:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 1 -- 172.21.15.46:0/262457285 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7fd0b00019f0 con 0x7fd0b80a44c0 2024-09-16T04:43:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bf282640 5 monclient: authenticate success, global_id 44637 2024-09-16T04:43:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 1 --2- 172.21.15.46:0/262457285 >> v2:172.21.15.114:3300/0 conn(0x7fd0b809c1c0 0x7fd0b80a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:43:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:43:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 10 monclient: dump: 2024-09-16T04:43:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:41.192 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.192 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.192 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:41.192 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.192 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:41.192 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:41.192 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:41.192 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:43:41.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:41.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 1 -- 172.21.15.46:0/262457285 >> v1:172.21.15.46:6789/0 conn(0x7fd0b80a44c0 legacy=0x7fd0b8135950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:41.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:43:41.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 1 --2- 172.21.15.46:0/262457285 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0b80a3ba0 0x7fd0a40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 10 monclient(hunting): picked mon.c con 0x7fd0b80a3ba0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:41.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 1 --2- 172.21.15.46:0/262457285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a4003f30 0x7fd0a4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 10 monclient(hunting): picked mon.a con 0x7fd0a4003f30 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:41.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 1 --2- 172.21.15.46:0/262457285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0a4004d40 0x7fd0a40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 10 monclient(hunting): picked mon.b con 0x7fd0a4004d40 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:41.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 1 -- 172.21.15.46:0/262457285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b809b860 con 0x7fd0a4003f30 2024-09-16T04:43:41.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 1 -- 172.21.15.46:0/262457285 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b809b2a0 con 0x7fd0a4004d40 2024-09-16T04:43:41.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 1 -- 172.21.15.46:0/262457285 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd0a400bac0 con 0x7fd0b80a3ba0 2024-09-16T04:43:41.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:41.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0affff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:43:41.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bf282640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:41.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bd01e640 1 --2- 172.21.15.46:0/262457285 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0b80a3ba0 0x7fd0a40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bd01e640 1 -- 172.21.15.46:0/262457285 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0b80a3ba0 msgr2=0x7fd0a40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:43:41.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bd01e640 1 -- 172.21.15.46:0/262457285 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0b80a3ba0 msgr2=0x7fd0a40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:41.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bd01e640 1 --2- 172.21.15.46:0/262457285 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0b80a3ba0 0x7fd0a40039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:41.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bd01e640 1 --2- 172.21.15.46:0/262457285 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0b80a3ba0 0x7fd0a40039b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:41.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 1 --2- 172.21.15.46:0/262457285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a4003f30 0x7fd0a4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bd81f640 1 --2- 172.21.15.46:0/262457285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0a4004d40 0x7fd0a40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 10 monclient(hunting): get_auth_request con 0x7fd0a4003f30 auth_method 0 2024-09-16T04:43:41.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:41.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:41.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bd81f640 10 monclient(hunting): get_auth_request con 0x7fd0a4004d40 auth_method 0 2024-09-16T04:43:41.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bd81f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:41.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bd81f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bd81f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:41.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:41.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:41.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:43:41.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bd81f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:41.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bd81f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:41.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bd81f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:43:41.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 10 monclient(hunting): handle_auth_done global_id 44637 payload 293 2024-09-16T04:43:41.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 1 -- 172.21.15.46:0/262457285 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0b80a3ba0 msgr2=0x7fd0a40039b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:41.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 1 --2- 172.21.15.46:0/262457285 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0b80a3ba0 0x7fd0a40039b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 1 -- 172.21.15.46:0/262457285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0a4004d40 msgr2=0x7fd0a40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:43:41.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 1 --2- 172.21.15.46:0/262457285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0a4004d40 0x7fd0a40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 10 monclient: _finish_hunting 0 2024-09-16T04:43:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 1 monclient: found mon.a 2024-09-16T04:43:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 1 -- 172.21.15.46:0/262457285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a4003520 con 0x7fd0a4003f30 2024-09-16T04:43:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 10 monclient: _finish_auth 0 2024-09-16T04:43:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 10 monclient: _check_auth_tickets 2024-09-16T04:43:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bc81d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.090+0000 7fd0bd81f640 1 --2- 172.21.15.46:0/262457285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0a4004d40 0x7fd0a40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:43:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.091+0000 7fd0bc81d640 1 --2- 172.21.15.46:0/262457285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a4003f30 0x7fd0a4004800 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8009cc0 tx=0x7fd0a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79b6a306a4397a1e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.091+0000 7fd0affff640 1 -- 172.21.15.46:0/262457285 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8036070 con 0x7fd0a4003f30 2024-09-16T04:43:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.091+0000 7fd0affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.091+0000 7fd0affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.091+0000 7fd0affff640 10 monclient: dump: 2024-09-16T04:43:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:41.197 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.091+0000 7fd0bf282640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.091+0000 7fd0affff640 1 -- 172.21.15.46:0/262457285 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0a8031400 con 0x7fd0a4003f30 2024-09-16T04:43:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.091+0000 7fd0affff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.091+0000 7fd0bf282640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.091+0000 7fd0bf282640 4 set_mon_vals no callback set 2024-09-16T04:43:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.091+0000 7fd0affff640 1 -- 172.21.15.46:0/262457285 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8009860 con 0x7fd0a4003f30 2024-09-16T04:43:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.091+0000 7fd0affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.091+0000 7fd0affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.091+0000 7fd0affff640 10 monclient: dump: 2024-09-16T04:43:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:41.199 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.091+0000 7fd0bf282640 10 monclient: shutdown 2024-09-16T04:43:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.091+0000 7fd0bf282640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.091+0000 7fd0bf282640 1 -- 172.21.15.46:0/262457285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a4003f30 msgr2=0x7fd0a4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.091+0000 7fd0bf282640 1 --2- 172.21.15.46:0/262457285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a4003f30 0x7fd0a4004800 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8009cc0 tx=0x7fd0a4005830 comp rx=0 tx=0).stop 2024-09-16T04:43:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.091+0000 7fd0bd81f640 1 -- 172.21.15.46:0/262457285 reap_dead start 2024-09-16T04:43:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.092+0000 7fd0bf282640 1 -- 172.21.15.46:0/262457285 shutdown_connections 2024-09-16T04:43:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.092+0000 7fd0bf282640 1 -- 172.21.15.46:0/262457285 >> 172.21.15.46:0/262457285 conn(0x7fd0b80a60c0 msgr2=0x7fd0b80a64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.092+0000 7fd0bf282640 1 -- 172.21.15.46:0/262457285 shutdown_connections 2024-09-16T04:43:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.092+0000 7fd0bf282640 1 -- 172.21.15.46:0/262457285 wait complete. 2024-09-16T04:43:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.092+0000 7fd0bf282640 10 monclient: build_initial_monmap 2024-09-16T04:43:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.092+0000 7fd0bf282640 10 monclient: monmap: 2024-09-16T04:43:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:41.200 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bf282640 1 Processor -- start 2024-09-16T04:43:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bf282640 1 -- start start 2024-09-16T04:43:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bf282640 10 monclient: init 2024-09-16T04:43:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bf282640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bf282640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:43:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bf282640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0a4003f30 0x7fd0b81cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bf282640 10 monclient(hunting): picked mon.noname-c con 0x7fd0a4003f30 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bf282640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0b81cc490 0x7fd0b8123fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bf282640 10 monclient(hunting): picked mon.noname-a con 0x7fd0b81cc490 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bf282640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0b81cc8f0 0x7fd0b8124500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bf282640 10 monclient(hunting): picked mon.noname-b con 0x7fd0b81cc8f0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bf282640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bf282640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b80a5540 con 0x7fd0b81cc490 2024-09-16T04:43:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bf282640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bf282640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b80a53c0 con 0x7fd0b81cc8f0 2024-09-16T04:43:41.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bf282640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bf282640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd0b809bc80 con 0x7fd0a4003f30 2024-09-16T04:43:41.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bf282640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:41.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bf282640 10 monclient(hunting): authenticate will time out at 1019.508605s 2024-09-16T04:43:41.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bd01e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0a4003f30 0x7fd0b81cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bc81d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0b81cc490 0x7fd0b8123fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bd01e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0a4003f30 0x7fd0b81cbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:42682/0 (socket says 172.21.15.46:42682) 2024-09-16T04:43:41.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bd01e640 1 -- 172.21.15.46:0/3423971049 learned_addr learned my addr 172.21.15.46:0/3423971049 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:41.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bd01e640 10 monclient(hunting): get_auth_request con 0x7fd0a4003f30 auth_method 0 2024-09-16T04:43:41.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bc81d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0b81cc490 0x7fd0b8123fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53432/0 (socket says 172.21.15.46:53432) 2024-09-16T04:43:41.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bd01e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:41.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bd01e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bd01e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:41.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bc81d640 10 monclient(hunting): get_auth_request con 0x7fd0b81cc490 auth_method 0 2024-09-16T04:43:41.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bc81d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:41.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bc81d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bc81d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:41.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bc81d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:41.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bc81d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:41.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bd01e640 1 -- 172.21.15.46:0/3423971049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0a4003f30 msgr2=0x7fd0b81cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:43:41.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bd01e640 1 -- 172.21.15.46:0/3423971049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0a4003f30 msgr2=0x7fd0b81cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:43:41.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bd01e640 1 --2- 172.21.15.46:0/3423971049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0a4003f30 0x7fd0b81cbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:41.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bc81d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:41.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.093+0000 7fd0bd01e640 1 --2- 172.21.15.46:0/3423971049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0a4003f30 0x7fd0b81cbf50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:41.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0bd81f640 1 --2- 172.21.15.46:0/3423971049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0b81cc8f0 0x7fd0b8124500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0bc81d640 10 monclient(hunting): handle_auth_done global_id 44640 payload 824 2024-09-16T04:43:41.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0bc81d640 1 -- 172.21.15.46:0/3423971049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0a4003f30 msgr2=0x7fd0b81cbf50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:43:41.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0bc81d640 1 --2- 172.21.15.46:0/3423971049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0a4003f30 0x7fd0b81cbf50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0bc81d640 1 -- 172.21.15.46:0/3423971049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0b81cc8f0 msgr2=0x7fd0b8124500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:41.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0bc81d640 1 --2- 172.21.15.46:0/3423971049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0b81cc8f0 0x7fd0b8124500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0bc81d640 10 monclient: _finish_hunting 0 2024-09-16T04:43:41.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0bc81d640 1 monclient: found mon.noname-a 2024-09-16T04:43:41.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0bc81d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:41.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0bc81d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:43:41.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0bc81d640 1 -- 172.21.15.46:0/3423971049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b8127f50 con 0x7fd0b81cc490 2024-09-16T04:43:41.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0bd81f640 10 monclient: get_auth_request con 0x7fd0b81cc8f0 auth_method 0 2024-09-16T04:43:41.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0bd81f640 1 --2- 172.21.15.46:0/3423971049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0b81cc8f0 0x7fd0b8124500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:43:41.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0bc81d640 1 --2- 172.21.15.46:0/3423971049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0b81cc490 0x7fd0b8123fc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8031000 tx=0x7fd0a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:41.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0adffb640 1 -- 172.21.15.46:0/3423971049 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8040070 con 0x7fd0b81cc490 2024-09-16T04:43:41.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:41.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0adffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:43:41.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0adffb640 10 monclient: dump: 2024-09-16T04:43:41.205 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:41.205 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:41.205 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.205 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.205 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:41.205 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.205 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:41.205 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:41.205 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:41.205 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0adffb640 10 monclient: _finish_auth 0 2024-09-16T04:43:41.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0adffb640 10 monclient: _check_auth_tickets 2024-09-16T04:43:41.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:41.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0bf282640 5 monclient: authenticate success, global_id 44640 2024-09-16T04:43:41.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0adffb640 1 -- 172.21.15.46:0/3423971049 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0a803b6a0 con 0x7fd0b81cc490 2024-09-16T04:43:41.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0adffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:41.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0adffb640 1 -- 172.21.15.46:0/3423971049 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8036040 con 0x7fd0b81cc490 2024-09-16T04:43:41.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:41.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0adffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:41.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0adffb640 10 monclient: dump: 2024-09-16T04:43:41.206 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:41.206 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:41.206 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.207 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.207 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:41.207 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.207 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:41.207 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:41.207 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:41.207 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0af7fe640 4 set_mon_vals no callback set 2024-09-16T04:43:41.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0bf282640 10 monclient: _renew_subs 2024-09-16T04:43:41.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0bf282640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:41.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0bf282640 1 -- 172.21.15.46:0/3423971049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0b80a31e0 con 0x7fd0b81cc490 2024-09-16T04:43:41.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0bf282640 10 monclient: _renew_subs 2024-09-16T04:43:41.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0bf282640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:41.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.094+0000 7fd0bf282640 1 -- 172.21.15.46:0/3423971049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0b8125c60 con 0x7fd0b81cc490 2024-09-16T04:43:41.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.095+0000 7fd0adffb640 1 -- 172.21.15.46:0/3423971049 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fd0a8043070 con 0x7fd0b81cc490 2024-09-16T04:43:41.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.095+0000 7fd0bf282640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:41.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.095+0000 7fd0bf282640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:41.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.095+0000 7fd0bf282640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:41.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.095+0000 7fd0bf282640 1 -- 172.21.15.46:0/3423971049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0b80a3f80 con 0x7fd0b81cc490 2024-09-16T04:43:41.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.095+0000 7fd0adffb640 1 --2- 172.21.15.46:0/3423971049 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd09c0383b0 0x7fd09c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.095+0000 7fd0adffb640 1 -- 172.21.15.46:0/3423971049 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(191..191 src has 1..191) ==== 4346+0+0 (secure 0 0 0) 0x7fd0a806be50 con 0x7fd0b81cc490 2024-09-16T04:43:41.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.096+0000 7fd0bd01e640 1 --2- 172.21.15.46:0/3423971049 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd09c0383b0 0x7fd09c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.096+0000 7fd0bd01e640 10 monclient: get_auth_request con 0x7fd09c0383b0 auth_method 0 2024-09-16T04:43:41.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.096+0000 7fd0bd01e640 1 --2- 172.21.15.46:0/3423971049 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd09c0383b0 0x7fd09c03a870 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4006a90 tx=0x7fd0b4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:41.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.101+0000 7fd0adffb640 1 -- 172.21.15.46:0/3423971049 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd0a803cdb0 con 0x7fd0b81cc490 2024-09-16T04:43:41.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.101+0000 7fd0adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:41.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.101+0000 7fd0adffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:41.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.349+0000 7fd0bf282640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:43:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.349+0000 7fd0bf282640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.349+0000 7fd0bf282640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.349+0000 7fd0bf282640 1 -- 172.21.15.46:0/3423971049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd0b8067e20 con 0x7fd0b81cc490 2024-09-16T04:43:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.350+0000 7fd0adffb640 1 -- 172.21.15.46:0/3423971049 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fd0a803bae0 con 0x7fd0b81cc490 2024-09-16T04:43:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.350+0000 7fd0adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.350+0000 7fd0adffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:41.352 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:43:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.352+0000 7fd0bf282640 1 -- 172.21.15.46:0/3423971049 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd09c0383b0 msgr2=0x7fd09c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.352+0000 7fd0bf282640 1 --2- 172.21.15.46:0/3423971049 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd09c0383b0 0x7fd09c03a870 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4006a90 tx=0x7fd0b4008040 comp rx=0 tx=0).stop 2024-09-16T04:43:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.352+0000 7fd0bf282640 10 monclient: shutdown 2024-09-16T04:43:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.352+0000 7fd0bf282640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.352+0000 7fd0bf282640 1 -- 172.21.15.46:0/3423971049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0b81cc490 msgr2=0x7fd0b8123fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.352+0000 7fd0bf282640 1 --2- 172.21.15.46:0/3423971049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0b81cc490 0x7fd0b8123fc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8031000 tx=0x7fd0a8004730 comp rx=0 tx=0).stop 2024-09-16T04:43:41.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.352+0000 7fd0bf282640 1 -- 172.21.15.46:0/3423971049 shutdown_connections 2024-09-16T04:43:41.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.352+0000 7fd0bf282640 1 --2- 172.21.15.46:0/3423971049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0b81cc8f0 0x7fd0b8124500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.352+0000 7fd0bf282640 1 --2- 172.21.15.46:0/3423971049 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd09c0383b0 0x7fd09c03a870 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.352+0000 7fd0bf282640 1 --2- 172.21.15.46:0/3423971049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0b81cc490 0x7fd0b8123fc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.352+0000 7fd0bf282640 1 --2- 172.21.15.46:0/3423971049 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd0a4003f30 0x7fd0b81cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.352+0000 7fd0bf282640 1 -- 172.21.15.46:0/3423971049 >> 172.21.15.46:0/3423971049 conn(0x7fd0b80a60c0 msgr2=0x7fd0b809f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:41.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.352+0000 7fd0bf282640 1 -- 172.21.15.46:0/3423971049 shutdown_connections 2024-09-16T04:43:41.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.352+0000 7fd0bf282640 1 -- 172.21.15.46:0/3423971049 wait complete. 2024-09-16T04:43:41.364 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:43:41.364 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:43:41.364 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:43:41.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.467+0000 7fe91bb11640 10 monclient: get_monmap_and_config 2024-09-16T04:43:41.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.467+0000 7fe91bb11640 10 monclient: build_initial_monmap 2024-09-16T04:43:41.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.467+0000 7fe91bb11640 10 monclient: monmap: 2024-09-16T04:43:41.469 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:41.469 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:41.469 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:41.468326+0000 2024-09-16T04:43:41.469 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:41.468326+0000 2024-09-16T04:43:41.469 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:41.469 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.469 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:41.469 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:41.469 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:41.469 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:41.470 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:41.470 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.468+0000 7fe91bb11640 1 Processor -- start 2024-09-16T04:43:41.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.468+0000 7fe91bb11640 1 -- start start 2024-09-16T04:43:41.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.468+0000 7fe91bb11640 10 monclient: init 2024-09-16T04:43:41.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91bb11640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:41.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91bb11640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-16T04:43:41.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91bb11640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe9140a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:43:41.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91bb11640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9140a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:43:41.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91bb11640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe91413cea0 0x7fe91413f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91bb11640 10 monclient(hunting): picked mon.noname-c con 0x7fe91413cea0 addr v2:172.21.15.114:3300/0 2024-09-16T04:43:41.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91bb11640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe91409b940 con 0x7fe9140a3c10 2024-09-16T04:43:41.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91bb11640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe91409ab00 con 0x7fe9140a3ff0 2024-09-16T04:43:41.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91bb11640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91bb11640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fe91409b030 con 0x7fe91413cea0 2024-09-16T04:43:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91bb11640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91bb11640 10 monclient(hunting): authenticate will time out at 1019.884152s 2024-09-16T04:43:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9198ad640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe9140a3c10 0x7fe91409bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37062/0 (socket says 172.21.15.46:37062) 2024-09-16T04:43:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9198ad640 1 -- 172.21.15.46:0/1887871372 learned_addr learned my addr 172.21.15.46:0/1887871372 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:43:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91a0ae640 1 --2- 172.21.15.46:0/1887871372 >> v2:172.21.15.114:3300/0 conn(0x7fe91413cea0 0x7fe91413f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 1 -- 172.21.15.46:0/1887871372 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1063115433 0 0) 0x7fe91409b940 con 0x7fe9140a3c10 2024-09-16T04:43:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 10 monclient(hunting): _negotiate 2024-09-16T04:43:41.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:41.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 10 monclient(hunting): my global_id is 44643 2024-09-16T04:43:41.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 1 -- 172.21.15.46:0/1887871372 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe900003520 con 0x7fe9140a3c10 2024-09-16T04:43:41.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91a0ae640 10 monclient(hunting): get_auth_request con 0x7fe91413cea0 auth_method 0 2024-09-16T04:43:41.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91a0ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:41.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91a0ae640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91a0ae640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:41.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91a0ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:41.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91a0ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:41.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91a0ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:41.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 1 -- 172.21.15.46:0/1887871372 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1549468706 0 0) 0x7fe900003520 con 0x7fe9140a3c10 2024-09-16T04:43:41.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 1 -- 172.21.15.46:0/1887871372 >> v2:172.21.15.114:3300/0 conn(0x7fe91413cea0 msgr2=0x7fe91413f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:43:41.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 1 --2- 172.21.15.46:0/1887871372 >> v2:172.21.15.114:3300/0 conn(0x7fe91413cea0 0x7fe91413f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 1 -- 172.21.15.46:0/1887871372 >> v1:172.21.15.114:6789/0 conn(0x7fe9140a3ff0 legacy=0x7fe9140a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 10 monclient: _finish_hunting 0 2024-09-16T04:43:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:43:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:43:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 1 -- 172.21.15.46:0/1887871372 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe91409b310 con 0x7fe9140a3c10 2024-09-16T04:43:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 10 monclient: _finish_auth 0 2024-09-16T04:43:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 10 monclient: _check_auth_tickets 2024-09-16T04:43:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91bb11640 5 monclient: authenticate success, global_id 44643 2024-09-16T04:43:41.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe91bb11640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:41.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 1 -- 172.21.15.46:0/1887871372 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7fe910001670 con 0x7fe9140a3c10 2024-09-16T04:43:41.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:41.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:43:41.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 10 monclient: dump: 2024-09-16T04:43:41.495 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:41.495 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:41.495 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.495 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.495 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:41.496 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:43:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 1 -- 172.21.15.46:0/1887871372 >> v1:172.21.15.46:6789/0 conn(0x7fe9140a3c10 legacy=0x7fe91409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:43:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 1 --2- 172.21.15.46:0/1887871372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9000039b0 0x7fe900003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 10 monclient(hunting): picked mon.b con 0x7fe9000039b0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:41.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 1 --2- 172.21.15.46:0/1887871372 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe900004800 0x7fe900004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 10 monclient(hunting): picked mon.c con 0x7fe900004800 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:41.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 1 --2- 172.21.15.46:0/1887871372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9000051c0 0x7fe90000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 10 monclient(hunting): picked mon.a con 0x7fe9000051c0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:41.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.469+0000 7fe9188ab640 1 -- 172.21.15.46:0/1887871372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe91409b030 con 0x7fe9000051c0 2024-09-16T04:43:41.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9188ab640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9188ab640 1 -- 172.21.15.46:0/1887871372 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe90000c160 con 0x7fe9000039b0 2024-09-16T04:43:41.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9188ab640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9188ab640 1 -- 172.21.15.46:0/1887871372 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe90000c360 con 0x7fe900004800 2024-09-16T04:43:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9188ab640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9188ab640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:43:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe91bb11640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9198ad640 1 --2- 172.21.15.46:0/1887871372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9000039b0 0x7fe900003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe91a0ae640 1 --2- 172.21.15.46:0/1887871372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9000051c0 0x7fe90000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9190ac640 1 --2- 172.21.15.46:0/1887871372 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe900004800 0x7fe900004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9198ad640 10 monclient(hunting): get_auth_request con 0x7fe9000039b0 auth_method 0 2024-09-16T04:43:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9198ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9198ad640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9198ad640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe91a0ae640 10 monclient(hunting): get_auth_request con 0x7fe9000051c0 auth_method 0 2024-09-16T04:43:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe91a0ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe91a0ae640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe91a0ae640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9190ac640 10 monclient(hunting): get_auth_request con 0x7fe900004800 auth_method 0 2024-09-16T04:43:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9190ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9190ac640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9190ac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9198ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9198ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9198ad640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:43:41.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe91a0ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:41.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe91a0ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:41.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe91a0ae640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:43:41.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9190ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:41.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9190ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:41.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9190ac640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:43:41.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9198ad640 10 monclient(hunting): handle_auth_done global_id 44643 payload 293 2024-09-16T04:43:41.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9198ad640 1 -- 172.21.15.46:0/1887871372 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe900004800 msgr2=0x7fe900004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:43:41.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9198ad640 1 --2- 172.21.15.46:0/1887871372 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe900004800 0x7fe900004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9198ad640 1 -- 172.21.15.46:0/1887871372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9000051c0 msgr2=0x7fe90000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:43:41.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9198ad640 1 --2- 172.21.15.46:0/1887871372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9000051c0 0x7fe90000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9198ad640 10 monclient: _finish_hunting 0 2024-09-16T04:43:41.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9198ad640 1 monclient: found mon.b 2024-09-16T04:43:41.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9198ad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:41.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9198ad640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:43:41.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9198ad640 1 -- 172.21.15.46:0/1887871372 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe900003520 con 0x7fe9000039b0 2024-09-16T04:43:41.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9198ad640 10 monclient: _finish_auth 0 2024-09-16T04:43:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9198ad640 10 monclient: _check_auth_tickets 2024-09-16T04:43:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9198ad640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe9190ac640 1 --2- 172.21.15.46:0/1887871372 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe900004800 0x7fe900004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:43:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe91a0ae640 1 --2- 172.21.15.46:0/1887871372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9000051c0 0x7fe90000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:43:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.470+0000 7fe91a0ae640 1 -- 172.21.15.46:0/1887871372 reap_dead start 2024-09-16T04:43:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.471+0000 7fe9198ad640 1 --2- 172.21.15.46:0/1887871372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9000039b0 0x7fe900003dd0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe91000c210 tx=0x7fe900005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2ecc645e1e9e6f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.471+0000 7fe9188ab640 1 -- 172.21.15.46:0/1887871372 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe910007d30 con 0x7fe9000039b0 2024-09-16T04:43:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.471+0000 7fe9188ab640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.471+0000 7fe9188ab640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:43:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.471+0000 7fe9188ab640 10 monclient: dump: 2024-09-16T04:43:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:41.504 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:41.504 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:41.504 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.471+0000 7fe9188ab640 1 -- 172.21.15.46:0/1887871372 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe910005ad0 con 0x7fe9000039b0 2024-09-16T04:43:41.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.471+0000 7fe91bb11640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:41.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.471+0000 7fe9188ab640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:41.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.471+0000 7fe9188ab640 1 -- 172.21.15.46:0/1887871372 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe910007070 con 0x7fe9000039b0 2024-09-16T04:43:41.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.471+0000 7fe91bb11640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:41.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.471+0000 7fe91bb11640 4 set_mon_vals no callback set 2024-09-16T04:43:41.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.471+0000 7fe9188ab640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:41.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.471+0000 7fe9188ab640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:43:41.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.471+0000 7fe9188ab640 10 monclient: dump: 2024-09-16T04:43:41.505 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:41.505 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:41.505 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.505 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.505 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:41.506 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.506 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:41.506 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:41.506 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:41.506 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.471+0000 7fe91bb11640 10 monclient: shutdown 2024-09-16T04:43:41.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.471+0000 7fe91bb11640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:41.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.471+0000 7fe91bb11640 1 -- 172.21.15.46:0/1887871372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9000039b0 msgr2=0x7fe900003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:41.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.471+0000 7fe91bb11640 1 --2- 172.21.15.46:0/1887871372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9000039b0 0x7fe900003dd0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe91000c210 tx=0x7fe900005830 comp rx=0 tx=0).stop 2024-09-16T04:43:41.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.471+0000 7fe91bb11640 1 -- 172.21.15.46:0/1887871372 shutdown_connections 2024-09-16T04:43:41.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.471+0000 7fe91bb11640 1 --2- 172.21.15.46:0/1887871372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9000039b0 0x7fe900003dd0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.471+0000 7fe91bb11640 1 -- 172.21.15.46:0/1887871372 >> 172.21.15.46:0/1887871372 conn(0x7fe9140a6340 msgr2=0x7fe91406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:41.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.472+0000 7fe91bb11640 1 -- 172.21.15.46:0/1887871372 shutdown_connections 2024-09-16T04:43:41.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.472+0000 7fe91bb11640 1 -- 172.21.15.46:0/1887871372 wait complete. 2024-09-16T04:43:41.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.472+0000 7fe91bb11640 10 monclient: build_initial_monmap 2024-09-16T04:43:41.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.472+0000 7fe91bb11640 10 monclient: monmap: 2024-09-16T04:43:41.507 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:41.507 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:41.508 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:41.508 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:41.508 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:41.508 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.508 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:41.508 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:41.508 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:41.508 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.472+0000 7fe91bb11640 1 Processor -- start 2024-09-16T04:43:41.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe91bb11640 1 -- start start 2024-09-16T04:43:41.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe91bb11640 10 monclient: init 2024-09-16T04:43:41.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe91bb11640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:41.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe91bb11640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:43:41.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe91bb11640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9000039b0 0x7fe91411f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe91bb11640 10 monclient(hunting): picked mon.noname-c con 0x7fe9000039b0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:41.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe91bb11640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe91411fda0 0x7fe914120200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe91bb11640 10 monclient(hunting): picked mon.noname-a con 0x7fe91411fda0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:41.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe91bb11640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9141d0050 0x7fe9141d2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe91bb11640 10 monclient(hunting): picked mon.noname-b con 0x7fe9141d0050 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:41.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe91bb11640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe91bb11640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe91409be70 con 0x7fe91411fda0 2024-09-16T04:43:41.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe91bb11640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe91bb11640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe91409bcf0 con 0x7fe9141d0050 2024-09-16T04:43:41.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe91bb11640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe91bb11640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe91409bff0 con 0x7fe9000039b0 2024-09-16T04:43:41.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe91bb11640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:41.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe91bb11640 10 monclient(hunting): authenticate will time out at 1019.888566s 2024-09-16T04:43:41.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9198ad640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9000039b0 0x7fe91411f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9190ac640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe91411fda0 0x7fe914120200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9198ad640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9000039b0 0x7fe91411f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:42704/0 (socket says 172.21.15.46:42704) 2024-09-16T04:43:41.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9190ac640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe91411fda0 0x7fe914120200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53450/0 (socket says 172.21.15.46:53450) 2024-09-16T04:43:41.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9198ad640 1 -- 172.21.15.46:0/4227879052 learned_addr learned my addr 172.21.15.46:0/4227879052 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:41.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9198ad640 10 monclient(hunting): get_auth_request con 0x7fe9000039b0 auth_method 0 2024-09-16T04:43:41.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9198ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:41.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9198ad640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9198ad640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:41.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9190ac640 10 monclient(hunting): get_auth_request con 0x7fe91411fda0 auth_method 0 2024-09-16T04:43:41.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9190ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:41.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9190ac640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9190ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:41.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9190ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:41.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9190ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:41.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9198ad640 1 -- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9000039b0 msgr2=0x7fe91411f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:43:41.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9198ad640 1 -- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9000039b0 msgr2=0x7fe91411f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:43:41.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9190ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:41.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9198ad640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9000039b0 0x7fe91411f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:41.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9198ad640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9000039b0 0x7fe91411f860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:41.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9190ac640 10 monclient(hunting): handle_auth_done global_id 44646 payload 824 2024-09-16T04:43:41.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe91a0ae640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9141d0050 0x7fe9141d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9190ac640 1 -- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9000039b0 msgr2=0x7fe91411f860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:43:41.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9190ac640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe9000039b0 0x7fe91411f860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9190ac640 1 -- 172.21.15.46:0/4227879052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9141d0050 msgr2=0x7fe9141d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:41.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9190ac640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9141d0050 0x7fe9141d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9190ac640 10 monclient: _finish_hunting 0 2024-09-16T04:43:41.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9190ac640 1 monclient: found mon.noname-a 2024-09-16T04:43:41.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9190ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:41.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9190ac640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:43:41.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe9190ac640 1 -- 172.21.15.46:0/4227879052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe914122cb0 con 0x7fe91411fda0 2024-09-16T04:43:41.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe91a0ae640 10 monclient: get_auth_request con 0x7fe9141d0050 auth_method 0 2024-09-16T04:43:41.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.473+0000 7fe91a0ae640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9141d0050 0x7fe9141d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:43:41.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe9190ac640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe91411fda0 0x7fe914120200 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe9140518b0 tx=0x7fe90800fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:41.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe9067fc640 1 -- 172.21.15.46:0/4227879052 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90801a070 con 0x7fe91411fda0 2024-09-16T04:43:41.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe9067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:41.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe9067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:43:41.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe9067fc640 10 monclient: dump: 2024-09-16T04:43:41.515 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:41.515 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:41.516 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.516 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.516 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:41.516 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.516 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:41.516 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:41.516 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:41.516 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe9067fc640 10 monclient: _finish_auth 0 2024-09-16T04:43:41.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe9067fc640 10 monclient: _check_auth_tickets 2024-09-16T04:43:41.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe9067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:41.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe9067fc640 1 -- 172.21.15.46:0/4227879052 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe90800fe50 con 0x7fe91411fda0 2024-09-16T04:43:41.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe91bb11640 5 monclient: authenticate success, global_id 44646 2024-09-16T04:43:41.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe91bb11640 10 monclient: _renew_subs 2024-09-16T04:43:41.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe91bb11640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:41.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe91bb11640 1 -- 172.21.15.46:0/4227879052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe914122f90 con 0x7fe91411fda0 2024-09-16T04:43:41.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe9067fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:41.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe9067fc640 1 -- 172.21.15.46:0/4227879052 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe908001370 con 0x7fe91411fda0 2024-09-16T04:43:41.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe9067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:41.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe9067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:41.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe9067fc640 10 monclient: dump: 2024-09-16T04:43:41.517 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:41.518 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:41.518 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.518 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.518 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:41.518 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.518 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:41.518 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:41.518 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:41.518 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe91bb11640 10 monclient: _renew_subs 2024-09-16T04:43:41.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe91bb11640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:41.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe907fff640 4 set_mon_vals no callback set 2024-09-16T04:43:41.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.474+0000 7fe91bb11640 1 -- 172.21.15.46:0/4227879052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9140a0b70 con 0x7fe91411fda0 2024-09-16T04:43:41.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe91bb11640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:41.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe9067fc640 1 -- 172.21.15.46:0/4227879052 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fe90801d070 con 0x7fe91411fda0 2024-09-16T04:43:41.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe91bb11640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe91409f120 0x7fe91409f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe91bb11640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe91bb11640 1 -- 172.21.15.46:0/4227879052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe91409cc20 con 0x7fe91409f120 2024-09-16T04:43:41.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe9198ad640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe91409f120 0x7fe91409f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe9198ad640 10 monclient: get_auth_request con 0x7fe91409f120 auth_method 0 2024-09-16T04:43:41.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe9198ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:41.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe9198ad640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe9198ad640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:41.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe9198ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:41.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe9198ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:41.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe9198ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:41.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe9067fc640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe8f80383b0 0x7fe8f803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe9067fc640 1 -- 172.21.15.46:0/4227879052 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 4738+0+0 (secure 0 0 0) 0x7fe90804aea0 con 0x7fe91411fda0 2024-09-16T04:43:41.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe9198ad640 10 monclient(hunting): handle_auth_done global_id 44649 payload 824 2024-09-16T04:43:41.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe9198ad640 1 -- 172.21.15.46:0/4227879052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe9140a0400 con 0x7fe91409f120 2024-09-16T04:43:41.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe91a0ae640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe8f80383b0 0x7fe8f803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe91a0ae640 10 monclient: get_auth_request con 0x7fe8f80383b0 auth_method 0 2024-09-16T04:43:41.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe9198ad640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe91409f120 0x7fe91409f500 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe910001200 tx=0x7fe910001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:41.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe9067fc640 1 -- 172.21.15.46:0/4227879052 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe910042020 con 0x7fe91409f120 2024-09-16T04:43:41.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe9067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:41.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe9067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:41.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.475+0000 7fe9067fc640 10 monclient: dump: 2024-09-16T04:43:41.522 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:41.522 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:41.522 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.523 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.523 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:41.523 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.523 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:41.523 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:41.523 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:41.523 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.476+0000 7fe91a0ae640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe8f80383b0 0x7fe8f803a870 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe914067ce0 tx=0x7fe90c00b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:41.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.476+0000 7fe9067fc640 1 -- 172.21.15.46:0/4227879052 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe910038040 con 0x7fe91409f120 2024-09-16T04:43:41.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.476+0000 7fe9067fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:41.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.476+0000 7fe9067fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:41.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.476+0000 7fe9067fc640 1 -- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe91409f120 msgr2=0x7fe91409f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:41.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.476+0000 7fe9067fc640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe91409f120 0x7fe91409f500 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe910001200 tx=0x7fe910001a40 comp rx=0 tx=0).stop 2024-09-16T04:43:41.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.476+0000 7fe9067fc640 1 -- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe91409f120 msgr2=0x7fe91409f500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:43:41.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.476+0000 7fe9067fc640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe91409f120 0x7fe91409f500 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.482+0000 7fe91bb11640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:43:41.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.482+0000 7fe91bb11640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe91409fa40 0x7fe9141d69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.482+0000 7fe91bb11640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.482+0000 7fe91bb11640 1 -- 172.21.15.46:0/4227879052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe9140a43d0 con 0x7fe91409fa40 2024-09-16T04:43:41.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.482+0000 7fe9198ad640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe91409fa40 0x7fe9141d69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.482+0000 7fe9198ad640 10 monclient: get_auth_request con 0x7fe91409fa40 auth_method 0 2024-09-16T04:43:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.482+0000 7fe9198ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.482+0000 7fe9198ad640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.482+0000 7fe9198ad640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.482+0000 7fe9198ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.482+0000 7fe9198ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.482+0000 7fe9198ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.482+0000 7fe9198ad640 10 monclient(hunting): handle_auth_done global_id 44652 payload 824 2024-09-16T04:43:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.482+0000 7fe9198ad640 1 -- 172.21.15.46:0/4227879052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe914121580 con 0x7fe91409fa40 2024-09-16T04:43:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.482+0000 7fe9198ad640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe91409fa40 0x7fe9141d69a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe910040000 tx=0x7fe91002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe9067fc640 1 -- 172.21.15.46:0/4227879052 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe910042050 con 0x7fe91409fa40 2024-09-16T04:43:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe9067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe9067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe9067fc640 10 monclient: dump: 2024-09-16T04:43:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:41.527 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe9067fc640 1 -- 172.21.15.46:0/4227879052 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7fe910007480 con 0x7fe91409fa40 2024-09-16T04:43:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe9067fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:43:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe9067fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe9067fc640 1 -- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe91409fa40 msgr2=0x7fe9141d69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe9067fc640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe91409fa40 0x7fe9141d69a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe910040000 tx=0x7fe91002e000 comp rx=0 tx=0).stop 2024-09-16T04:43:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe9067fc640 1 -- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe91409fa40 msgr2=0x7fe9141d69a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:43:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe9067fc640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe91409fa40 0x7fe9141d69a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.528 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:43:41.528 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:43:41.528 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:43:41.528 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:43:41.528 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 58, 2024-09-16T04:43:41.528 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 101230, 2024-09-16T04:43:41.529 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:43:41.529 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:43:41.529 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:43:41.529 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:43:41.529 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:41.529 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 2, 2024-09-16T04:43:41.529 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:43:41.529 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:43:41.529 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:43:41.529 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:41.529 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:41.530 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:41.530 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:41.530 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:41.530 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:41.530 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:41.530 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:41.530 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:41.530 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:41.530 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:41.530 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:41.530 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:43:41.531 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:43:41.531 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:41.531 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:41.531 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:41.531 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:41.531 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:41.531 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:41.531 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:41.531 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:41.531 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:41.532 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:41.532 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:41.532 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:41.532 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:41.532 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:43:41.532 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:43:41.532 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:43:41.532 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:43:41.533 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:43:41.533 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:43:41.533 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:43:41.533 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:43:41.533 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:43:41.533 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:43:41.533 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:43:41.533 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:43:41.533 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:43:41.534 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:43:41.534 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:43:41.534 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:43:41.534 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:43:41.534 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:41.534 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:41.534 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:41.534 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:41.534 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:41.534 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:41.535 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:41.535 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:41.535 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:41.535 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:41.535 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:41.535 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:41.535 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:43:41.535 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:41.535 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:43:41.535 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:41.536 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:43:41.536 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:43:41.536 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:41.536 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:41.536 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:41.536 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:41.536 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:43:41.536 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:41.536 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:41.537 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:41.537 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:41.537 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:43:41.537 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:41.537 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:41.537 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:41.537 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:41.537 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:43:41.537 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:43:41.537 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:41.537 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:41.538 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:41.538 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:41.538 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:41.538 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:43:41.538 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:43:41.538 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:41.538 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:41.538 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:41.539 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:41.539 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:43:41.539 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:41.539 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:41.539 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:41.539 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:41.539 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:43:41.540 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:41.540 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:41.540 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:41.540 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:41.540 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:43:41.540 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:43:41.540 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:41.540 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:41.541 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:41.541 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:41.541 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:41.541 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:43:41.541 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:43:41.541 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:41.541 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:41.541 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:41.541 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:41.542 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:43:41.542 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:41.542 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:41.542 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:41.542 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:41.542 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:43:41.542 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:41.543 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:41.543 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:41.543 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:41.543 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:43:41.543 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:43:41.543 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:41.543 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:41.543 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:41.543 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:41.544 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:41.544 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:41.544 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:43:41.544 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:43:41.544 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:41.544 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:41.544 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:41.544 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:43:41.544 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:41.544 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:41.544 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:43:41.544 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:41.545 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:41.545 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:41.545 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:43:41.545 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:41.545 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:41.545 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:43:41.545 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:41.545 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:41.545 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:41.545 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:43:41.545 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:41.546 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:41.546 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:41.546 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:43:41.546 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:43:41.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe91bb11640 1 -- 172.21.15.46:0/4227879052 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe8f80383b0 msgr2=0x7fe8f803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:41.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe91bb11640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe8f80383b0 0x7fe8f803a870 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe914067ce0 tx=0x7fe90c00b040 comp rx=0 tx=0).stop 2024-09-16T04:43:41.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe91bb11640 10 monclient: shutdown 2024-09-16T04:43:41.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe91bb11640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:41.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe91bb11640 1 -- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe91411fda0 msgr2=0x7fe914120200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:41.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe91bb11640 1 --2- 172.21.15.46:0/4227879052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe91411fda0 0x7fe914120200 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe9140518b0 tx=0x7fe90800fc00 comp rx=0 tx=0).stop 2024-09-16T04:43:41.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe91a0ae640 1 -- 172.21.15.46:0/4227879052 reap_dead start 2024-09-16T04:43:41.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe91a0ae640 1 -- 172.21.15.46:0/4227879052 reap_dead start 2024-09-16T04:43:41.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe91bb11640 1 -- 172.21.15.46:0/4227879052 shutdown_connections 2024-09-16T04:43:41.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe91bb11640 1 -- 172.21.15.46:0/4227879052 >> 172.21.15.46:0/4227879052 conn(0x7fe9140a6340 msgr2=0x7fe91413c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:41.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe91bb11640 1 -- 172.21.15.46:0/4227879052 shutdown_connections 2024-09-16T04:43:41.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.483+0000 7fe91bb11640 1 -- 172.21.15.46:0/4227879052 wait complete. 2024-09-16T04:43:41.549 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:43:41.549 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:43:41.550 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:43:41.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.652+0000 7f14a3a37640 10 monclient: get_monmap_and_config 2024-09-16T04:43:41.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.652+0000 7f14a3a37640 10 monclient: build_initial_monmap 2024-09-16T04:43:41.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.652+0000 7f14a3a37640 10 monclient: monmap: 2024-09-16T04:43:41.653 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:41.653 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:41.653 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:41.652613+0000 2024-09-16T04:43:41.653 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:41.652613+0000 2024-09-16T04:43:41.653 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:41.654 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.654 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:41.654 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:41.654 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:41.654 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:41.654 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:41.654 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.652+0000 7f14a3a37640 1 Processor -- start 2024-09-16T04:43:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.652+0000 7f14a3a37640 1 -- start start 2024-09-16T04:43:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.652+0000 7f14a3a37640 10 monclient: init 2024-09-16T04:43:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a3a37640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a3a37640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-16T04:43:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a3a37640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f149c0a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:43:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a3a37640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c0a3ff0 0x7f149c0a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a3a37640 10 monclient(hunting): picked mon.noname-b con 0x7f149c0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a3a37640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f149c13be90 0x7f149c13e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a3a37640 10 monclient(hunting): picked mon.noname-a con 0x7f149c13be90 addr v2:172.21.15.46:3300/0 2024-09-16T04:43:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a3a37640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f149c09ab70 con 0x7f149c0a3c10 2024-09-16T04:43:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a3a37640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a3a37640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f149c0a4d60 con 0x7f149c13be90 2024-09-16T04:43:41.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a3a37640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a3a37640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f149c09b030 con 0x7f149c0a3ff0 2024-09-16T04:43:41.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a3a37640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:41.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a3a37640 10 monclient(hunting): authenticate will time out at 1020.068420s 2024-09-16T04:43:41.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a1fd4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f149c13be90 0x7f149c13e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a0fd2640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c0a3ff0 0x7f149c0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a0fd2640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c0a3ff0 0x7f149c0a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:42714/0 (socket says 172.21.15.46:42714) 2024-09-16T04:43:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a17d3640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f149c0a3c10 0x7f149c09bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.46:57276/0 (socket says 172.21.15.46:57276) 2024-09-16T04:43:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a1fd4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f149c13be90 0x7f149c13e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53474/0 (socket says 172.21.15.46:53474) 2024-09-16T04:43:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a0fd2640 1 -- 172.21.15.46:0/1475094405 learned_addr learned my addr 172.21.15.46:0/1475094405 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a0fd2640 10 monclient(hunting): get_auth_request con 0x7f149c0a3ff0 auth_method 0 2024-09-16T04:43:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a0fd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a0fd2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a0fd2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a1fd4640 10 monclient(hunting): get_auth_request con 0x7f149c13be90 auth_method 0 2024-09-16T04:43:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a1fd4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a1fd4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a1fd4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a1fd4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a1fd4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a0fd2640 1 -- 172.21.15.46:0/1475094405 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c0a3ff0 msgr2=0x7f149c0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:43:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a0fd2640 1 -- 172.21.15.46:0/1475094405 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c0a3ff0 msgr2=0x7f149c0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a0fd2640 1 --2- 172.21.15.46:0/1475094405 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c0a3ff0 0x7f149c0a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a1fd4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a0fd2640 1 --2- 172.21.15.46:0/1475094405 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c0a3ff0 0x7f149c0a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a1fd4640 10 monclient(hunting): handle_auth_done global_id 44655 payload 274 2024-09-16T04:43:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a1fd4640 1 -- 172.21.15.46:0/1475094405 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c0a3ff0 msgr2=0x7f149c0a43d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a1fd4640 1 --2- 172.21.15.46:0/1475094405 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c0a3ff0 0x7f149c0a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f1493fff640 1 -- 172.21.15.46:0/1475094405 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1401507896 0 0) 0x7f149c09ab70 con 0x7f149c0a3c10 2024-09-16T04:43:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a1fd4640 1 -- 172.21.15.46:0/1475094405 >> v1:172.21.15.114:6789/0 conn(0x7f149c0a3c10 legacy=0x7f149c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a1fd4640 10 monclient: _finish_hunting 0 2024-09-16T04:43:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a1fd4640 1 monclient: found mon.noname-a 2024-09-16T04:43:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a1fd4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a1fd4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:43:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f14a1fd4640 1 -- 172.21.15.46:0/1475094405 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f149c09b310 con 0x7f149c13be90 2024-09-16T04:43:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.653+0000 7f1493fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:43:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.654+0000 7f14a1fd4640 1 --2- 172.21.15.46:0/1475094405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f149c13be90 0x7f149c13e280 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1494009d90 tx=0x7f149402a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2e05b6063a0449a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.654+0000 7f1493fff640 1 -- 172.21.15.46:0/1475094405 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1494007740 con 0x7f149c13be90 2024-09-16T04:43:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.654+0000 7f1493fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.654+0000 7f1493fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:43:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.654+0000 7f1493fff640 10 monclient: dump: 2024-09-16T04:43:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:41.671 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.654+0000 7f1493fff640 10 monclient: _finish_auth 0 2024-09-16T04:43:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.654+0000 7f1493fff640 10 monclient: _check_auth_tickets 2024-09-16T04:43:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.654+0000 7f1493fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.654+0000 7f1493fff640 1 -- 172.21.15.46:0/1475094405 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14940078e0 con 0x7f149c13be90 2024-09-16T04:43:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.654+0000 7f14a3a37640 5 monclient: authenticate success, global_id 44655 2024-09-16T04:43:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.654+0000 7f14a3a37640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.654+0000 7f1493fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.654+0000 7f14a3a37640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.654+0000 7f14a3a37640 4 set_mon_vals no callback set 2024-09-16T04:43:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.654+0000 7f1493fff640 1 -- 172.21.15.46:0/1475094405 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1494007cb0 con 0x7f149c13be90 2024-09-16T04:43:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.654+0000 7f1493fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.654+0000 7f1493fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.654+0000 7f1493fff640 10 monclient: dump: 2024-09-16T04:43:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:41.673 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.655+0000 7f14a3a37640 10 monclient: shutdown 2024-09-16T04:43:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.655+0000 7f14a3a37640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.655+0000 7f14a3a37640 1 -- 172.21.15.46:0/1475094405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f149c13be90 msgr2=0x7f149c13e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.655+0000 7f14a3a37640 1 --2- 172.21.15.46:0/1475094405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f149c13be90 0x7f149c13e280 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1494009d90 tx=0x7f149402a530 comp rx=0 tx=0).stop 2024-09-16T04:43:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.655+0000 7f14a3a37640 1 -- 172.21.15.46:0/1475094405 shutdown_connections 2024-09-16T04:43:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.655+0000 7f14a3a37640 1 --2- 172.21.15.46:0/1475094405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f149c13be90 0x7f149c13e280 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.655+0000 7f14a3a37640 1 --2- 172.21.15.46:0/1475094405 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c0a3ff0 0x7f149c0a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.655+0000 7f14a3a37640 1 -- 172.21.15.46:0/1475094405 >> 172.21.15.46:0/1475094405 conn(0x7f149c0a5d10 msgr2=0x7f149c0a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.655+0000 7f14a3a37640 1 -- 172.21.15.46:0/1475094405 shutdown_connections 2024-09-16T04:43:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.655+0000 7f14a3a37640 1 -- 172.21.15.46:0/1475094405 wait complete. 2024-09-16T04:43:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.655+0000 7f14a3a37640 10 monclient: build_initial_monmap 2024-09-16T04:43:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.655+0000 7f14a3a37640 10 monclient: monmap: 2024-09-16T04:43:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:41.675 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.675 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:41.675 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:41.675 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:41.675 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a3a37640 1 Processor -- start 2024-09-16T04:43:41.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a3a37640 1 -- start start 2024-09-16T04:43:41.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a3a37640 10 monclient: init 2024-09-16T04:43:41.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a3a37640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:41.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a3a37640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:43:41.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a3a37640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f149c0a3c10 0x7f149c1cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a3a37640 10 monclient(hunting): picked mon.noname-a con 0x7f149c0a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:41.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a3a37640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f149c0a3ff0 0x7f149c136d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a3a37640 10 monclient(hunting): picked mon.noname-b con 0x7f149c0a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:41.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a3a37640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c13be90 0x7f149c0a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a3a37640 10 monclient(hunting): picked mon.noname-c con 0x7f149c13be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:41.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a3a37640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a3a37640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f149c06e550 con 0x7f149c0a3c10 2024-09-16T04:43:41.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a3a37640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a3a37640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f149c06e3d0 con 0x7f149c0a3ff0 2024-09-16T04:43:41.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a3a37640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a3a37640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f149c13e7c0 con 0x7f149c13be90 2024-09-16T04:43:41.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a3a37640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:41.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a3a37640 10 monclient(hunting): authenticate will time out at 1020.071838s 2024-09-16T04:43:41.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a17d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f149c0a3c10 0x7f149c1cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a17d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f149c0a3c10 0x7f149c1cb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53490/0 (socket says 172.21.15.46:53490) 2024-09-16T04:43:41.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a17d3640 1 -- 172.21.15.46:0/2915569685 learned_addr learned my addr 172.21.15.46:0/2915569685 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:41.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a17d3640 10 monclient(hunting): get_auth_request con 0x7f149c0a3c10 auth_method 0 2024-09-16T04:43:41.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a17d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:41.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a17d3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a17d3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:41.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a1fd4640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c13be90 0x7f149c0a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a1fd4640 10 monclient(hunting): get_auth_request con 0x7f149c13be90 auth_method 0 2024-09-16T04:43:41.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a1fd4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:41.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a1fd4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.656+0000 7f14a1fd4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:41.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a17d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:41.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a17d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:41.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a17d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:41.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a1fd4640 1 -- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c13be90 msgr2=0x7f149c0a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:43:41.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a1fd4640 1 -- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c13be90 msgr2=0x7f149c0a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:43:41.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a1fd4640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c13be90 0x7f149c0a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:41.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a1fd4640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c13be90 0x7f149c0a1720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:41.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a17d3640 10 monclient(hunting): handle_auth_done global_id 44658 payload 824 2024-09-16T04:43:41.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a0fd2640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f149c0a3ff0 0x7f149c136d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a17d3640 1 -- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c13be90 msgr2=0x7f149c0a1720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:43:41.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a17d3640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c13be90 0x7f149c0a1720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a17d3640 1 -- 172.21.15.46:0/2915569685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f149c0a3ff0 msgr2=0x7f149c136d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:41.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a17d3640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f149c0a3ff0 0x7f149c136d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:41.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a17d3640 10 monclient: _finish_hunting 0 2024-09-16T04:43:41.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a17d3640 1 monclient: found mon.noname-a 2024-09-16T04:43:41.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a17d3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:41.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a17d3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:43:41.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a17d3640 1 -- 172.21.15.46:0/2915569685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f149c0a0400 con 0x7f149c0a3c10 2024-09-16T04:43:41.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a0fd2640 10 monclient: get_auth_request con 0x7f149c0a3ff0 auth_method 0 2024-09-16T04:43:41.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a0fd2640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f149c0a3ff0 0x7f149c136d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:43:41.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a17d3640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f149c0a3c10 0x7f149c1cb7d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f149800c430 tx=0x7f149800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:41.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f1491ffb640 1 -- 172.21.15.46:0/2915569685 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14980074e0 con 0x7f149c0a3c10 2024-09-16T04:43:41.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f1491ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:41.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f1491ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:43:41.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f1491ffb640 10 monclient: dump: 2024-09-16T04:43:41.680 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:41.680 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:41.680 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.680 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.680 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:41.680 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.680 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:41.680 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:41.680 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:41.680 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f1491ffb640 10 monclient: _finish_auth 0 2024-09-16T04:43:41.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f1491ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:43:41.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f1491ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:41.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a3a37640 5 monclient: authenticate success, global_id 44658 2024-09-16T04:43:41.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f1491ffb640 1 -- 172.21.15.46:0/2915569685 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14980047e0 con 0x7f149c0a3c10 2024-09-16T04:43:41.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f1491ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:41.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f1491ffb640 1 -- 172.21.15.46:0/2915569685 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1498004bd0 con 0x7f149c0a3c10 2024-09-16T04:43:41.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a3a37640 10 monclient: _renew_subs 2024-09-16T04:43:41.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a3a37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:41.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a3a37640 1 -- 172.21.15.46:0/2915569685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f149c13c7e0 con 0x7f149c0a3c10 2024-09-16T04:43:41.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14937fe640 4 set_mon_vals no callback set 2024-09-16T04:43:41.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f1491ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:41.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f1491ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:41.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f1491ffb640 10 monclient: dump: 2024-09-16T04:43:41.681 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:41.681 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:41.681 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:41.682 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a3a37640 10 monclient: _renew_subs 2024-09-16T04:43:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a3a37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.657+0000 7f14a3a37640 1 -- 172.21.15.46:0/2915569685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f149c121110 con 0x7f149c0a3c10 2024-09-16T04:43:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.658+0000 7f1491ffb640 1 -- 172.21.15.46:0/2915569685 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f1498011070 con 0x7f149c0a3c10 2024-09-16T04:43:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.658+0000 7f14a3a37640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.658+0000 7f14a3a37640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c137280 0x7f149c121350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.658+0000 7f14a3a37640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.658+0000 7f14a3a37640 1 -- 172.21.15.46:0/2915569685 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f149c0a43d0 con 0x7f149c137280 2024-09-16T04:43:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.658+0000 7f1491ffb640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f14800383e0 0x7f148003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.658+0000 7f14a0fd2640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c137280 0x7f149c121350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.658+0000 7f1491ffb640 1 -- 172.21.15.46:0/2915569685 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 4738+0+0 (secure 0 0 0) 0x7f149804b360 con 0x7f149c0a3c10 2024-09-16T04:43:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.658+0000 7f14a0fd2640 10 monclient: get_auth_request con 0x7f149c137280 auth_method 0 2024-09-16T04:43:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.658+0000 7f14a0fd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.658+0000 7f14a0fd2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.658+0000 7f14a0fd2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.658+0000 7f14a0fd2640 1 -- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c137280 msgr2=0x7f149c121350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:43:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.658+0000 7f14a0fd2640 1 -- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c137280 msgr2=0x7f149c121350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:43:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.658+0000 7f14a0fd2640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c137280 0x7f149c121350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.658+0000 7f14a0fd2640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c137280 0x7f149c121350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.658+0000 7f14a1fd4640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f14800383e0 0x7f148003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.658+0000 7f14a1fd4640 10 monclient: get_auth_request con 0x7f14800383e0 auth_method 0 2024-09-16T04:43:41.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.659+0000 7f14a1fd4640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f14800383e0 0x7f148003a8a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1494034000 tx=0x7f149402f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.859+0000 7f14a0fd2640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c137280 0x7f149c121350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.859+0000 7f14a0fd2640 10 monclient: get_auth_request con 0x7f149c137280 auth_method 0 2024-09-16T04:43:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.859+0000 7f14a0fd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:41.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.859+0000 7f14a0fd2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:41.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.859+0000 7f14a0fd2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:41.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.859+0000 7f14a0fd2640 1 -- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c137280 msgr2=0x7f149c121350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:43:41.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.859+0000 7f14a0fd2640 1 -- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c137280 msgr2=0x7f149c121350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:43:41.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.859+0000 7f14a0fd2640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c137280 0x7f149c121350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:41.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:41.859+0000 7f14a0fd2640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c137280 0x7f149c121350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:43:42.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.260+0000 7f14a0fd2640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c137280 0x7f149c121350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.260+0000 7f14a0fd2640 10 monclient: get_auth_request con 0x7f149c137280 auth_method 0 2024-09-16T04:43:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.260+0000 7f14a0fd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.260+0000 7f14a0fd2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.260+0000 7f14a0fd2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.260+0000 7f14a0fd2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.260+0000 7f14a0fd2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.260+0000 7f14a0fd2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.260+0000 7f14a0fd2640 10 monclient(hunting): handle_auth_done global_id 64100 payload 824 2024-09-16T04:43:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.260+0000 7f14a0fd2640 1 -- 172.21.15.46:0/2915569685 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f149c069150 con 0x7f149c137280 2024-09-16T04:43:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.260+0000 7f14a0fd2640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c137280 0x7f149c121350 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f149c0a2ea0 tx=0x7f148c013d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.261+0000 7f1491ffb640 1 -- 172.21.15.46:0/2915569685 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148c013e40 con 0x7f149c137280 2024-09-16T04:43:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.261+0000 7f1491ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.261+0000 7f1491ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:43:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.261+0000 7f1491ffb640 10 monclient: dump: 2024-09-16T04:43:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:42.263 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.261+0000 7f1491ffb640 1 -- 172.21.15.46:0/2915569685 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f148c013e40 con 0x7f149c137280 2024-09-16T04:43:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.261+0000 7f1491ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.261+0000 7f1491ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.261+0000 7f1491ffb640 1 -- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c137280 msgr2=0x7f149c121350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.261+0000 7f1491ffb640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c137280 0x7f149c121350 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f149c0a2ea0 tx=0x7f148c013d10 comp rx=0 tx=0).stop 2024-09-16T04:43:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.261+0000 7f1491ffb640 1 -- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c137280 msgr2=0x7f149c121350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:43:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.261+0000 7f1491ffb640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c137280 0x7f149c121350 secure :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f149c0a2ea0 tx=0x7f148c013d10 comp rx=0 tx=0).stop 2024-09-16T04:43:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.268+0000 7f14a3a37640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:43:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.268+0000 7f14a3a37640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c0a1c60 0x7f149c0a2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.268+0000 7f14a3a37640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.268+0000 7f14a3a37640 1 -- 172.21.15.46:0/2915569685 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f149c0a25b0 con 0x7f149c0a1c60 2024-09-16T04:43:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.268+0000 7f14a0fd2640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c0a1c60 0x7f149c0a2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.268+0000 7f14a0fd2640 10 monclient: get_auth_request con 0x7f149c0a1c60 auth_method 0 2024-09-16T04:43:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.268+0000 7f14a0fd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.268+0000 7f14a0fd2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.268+0000 7f14a0fd2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.268+0000 7f14a0fd2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.268+0000 7f14a0fd2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.268+0000 7f14a0fd2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.268+0000 7f14a0fd2640 10 monclient(hunting): handle_auth_done global_id 64103 payload 824 2024-09-16T04:43:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.268+0000 7f14a0fd2640 1 -- 172.21.15.46:0/2915569685 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f149c067e40 con 0x7f149c0a1c60 2024-09-16T04:43:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.269+0000 7f14a0fd2640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c0a1c60 0x7f149c0a2070 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f148c01d000 tx=0x7f148c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.269+0000 7f1491ffb640 1 -- 172.21.15.46:0/2915569685 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148c02a630 con 0x7f149c0a1c60 2024-09-16T04:43:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.269+0000 7f1491ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.269+0000 7f1491ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:43:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.269+0000 7f1491ffb640 10 monclient: dump: 2024-09-16T04:43:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:42.273 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.269+0000 7f1491ffb640 1 -- 172.21.15.46:0/2915569685 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f148c02a630 con 0x7f149c0a1c60 2024-09-16T04:43:42.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.269+0000 7f1491ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:43:42.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.269+0000 7f1491ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:42.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.269+0000 7f1491ffb640 1 -- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c0a1c60 msgr2=0x7f149c0a2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:42.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.269+0000 7f1491ffb640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c0a1c60 0x7f149c0a2070 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f148c01d000 tx=0x7f148c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:43:42.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.269+0000 7f1491ffb640 1 -- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c0a1c60 msgr2=0x7f149c0a2070 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:43:42.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.269+0000 7f1491ffb640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f149c0a1c60 0x7f149c0a2070 secure :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f148c01d000 tx=0x7f148c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:43:42.274 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:43:42.275 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:43:42.275 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:43:42.275 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:43:42.275 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 58, 2024-09-16T04:43:42.275 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 10205, 2024-09-16T04:43:42.275 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:43:42.275 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:43:42.275 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:43:42.275 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:43:42.275 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:42.275 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 3, 2024-09-16T04:43:42.275 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:43:42.275 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:43:42.276 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:43:42.276 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:42.276 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:42.276 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:42.276 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:42.276 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:42.276 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:42.276 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:42.276 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:42.276 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:42.276 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:42.276 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:42.277 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:42.277 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:43:42.277 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:43:42.277 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:42.277 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:42.277 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:42.277 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:42.277 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:42.277 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:42.277 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:42.277 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:42.277 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:42.277 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:42.278 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:42.278 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:42.278 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.278 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:43:42.278 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:43:42.278 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:43:42.278 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:43:42.278 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:43:42.278 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:43:42.278 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:43:42.278 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:43:42.278 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:43:42.279 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:43:42.279 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:43:42.279 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:43:42.279 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:43:42.279 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:43:42.279 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:43:42.279 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:43:42.279 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:43:42.279 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:42.279 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:42.279 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:42.279 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:42.279 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:42.279 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:42.280 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:42.280 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:42.280 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:42.280 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:42.280 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:42.280 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:42.280 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:43:42.280 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.280 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:43:42.280 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.280 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:43:42.280 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:43:42.281 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:42.281 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:42.281 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.281 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:42.281 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:43:42.281 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:42.281 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.281 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.281 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:42.281 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:43:42.281 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:42.281 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:42.281 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:42.282 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.282 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:43:42.282 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:43:42.282 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:42.282 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:42.282 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:42.282 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.282 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.282 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:43:42.282 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:43:42.282 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:42.282 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:42.282 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.282 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:42.283 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:43:42.283 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:42.283 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.283 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.283 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:42.283 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:43:42.283 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:42.283 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:42.283 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:42.283 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.283 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:43:42.283 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:43:42.284 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:42.284 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:42.284 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:42.284 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.284 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.284 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:43:42.284 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:43:42.284 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:42.284 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:42.284 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.284 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:42.284 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:43:42.284 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:42.285 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.285 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.285 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:42.285 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:43:42.285 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:42.285 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:42.285 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:42.285 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.285 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:43:42.285 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:43:42.285 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:42.285 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:42.285 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:42.286 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:42.286 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:42.286 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.286 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:43:42.286 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:43:42.286 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.286 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:42.286 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:42.286 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:43:42.286 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:42.286 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:42.286 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:43:42.286 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.286 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:42.287 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:42.287 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:43:42.287 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:42.287 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:42.287 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.287 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:43:42.287 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:43:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.270+0000 7f14a3a37640 1 -- 172.21.15.46:0/2915569685 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f14800383e0 msgr2=0x7f148003a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.270+0000 7f14a3a37640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f14800383e0 0x7f148003a8a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1494034000 tx=0x7f149402f000 comp rx=0 tx=0).stop 2024-09-16T04:43:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.270+0000 7f14a3a37640 10 monclient: shutdown 2024-09-16T04:43:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.270+0000 7f14a3a37640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.270+0000 7f14a3a37640 1 -- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f149c0a3c10 msgr2=0x7f149c1cb7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.270+0000 7f14a3a37640 1 --2- 172.21.15.46:0/2915569685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f149c0a3c10 0x7f149c1cb7d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f149800c430 tx=0x7f149800c8f0 comp rx=0 tx=0).stop 2024-09-16T04:43:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.270+0000 7f14a1fd4640 1 -- 172.21.15.46:0/2915569685 reap_dead start 2024-09-16T04:43:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.270+0000 7f14a1fd4640 1 -- 172.21.15.46:0/2915569685 reap_dead start 2024-09-16T04:43:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.270+0000 7f14a3a37640 1 -- 172.21.15.46:0/2915569685 shutdown_connections 2024-09-16T04:43:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.270+0000 7f14a3a37640 1 -- 172.21.15.46:0/2915569685 >> 172.21.15.46:0/2915569685 conn(0x7f149c0a5d10 msgr2=0x7f149c09f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.270+0000 7f14a3a37640 1 -- 172.21.15.46:0/2915569685 shutdown_connections 2024-09-16T04:43:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.270+0000 7f14a3a37640 1 -- 172.21.15.46:0/2915569685 wait complete. 2024-09-16T04:43:42.291 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:43:42.291 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:43:42.291 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:43:42.374 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:42.375 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38287]: starting. 2024-09-16T04:43:42.375 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38287]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-38252 2024-09-16T04:43:42.375 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38287]: created object 0... 2024-09-16T04:43:42.375 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38287]: created object 25... 2024-09-16T04:43:42.375 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38287]: created object 49... 2024-09-16T04:43:42.375 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38287]: finishing. 2024-09-16T04:43:42.375 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38287]: shutting down. 2024-09-16T04:43:42.387 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:42.387 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38289]: starting. 2024-09-16T04:43:42.387 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38289]: listing objects. 2024-09-16T04:43:42.387 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38289]: listed object 0... 2024-09-16T04:43:42.387 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38289]: listed object 25... 2024-09-16T04:43:42.387 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38289]: saw 50 objects 2024-09-16T04:43:42.388 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38289]: shutting down. 2024-09-16T04:43:42.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.411+0000 7f1734bcf640 10 monclient: get_monmap_and_config 2024-09-16T04:43:42.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.411+0000 7f1734bcf640 10 monclient: build_initial_monmap 2024-09-16T04:43:42.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.411+0000 7f1734bcf640 10 monclient: monmap: 2024-09-16T04:43:42.413 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:42.413 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:42.413 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:42.412388+0000 2024-09-16T04:43:42.413 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:42.412388+0000 2024-09-16T04:43:42.413 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:42.413 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:42.413 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:42.413 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:42.413 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:42.413 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:42.413 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:42.414 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:42.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.412+0000 7f1734bcf640 1 Processor -- start 2024-09-16T04:43:42.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.412+0000 7f1734bcf640 1 -- start start 2024-09-16T04:43:42.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.412+0000 7f1734bcf640 10 monclient: init 2024-09-16T04:43:42.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f1734bcf640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:42.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f1734bcf640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-16T04:43:42.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f1734bcf640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f173006e160 0x7f17300a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:42.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f1734bcf640 10 monclient(hunting): picked mon.noname-b con 0x7f173006e160 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:42.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f1734bcf640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f17300a2ee0 0x7f1730136940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:42.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f1734bcf640 10 monclient(hunting): picked mon.noname-c con 0x7f17300a2ee0 addr v2:172.21.15.114:3300/0 2024-09-16T04:43:42.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f1734bcf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1730136e80 addr v1:172.21.15.46:6789/0 2024-09-16T04:43:42.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f1734bcf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17300a3fd0 con 0x7f1730136e80 2024-09-16T04:43:42.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f1734bcf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:42.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f1734bcf640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f17300a3300 con 0x7f17300a2ee0 2024-09-16T04:43:42.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f1734bcf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:42.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f1734bcf640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f173006e540 con 0x7f173006e160 2024-09-16T04:43:42.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f1734bcf640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:42.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f1734bcf640 10 monclient(hunting): authenticate will time out at 1020.828353s 2024-09-16T04:43:42.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172e59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f173006e160 0x7f17300a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:42.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172e59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f173006e160 0x7f17300a29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:42782/0 (socket says 172.21.15.46:42782) 2024-09-16T04:43:42.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172ed9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1730136e80 0x7f173013dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37074/0 (socket says 172.21.15.46:37074) 2024-09-16T04:43:42.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172e59c640 1 -- 172.21.15.46:0/2263685139 learned_addr learned my addr 172.21.15.46:0/2263685139 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:42.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172e59c640 10 monclient(hunting): get_auth_request con 0x7f173006e160 auth_method 0 2024-09-16T04:43:42.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:42.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:42.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:42.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:42.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:42.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:42.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172dd9b640 1 --2- 172.21.15.46:0/2263685139 >> v2:172.21.15.114:3300/0 conn(0x7f17300a2ee0 0x7f1730136940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:42.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172dd9b640 10 monclient(hunting): get_auth_request con 0x7f17300a2ee0 auth_method 0 2024-09-16T04:43:42.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:42.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:42.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:42.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172e59c640 10 monclient(hunting): handle_auth_done global_id 64106 payload 274 2024-09-16T04:43:42.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172e59c640 1 -- 172.21.15.46:0/2263685139 >> v2:172.21.15.114:3300/0 conn(0x7f17300a2ee0 msgr2=0x7f1730136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:43:42.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172e59c640 1 --2- 172.21.15.46:0/2263685139 >> v2:172.21.15.114:3300/0 conn(0x7f17300a2ee0 0x7f1730136940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:42.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172e59c640 1 -- 172.21.15.46:0/2263685139 >> v1:172.21.15.46:6789/0 conn(0x7f1730136e80 legacy=0x7f173013dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:42.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172e59c640 10 monclient: _finish_hunting 0 2024-09-16T04:43:42.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172e59c640 1 monclient: found mon.noname-b 2024-09-16T04:43:42.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:42.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:43:42.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172e59c640 1 -- 172.21.15.46:0/2263685139 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17300a0bd0 con 0x7f173006e160 2024-09-16T04:43:42.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172d59a640 1 -- 172.21.15.46:0/2263685139 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1870867243 0 0) 0x7f1728001490 con 0x7f1730136e80 2024-09-16T04:43:42.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172d59a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:43:42.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.413+0000 7f172dd9b640 1 --2- 172.21.15.46:0/2263685139 >> v2:172.21.15.114:3300/0 conn(0x7f17300a2ee0 0x7f1730136940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:43:42.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.414+0000 7f172e59c640 1 --2- 172.21.15.46:0/2263685139 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f173006e160 0x7f17300a29a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1720009870 tx=0x7f172002a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59ea6102ed65f6b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:42.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.414+0000 7f172d59a640 1 -- 172.21.15.46:0/2263685139 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1720037070 con 0x7f173006e160 2024-09-16T04:43:42.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.414+0000 7f172d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:42.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.414+0000 7f172d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:43:42.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.414+0000 7f172d59a640 10 monclient: dump: 2024-09-16T04:43:42.427 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:42.427 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:42.427 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.427 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.428 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:42.428 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:42.428 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:42.428 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:42.428 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:42.428 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:42.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.414+0000 7f172d59a640 10 monclient: _finish_auth 0 2024-09-16T04:43:42.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.414+0000 7f172d59a640 10 monclient: _check_auth_tickets 2024-09-16T04:43:42.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.414+0000 7f172d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:42.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.414+0000 7f1734bcf640 5 monclient: authenticate success, global_id 64106 2024-09-16T04:43:42.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.414+0000 7f172d59a640 1 -- 172.21.15.46:0/2263685139 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1720032440 con 0x7f173006e160 2024-09-16T04:43:42.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.414+0000 7f1734bcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:42.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.414+0000 7f172d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:42.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.414+0000 7f1734bcf640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:42.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.414+0000 7f1734bcf640 4 set_mon_vals no callback set 2024-09-16T04:43:42.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.414+0000 7f172d59a640 1 -- 172.21.15.46:0/2263685139 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1720032810 con 0x7f173006e160 2024-09-16T04:43:42.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.414+0000 7f172d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:42.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.414+0000 7f172d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:43:42.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.414+0000 7f172d59a640 10 monclient: dump: 2024-09-16T04:43:42.429 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:42.429 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:42.429 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.429 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.429 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:42.429 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:42.429 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:42.430 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.415+0000 7f1734bcf640 10 monclient: shutdown 2024-09-16T04:43:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.415+0000 7f1734bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.415+0000 7f1734bcf640 1 -- 172.21.15.46:0/2263685139 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f173006e160 msgr2=0x7f17300a29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.415+0000 7f1734bcf640 1 --2- 172.21.15.46:0/2263685139 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f173006e160 0x7f17300a29a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1720009870 tx=0x7f172002a130 comp rx=0 tx=0).stop 2024-09-16T04:43:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.415+0000 7f1734bcf640 1 -- 172.21.15.46:0/2263685139 shutdown_connections 2024-09-16T04:43:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.415+0000 7f1734bcf640 1 --2- 172.21.15.46:0/2263685139 >> v2:172.21.15.114:3300/0 conn(0x7f17300a2ee0 0x7f1730136940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.415+0000 7f1734bcf640 1 --2- 172.21.15.46:0/2263685139 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f173006e160 0x7f17300a29a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.415+0000 7f1734bcf640 1 -- 172.21.15.46:0/2263685139 >> 172.21.15.46:0/2263685139 conn(0x7f17300a5c10 msgr2=0x7f17300a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.415+0000 7f1734bcf640 1 -- 172.21.15.46:0/2263685139 shutdown_connections 2024-09-16T04:43:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.415+0000 7f1734bcf640 1 -- 172.21.15.46:0/2263685139 wait complete. 2024-09-16T04:43:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.416+0000 7f1734bcf640 10 monclient: build_initial_monmap 2024-09-16T04:43:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.416+0000 7f1734bcf640 10 monclient: monmap: 2024-09-16T04:43:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:42.431 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.416+0000 7f1734bcf640 1 Processor -- start 2024-09-16T04:43:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.416+0000 7f1734bcf640 1 -- start start 2024-09-16T04:43:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.416+0000 7f1734bcf640 10 monclient: init 2024-09-16T04:43:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.416+0000 7f1734bcf640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.416+0000 7f1734bcf640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:43:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.416+0000 7f1734bcf640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f173006e160 0x7f1730136120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.416+0000 7f1734bcf640 10 monclient(hunting): picked mon.noname-c con 0x7f173006e160 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.416+0000 7f1734bcf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17300a2ee0 0x7f17301cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.416+0000 7f1734bcf640 10 monclient(hunting): picked mon.noname-a con 0x7f17300a2ee0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.416+0000 7f1734bcf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1730136e80 0x7f17301cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.416+0000 7f1734bcf640 10 monclient(hunting): picked mon.noname-b con 0x7f1730136e80 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.416+0000 7f1734bcf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.416+0000 7f1734bcf640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f17300a34a0 con 0x7f17300a2ee0 2024-09-16T04:43:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.416+0000 7f1734bcf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.416+0000 7f1734bcf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f17300a5130 con 0x7f1730136e80 2024-09-16T04:43:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.416+0000 7f1734bcf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.416+0000 7f1734bcf640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f1730120f40 con 0x7f173006e160 2024-09-16T04:43:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.416+0000 7f1734bcf640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.416+0000 7f1734bcf640 10 monclient(hunting): authenticate will time out at 1020.831971s 2024-09-16T04:43:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17300a2ee0 0x7f17301cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172e59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f173006e160 0x7f1730136120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17300a2ee0 0x7f17301cbb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53492/0 (socket says 172.21.15.46:53492) 2024-09-16T04:43:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172dd9b640 1 -- 172.21.15.46:0/1101586792 learned_addr learned my addr 172.21.15.46:0/1101586792 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172dd9b640 10 monclient(hunting): get_auth_request con 0x7f17300a2ee0 auth_method 0 2024-09-16T04:43:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172e59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f173006e160 0x7f1730136120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:42796/0 (socket says 172.21.15.46:42796) 2024-09-16T04:43:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172e59c640 10 monclient(hunting): get_auth_request con 0x7f173006e160 auth_method 0 2024-09-16T04:43:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172ed9d640 1 --2- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1730136e80 0x7f17301cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172ed9d640 10 monclient(hunting): get_auth_request con 0x7f1730136e80 auth_method 0 2024-09-16T04:43:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172e59c640 10 monclient(hunting): handle_auth_done global_id 64109 payload 824 2024-09-16T04:43:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172e59c640 1 -- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1730136e80 msgr2=0x7f17301cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:42.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172e59c640 1 --2- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1730136e80 0x7f17301cc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:42.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172e59c640 1 -- 172.21.15.46:0/1101586792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17300a2ee0 msgr2=0x7f17301cbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:42.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172e59c640 1 --2- 172.21.15.46:0/1101586792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17300a2ee0 0x7f17301cbb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:42.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172e59c640 10 monclient: _finish_hunting 0 2024-09-16T04:43:42.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172e59c640 1 monclient: found mon.noname-c 2024-09-16T04:43:42.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:42.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:43:42.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172e59c640 1 -- 172.21.15.46:0/1101586792 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17300a0300 con 0x7f173006e160 2024-09-16T04:43:42.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172dd9b640 1 --2- 172.21.15.46:0/1101586792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17300a2ee0 0x7f17301cbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:43:42.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172e59c640 1 --2- 172.21.15.46:0/1101586792 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f173006e160 0x7f1730136120 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1730068140 tx=0x7f17200046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:42.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f171effd640 1 -- 172.21.15.46:0/1101586792 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1720041070 con 0x7f173006e160 2024-09-16T04:43:42.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f171effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:42.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f171effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:43:42.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f171effd640 10 monclient: dump: 2024-09-16T04:43:42.437 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:42.437 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:42.437 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.437 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.437 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:42.437 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:42.437 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:42.437 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:42.437 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:42.437 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:42.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f171effd640 10 monclient: _finish_auth 0 2024-09-16T04:43:42.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f171effd640 10 monclient: _check_auth_tickets 2024-09-16T04:43:42.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f171effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:42.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f171effd640 1 -- 172.21.15.46:0/1101586792 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f172003c930 con 0x7f173006e160 2024-09-16T04:43:42.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f1734bcf640 5 monclient: authenticate success, global_id 64109 2024-09-16T04:43:42.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f1734bcf640 10 monclient: _renew_subs 2024-09-16T04:43:42.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f1734bcf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:42.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f1734bcf640 1 -- 172.21.15.46:0/1101586792 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17301235d0 con 0x7f173006e160 2024-09-16T04:43:42.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f171effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:42.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f171effd640 1 -- 172.21.15.46:0/1101586792 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1720037040 con 0x7f173006e160 2024-09-16T04:43:42.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f171effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:42.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f171effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:43:42.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f171effd640 10 monclient: dump: 2024-09-16T04:43:42.438 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:42.438 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:42.438 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.439 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.439 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:42.439 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:42.439 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:42.439 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:42.439 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:42.439 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:42.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f172cd99640 4 set_mon_vals no callback set 2024-09-16T04:43:42.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f1734bcf640 10 monclient: _renew_subs 2024-09-16T04:43:42.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f1734bcf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:42.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.417+0000 7f1734bcf640 1 -- 172.21.15.46:0/1101586792 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f17301222f0 con 0x7f173006e160 2024-09-16T04:43:42.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.418+0000 7f1734bcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:42.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.418+0000 7f1734bcf640 1 --2- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17301c90c0 0x7f17301cc600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:42.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.418+0000 7f1734bcf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:42.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.418+0000 7f171effd640 1 -- 172.21.15.46:0/1101586792 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f1720044070 con 0x7f173006e160 2024-09-16T04:43:42.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.418+0000 7f1734bcf640 1 -- 172.21.15.46:0/1101586792 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1730123250 con 0x7f17301c90c0 2024-09-16T04:43:42.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.418+0000 7f172dd9b640 1 --2- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17301c90c0 0x7f17301cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:42.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.418+0000 7f171effd640 1 --2- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f17100383e0 0x7f171003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:42.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.418+0000 7f172dd9b640 10 monclient: get_auth_request con 0x7f17301c90c0 auth_method 0 2024-09-16T04:43:42.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.418+0000 7f172dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:42.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.418+0000 7f172dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:42.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.418+0000 7f172dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:42.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.418+0000 7f171effd640 1 -- 172.21.15.46:0/1101586792 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(193..193 src has 1..193) ==== 4738+0+0 (secure 0 0 0) 0x7f172006c150 con 0x7f173006e160 2024-09-16T04:43:42.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.418+0000 7f172dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:42.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.418+0000 7f172dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:42.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.418+0000 7f172dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:42.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.419+0000 7f172ed9d640 1 --2- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f17100383e0 0x7f171003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:42.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.419+0000 7f172ed9d640 10 monclient: get_auth_request con 0x7f17100383e0 auth_method 0 2024-09-16T04:43:42.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.419+0000 7f172dd9b640 10 monclient(hunting): handle_auth_done global_id 5935 payload 824 2024-09-16T04:43:42.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.419+0000 7f172dd9b640 1 -- 172.21.15.46:0/1101586792 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f17300a29a0 con 0x7f17301c90c0 2024-09-16T04:43:42.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.419+0000 7f172dd9b640 1 --2- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17301c90c0 0x7f17301cc600 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f1724006a90 tx=0x7f1724011f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:42.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.419+0000 7f171effd640 1 -- 172.21.15.46:0/1101586792 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1724019070 con 0x7f17301c90c0 2024-09-16T04:43:42.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.419+0000 7f171effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:42.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.419+0000 7f171effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:43:42.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.419+0000 7f171effd640 10 monclient: dump: 2024-09-16T04:43:42.441 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:42.441 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:42.441 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.441 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.441 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:42.442 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:42.442 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:42.442 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:42.442 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:42.442 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:42.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.419+0000 7f172ed9d640 1 --2- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f17100383e0 0x7f171003a8a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f17280012e0 tx=0x7f1728007e50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:42.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.419+0000 7f171effd640 1 -- 172.21.15.46:0/1101586792 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f172400fe10 con 0x7f17301c90c0 2024-09-16T04:43:42.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.419+0000 7f171effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:42.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.419+0000 7f171effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:42.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.419+0000 7f171effd640 1 -- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17301c90c0 msgr2=0x7f17301cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:42.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.419+0000 7f171effd640 1 --2- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17301c90c0 0x7f17301cc600 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f1724006a90 tx=0x7f1724011f10 comp rx=0 tx=0).stop 2024-09-16T04:43:42.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.419+0000 7f171effd640 1 -- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17301c90c0 msgr2=0x7f17301cc600 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:43:42.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.419+0000 7f171effd640 1 --2- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17301c90c0 0x7f17301cc600 secure :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f1724006a90 tx=0x7f1724011f10 comp rx=0 tx=0).stop 2024-09-16T04:43:42.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.425+0000 7f1734bcf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:43:42.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.425+0000 7f1734bcf640 1 --2- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1730136660 0x7f173009b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:42.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.425+0000 7f1734bcf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:42.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.425+0000 7f1734bcf640 1 -- 172.21.15.46:0/1101586792 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f17300a3620 con 0x7f1730136660 2024-09-16T04:43:42.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.426+0000 7f172dd9b640 1 --2- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1730136660 0x7f173009b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:42.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.426+0000 7f172dd9b640 10 monclient: get_auth_request con 0x7f1730136660 auth_method 0 2024-09-16T04:43:42.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.426+0000 7f172dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:42.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.426+0000 7f172dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:42.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.426+0000 7f172dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:42.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.426+0000 7f172dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:42.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.426+0000 7f172dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:42.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.426+0000 7f172dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:42.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.426+0000 7f172dd9b640 10 monclient(hunting): handle_auth_done global_id 5938 payload 824 2024-09-16T04:43:42.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.426+0000 7f172dd9b640 1 -- 172.21.15.46:0/1101586792 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1730067e00 con 0x7f1730136660 2024-09-16T04:43:42.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.426+0000 7f172dd9b640 1 --2- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1730136660 0x7f173009b590 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1724006a90 tx=0x7f17240046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:42.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.426+0000 7f171effd640 1 -- 172.21.15.46:0/1101586792 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172402b020 con 0x7f1730136660 2024-09-16T04:43:42.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.426+0000 7f171effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:42.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.426+0000 7f171effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:43:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.426+0000 7f171effd640 10 monclient: dump: 2024-09-16T04:43:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:42.446 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.427+0000 7f171effd640 1 -- 172.21.15.46:0/1101586792 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f1724015660 con 0x7f1730136660 2024-09-16T04:43:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.427+0000 7f171effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:43:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.427+0000 7f171effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:42.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.427+0000 7f171effd640 1 -- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1730136660 msgr2=0x7f173009b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:42.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.427+0000 7f171effd640 1 --2- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1730136660 0x7f173009b590 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1724006a90 tx=0x7f17240046c0 comp rx=0 tx=0).stop 2024-09-16T04:43:42.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.427+0000 7f171effd640 1 -- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1730136660 msgr2=0x7f173009b590 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:43:42.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.427+0000 7f171effd640 1 --2- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1730136660 0x7f173009b590 secure :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1724006a90 tx=0x7f17240046c0 comp rx=0 tx=0).stop 2024-09-16T04:43:42.447 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:43:42.447 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:43:42.447 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:43:42.447 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:43:42.447 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 58, 2024-09-16T04:43:42.448 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 308328, 2024-09-16T04:43:42.448 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:43:42.448 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:43:42.448 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:43:42.448 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:43:42.448 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:42.448 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 3, 2024-09-16T04:43:42.448 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:43:42.448 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:43:42.448 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:43:42.448 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:42.448 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:42.449 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:42.449 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:42.449 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:42.449 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:42.449 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:42.449 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:42.449 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:42.449 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:42.449 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:42.449 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:42.449 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:43:42.449 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:43:42.449 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:42.450 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:42.450 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:42.450 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:42.450 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:42.450 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:42.450 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:42.450 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:42.450 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:42.450 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:42.450 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:42.450 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:42.450 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.450 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:43:42.451 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:43:42.451 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:43:42.451 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:43:42.451 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:43:42.451 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:43:42.451 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:43:42.451 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:43:42.451 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:43:42.451 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:43:42.451 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:43:42.451 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:43:42.451 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:43:42.451 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:43:42.451 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:43:42.452 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:43:42.452 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:43:42.452 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:42.452 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:42.452 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:42.452 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:42.452 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:42.452 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:42.452 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:42.452 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:42.452 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:42.452 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:42.452 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:42.453 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:42.453 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:43:42.453 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.453 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:43:42.453 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.453 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:43:42.453 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:43:42.453 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:42.453 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:42.453 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.453 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:42.453 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:43:42.453 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:42.453 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.454 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.454 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:42.454 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:43:42.454 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:42.454 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:42.454 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:42.454 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.454 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:43:42.454 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:43:42.454 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:42.454 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:42.454 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:42.454 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.454 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.455 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:43:42.455 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:43:42.455 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:42.455 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:42.455 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.455 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:42.455 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:43:42.455 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:42.455 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.455 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.455 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:42.456 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:43:42.456 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:42.456 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:42.456 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:42.456 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.456 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:43:42.456 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:43:42.456 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:42.456 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:42.456 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:42.456 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.456 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.456 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:43:42.457 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:43:42.457 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:42.457 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:42.457 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.457 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:42.457 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:43:42.457 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:42.457 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.457 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.457 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:42.457 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:43:42.457 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:42.457 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:42.457 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:42.458 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.458 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:43:42.458 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:43:42.458 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:42.458 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:42.458 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:42.458 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:42.458 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:42.458 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.458 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:43:42.458 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:43:42.458 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.459 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:42.459 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:42.459 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:43:42.459 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:42.459 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:42.459 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:43:42.459 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.459 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:42.459 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:42.459 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4 2024-09-16T04:43:42.459 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:42.459 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:42.459 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:43:42.459 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.460 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:42.460 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:42.460 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6 2024-09-16T04:43:42.460 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:42.460 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:42.460 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:43:42.460 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:42.460 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:42.460 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:42.460 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:43:42.460 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:42.460 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:42.460 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:42.460 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:43:42.461 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:43:42.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.427+0000 7f1734bcf640 1 -- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f17100383e0 msgr2=0x7f171003a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:42.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.427+0000 7f1734bcf640 1 --2- 172.21.15.46:0/1101586792 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f17100383e0 0x7f171003a8a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f17280012e0 tx=0x7f1728007e50 comp rx=0 tx=0).stop 2024-09-16T04:43:42.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.427+0000 7f1734bcf640 10 monclient: shutdown 2024-09-16T04:43:42.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.427+0000 7f1734bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:42.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.427+0000 7f1734bcf640 1 -- 172.21.15.46:0/1101586792 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f173006e160 msgr2=0x7f1730136120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:42.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.427+0000 7f1734bcf640 1 --2- 172.21.15.46:0/1101586792 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f173006e160 0x7f1730136120 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1730068140 tx=0x7f17200046c0 comp rx=0 tx=0).stop 2024-09-16T04:43:42.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.427+0000 7f172ed9d640 1 -- 172.21.15.46:0/1101586792 reap_dead start 2024-09-16T04:43:42.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.427+0000 7f172ed9d640 1 -- 172.21.15.46:0/1101586792 reap_dead start 2024-09-16T04:43:42.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.427+0000 7f1734bcf640 1 -- 172.21.15.46:0/1101586792 shutdown_connections 2024-09-16T04:43:42.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.427+0000 7f1734bcf640 1 -- 172.21.15.46:0/1101586792 >> 172.21.15.46:0/1101586792 conn(0x7f17300a5c10 msgr2=0x7f173009ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:42.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.427+0000 7f1734bcf640 1 -- 172.21.15.46:0/1101586792 shutdown_connections 2024-09-16T04:43:42.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.427+0000 7f1734bcf640 1 -- 172.21.15.46:0/1101586792 wait complete. 2024-09-16T04:43:42.464 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:43:42.464 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:43:42.464 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:43:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.563+0000 7fa261dd8640 10 monclient: get_monmap_and_config 2024-09-16T04:43:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.563+0000 7fa261dd8640 10 monclient: build_initial_monmap 2024-09-16T04:43:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.563+0000 7fa261dd8640 10 monclient: monmap: 2024-09-16T04:43:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:42.564378+0000 2024-09-16T04:43:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:42.564378+0000 2024-09-16T04:43:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:42.566 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:42.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.564+0000 7fa261dd8640 1 Processor -- start 2024-09-16T04:43:42.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.564+0000 7fa261dd8640 1 -- start start 2024-09-16T04:43:42.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.564+0000 7fa261dd8640 10 monclient: init 2024-09-16T04:43:42.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa261dd8640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:42.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa261dd8640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-16T04:43:42.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa261dd8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa25c0a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:43:42.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa261dd8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa25c0a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:43:42.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa261dd8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa25c13cea0 0x7fa25c13f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:42.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa261dd8640 10 monclient(hunting): picked mon.noname-a con 0x7fa25c13cea0 addr v2:172.21.15.46:3300/0 2024-09-16T04:43:42.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa261dd8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa25c09b940 con 0x7fa25c0a3c10 2024-09-16T04:43:42.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa261dd8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa25c09ab00 con 0x7fa25c0a3ff0 2024-09-16T04:43:42.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa261dd8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:42.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa261dd8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa25c09b030 con 0x7fa25c13cea0 2024-09-16T04:43:42.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa261dd8640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:42.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa261dd8640 10 monclient(hunting): authenticate will time out at 1020.980190s 2024-09-16T04:43:42.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa25b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa25c0a3c10 0x7fa25c09bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37090/0 (socket says 172.21.15.46:37090) 2024-09-16T04:43:42.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa25b7fe640 1 -- 172.21.15.46:0/2508882717 learned_addr learned my addr 172.21.15.46:0/2508882717 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:43:42.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa25bfff640 1 --2- 172.21.15.46:0/2508882717 >> v2:172.21.15.46:3300/0 conn(0x7fa25c13cea0 0x7fa25c13f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:42.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa25bfff640 10 monclient(hunting): get_auth_request con 0x7fa25c13cea0 auth_method 0 2024-09-16T04:43:42.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa25bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:42.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa25bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:42.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa25bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:42.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa25bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:42.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa25bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:42.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa25bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:42.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa25bfff640 10 monclient(hunting): handle_auth_done global_id 44667 payload 274 2024-09-16T04:43:42.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa25bfff640 1 -- 172.21.15.46:0/2508882717 >> v1:172.21.15.114:6789/0 conn(0x7fa25c0a3ff0 legacy=0x7fa25c0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:42.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa25bfff640 1 -- 172.21.15.46:0/2508882717 >> v1:172.21.15.46:6789/0 conn(0x7fa25c0a3c10 legacy=0x7fa25c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:42.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa25bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:43:42.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa25bfff640 1 monclient: found mon.noname-a 2024-09-16T04:43:42.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa25bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:42.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa25bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:43:42.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.565+0000 7fa25bfff640 1 -- 172.21.15.46:0/2508882717 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25c09b310 con 0x7fa25c13cea0 2024-09-16T04:43:42.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa25bfff640 1 --2- 172.21.15.46:0/2508882717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa25c13cea0 0x7fa25c13f290 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa254009d90 tx=0x7fa25402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f559086f7b18fe0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:42.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa25a7fc640 1 -- 172.21.15.46:0/2508882717 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa254007740 con 0x7fa25c13cea0 2024-09-16T04:43:42.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa25a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:42.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa25a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:43:42.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa25a7fc640 10 monclient: dump: 2024-09-16T04:43:42.576 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:42.576 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:42.576 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.576 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.576 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:42.576 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:42.576 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:42.576 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:42.576 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:42.576 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:42.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa25a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:43:42.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa25a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:43:42.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa25a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:42.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa261dd8640 5 monclient: authenticate success, global_id 44667 2024-09-16T04:43:42.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa25a7fc640 1 -- 172.21.15.46:0/2508882717 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2540078e0 con 0x7fa25c13cea0 2024-09-16T04:43:42.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa261dd8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:42.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa25a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:42.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa261dd8640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:42.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa25a7fc640 1 -- 172.21.15.46:0/2508882717 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa254007cb0 con 0x7fa25c13cea0 2024-09-16T04:43:42.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa261dd8640 4 set_mon_vals no callback set 2024-09-16T04:43:42.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa25a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:42.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa25a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:42.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa25a7fc640 10 monclient: dump: 2024-09-16T04:43:42.577 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:42.577 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:42.577 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.577 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.578 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:42.578 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:42.578 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:42.578 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:42.578 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:42.578 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:42.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa261dd8640 10 monclient: shutdown 2024-09-16T04:43:42.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa261dd8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:42.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa261dd8640 1 -- 172.21.15.46:0/2508882717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa25c13cea0 msgr2=0x7fa25c13f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:42.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa261dd8640 1 --2- 172.21.15.46:0/2508882717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa25c13cea0 0x7fa25c13f290 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa254009d90 tx=0x7fa25402a660 comp rx=0 tx=0).stop 2024-09-16T04:43:42.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa261dd8640 1 -- 172.21.15.46:0/2508882717 shutdown_connections 2024-09-16T04:43:42.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa261dd8640 1 --2- 172.21.15.46:0/2508882717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa25c13cea0 0x7fa25c13f290 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:42.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.566+0000 7fa261dd8640 1 -- 172.21.15.46:0/2508882717 >> 172.21.15.46:0/2508882717 conn(0x7fa25c0a6340 msgr2=0x7fa25c06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:42.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.567+0000 7fa261dd8640 1 -- 172.21.15.46:0/2508882717 shutdown_connections 2024-09-16T04:43:42.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.567+0000 7fa261dd8640 1 -- 172.21.15.46:0/2508882717 wait complete. 2024-09-16T04:43:42.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.567+0000 7fa261dd8640 10 monclient: build_initial_monmap 2024-09-16T04:43:42.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.567+0000 7fa261dd8640 10 monclient: monmap: 2024-09-16T04:43:42.580 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:42.580 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:42.580 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:42.580 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:42.580 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:42.580 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:42.580 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:42.580 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:42.580 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:42.580 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:42.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.567+0000 7fa261dd8640 1 Processor -- start 2024-09-16T04:43:42.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa261dd8640 1 -- start start 2024-09-16T04:43:42.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa261dd8640 10 monclient: init 2024-09-16T04:43:42.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa261dd8640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:42.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa261dd8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:43:42.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa261dd8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa25c0a3c10 0x7fa25c1cbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:42.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa261dd8640 10 monclient(hunting): picked mon.noname-b con 0x7fa25c0a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:42.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa261dd8640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa25c0a3ff0 0x7fa25c136e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:42.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa261dd8640 10 monclient(hunting): picked mon.noname-c con 0x7fa25c0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:42.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa261dd8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa25c13cea0 0x7fa25c139e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:42.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa261dd8640 10 monclient(hunting): picked mon.noname-a con 0x7fa25c13cea0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:42.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa261dd8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:42.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa261dd8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa25c13f950 con 0x7fa25c13cea0 2024-09-16T04:43:42.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa261dd8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:42.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa261dd8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa25c13f7d0 con 0x7fa25c0a3c10 2024-09-16T04:43:42.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa261dd8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:42.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa261dd8640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fa25c13fad0 con 0x7fa25c0a3ff0 2024-09-16T04:43:42.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa261dd8640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:42.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa261dd8640 10 monclient(hunting): authenticate will time out at 1020.983638s 2024-09-16T04:43:42.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25affd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa25c0a3ff0 0x7fa25c136e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:42.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa25c13cea0 0x7fa25c139e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:42.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa25c13cea0 0x7fa25c139e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53510/0 (socket says 172.21.15.46:53510) 2024-09-16T04:43:42.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25bfff640 1 -- 172.21.15.46:0/3615273251 learned_addr learned my addr 172.21.15.46:0/3615273251 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:42.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25affd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa25c0a3ff0 0x7fa25c136e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:42806/0 (socket says 172.21.15.46:42806) 2024-09-16T04:43:42.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25bfff640 10 monclient(hunting): get_auth_request con 0x7fa25c13cea0 auth_method 0 2024-09-16T04:43:42.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:42.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:42.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:42.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25affd640 10 monclient(hunting): get_auth_request con 0x7fa25c0a3ff0 auth_method 0 2024-09-16T04:43:42.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25b7fe640 1 --2- 172.21.15.46:0/3615273251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa25c0a3c10 0x7fa25c1cbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:42.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:42.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:42.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:42.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25b7fe640 10 monclient(hunting): get_auth_request con 0x7fa25c0a3c10 auth_method 0 2024-09-16T04:43:42.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:42.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:42.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:42.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:42.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:42.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:42.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:42.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:42.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:42.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:42.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:42.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.568+0000 7fa25affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:42.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa25bfff640 10 monclient(hunting): handle_auth_done global_id 44673 payload 824 2024-09-16T04:43:42.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa25bfff640 1 -- 172.21.15.46:0/3615273251 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa25c0a3ff0 msgr2=0x7fa25c136e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:42.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa25bfff640 1 --2- 172.21.15.46:0/3615273251 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa25c0a3ff0 0x7fa25c136e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:42.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa25bfff640 1 -- 172.21.15.46:0/3615273251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa25c0a3c10 msgr2=0x7fa25c1cbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:42.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa25bfff640 1 --2- 172.21.15.46:0/3615273251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa25c0a3c10 0x7fa25c1cbbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:42.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa25bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:43:42.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa25bfff640 1 monclient: found mon.noname-a 2024-09-16T04:43:42.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa25bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:42.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa25bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:43:42.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa25bfff640 1 -- 172.21.15.46:0/3615273251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25c13c670 con 0x7fa25c13cea0 2024-09-16T04:43:42.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa25b7fe640 1 --2- 172.21.15.46:0/3615273251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa25c0a3c10 0x7fa25c1cbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:43:42.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa25bfff640 1 --2- 172.21.15.46:0/3615273251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa25c13cea0 0x7fa25c139e70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa254009eb0 tx=0x7fa254004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:42.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa243fff640 1 -- 172.21.15.46:0/3615273251 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa254040070 con 0x7fa25c13cea0 2024-09-16T04:43:42.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa243fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:42.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa243fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:43:42.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa243fff640 10 monclient: dump: 2024-09-16T04:43:42.585 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:42.585 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:42.585 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.585 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.585 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:42.585 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:42.586 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:42.586 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:42.586 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:42.586 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:42.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa243fff640 10 monclient: _finish_auth 0 2024-09-16T04:43:42.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa243fff640 10 monclient: _check_auth_tickets 2024-09-16T04:43:42.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa243fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:42.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa261dd8640 5 monclient: authenticate success, global_id 44673 2024-09-16T04:43:42.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa243fff640 1 -- 172.21.15.46:0/3615273251 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa254031bc0 con 0x7fa25c13cea0 2024-09-16T04:43:42.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa261dd8640 10 monclient: _renew_subs 2024-09-16T04:43:42.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa261dd8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:42.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa261dd8640 1 -- 172.21.15.46:0/3615273251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa25c120510 con 0x7fa25c13cea0 2024-09-16T04:43:42.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa243fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:42.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa243fff640 1 -- 172.21.15.46:0/3615273251 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa254045460 con 0x7fa25c13cea0 2024-09-16T04:43:42.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa243fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:42.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa243fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:42.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa243fff640 10 monclient: dump: 2024-09-16T04:43:42.587 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:42.587 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:42.587 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.587 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:42.587 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:42.587 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:42.587 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:42.587 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:42.587 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:42.587 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:42.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa261dd8640 10 monclient: _renew_subs 2024-09-16T04:43:42.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa261dd8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:42.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa261dd8640 1 -- 172.21.15.46:0/3615273251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa25c13c3f0 con 0x7fa25c13cea0 2024-09-16T04:43:42.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.569+0000 7fa259ffb640 4 set_mon_vals no callback set 2024-09-16T04:43:42.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.570+0000 7fa243fff640 1 -- 172.21.15.46:0/3615273251 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fa254039040 con 0x7fa25c13cea0 2024-09-16T04:43:42.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.570+0000 7fa261dd8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:42.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.570+0000 7fa261dd8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:42.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.570+0000 7fa261dd8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:42.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.570+0000 7fa261dd8640 1 -- 172.21.15.46:0/3615273251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa25c0a0400 con 0x7fa25c13cea0 2024-09-16T04:43:42.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.570+0000 7fa243fff640 1 --2- 172.21.15.46:0/3615273251 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa2480383b0 0x7fa24803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:42.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.570+0000 7fa243fff640 1 -- 172.21.15.46:0/3615273251 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(193..193 src has 1..193) ==== 4738+0+0 (secure 0 0 0) 0x7fa25406d1b0 con 0x7fa25c13cea0 2024-09-16T04:43:42.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.570+0000 7fa25b7fe640 1 --2- 172.21.15.46:0/3615273251 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa2480383b0 0x7fa24803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:42.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.571+0000 7fa25b7fe640 10 monclient: get_auth_request con 0x7fa2480383b0 auth_method 0 2024-09-16T04:43:42.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.571+0000 7fa25b7fe640 1 --2- 172.21.15.46:0/3615273251 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa2480383b0 0x7fa24803a870 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa24c005880 tx=0x7fa24c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:42.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.575+0000 7fa243fff640 1 -- 172.21.15.46:0/3615273251 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa25c0a0400 con 0x7fa25c13cea0 2024-09-16T04:43:42.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.575+0000 7fa243fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:42.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.575+0000 7fa243fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:42.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.819+0000 7fa261dd8640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:43:42.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.819+0000 7fa261dd8640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:43:42.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.819+0000 7fa261dd8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:42.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.819+0000 7fa261dd8640 1 -- 172.21.15.46:0/3615273251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fa25c09bcf0 con 0x7fa25c13cea0 2024-09-16T04:43:42.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.820+0000 7fa243fff640 1 -- 172.21.15.46:0/3615273251 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fa25403be00 con 0x7fa25c13cea0 2024-09-16T04:43:42.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.820+0000 7fa243fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:43:42.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.820+0000 7fa243fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:42.824 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.820+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.824 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.820+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=1,health=11,kv=12,logm=64} crc {auth=1375660361,config=555582061,health=3600961046,kv=3692544301,logm=4020088561}) 2024-09-16T04:43:42.824 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.820+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=12,kv=12,logm=64} crc {auth=1519185458,config=3541143443,health=451269054,kv=4017288519,logm=1405135302}) 2024-09-16T04:43:42.824 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.820+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.825 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.820+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=1,health=11,kv=12,logm=64} crc {auth=1375660361,config=555582061,health=3600961046,kv=3692544301,logm=4020088561}) 2024-09-16T04:43:42.825 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.820+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=11,kv=12,logm=63} crc {auth=2199599635,config=3541143443,health=373070800,kv=4017288519,logm=3355026235}) 2024-09-16T04:43:42.825 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.821+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.825 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.821+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1187099194}) 2024-09-16T04:43:42.825 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.821+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2647182524}) 2024-09-16T04:43:42.825 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.822+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.825 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.822+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1187099194}) 2024-09-16T04:43:42.825 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.822+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3364215996}) 2024-09-16T04:43:42.825 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.823+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.825 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.823+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=703726621}) 2024-09-16T04:43:42.825 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.823+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1978701564}) 2024-09-16T04:43:42.825 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.823+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.826 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.823+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=703726621}) 2024-09-16T04:43:42.826 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.823+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1269515155}) 2024-09-16T04:43:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.823+0000 7fa261dd8640 1 -- 172.21.15.46:0/3615273251 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa2480383b0 msgr2=0x7fa24803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.823+0000 7fa261dd8640 1 --2- 172.21.15.46:0/3615273251 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa2480383b0 0x7fa24803a870 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa24c005880 tx=0x7fa24c009040 comp rx=0 tx=0).stop 2024-09-16T04:43:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.823+0000 7fa261dd8640 10 monclient: shutdown 2024-09-16T04:43:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.823+0000 7fa261dd8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.823+0000 7fa261dd8640 1 -- 172.21.15.46:0/3615273251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa25c13cea0 msgr2=0x7fa25c139e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.823+0000 7fa261dd8640 1 --2- 172.21.15.46:0/3615273251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa25c13cea0 0x7fa25c139e70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa254009eb0 tx=0x7fa254004a40 comp rx=0 tx=0).stop 2024-09-16T04:43:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.823+0000 7fa261dd8640 1 -- 172.21.15.46:0/3615273251 shutdown_connections 2024-09-16T04:43:42.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.823+0000 7fa261dd8640 1 --2- 172.21.15.46:0/3615273251 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa2480383b0 0x7fa24803a870 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:42.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.823+0000 7fa261dd8640 1 --2- 172.21.15.46:0/3615273251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa25c13cea0 0x7fa25c139e70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:42.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.823+0000 7fa261dd8640 1 --2- 172.21.15.46:0/3615273251 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa25c0a3ff0 0x7fa25c136e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:42.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.823+0000 7fa261dd8640 1 --2- 172.21.15.46:0/3615273251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa25c0a3c10 0x7fa25c1cbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:42.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.823+0000 7fa261dd8640 1 -- 172.21.15.46:0/3615273251 >> 172.21.15.46:0/3615273251 conn(0x7fa25c0a6340 msgr2=0x7fa25c09f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:42.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.823+0000 7fa261dd8640 1 -- 172.21.15.46:0/3615273251 shutdown_connections 2024-09-16T04:43:42.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:42.823+0000 7fa261dd8640 1 -- 172.21.15.46:0/3615273251 wait complete. 2024-09-16T04:43:42.834 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.824+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.834 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.824+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2224527979}) 2024-09-16T04:43:42.834 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.824+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4212379010}) 2024-09-16T04:43:42.834 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.824+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.834 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.824+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2224527979}) 2024-09-16T04:43:42.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.824+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3076231377}) 2024-09-16T04:43:42.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.825+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.825+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=930166667}) 2024-09-16T04:43:42.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.825+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3663822781}) 2024-09-16T04:43:42.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.825+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.825+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=930166667}) 2024-09-16T04:43:42.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.825+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794142684}) 2024-09-16T04:43:42.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.825+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.826+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1622825134}) 2024-09-16T04:43:42.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.826+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1761537450}) 2024-09-16T04:43:42.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.826+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.826+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1622825134}) 2024-09-16T04:43:42.836 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.826+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=312520087}) 2024-09-16T04:43:42.836 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.826+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.836 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.827+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1487360541}) 2024-09-16T04:43:42.836 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.827+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=790477652}) 2024-09-16T04:43:42.836 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.827+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.836 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.827+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1487360541}) 2024-09-16T04:43:42.836 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.827+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3701470689}) 2024-09-16T04:43:42.836 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.828+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.836 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.828+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2772935336}) 2024-09-16T04:43:42.836 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.828+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1928440142}) 2024-09-16T04:43:42.836 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.828+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.837 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.828+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2772935336}) 2024-09-16T04:43:42.837 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.828+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3631209587}) 2024-09-16T04:43:42.837 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.829+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.837 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.829+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197550941}) 2024-09-16T04:43:42.837 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.829+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3611510349}) 2024-09-16T04:43:42.837 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.829+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.837 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.829+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197550941}) 2024-09-16T04:43:42.837 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.829+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2118996073}) 2024-09-16T04:43:42.837 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.829+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.837 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.829+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2548448896}) 2024-09-16T04:43:42.837 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.830+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2888294097}) 2024-09-16T04:43:42.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.830+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.830+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2548448896}) 2024-09-16T04:43:42.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.830+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2649263686}) 2024-09-16T04:43:42.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.831+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.831+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=477915687}) 2024-09-16T04:43:42.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.831+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1468271304}) 2024-09-16T04:43:42.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.831+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.831+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=477915687}) 2024-09-16T04:43:42.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.831+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=181266151}) 2024-09-16T04:43:42.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.832+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.832+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2539828995}) 2024-09-16T04:43:42.839 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.832+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1855597587}) 2024-09-16T04:43:42.839 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.832+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.839 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.832+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2539828995}) 2024-09-16T04:43:42.839 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.832+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1110471604}) 2024-09-16T04:43:42.839 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.833+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.839 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.833+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3820994387}) 2024-09-16T04:43:42.839 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.833+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2677919391}) 2024-09-16T04:43:42.839 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.833+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.839 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.833+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3820994387}) 2024-09-16T04:43:42.839 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.833+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=408561071}) 2024-09-16T04:43:42.842 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.834+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.842 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.834+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=340612538,mdsmap=3956149183}) 2024-09-16T04:43:42.842 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.834+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1593282584}) 2024-09-16T04:43:42.843 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.834+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.843 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.834+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=340612538,mdsmap=3956149183}) 2024-09-16T04:43:42.843 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.834+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=3540182325,mdsmap=3956149183,mgr=542623631}) 2024-09-16T04:43:42.843 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.835+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.843 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.835+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=12,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=57} crc {mgr=1081907435,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=3384302737,mon_config_key=2008167605,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=1061002559,osd_snap=723727393,osdmap=262175387}) 2024-09-16T04:43:42.843 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.835+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=13,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=59} crc {mgr=1652149447,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=3743212093,mon_config_key=2723417724,monmap=3088247691,osd_metadata=3229151404,osd_pg_creating=1061002559,osd_snap=723727393,osdmap=834711091}) 2024-09-16T04:43:42.843 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.835+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.843 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.835+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=12,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=57} crc {mgr=1081907435,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=3384302737,mon_config_key=2008167605,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=1061002559,osd_snap=723727393,osdmap=262175387}) 2024-09-16T04:43:42.843 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.835+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=58} crc {mgr=1130647210,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=2216919489,mon_config_key=1169725086,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=1061002559,osd_snap=723727393,osdmap=235825295}) 2024-09-16T04:43:42.843 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.836+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.843 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.836+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1188799384}) 2024-09-16T04:43:42.844 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.836+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4197763295}) 2024-09-16T04:43:42.844 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.836+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.844 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.836+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1188799384}) 2024-09-16T04:43:42.844 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.836+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3403101784}) 2024-09-16T04:43:42.844 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.837+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.844 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.837+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=179045034}) 2024-09-16T04:43:42.844 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.837+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3140048353}) 2024-09-16T04:43:42.844 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.837+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.844 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.837+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=179045034}) 2024-09-16T04:43:42.844 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.837+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2577504312}) 2024-09-16T04:43:42.845 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:43:42.846 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.841+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.846 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.841+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=40} crc {osdmap=2438180551}) 2024-09-16T04:43:42.846 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.841+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=36} crc {osdmap=964100046}) 2024-09-16T04:43:42.846 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.841+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:43:42.846 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.841+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=40} crc {osdmap=2438180551}) 2024-09-16T04:43:42.846 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:43:42.841+0000 7fd9dbfaa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=38} crc {osdmap=3077427009}) 2024-09-16T04:43:43.293 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:43.293 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38288]: starting. 2024-09-16T04:43:43.293 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38288]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-38252 2024-09-16T04:43:43.293 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38288]: shutting down. 2024-09-16T04:43:43.294 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:43.295 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:43:43.295 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:43:43.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:43:43.846 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:43:43.847 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:43:43.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.959+0000 7f86a518d640 10 monclient: get_monmap_and_config 2024-09-16T04:43:43.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.959+0000 7f86a518d640 10 monclient: build_initial_monmap 2024-09-16T04:43:43.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.959+0000 7f86a518d640 10 monclient: monmap: 2024-09-16T04:43:43.961 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:43.961 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:43.961 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:43.959929+0000 2024-09-16T04:43:43.961 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:43.959929+0000 2024-09-16T04:43:43.961 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:43.961 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:43.961 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:43.961 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:43.961 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:43.961 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:43.961 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:43.961 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f86a518d640 1 Processor -- start 2024-09-16T04:43:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f86a518d640 1 -- start start 2024-09-16T04:43:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f86a518d640 10 monclient: init 2024-09-16T04:43:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f86a518d640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f86a518d640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-16T04:43:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f86a518d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f86a0136e50 0x7f86a0137230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f86a518d640 10 monclient(hunting): picked mon.noname-b con 0x7f86a0136e50 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f86a518d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f86a00a3500 0x7f86a00a3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f86a518d640 10 monclient(hunting): picked mon.noname-c con 0x7f86a00a3500 addr v2:172.21.15.114:3300/0 2024-09-16T04:43:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f86a518d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f86a00a4180 addr v1:172.21.15.114:6789/0 2024-09-16T04:43:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f86a518d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86a00a4e90 con 0x7f86a00a4180 2024-09-16T04:43:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f86a518d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:43.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f86a518d640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f86a00a5410 con 0x7f86a00a3500 2024-09-16T04:43:43.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f86a518d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:43.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f86a518d640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f86a00a0dc0 con 0x7f86a0136e50 2024-09-16T04:43:43.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f86a518d640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:43.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f86a518d640 10 monclient(hunting): authenticate will time out at 1022.375672s 2024-09-16T04:43:43.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f869ed9d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f86a0136e50 0x7f86a0137230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:43.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f869ed9d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f86a0136e50 0x7f86a0137230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:42810/0 (socket says 172.21.15.46:42810) 2024-09-16T04:43:43.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f869ed9d640 1 -- 172.21.15.46:0/2186116248 learned_addr learned my addr 172.21.15.46:0/2186116248 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:43.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f869ed9d640 10 monclient(hunting): get_auth_request con 0x7f86a0136e50 auth_method 0 2024-09-16T04:43:43.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f869ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:43.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f869ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:43.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f869ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:43.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f869ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:43.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f869ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:43.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f869ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:43.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.960+0000 7f869e59c640 1 --2- 172.21.15.46:0/2186116248 >> v2:172.21.15.114:3300/0 conn(0x7f86a00a3500 0x7f86a00a3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:43.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.961+0000 7f869e59c640 10 monclient(hunting): get_auth_request con 0x7f86a00a3500 auth_method 0 2024-09-16T04:43:43.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.961+0000 7f869e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:43.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.961+0000 7f869e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:43.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.961+0000 7f869e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:43.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.961+0000 7f869ed9d640 10 monclient(hunting): handle_auth_done global_id 64118 payload 274 2024-09-16T04:43:43.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.961+0000 7f869ed9d640 1 -- 172.21.15.46:0/2186116248 >> v2:172.21.15.114:3300/0 conn(0x7f86a00a3500 msgr2=0x7f86a00a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:43:43.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.961+0000 7f869ed9d640 1 --2- 172.21.15.46:0/2186116248 >> v2:172.21.15.114:3300/0 conn(0x7f86a00a3500 0x7f86a00a3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:43.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.961+0000 7f869ed9d640 1 -- 172.21.15.46:0/2186116248 >> v1:172.21.15.114:6789/0 conn(0x7f86a00a4180 legacy=0x7f86a00a45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:43.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.961+0000 7f869ed9d640 10 monclient: _finish_hunting 0 2024-09-16T04:43:43.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.961+0000 7f869ed9d640 1 monclient: found mon.noname-b 2024-09-16T04:43:43.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.961+0000 7f869ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:43.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.961+0000 7f869ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:43:43.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.961+0000 7f869ed9d640 1 -- 172.21.15.46:0/2186116248 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a00a10a0 con 0x7f86a0136e50 2024-09-16T04:43:43.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.961+0000 7f869ed9d640 1 --2- 172.21.15.46:0/2186116248 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f86a0136e50 0x7f86a0137230 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8690009870 tx=0x7f869002a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5977d6d22fc63ad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:43.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f869dd9b640 1 -- 172.21.15.46:0/2186116248 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8690037070 con 0x7f86a0136e50 2024-09-16T04:43:43.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f869dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:43.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f869dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:43:43.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f869dd9b640 10 monclient: dump: 2024-09-16T04:43:43.972 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:43.972 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:43.972 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:43.972 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:43.972 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:43.973 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:43.973 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:43.973 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:43.973 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:43.973 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:43.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f869dd9b640 10 monclient: _finish_auth 0 2024-09-16T04:43:43.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f869dd9b640 10 monclient: _check_auth_tickets 2024-09-16T04:43:43.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f869dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:43.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f869dd9b640 1 -- 172.21.15.46:0/2186116248 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8690032440 con 0x7f86a0136e50 2024-09-16T04:43:43.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f86a518d640 5 monclient: authenticate success, global_id 64118 2024-09-16T04:43:43.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f86a518d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:43.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f869dd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:43.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f869dd9b640 1 -- 172.21.15.46:0/2186116248 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8690032810 con 0x7f86a0136e50 2024-09-16T04:43:43.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f86a518d640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:43.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f86a518d640 4 set_mon_vals no callback set 2024-09-16T04:43:43.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f869dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:43.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f869dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:43:43.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f869dd9b640 10 monclient: dump: 2024-09-16T04:43:43.974 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:43.974 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:43.974 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:43.974 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:43.974 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:43.974 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:43.974 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:43.974 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:43.974 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:43.974 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:43.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f86a518d640 10 monclient: shutdown 2024-09-16T04:43:43.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f86a518d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:43.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f86a518d640 1 -- 172.21.15.46:0/2186116248 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f86a0136e50 msgr2=0x7f86a0137230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:43.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f86a518d640 1 --2- 172.21.15.46:0/2186116248 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f86a0136e50 0x7f86a0137230 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8690009870 tx=0x7f869002a130 comp rx=0 tx=0).stop 2024-09-16T04:43:43.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f86a518d640 1 -- 172.21.15.46:0/2186116248 shutdown_connections 2024-09-16T04:43:43.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f86a518d640 1 --2- 172.21.15.46:0/2186116248 >> v2:172.21.15.114:3300/0 conn(0x7f86a00a3500 0x7f86a00a3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:43.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f86a518d640 1 --2- 172.21.15.46:0/2186116248 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f86a0136e50 0x7f86a0137230 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:43.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f86a518d640 1 -- 172.21.15.46:0/2186116248 >> 172.21.15.46:0/2186116248 conn(0x7f86a00a5f20 msgr2=0x7f86a00a6330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:43.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f86a518d640 1 -- 172.21.15.46:0/2186116248 shutdown_connections 2024-09-16T04:43:43.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.962+0000 7f86a518d640 1 -- 172.21.15.46:0/2186116248 wait complete. 2024-09-16T04:43:43.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.963+0000 7f86a518d640 10 monclient: build_initial_monmap 2024-09-16T04:43:43.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.963+0000 7f86a518d640 10 monclient: monmap: 2024-09-16T04:43:43.975 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:43.975 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:43.976 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.963+0000 7f86a518d640 1 Processor -- start 2024-09-16T04:43:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f86a518d640 1 -- start start 2024-09-16T04:43:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f86a518d640 10 monclient: init 2024-09-16T04:43:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f86a518d640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:43.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f86a518d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:43:43.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f86a518d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86a00a3500 0x7f86a01cbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:43.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f86a518d640 10 monclient(hunting): picked mon.noname-a con 0x7f86a00a3500 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:43.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f86a518d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f86a00a4180 0x7f86a0135d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:43.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f86a518d640 10 monclient(hunting): picked mon.noname-c con 0x7f86a00a4180 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:43.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f86a518d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86a0136e50 0x7f86a0136290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:43.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f86a518d640 10 monclient(hunting): picked mon.noname-b con 0x7f86a0136e50 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:43.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f86a518d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:43.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f86a518d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f86a00a0b30 con 0x7f86a00a3500 2024-09-16T04:43:43.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f86a518d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:43.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f86a518d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f86a00a09b0 con 0x7f86a0136e50 2024-09-16T04:43:43.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f86a518d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:43.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f86a518d640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f86a00a1b60 con 0x7f86a00a4180 2024-09-16T04:43:43.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f86a518d640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:43.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f86a518d640 10 monclient(hunting): authenticate will time out at 1022.379606s 2024-09-16T04:43:43.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f869e59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f86a00a4180 0x7f86a0135d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:43.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f869ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86a00a3500 0x7f86a01cbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:43.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f869e59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f86a00a4180 0x7f86a0135d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:42822/0 (socket says 172.21.15.46:42822) 2024-09-16T04:43:43.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f869e59c640 1 -- 172.21.15.46:0/1270789190 learned_addr learned my addr 172.21.15.46:0/1270789190 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:43.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f869e59c640 10 monclient(hunting): get_auth_request con 0x7f86a00a4180 auth_method 0 2024-09-16T04:43:43.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f869e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:43.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f869e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:43.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f869e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:43.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f869ed9d640 10 monclient(hunting): get_auth_request con 0x7f86a00a3500 auth_method 0 2024-09-16T04:43:43.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f869ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:43.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f869ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:43.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f869ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:43.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f869e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:43.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f869e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:43.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f869e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:43.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f869ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:43.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f869ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:43.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f869ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:43.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.964+0000 7f869f59e640 1 --2- 172.21.15.46:0/1270789190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86a0136e50 0x7f86a0136290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:43.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f869e59c640 10 monclient(hunting): handle_auth_done global_id 64121 payload 824 2024-09-16T04:43:43.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f869e59c640 1 -- 172.21.15.46:0/1270789190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86a0136e50 msgr2=0x7f86a0136290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:43.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f869e59c640 1 --2- 172.21.15.46:0/1270789190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86a0136e50 0x7f86a0136290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:43.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f869e59c640 1 -- 172.21.15.46:0/1270789190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86a00a3500 msgr2=0x7f86a01cbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:43.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f869e59c640 1 --2- 172.21.15.46:0/1270789190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86a00a3500 0x7f86a01cbd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:43.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f869e59c640 10 monclient: _finish_hunting 0 2024-09-16T04:43:43.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f869e59c640 1 monclient: found mon.noname-c 2024-09-16T04:43:43.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f869e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:43.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f869e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:43:43.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f869e59c640 1 -- 172.21.15.46:0/1270789190 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a0125f50 con 0x7f86a00a4180 2024-09-16T04:43:43.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f869f59e640 10 monclient: get_auth_request con 0x7f86a0136e50 auth_method 0 2024-09-16T04:43:43.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f869f59e640 1 --2- 172.21.15.46:0/1270789190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86a0136e50 0x7f86a0136290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:43:43.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f869e59c640 1 --2- 172.21.15.46:0/1270789190 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f86a00a4180 0x7f86a0135d50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f869400d800 tx=0x7f869400dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:43.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f86877fe640 1 -- 172.21.15.46:0/1270789190 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8694004020 con 0x7f86a00a4180 2024-09-16T04:43:43.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f86877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:43.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f86877fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:43:43.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f86877fe640 10 monclient: dump: 2024-09-16T04:43:43.981 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:43.981 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:43.981 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:43.982 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:43.982 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:43.982 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:43.982 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:43.982 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:43.982 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:43.982 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:43.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f86877fe640 10 monclient: _finish_auth 0 2024-09-16T04:43:43.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f86877fe640 10 monclient: _check_auth_tickets 2024-09-16T04:43:43.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f86877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:43.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f86a518d640 5 monclient: authenticate success, global_id 64121 2024-09-16T04:43:43.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f86877fe640 1 -- 172.21.15.46:0/1270789190 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86940041c0 con 0x7f86a00a4180 2024-09-16T04:43:43.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f86877fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:43.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f86a518d640 10 monclient: _renew_subs 2024-09-16T04:43:43.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f86a518d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:43.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f86877fe640 1 -- 172.21.15.46:0/1270789190 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8694004390 con 0x7f86a00a4180 2024-09-16T04:43:43.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f86a518d640 1 -- 172.21.15.46:0/1270789190 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a00a0430 con 0x7f86a00a4180 2024-09-16T04:43:43.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f86877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:43.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f86877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:43:43.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f86877fe640 10 monclient: dump: 2024-09-16T04:43:43.983 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:43.983 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:43.983 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:43.983 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:43.983 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:43.983 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:43.984 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:43.984 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:43.984 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:43.984 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:43.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f869d59a640 4 set_mon_vals no callback set 2024-09-16T04:43:43.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f86a518d640 10 monclient: _renew_subs 2024-09-16T04:43:43.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f86a518d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:43.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.965+0000 7f86a518d640 1 -- 172.21.15.46:0/1270789190 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a0127470 con 0x7f86a00a4180 2024-09-16T04:43:43.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.966+0000 7f86877fe640 1 -- 172.21.15.46:0/1270789190 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f869401b020 con 0x7f86a00a4180 2024-09-16T04:43:43.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.966+0000 7f86877fe640 1 --2- 172.21.15.46:0/1270789190 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f868c0383b0 0x7f868c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:43.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.966+0000 7f86a518d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:43.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.966+0000 7f86a518d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:43.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.966+0000 7f86a518d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:43.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.966+0000 7f86877fe640 1 -- 172.21.15.46:0/1270789190 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(194..194 src has 1..194) ==== 4346+0+0 (secure 0 0 0) 0x7f869404b320 con 0x7f86a00a4180 2024-09-16T04:43:43.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.966+0000 7f86a518d640 1 -- 172.21.15.46:0/1270789190 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86a00a3c40 con 0x7f86a00a4180 2024-09-16T04:43:43.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.966+0000 7f869ed9d640 1 --2- 172.21.15.46:0/1270789190 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f868c0383b0 0x7f868c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:43.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.966+0000 7f869ed9d640 10 monclient: get_auth_request con 0x7f868c0383b0 auth_method 0 2024-09-16T04:43:43.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.967+0000 7f869ed9d640 1 --2- 172.21.15.46:0/1270789190 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f868c0383b0 0x7f868c03a870 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8690034000 tx=0x7f869002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:43.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.971+0000 7f86877fe640 1 -- 172.21.15.46:0/1270789190 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f869401d810 con 0x7f86a00a4180 2024-09-16T04:43:43.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.971+0000 7f86877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:43.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:43.971+0000 7f86877fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:44.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.217+0000 7f86a518d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:43:44.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.217+0000 7f86a518d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:44.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.217+0000 7f86a518d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:44.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.217+0000 7f86a518d640 1 -- 172.21.15.46:0/1270789190 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f86a0126eb0 con 0x7f86a00a4180 2024-09-16T04:43:44.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.217+0000 7f86877fe640 1 -- 172.21.15.46:0/1270789190 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f869401da70 con 0x7f86a00a4180 2024-09-16T04:43:44.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.217+0000 7f86877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:44.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.217+0000 7f86877fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:44.220 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:43:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.219+0000 7f86a518d640 1 -- 172.21.15.46:0/1270789190 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f868c0383b0 msgr2=0x7f868c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.219+0000 7f86a518d640 1 --2- 172.21.15.46:0/1270789190 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f868c0383b0 0x7f868c03a870 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8690034000 tx=0x7f869002e000 comp rx=0 tx=0).stop 2024-09-16T04:43:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.219+0000 7f86a518d640 10 monclient: shutdown 2024-09-16T04:43:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.219+0000 7f86a518d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.219+0000 7f86a518d640 1 -- 172.21.15.46:0/1270789190 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f86a00a4180 msgr2=0x7f86a0135d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.219+0000 7f86a518d640 1 --2- 172.21.15.46:0/1270789190 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f86a00a4180 0x7f86a0135d50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f869400d800 tx=0x7f869400dcc0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.219+0000 7f86a518d640 1 -- 172.21.15.46:0/1270789190 shutdown_connections 2024-09-16T04:43:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.219+0000 7f86a518d640 1 --2- 172.21.15.46:0/1270789190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86a0136e50 0x7f86a0136290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.219+0000 7f86a518d640 1 --2- 172.21.15.46:0/1270789190 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f86a00a4180 0x7f86a0135d50 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.219+0000 7f86a518d640 1 --2- 172.21.15.46:0/1270789190 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f868c0383b0 0x7f868c03a870 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.219+0000 7f86a518d640 1 --2- 172.21.15.46:0/1270789190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86a00a3500 0x7f86a01cbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.219+0000 7f86a518d640 1 -- 172.21.15.46:0/1270789190 >> 172.21.15.46:0/1270789190 conn(0x7f86a00a5f20 msgr2=0x7f86a00a78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:44.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.219+0000 7f86a518d640 1 -- 172.21.15.46:0/1270789190 shutdown_connections 2024-09-16T04:43:44.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.219+0000 7f86a518d640 1 -- 172.21.15.46:0/1270789190 wait complete. 2024-09-16T04:43:44.231 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:43:44.231 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:43:44.231 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:43:44.231 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:43:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.335+0000 7f3381c74640 10 monclient: get_monmap_and_config 2024-09-16T04:43:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.335+0000 7f3381c74640 10 monclient: build_initial_monmap 2024-09-16T04:43:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.335+0000 7f3381c74640 10 monclient: monmap: 2024-09-16T04:43:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:44.336075+0000 2024-09-16T04:43:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:44.336075+0000 2024-09-16T04:43:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:44.338 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:44.338 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:44.338 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:44.338 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.336+0000 7f3381c74640 1 Processor -- start 2024-09-16T04:43:44.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.336+0000 7f3381c74640 1 -- start start 2024-09-16T04:43:44.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.336+0000 7f3381c74640 10 monclient: init 2024-09-16T04:43:44.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.336+0000 7f3381c74640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:44.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.336+0000 7f3381c74640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-16T04:43:44.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.336+0000 7f3381c74640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f337c06e160 0x7f337c0a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.336+0000 7f3381c74640 10 monclient(hunting): picked mon.noname-a con 0x7f337c06e160 addr v2:172.21.15.46:3300/0 2024-09-16T04:43:44.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.336+0000 7f3381c74640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f337c0a2ee0 0x7f337c136940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.336+0000 7f3381c74640 10 monclient(hunting): picked mon.noname-b con 0x7f337c0a2ee0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:44.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.336+0000 7f3381c74640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f337c136e80 addr v1:172.21.15.114:6789/0 2024-09-16T04:43:44.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.336+0000 7f3381c74640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f337c0a3fd0 con 0x7f337c136e80 2024-09-16T04:43:44.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.336+0000 7f3381c74640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.336+0000 7f3381c74640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f337c0a3300 con 0x7f337c06e160 2024-09-16T04:43:44.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.336+0000 7f3381c74640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.336+0000 7f3381c74640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f337c06e540 con 0x7f337c0a2ee0 2024-09-16T04:43:44.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.336+0000 7f3381c74640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:44.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.336+0000 7f3381c74640 10 monclient(hunting): authenticate will time out at 1022.751911s 2024-09-16T04:43:44.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337affd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f337c0a2ee0 0x7f337c136940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f337c06e160 0x7f337c0a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337affd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f337c0a2ee0 0x7f337c136940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:42830/0 (socket says 172.21.15.46:42830) 2024-09-16T04:43:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337b7fe640 10 monclient(hunting): get_auth_request con 0x7f337c06e160 auth_method 0 2024-09-16T04:43:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337affd640 1 -- 172.21.15.46:0/60089624 learned_addr learned my addr 172.21.15.46:0/60089624 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337affd640 10 monclient(hunting): get_auth_request con 0x7f337c0a2ee0 auth_method 0 2024-09-16T04:43:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337b7fe640 10 monclient(hunting): handle_auth_done global_id 44694 payload 274 2024-09-16T04:43:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337b7fe640 1 -- 172.21.15.46:0/60089624 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f337c0a2ee0 msgr2=0x7f337c136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:43:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337b7fe640 1 --2- 172.21.15.46:0/60089624 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f337c0a2ee0 0x7f337c136940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337b7fe640 1 -- 172.21.15.46:0/60089624 >> v1:172.21.15.114:6789/0 conn(0x7f337c136e80 legacy=0x7f337c13dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:43:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:43:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337b7fe640 1 monclient: found mon.noname-a 2024-09-16T04:43:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:43:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337b7fe640 1 -- 172.21.15.46:0/60089624 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337c0a0bd0 con 0x7f337c06e160 2024-09-16T04:43:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.337+0000 7f337affd640 1 --2- 172.21.15.46:0/60089624 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f337c0a2ee0 0x7f337c136940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:43:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f337b7fe640 1 --2- 172.21.15.46:0/60089624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c06e160 0x7f337c0a29a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f336c0097d0 tx=0x7f336c02a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52cc995ba19f827a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:44.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f337a7fc640 1 -- 172.21.15.46:0/60089624 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c037070 con 0x7f337c06e160 2024-09-16T04:43:44.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f337a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f337a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:43:44.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f337a7fc640 10 monclient: dump: 2024-09-16T04:43:44.353 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.353 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.353 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.353 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.353 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.353 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.353 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.353 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.353 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.353 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f337a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:43:44.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f337a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:43:44.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f337a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:44.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f337a7fc640 1 -- 172.21.15.46:0/60089624 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f336c02ad50 con 0x7f337c06e160 2024-09-16T04:43:44.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f3381c74640 5 monclient: authenticate success, global_id 44694 2024-09-16T04:43:44.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f3381c74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:44.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f337a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:44.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f3381c74640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:44.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f337a7fc640 1 -- 172.21.15.46:0/60089624 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c032660 con 0x7f337c06e160 2024-09-16T04:43:44.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f3381c74640 4 set_mon_vals no callback set 2024-09-16T04:43:44.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f337a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f337a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:44.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f337a7fc640 10 monclient: dump: 2024-09-16T04:43:44.355 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.355 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.355 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.355 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.355 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.355 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.355 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.355 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.355 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.355 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f3381c74640 10 monclient: shutdown 2024-09-16T04:43:44.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f3381c74640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:44.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f3381c74640 1 -- 172.21.15.46:0/60089624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c06e160 msgr2=0x7f337c0a29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f3381c74640 1 --2- 172.21.15.46:0/60089624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c06e160 0x7f337c0a29a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f336c0097d0 tx=0x7f336c02a040 comp rx=0 tx=0).stop 2024-09-16T04:43:44.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f3381c74640 1 -- 172.21.15.46:0/60089624 shutdown_connections 2024-09-16T04:43:44.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f3381c74640 1 --2- 172.21.15.46:0/60089624 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f337c0a2ee0 0x7f337c136940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f3381c74640 1 --2- 172.21.15.46:0/60089624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c06e160 0x7f337c0a29a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f3381c74640 1 -- 172.21.15.46:0/60089624 >> 172.21.15.46:0/60089624 conn(0x7f337c0a5c10 msgr2=0x7f337c0a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:44.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f3381c74640 1 -- 172.21.15.46:0/60089624 shutdown_connections 2024-09-16T04:43:44.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.338+0000 7f3381c74640 1 -- 172.21.15.46:0/60089624 wait complete. 2024-09-16T04:43:44.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.339+0000 7f3381c74640 10 monclient: build_initial_monmap 2024-09-16T04:43:44.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.339+0000 7f3381c74640 10 monclient: monmap: 2024-09-16T04:43:44.356 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:44.356 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:44.356 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:44.356 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:44.356 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:44.356 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.357 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:44.357 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:44.357 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:44.357 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.339+0000 7f3381c74640 1 Processor -- start 2024-09-16T04:43:44.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.339+0000 7f3381c74640 1 -- start start 2024-09-16T04:43:44.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.339+0000 7f3381c74640 10 monclient: init 2024-09-16T04:43:44.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f3381c74640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:44.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f3381c74640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:43:44.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f3381c74640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f337c06e160 0x7f337c135fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f3381c74640 10 monclient(hunting): picked mon.noname-c con 0x7f337c06e160 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:44.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f3381c74640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c0a2ee0 0x7f337c136520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f3381c74640 10 monclient(hunting): picked mon.noname-a con 0x7f337c0a2ee0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:44.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f3381c74640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f337c136e80 0x7f337c1cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f3381c74640 10 monclient(hunting): picked mon.noname-b con 0x7f337c136e80 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:44.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f3381c74640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f3381c74640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f337c0a34a0 con 0x7f337c0a2ee0 2024-09-16T04:43:44.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f3381c74640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f3381c74640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f337c0a5130 con 0x7f337c136e80 2024-09-16T04:43:44.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f3381c74640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f3381c74640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f337c121f30 con 0x7f337c06e160 2024-09-16T04:43:44.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f3381c74640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:44.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f3381c74640 10 monclient(hunting): authenticate will time out at 1022.755377s 2024-09-16T04:43:44.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c0a2ee0 0x7f337c136520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337b7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f337c06e160 0x7f337c135fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c0a2ee0 0x7f337c136520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53530/0 (socket says 172.21.15.46:53530) 2024-09-16T04:43:44.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337affd640 1 -- 172.21.15.46:0/4182209543 learned_addr learned my addr 172.21.15.46:0/4182209543 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:44.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337affd640 10 monclient(hunting): get_auth_request con 0x7f337c0a2ee0 auth_method 0 2024-09-16T04:43:44.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337b7fe640 10 monclient(hunting): get_auth_request con 0x7f337c06e160 auth_method 0 2024-09-16T04:43:44.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:44.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:44.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:44.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:44.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:44.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:44.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337bfff640 1 --2- 172.21.15.46:0/4182209543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f337c136e80 0x7f337c1cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337bfff640 10 monclient(hunting): get_auth_request con 0x7f337c136e80 auth_method 0 2024-09-16T04:43:44.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337affd640 10 monclient(hunting): handle_auth_done global_id 44697 payload 824 2024-09-16T04:43:44.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337affd640 1 -- 172.21.15.46:0/4182209543 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f337c06e160 msgr2=0x7f337c135fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337affd640 1 --2- 172.21.15.46:0/4182209543 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f337c06e160 0x7f337c135fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337affd640 1 -- 172.21.15.46:0/4182209543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f337c136e80 msgr2=0x7f337c1cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337affd640 1 --2- 172.21.15.46:0/4182209543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f337c136e80 0x7f337c1cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337affd640 10 monclient: _finish_hunting 0 2024-09-16T04:43:44.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337affd640 1 monclient: found mon.noname-a 2024-09-16T04:43:44.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:44.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:43:44.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337affd640 1 -- 172.21.15.46:0/4182209543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337c1c8fc0 con 0x7f337c0a2ee0 2024-09-16T04:43:44.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337b7fe640 1 --2- 172.21.15.46:0/4182209543 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f337c06e160 0x7f337c135fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:43:44.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337bfff640 1 --2- 172.21.15.46:0/4182209543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f337c136e80 0x7f337c1cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:43:44.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.340+0000 7f337affd640 1 --2- 172.21.15.46:0/4182209543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c0a2ee0 0x7f337c136520 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f337000e9b0 tx=0x7f337000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:44.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3363fff640 1 -- 172.21.15.46:0/4182209543 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3370017070 con 0x7f337c0a2ee0 2024-09-16T04:43:44.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3363fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3363fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:43:44.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3363fff640 10 monclient: dump: 2024-09-16T04:43:44.361 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.361 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.361 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.361 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.362 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.362 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.362 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.362 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.362 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.362 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3363fff640 10 monclient: _finish_auth 0 2024-09-16T04:43:44.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3363fff640 10 monclient: _check_auth_tickets 2024-09-16T04:43:44.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3363fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:44.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3363fff640 1 -- 172.21.15.46:0/4182209543 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f337000ca00 con 0x7f337c0a2ee0 2024-09-16T04:43:44.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3381c74640 5 monclient: authenticate success, global_id 44697 2024-09-16T04:43:44.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3381c74640 10 monclient: _renew_subs 2024-09-16T04:43:44.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3381c74640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:44.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3381c74640 1 -- 172.21.15.46:0/4182209543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f337c0a8040 con 0x7f337c0a2ee0 2024-09-16T04:43:44.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3363fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:44.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3363fff640 1 -- 172.21.15.46:0/4182209543 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337000cdf0 con 0x7f337c0a2ee0 2024-09-16T04:43:44.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3363fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3363fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:44.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3363fff640 10 monclient: dump: 2024-09-16T04:43:44.363 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.363 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.363 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.363 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.363 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.363 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.363 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.363 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.363 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.364 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3381c74640 10 monclient: _renew_subs 2024-09-16T04:43:44.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3381c74640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:44.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3381c74640 1 -- 172.21.15.46:0/4182209543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f337c1246d0 con 0x7f337c0a2ee0 2024-09-16T04:43:44.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3379ffb640 4 set_mon_vals no callback set 2024-09-16T04:43:44.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3381c74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:44.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3381c74640 1 --2- 172.21.15.46:0/4182209543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c1cbfc0 0x7f337c1cc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3381c74640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.341+0000 7f3381c74640 1 -- 172.21.15.46:0/4182209543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f337c0a3620 con 0x7f337c1cbfc0 2024-09-16T04:43:44.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.342+0000 7f3363fff640 1 -- 172.21.15.46:0/4182209543 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f337001a070 con 0x7f337c0a2ee0 2024-09-16T04:43:44.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.342+0000 7f337b7fe640 1 --2- 172.21.15.46:0/4182209543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c1cbfc0 0x7f337c1cc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.342+0000 7f337b7fe640 10 monclient: get_auth_request con 0x7f337c1cbfc0 auth_method 0 2024-09-16T04:43:44.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.342+0000 7f337b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.342+0000 7f337b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.342+0000 7f337b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.342+0000 7f337b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:44.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.342+0000 7f337b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:44.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.342+0000 7f337b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:44.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.342+0000 7f3363fff640 1 --2- 172.21.15.46:0/4182209543 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f33680383b0 0x7f336803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.342+0000 7f3363fff640 1 -- 172.21.15.46:0/4182209543 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 4738+0+0 (secure 0 0 0) 0x7f337004b6e0 con 0x7f337c0a2ee0 2024-09-16T04:43:44.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.342+0000 7f337b7fe640 10 monclient(hunting): handle_auth_done global_id 44700 payload 824 2024-09-16T04:43:44.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.342+0000 7f337b7fe640 1 -- 172.21.15.46:0/4182209543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f337c1cc8e0 con 0x7f337c1cbfc0 2024-09-16T04:43:44.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.342+0000 7f337b7fe640 1 --2- 172.21.15.46:0/4182209543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c1cbfc0 0x7f337c1cc3a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f337c068140 tx=0x7f336c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:44.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.342+0000 7f337bfff640 1 --2- 172.21.15.46:0/4182209543 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f33680383b0 0x7f336803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.342+0000 7f337bfff640 10 monclient: get_auth_request con 0x7f33680383b0 auth_method 0 2024-09-16T04:43:44.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.342+0000 7f3363fff640 1 -- 172.21.15.46:0/4182209543 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c037040 con 0x7f337c1cbfc0 2024-09-16T04:43:44.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.342+0000 7f3363fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.342+0000 7f3363fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:44.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.342+0000 7f3363fff640 10 monclient: dump: 2024-09-16T04:43:44.366 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.366 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.366 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.366 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.366 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.366 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.366 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.366 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.366 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.367 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.343+0000 7f337bfff640 1 --2- 172.21.15.46:0/4182209543 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f33680383b0 0x7f336803a870 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3374004530 tx=0x7f3374009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:44.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.343+0000 7f3363fff640 1 -- 172.21.15.46:0/4182209543 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f336c032e40 con 0x7f337c1cbfc0 2024-09-16T04:43:44.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.343+0000 7f3363fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:44.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.343+0000 7f3363fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:44.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.343+0000 7f3363fff640 1 -- 172.21.15.46:0/4182209543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c1cbfc0 msgr2=0x7f337c1cc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.343+0000 7f3363fff640 1 --2- 172.21.15.46:0/4182209543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c1cbfc0 0x7f337c1cc3a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f337c068140 tx=0x7f336c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.343+0000 7f3363fff640 1 -- 172.21.15.46:0/4182209543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c1cbfc0 msgr2=0x7f337c1cc3a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:43:44.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.343+0000 7f3363fff640 1 --2- 172.21.15.46:0/4182209543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c1cbfc0 0x7f337c1cc3a0 secure :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f337c068140 tx=0x7f336c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f3381c74640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:43:44.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f3381c74640 1 --2- 172.21.15.46:0/4182209543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c09cf10 0x7f337c09d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f3381c74640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f3381c74640 1 -- 172.21.15.46:0/4182209543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f337c0a29a0 con 0x7f337c09cf10 2024-09-16T04:43:44.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f337b7fe640 1 --2- 172.21.15.46:0/4182209543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c09cf10 0x7f337c09d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f337b7fe640 10 monclient: get_auth_request con 0x7f337c09cf10 auth_method 0 2024-09-16T04:43:44.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f337b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f337b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f337b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f337b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:44.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f337b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:44.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f337b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:44.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f337b7fe640 10 monclient(hunting): handle_auth_done global_id 44703 payload 824 2024-09-16T04:43:44.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f337b7fe640 1 -- 172.21.15.46:0/4182209543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f337c067e00 con 0x7f337c09cf10 2024-09-16T04:43:44.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f337b7fe640 1 --2- 172.21.15.46:0/4182209543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c09cf10 0x7f337c09d320 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f336c000c00 tx=0x7f336c008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:44.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f3363fff640 1 -- 172.21.15.46:0/4182209543 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c037040 con 0x7f337c09cf10 2024-09-16T04:43:44.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f3363fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f3363fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:44.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f3363fff640 10 monclient: dump: 2024-09-16T04:43:44.370 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.370 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.370 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.371 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.371 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.371 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.371 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.371 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.371 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.371 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f3363fff640 1 -- 172.21.15.46:0/4182209543 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7f336c03c840 con 0x7f337c09cf10 2024-09-16T04:43:44.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f3363fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:43:44.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f3363fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:44.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f3363fff640 1 -- 172.21.15.46:0/4182209543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c09cf10 msgr2=0x7f337c09d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.349+0000 7f3363fff640 1 --2- 172.21.15.46:0/4182209543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c09cf10 0x7f337c09d320 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f336c000c00 tx=0x7f336c008030 comp rx=0 tx=0).stop 2024-09-16T04:43:44.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.350+0000 7f3363fff640 1 -- 172.21.15.46:0/4182209543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c09cf10 msgr2=0x7f337c09d320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:43:44.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.350+0000 7f3363fff640 1 --2- 172.21.15.46:0/4182209543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c09cf10 0x7f337c09d320 secure :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f336c000c00 tx=0x7f336c008030 comp rx=0 tx=0).stop 2024-09-16T04:43:44.372 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:43:44.372 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:43:44.372 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:43:44.372 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:43:44.372 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 58, 2024-09-16T04:43:44.372 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 104096, 2024-09-16T04:43:44.372 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:43:44.372 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:43:44.372 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:43:44.373 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:43:44.373 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:44.373 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 5, 2024-09-16T04:43:44.373 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:43:44.373 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:43:44.373 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:43:44.373 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:44.373 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:44.373 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:44.373 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:44.373 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:44.373 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:44.373 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:44.373 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:44.374 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:44.374 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:44.374 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:44.374 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:44.374 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:43:44.374 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:43:44.374 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:44.374 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:44.374 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:44.374 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:44.374 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:44.374 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:44.374 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:44.374 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:44.375 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:44.375 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:44.375 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:44.375 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:44.375 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.375 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:43:44.375 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:43:44.375 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:43:44.375 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:43:44.375 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:43:44.375 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:43:44.375 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:43:44.375 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:43:44.375 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:43:44.376 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:43:44.376 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:43:44.376 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:43:44.376 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:43:44.376 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:43:44.376 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:43:44.376 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:43:44.376 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:43:44.376 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:44.376 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:44.376 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:44.376 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:44.376 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:44.377 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:44.377 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:44.377 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:44.377 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:44.377 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:44.377 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:44.377 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:44.377 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:43:44.377 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.377 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:43:44.377 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.377 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:43:44.377 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:43:44.378 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:44.378 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:44.378 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.378 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:44.378 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:43:44.378 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:44.378 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.378 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.378 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:44.378 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:43:44.378 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:44.378 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:44.378 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:44.378 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.379 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:43:44.379 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:43:44.379 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:44.379 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:44.379 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:44.379 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.379 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.379 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:43:44.379 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:43:44.379 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:44.379 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:44.379 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.379 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:44.379 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:43:44.380 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:44.380 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.380 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.380 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:44.380 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:43:44.380 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:44.380 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:44.380 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:44.380 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.380 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:43:44.380 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:43:44.380 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:44.380 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:44.381 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:44.381 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.381 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.381 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:43:44.381 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:43:44.381 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:44.381 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:44.381 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.381 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:44.381 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:43:44.381 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:44.381 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.381 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.382 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:44.382 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:43:44.382 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:44.382 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:44.382 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:44.382 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.382 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:43:44.382 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:43:44.382 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:44.382 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:44.382 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:44.382 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:44.382 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:44.382 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.383 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:43:44.383 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:43:44.383 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.383 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:44.383 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:44.383 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:43:44.383 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:44.383 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:44.383 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:43:44.383 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.383 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:44.383 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:44.383 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:43:44.384 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:44.384 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:44.384 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:43:44.384 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.384 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:44.384 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:44.384 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:43:44.384 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:44.384 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:44.384 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.384 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:43:44.384 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:43:44.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.350+0000 7f3381c74640 1 -- 172.21.15.46:0/4182209543 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f33680383b0 msgr2=0x7f336803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.350+0000 7f3381c74640 1 --2- 172.21.15.46:0/4182209543 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f33680383b0 0x7f336803a870 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3374004530 tx=0x7f3374009040 comp rx=0 tx=0).stop 2024-09-16T04:43:44.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.350+0000 7f3381c74640 10 monclient: shutdown 2024-09-16T04:43:44.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.350+0000 7f3381c74640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:44.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.350+0000 7f3381c74640 1 -- 172.21.15.46:0/4182209543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c0a2ee0 msgr2=0x7f337c136520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.350+0000 7f3381c74640 1 --2- 172.21.15.46:0/4182209543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f337c0a2ee0 0x7f337c136520 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f337000e9b0 tx=0x7f337000ee70 comp rx=0 tx=0).stop 2024-09-16T04:43:44.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.350+0000 7f337bfff640 1 -- 172.21.15.46:0/4182209543 reap_dead start 2024-09-16T04:43:44.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.350+0000 7f337bfff640 1 -- 172.21.15.46:0/4182209543 reap_dead start 2024-09-16T04:43:44.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.350+0000 7f3381c74640 1 -- 172.21.15.46:0/4182209543 shutdown_connections 2024-09-16T04:43:44.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.350+0000 7f3381c74640 1 -- 172.21.15.46:0/4182209543 >> 172.21.15.46:0/4182209543 conn(0x7f337c0a5c10 msgr2=0x7f337c0a6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:44.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.350+0000 7f3381c74640 1 -- 172.21.15.46:0/4182209543 shutdown_connections 2024-09-16T04:43:44.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.350+0000 7f3381c74640 1 -- 172.21.15.46:0/4182209543 wait complete. 2024-09-16T04:43:44.387 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:43:44.387 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:43:44.387 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:43:44.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.488+0000 7faefc665640 10 monclient: get_monmap_and_config 2024-09-16T04:43:44.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.488+0000 7faefc665640 10 monclient: build_initial_monmap 2024-09-16T04:43:44.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.488+0000 7faefc665640 10 monclient: monmap: 2024-09-16T04:43:44.490 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:44.490 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.490 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:44.488843+0000 2024-09-16T04:43:44.490 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:44.488843+0000 2024-09-16T04:43:44.490 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:44.490 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.490 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:44.490 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:44.490 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:44.490 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:44.490 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:44.491 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.489+0000 7faefc665640 1 Processor -- start 2024-09-16T04:43:44.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.489+0000 7faefc665640 1 -- start start 2024-09-16T04:43:44.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.489+0000 7faefc665640 10 monclient: init 2024-09-16T04:43:44.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.489+0000 7faefc665640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:44.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.489+0000 7faefc665640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-16T04:43:44.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.489+0000 7faefc665640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faef40a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:43:44.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.489+0000 7faefc665640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faef40a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:43:44.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.489+0000 7faefc665640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faef413cea0 0x7faef413f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.489+0000 7faefc665640 10 monclient(hunting): picked mon.noname-a con 0x7faef413cea0 addr v2:172.21.15.46:3300/0 2024-09-16T04:43:44.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.489+0000 7faefc665640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faef409b940 con 0x7faef40a3c10 2024-09-16T04:43:44.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.489+0000 7faefc665640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faef409ab00 con 0x7faef40a3ff0 2024-09-16T04:43:44.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.489+0000 7faefc665640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.489+0000 7faefc665640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7faef409b030 con 0x7faef413cea0 2024-09-16T04:43:44.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.489+0000 7faefc665640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:44.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.489+0000 7faefc665640 10 monclient(hunting): authenticate will time out at 1022.904985s 2024-09-16T04:43:44.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.490+0000 7faef9c00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7faef40a3ff0 0x7faef40a43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.46:34918/0 (socket says 172.21.15.46:34918) 2024-09-16T04:43:44.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.490+0000 7faef9c00640 1 -- 172.21.15.46:0/720877124 learned_addr learned my addr 172.21.15.46:0/720877124 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:43:44.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.490+0000 7faefac02640 1 --2- 172.21.15.46:0/720877124 >> v2:172.21.15.46:3300/0 conn(0x7faef413cea0 0x7faef413f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.490+0000 7faefac02640 10 monclient(hunting): get_auth_request con 0x7faef413cea0 auth_method 0 2024-09-16T04:43:44.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.490+0000 7faefac02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.490+0000 7faefac02640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.490+0000 7faefac02640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.490+0000 7faefac02640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:44.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.490+0000 7faefac02640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:44.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.490+0000 7faefac02640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:44.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.490+0000 7faefac02640 10 monclient(hunting): handle_auth_done global_id 44706 payload 274 2024-09-16T04:43:44.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.490+0000 7faefac02640 1 -- 172.21.15.46:0/720877124 >> v1:172.21.15.114:6789/0 conn(0x7faef40a3ff0 legacy=0x7faef40a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.490+0000 7faefac02640 1 -- 172.21.15.46:0/720877124 >> v1:172.21.15.46:6789/0 conn(0x7faef40a3c10 legacy=0x7faef409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.490+0000 7faefac02640 10 monclient: _finish_hunting 0 2024-09-16T04:43:44.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.490+0000 7faefac02640 1 monclient: found mon.noname-a 2024-09-16T04:43:44.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.490+0000 7faef93ff640 1 -- 172.21.15.46:0/720877124 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2448290859 0 0) 0x7faef409ab00 con 0x7faef40a3ff0 2024-09-16T04:43:44.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.490+0000 7faefac02640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:44.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.490+0000 7faefac02640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:43:44.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.490+0000 7faefac02640 1 -- 172.21.15.46:0/720877124 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef409b310 con 0x7faef413cea0 2024-09-16T04:43:44.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.490+0000 7faef93ff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:43:44.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.491+0000 7faefac02640 1 --2- 172.21.15.46:0/720877124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faef413cea0 0x7faef413f290 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7faeec009d90 tx=0x7faeec02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b42843a8ae4d815 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:44.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.491+0000 7faef93ff640 1 -- 172.21.15.46:0/720877124 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeec007740 con 0x7faef413cea0 2024-09-16T04:43:44.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.491+0000 7faef93ff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.491+0000 7faef93ff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:43:44.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.491+0000 7faef93ff640 10 monclient: dump: 2024-09-16T04:43:44.502 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.502 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.502 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.502 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.502 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.503 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.503 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.503 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.503 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.503 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.491+0000 7faef93ff640 10 monclient: _finish_auth 0 2024-09-16T04:43:44.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.491+0000 7faef93ff640 10 monclient: _check_auth_tickets 2024-09-16T04:43:44.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.491+0000 7faef93ff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:44.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.491+0000 7faefc665640 5 monclient: authenticate success, global_id 44706 2024-09-16T04:43:44.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.491+0000 7faef93ff640 1 -- 172.21.15.46:0/720877124 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faeec0078e0 con 0x7faef413cea0 2024-09-16T04:43:44.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.491+0000 7faefc665640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:44.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.491+0000 7faef93ff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:44.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.491+0000 7faefc665640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:44.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.491+0000 7faefc665640 4 set_mon_vals no callback set 2024-09-16T04:43:44.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.491+0000 7faef93ff640 1 -- 172.21.15.46:0/720877124 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeec007cb0 con 0x7faef413cea0 2024-09-16T04:43:44.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.491+0000 7faef93ff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.491+0000 7faef93ff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:44.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.491+0000 7faef93ff640 10 monclient: dump: 2024-09-16T04:43:44.504 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.504 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.504 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.504 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.504 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.504 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.504 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.504 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.504 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.504 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.492+0000 7faefc665640 10 monclient: shutdown 2024-09-16T04:43:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.492+0000 7faefc665640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.492+0000 7faefc665640 1 -- 172.21.15.46:0/720877124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faef413cea0 msgr2=0x7faef413f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.492+0000 7faefc665640 1 --2- 172.21.15.46:0/720877124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faef413cea0 0x7faef413f290 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7faeec009d90 tx=0x7faeec02a660 comp rx=0 tx=0).stop 2024-09-16T04:43:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.492+0000 7faefc665640 1 -- 172.21.15.46:0/720877124 shutdown_connections 2024-09-16T04:43:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.492+0000 7faefc665640 1 --2- 172.21.15.46:0/720877124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faef413cea0 0x7faef413f290 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.492+0000 7faefc665640 1 -- 172.21.15.46:0/720877124 >> 172.21.15.46:0/720877124 conn(0x7faef40a6340 msgr2=0x7faef406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.492+0000 7faefc665640 1 -- 172.21.15.46:0/720877124 shutdown_connections 2024-09-16T04:43:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.492+0000 7faefc665640 1 -- 172.21.15.46:0/720877124 wait complete. 2024-09-16T04:43:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.492+0000 7faefc665640 10 monclient: build_initial_monmap 2024-09-16T04:43:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.492+0000 7faefc665640 10 monclient: monmap: 2024-09-16T04:43:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:44.506 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:44.506 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:44.506 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.506 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:44.506 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:44.506 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:44.506 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefc665640 1 Processor -- start 2024-09-16T04:43:44.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefc665640 1 -- start start 2024-09-16T04:43:44.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefc665640 10 monclient: init 2024-09-16T04:43:44.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefc665640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:44.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefc665640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:43:44.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefc665640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faef40a3c10 0x7faef41369c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefc665640 10 monclient(hunting): picked mon.noname-b con 0x7faef40a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:44.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefc665640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faef40a3ff0 0x7faef4136f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefc665640 10 monclient(hunting): picked mon.noname-a con 0x7faef40a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:44.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefc665640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faef413cea0 0x7faef4139cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefc665640 10 monclient(hunting): picked mon.noname-c con 0x7faef413cea0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:44.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefc665640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefc665640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faef413f950 con 0x7faef40a3ff0 2024-09-16T04:43:44.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefc665640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefc665640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7faef413f7d0 con 0x7faef40a3c10 2024-09-16T04:43:44.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefc665640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefc665640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7faef413fad0 con 0x7faef413cea0 2024-09-16T04:43:44.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefc665640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:44.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefc665640 10 monclient(hunting): authenticate will time out at 1022.908712s 2024-09-16T04:43:44.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faef9c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faef40a3ff0 0x7faef4136f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefac02640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faef413cea0 0x7faef4139cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faef9c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faef40a3ff0 0x7faef4136f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56204/0 (socket says 172.21.15.46:56204) 2024-09-16T04:43:44.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefac02640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faef413cea0 0x7faef4139cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:52394/0 (socket says 172.21.15.46:52394) 2024-09-16T04:43:44.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faef9c00640 1 -- 172.21.15.46:0/496907054 learned_addr learned my addr 172.21.15.46:0/496907054 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:44.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faef9c00640 10 monclient(hunting): get_auth_request con 0x7faef40a3ff0 auth_method 0 2024-09-16T04:43:44.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faef9c00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faef9c00640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faef9c00640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefac02640 10 monclient(hunting): get_auth_request con 0x7faef413cea0 auth_method 0 2024-09-16T04:43:44.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefac02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefac02640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefac02640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faef9c00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:44.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faef9c00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:44.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faef9c00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:44.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefac02640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:44.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefac02640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:44.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.493+0000 7faefac02640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:44.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faefa401640 1 --2- 172.21.15.46:0/496907054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faef40a3c10 0x7faef41369c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faefa401640 10 monclient(hunting): get_auth_request con 0x7faef40a3c10 auth_method 0 2024-09-16T04:43:44.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faefa401640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faefa401640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faefa401640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faef9c00640 10 monclient(hunting): handle_auth_done global_id 44712 payload 824 2024-09-16T04:43:44.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faef9c00640 1 -- 172.21.15.46:0/496907054 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faef413cea0 msgr2=0x7faef4139cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faef9c00640 1 --2- 172.21.15.46:0/496907054 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faef413cea0 0x7faef4139cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faef9c00640 1 -- 172.21.15.46:0/496907054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faef40a3c10 msgr2=0x7faef41369c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faef9c00640 1 --2- 172.21.15.46:0/496907054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faef40a3c10 0x7faef41369c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faef9c00640 10 monclient: _finish_hunting 0 2024-09-16T04:43:44.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faef9c00640 1 monclient: found mon.noname-a 2024-09-16T04:43:44.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faef9c00640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:44.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faef9c00640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:43:44.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faef9c00640 1 -- 172.21.15.46:0/496907054 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef413d6d0 con 0x7faef40a3ff0 2024-09-16T04:43:44.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faefac02640 1 --2- 172.21.15.46:0/496907054 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faef413cea0 0x7faef4139cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:43:44.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faef9c00640 1 --2- 172.21.15.46:0/496907054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faef40a3ff0 0x7faef4136f00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7faee400c520 tx=0x7faee400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:44.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faeeaffd640 1 -- 172.21.15.46:0/496907054 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee4018070 con 0x7faef40a3ff0 2024-09-16T04:43:44.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faeeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faeeaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:43:44.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faeeaffd640 10 monclient: dump: 2024-09-16T04:43:44.510 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.511 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.511 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.511 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.511 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.511 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.511 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.511 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.511 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.511 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faeeaffd640 10 monclient: _finish_auth 0 2024-09-16T04:43:44.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faeeaffd640 10 monclient: _check_auth_tickets 2024-09-16T04:43:44.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faeeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:44.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faeeaffd640 1 -- 172.21.15.46:0/496907054 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faee40074e0 con 0x7faef40a3ff0 2024-09-16T04:43:44.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faefc665640 5 monclient: authenticate success, global_id 44712 2024-09-16T04:43:44.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faeeaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:44.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faeeaffd640 1 -- 172.21.15.46:0/496907054 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee40049e0 con 0x7faef40a3ff0 2024-09-16T04:43:44.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faeeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faeeaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:44.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faeeaffd640 10 monclient: dump: 2024-09-16T04:43:44.512 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.512 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.512 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.512 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.512 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.512 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.512 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.512 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.513 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.513 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faefc665640 10 monclient: _renew_subs 2024-09-16T04:43:44.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faefc665640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:44.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faefc665640 1 -- 172.21.15.46:0/496907054 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faef413d280 con 0x7faef40a3ff0 2024-09-16T04:43:44.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faeebfff640 4 set_mon_vals no callback set 2024-09-16T04:43:44.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faefc665640 10 monclient: _renew_subs 2024-09-16T04:43:44.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faefc665640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:44.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.494+0000 7faefc665640 1 -- 172.21.15.46:0/496907054 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faef4120330 con 0x7faef40a3ff0 2024-09-16T04:43:44.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.495+0000 7faefc665640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:44.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.495+0000 7faefc665640 1 --2- 172.21.15.46:0/496907054 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faef402c640 0x7faef402ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.495+0000 7faefc665640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.495+0000 7faefc665640 1 -- 172.21.15.46:0/496907054 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7faef409bcf0 con 0x7faef402c640 2024-09-16T04:43:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.495+0000 7faeeaffd640 1 -- 172.21.15.46:0/496907054 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7faee4004b80 con 0x7faef40a3ff0 2024-09-16T04:43:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.495+0000 7faefa401640 1 --2- 172.21.15.46:0/496907054 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faef402c640 0x7faef402ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.495+0000 7faefa401640 10 monclient: get_auth_request con 0x7faef402c640 auth_method 0 2024-09-16T04:43:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.495+0000 7faefa401640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.495+0000 7faefa401640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.495+0000 7faefa401640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.495+0000 7faefa401640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.495+0000 7faefa401640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.495+0000 7faefa401640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.495+0000 7faeeaffd640 1 --2- 172.21.15.46:0/496907054 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7faed80383b0 0x7faed803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.495+0000 7faeeaffd640 1 -- 172.21.15.46:0/496907054 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 4738+0+0 (secure 0 0 0) 0x7faee404a900 con 0x7faef40a3ff0 2024-09-16T04:43:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.495+0000 7faefa401640 10 monclient(hunting): handle_auth_done global_id 64133 payload 824 2024-09-16T04:43:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.495+0000 7faefa401640 1 -- 172.21.15.46:0/496907054 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faef409e820 con 0x7faef402c640 2024-09-16T04:43:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.495+0000 7faefac02640 1 --2- 172.21.15.46:0/496907054 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7faed80383b0 0x7faed803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.495+0000 7faefa401640 1 --2- 172.21.15.46:0/496907054 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faef402c640 0x7faef402ca20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7faef000e4d0 tx=0x7faef000e990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.495+0000 7faefac02640 10 monclient: get_auth_request con 0x7faed80383b0 auth_method 0 2024-09-16T04:43:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.496+0000 7faeeaffd640 1 -- 172.21.15.46:0/496907054 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef0018070 con 0x7faef402c640 2024-09-16T04:43:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.496+0000 7faeeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.496+0000 7faeeaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:43:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.496+0000 7faeeaffd640 10 monclient: dump: 2024-09-16T04:43:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.516 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.496+0000 7faeeaffd640 1 -- 172.21.15.46:0/496907054 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faef00052e0 con 0x7faef402c640 2024-09-16T04:43:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.496+0000 7faeeaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.496+0000 7faeeaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.496+0000 7faefac02640 1 --2- 172.21.15.46:0/496907054 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7faed80383b0 0x7faed803a870 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7faeec034000 tx=0x7faeec02f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.496+0000 7faeeaffd640 1 -- 172.21.15.46:0/496907054 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faef402c640 msgr2=0x7faef402ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.496+0000 7faeeaffd640 1 --2- 172.21.15.46:0/496907054 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faef402c640 0x7faef402ca20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7faef000e4d0 tx=0x7faef000e990 comp rx=0 tx=0).stop 2024-09-16T04:43:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.496+0000 7faeeaffd640 1 -- 172.21.15.46:0/496907054 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faef402c640 msgr2=0x7faef402ca20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:43:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.496+0000 7faeeaffd640 1 --2- 172.21.15.46:0/496907054 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faef402c640 0x7faef402ca20 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7faef000e4d0 tx=0x7faef000e990 comp rx=0 tx=0).stop 2024-09-16T04:43:44.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.502+0000 7faefc665640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:43:44.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.502+0000 7faefc665640 1 --2- 172.21.15.46:0/496907054 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faef4120500 0x7faef402cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.502+0000 7faefc665640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.502+0000 7faefc665640 1 -- 172.21.15.46:0/496907054 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7faef4067e20 con 0x7faef4120500 2024-09-16T04:43:44.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.502+0000 7faefa401640 1 --2- 172.21.15.46:0/496907054 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faef4120500 0x7faef402cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.502+0000 7faefa401640 10 monclient: get_auth_request con 0x7faef4120500 auth_method 0 2024-09-16T04:43:44.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.502+0000 7faefa401640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.502+0000 7faefa401640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.502+0000 7faefa401640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.502+0000 7faefa401640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.502+0000 7faefa401640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.502+0000 7faefa401640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.502+0000 7faefa401640 10 monclient(hunting): handle_auth_done global_id 64136 payload 824 2024-09-16T04:43:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.502+0000 7faefa401640 1 -- 172.21.15.46:0/496907054 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faef413a9b0 con 0x7faef4120500 2024-09-16T04:43:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.502+0000 7faefa401640 1 --2- 172.21.15.46:0/496907054 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faef4120500 0x7faef402cf60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faef0009010 tx=0x7faef0001120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faeeaffd640 1 -- 172.21.15.46:0/496907054 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef0018040 con 0x7faef4120500 2024-09-16T04:43:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faeeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faeeaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:43:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faeeaffd640 10 monclient: dump: 2024-09-16T04:43:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.521 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faeeaffd640 1 -- 172.21.15.46:0/496907054 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7faef0007630 con 0x7faef4120500 2024-09-16T04:43:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faeeaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:43:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faeeaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faeeaffd640 1 -- 172.21.15.46:0/496907054 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faef4120500 msgr2=0x7faef402cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faeeaffd640 1 --2- 172.21.15.46:0/496907054 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faef4120500 0x7faef402cf60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faef0009010 tx=0x7faef0001120 comp rx=0 tx=0).stop 2024-09-16T04:43:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faeeaffd640 1 -- 172.21.15.46:0/496907054 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faef4120500 msgr2=0x7faef402cf60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:43:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faeeaffd640 1 --2- 172.21.15.46:0/496907054 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7faef4120500 0x7faef402cf60 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faef0009010 tx=0x7faef0001120 comp rx=0 tx=0).stop 2024-09-16T04:43:44.521 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:43:44.521 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:43:44.522 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:43:44.522 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:43:44.522 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 58, 2024-09-16T04:43:44.522 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 12439, 2024-09-16T04:43:44.522 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:43:44.522 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:43:44.522 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:43:44.522 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:43:44.522 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:44.522 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 5, 2024-09-16T04:43:44.522 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:43:44.523 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:43:44.523 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:43:44.523 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:44.523 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:44.523 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:44.523 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:44.523 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:44.523 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:44.523 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:44.523 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:44.523 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:44.524 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:44.524 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:44.524 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:44.524 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:43:44.524 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:43:44.524 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:44.524 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:44.524 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:44.524 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:44.524 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:44.524 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:44.524 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:44.524 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:44.525 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:44.525 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:44.525 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:44.525 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:44.525 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.525 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:43:44.525 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:43:44.525 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:43:44.525 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:43:44.525 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:43:44.525 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:43:44.525 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:43:44.525 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:43:44.525 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:43:44.526 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:43:44.526 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:43:44.526 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:43:44.526 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:43:44.526 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:43:44.526 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:43:44.526 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:43:44.526 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:43:44.526 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:44.526 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:44.526 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:44.526 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:44.526 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:44.526 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:44.527 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:44.527 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:44.527 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:44.527 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:44.527 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:44.527 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:44.527 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:43:44.527 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.527 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:43:44.527 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.527 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:43:44.527 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:43:44.527 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:44.528 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:44.528 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.528 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:44.528 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:43:44.528 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:44.528 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.528 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.528 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:44.528 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:43:44.528 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:44.528 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:44.528 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:44.528 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.529 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:43:44.529 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:43:44.529 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:44.529 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:44.529 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:44.529 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.529 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.529 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:43:44.529 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:43:44.529 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:44.529 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:44.529 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.529 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:44.529 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:43:44.530 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:44.530 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.530 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.530 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:44.530 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:43:44.530 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:44.530 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:44.530 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:44.530 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.530 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:43:44.530 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:43:44.530 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:44.530 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:44.530 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:44.531 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.531 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.531 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:43:44.531 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:43:44.531 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:44.531 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:44.531 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.531 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:44.531 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:43:44.531 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:44.531 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.531 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.531 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:44.531 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:43:44.532 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:44.532 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:44.532 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:44.532 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.532 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:43:44.532 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:43:44.532 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:44.532 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:44.532 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:44.532 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:44.532 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:44.532 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.532 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:43:44.532 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:43:44.533 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.533 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:44.533 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:44.533 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:43:44.533 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:44.533 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:44.533 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:43:44.533 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.533 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:44.533 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:44.533 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:43:44.533 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:44.533 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:44.533 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.534 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:43:44.534 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:43:44.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faefc665640 1 -- 172.21.15.46:0/496907054 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7faed80383b0 msgr2=0x7faed803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faefc665640 1 --2- 172.21.15.46:0/496907054 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7faed80383b0 0x7faed803a870 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7faeec034000 tx=0x7faeec02f000 comp rx=0 tx=0).stop 2024-09-16T04:43:44.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faefc665640 10 monclient: shutdown 2024-09-16T04:43:44.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faefc665640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:44.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faefc665640 1 -- 172.21.15.46:0/496907054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faef40a3ff0 msgr2=0x7faef4136f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faefc665640 1 --2- 172.21.15.46:0/496907054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faef40a3ff0 0x7faef4136f00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7faee400c520 tx=0x7faee400c9e0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faefac02640 1 -- 172.21.15.46:0/496907054 reap_dead start 2024-09-16T04:43:44.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faefac02640 1 -- 172.21.15.46:0/496907054 reap_dead start 2024-09-16T04:43:44.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faefc665640 1 -- 172.21.15.46:0/496907054 shutdown_connections 2024-09-16T04:43:44.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faefc665640 1 -- 172.21.15.46:0/496907054 >> 172.21.15.46:0/496907054 conn(0x7faef40a6340 msgr2=0x7faef40a2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:44.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faefc665640 1 -- 172.21.15.46:0/496907054 shutdown_connections 2024-09-16T04:43:44.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.503+0000 7faefc665640 1 -- 172.21.15.46:0/496907054 wait complete. 2024-09-16T04:43:44.536 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:43:44.536 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:43:44.536 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:43:44.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.637+0000 7faff468d640 10 monclient: get_monmap_and_config 2024-09-16T04:43:44.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.637+0000 7faff468d640 10 monclient: build_initial_monmap 2024-09-16T04:43:44.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.637+0000 7faff468d640 10 monclient: monmap: 2024-09-16T04:43:44.638 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:44.638 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.638 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:44.637765+0000 2024-09-16T04:43:44.639 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:44.637765+0000 2024-09-16T04:43:44.639 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:44.639 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.639 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:44.639 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:44.639 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:44.639 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:44.639 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:44.639 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.637+0000 7faff468d640 1 Processor -- start 2024-09-16T04:43:44.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.637+0000 7faff468d640 1 -- start start 2024-09-16T04:43:44.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.637+0000 7faff468d640 10 monclient: init 2024-09-16T04:43:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff468d640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff468d640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-16T04:43:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff468d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fafec0a3c10 0x7fafec09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff468d640 10 monclient(hunting): picked mon.noname-a con 0x7fafec0a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:43:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff468d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fafec0a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:43:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff468d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fafec13be90 0x7fafec13e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff468d640 10 monclient(hunting): picked mon.noname-c con 0x7fafec13be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:43:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff468d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafec09b940 con 0x7fafec0a3ff0 2024-09-16T04:43:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff468d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff468d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fafec09ab00 con 0x7fafec0a3c10 2024-09-16T04:43:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff468d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff468d640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fafec09b030 con 0x7fafec13be90 2024-09-16T04:43:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff468d640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:44.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff468d640 10 monclient(hunting): authenticate will time out at 1023.053581s 2024-09-16T04:43:44.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff2429640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fafec0a3c10 0x7fafec09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff2429640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fafec0a3c10 0x7fafec09bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56216/0 (socket says 172.21.15.46:56216) 2024-09-16T04:43:44.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff2429640 1 -- 172.21.15.46:0/262216535 learned_addr learned my addr 172.21.15.46:0/262216535 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:44.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff2429640 10 monclient(hunting): get_auth_request con 0x7fafec0a3c10 auth_method 0 2024-09-16T04:43:44.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff2429640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff2429640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff2429640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff2429640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:44.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff2429640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:44.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff2429640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:44.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff2c2a640 1 --2- 172.21.15.46:0/262216535 >> v2:172.21.15.114:3300/0 conn(0x7fafec13be90 0x7fafec13e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff2c2a640 10 monclient(hunting): get_auth_request con 0x7fafec13be90 auth_method 0 2024-09-16T04:43:44.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff2c2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff2c2a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff2c2a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff2429640 10 monclient(hunting): handle_auth_done global_id 44715 payload 274 2024-09-16T04:43:44.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff2429640 1 -- 172.21.15.46:0/262216535 >> v2:172.21.15.114:3300/0 conn(0x7fafec13be90 msgr2=0x7fafec13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:43:44.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff2429640 1 --2- 172.21.15.46:0/262216535 >> v2:172.21.15.114:3300/0 conn(0x7fafec13be90 0x7fafec13e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.638+0000 7faff2429640 1 -- 172.21.15.46:0/262216535 >> v1:172.21.15.114:6789/0 conn(0x7fafec0a3ff0 legacy=0x7fafec0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:43:44.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff2429640 10 monclient: _finish_hunting 0 2024-09-16T04:43:44.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff2429640 1 monclient: found mon.noname-a 2024-09-16T04:43:44.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff2429640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:44.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff2429640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:43:44.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff2429640 1 -- 172.21.15.46:0/262216535 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafec09b310 con 0x7fafec0a3c10 2024-09-16T04:43:44.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff2429640 1 --2- 172.21.15.46:0/262216535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafec0a3c10 0x7fafec09bcf0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fafec06ee70 tx=0x7fafe402a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38f179b8071714e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:44.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff1427640 1 -- 172.21.15.46:0/262216535 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe4037070 con 0x7fafec0a3c10 2024-09-16T04:43:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff1427640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff1427640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:43:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff1427640 10 monclient: dump: 2024-09-16T04:43:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.651 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff1427640 10 monclient: _finish_auth 0 2024-09-16T04:43:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff1427640 10 monclient: _check_auth_tickets 2024-09-16T04:43:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff1427640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff1427640 1 -- 172.21.15.46:0/262216535 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafe4032440 con 0x7fafec0a3c10 2024-09-16T04:43:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff468d640 5 monclient: authenticate success, global_id 44715 2024-09-16T04:43:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff468d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff1427640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff468d640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff1427640 1 -- 172.21.15.46:0/262216535 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe4032810 con 0x7fafec0a3c10 2024-09-16T04:43:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff468d640 4 set_mon_vals no callback set 2024-09-16T04:43:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff1427640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff1427640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.639+0000 7faff1427640 10 monclient: dump: 2024-09-16T04:43:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.653 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.640+0000 7faff468d640 10 monclient: shutdown 2024-09-16T04:43:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.640+0000 7faff468d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.640+0000 7faff468d640 1 -- 172.21.15.46:0/262216535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafec0a3c10 msgr2=0x7fafec09bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.640+0000 7faff468d640 1 --2- 172.21.15.46:0/262216535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafec0a3c10 0x7fafec09bcf0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fafec06ee70 tx=0x7fafe402a150 comp rx=0 tx=0).stop 2024-09-16T04:43:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.640+0000 7faff468d640 1 -- 172.21.15.46:0/262216535 shutdown_connections 2024-09-16T04:43:44.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.640+0000 7faff468d640 1 --2- 172.21.15.46:0/262216535 >> v2:172.21.15.114:3300/0 conn(0x7fafec13be90 0x7fafec13e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.640+0000 7faff468d640 1 --2- 172.21.15.46:0/262216535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafec0a3c10 0x7fafec09bcf0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.640+0000 7faff468d640 1 -- 172.21.15.46:0/262216535 >> 172.21.15.46:0/262216535 conn(0x7fafec0a6340 msgr2=0x7fafec06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:44.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.640+0000 7faff468d640 1 -- 172.21.15.46:0/262216535 shutdown_connections 2024-09-16T04:43:44.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.640+0000 7faff468d640 1 -- 172.21.15.46:0/262216535 wait complete. 2024-09-16T04:43:44.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff468d640 10 monclient: build_initial_monmap 2024-09-16T04:43:44.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff468d640 10 monclient: monmap: 2024-09-16T04:43:44.654 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:44.654 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:44.654 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:44.654 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:44.654 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:44.654 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.655 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:44.655 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:44.655 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:44.655 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff468d640 1 Processor -- start 2024-09-16T04:43:44.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff468d640 1 -- start start 2024-09-16T04:43:44.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff468d640 10 monclient: init 2024-09-16T04:43:44.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff468d640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:44.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff468d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:43:44.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff468d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fafec0a3c10 0x7fafec1cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff468d640 10 monclient(hunting): picked mon.noname-c con 0x7fafec0a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:44.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff468d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafec0a3ff0 0x7fafec1cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff468d640 10 monclient(hunting): picked mon.noname-b con 0x7fafec0a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff468d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafec13be90 0x7fafec1cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff468d640 10 monclient(hunting): picked mon.noname-a con 0x7fafec13be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff468d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff468d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fafec0a6b40 con 0x7fafec13be90 2024-09-16T04:43:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff468d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff468d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fafec0a69c0 con 0x7fafec0a3ff0 2024-09-16T04:43:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff468d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff468d640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fafec13e7c0 con 0x7fafec0a3c10 2024-09-16T04:43:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff468d640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff468d640 10 monclient(hunting): authenticate will time out at 1023.056912s 2024-09-16T04:43:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.641+0000 7faff2c2a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafec13be90 0x7fafec1cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2c2a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafec13be90 0x7fafec1cc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56224/0 (socket says 172.21.15.46:56224) 2024-09-16T04:43:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2429640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fafec0a3c10 0x7fafec1cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2c2a640 1 -- 172.21.15.46:0/4037072279 learned_addr learned my addr 172.21.15.46:0/4037072279 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2c2a640 10 monclient(hunting): get_auth_request con 0x7fafec13be90 auth_method 0 2024-09-16T04:43:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2c2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2c2a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2c2a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2429640 10 monclient(hunting): get_auth_request con 0x7fafec0a3c10 auth_method 0 2024-09-16T04:43:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2429640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2429640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2429640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff1c28640 1 --2- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafec0a3ff0 0x7fafec1cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2c2a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2c2a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2c2a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2429640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:44.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2429640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:44.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2429640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:44.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff1c28640 10 monclient(hunting): get_auth_request con 0x7fafec0a3ff0 auth_method 0 2024-09-16T04:43:44.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff1c28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff1c28640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff1c28640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff1c28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:44.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff1c28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:44.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff1c28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:44.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2c2a640 10 monclient(hunting): handle_auth_done global_id 44718 payload 824 2024-09-16T04:43:44.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2c2a640 1 -- 172.21.15.46:0/4037072279 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fafec0a3c10 msgr2=0x7fafec1cb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2c2a640 1 --2- 172.21.15.46:0/4037072279 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fafec0a3c10 0x7fafec1cb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2c2a640 1 -- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafec0a3ff0 msgr2=0x7fafec1cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2c2a640 1 --2- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafec0a3ff0 0x7fafec1cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2c2a640 10 monclient: _finish_hunting 0 2024-09-16T04:43:44.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2c2a640 1 monclient: found mon.noname-a 2024-09-16T04:43:44.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2c2a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:44.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2c2a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:43:44.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2c2a640 1 -- 172.21.15.46:0/4037072279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafec13a0b0 con 0x7fafec13be90 2024-09-16T04:43:44.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2429640 1 --2- 172.21.15.46:0/4037072279 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fafec0a3c10 0x7fafec1cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:43:44.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff2c2a640 1 --2- 172.21.15.46:0/4037072279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafec13be90 0x7fafec1cc3c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fafdc007d90 tx=0x7fafdc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:44.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7fafeaffd640 1 -- 172.21.15.46:0/4037072279 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafdc014420 con 0x7fafec13be90 2024-09-16T04:43:44.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7fafeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7fafeaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:43:44.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7fafeaffd640 10 monclient: dump: 2024-09-16T04:43:44.659 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.660 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.660 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.660 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.660 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.660 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.660 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.660 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.660 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.660 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7fafeaffd640 10 monclient: _finish_auth 0 2024-09-16T04:43:44.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7fafeaffd640 10 monclient: _check_auth_tickets 2024-09-16T04:43:44.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7fafeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:44.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7fafeaffd640 1 -- 172.21.15.46:0/4037072279 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafdc019070 con 0x7fafec13be90 2024-09-16T04:43:44.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff468d640 5 monclient: authenticate success, global_id 44718 2024-09-16T04:43:44.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7fafeaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:44.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7fafeaffd640 1 -- 172.21.15.46:0/4037072279 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafdc014820 con 0x7fafec13be90 2024-09-16T04:43:44.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7fafeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7fafeaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:43:44.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7fafeaffd640 10 monclient: dump: 2024-09-16T04:43:44.661 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.661 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.661 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.661 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.661 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.661 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.661 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.662 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff468d640 10 monclient: _renew_subs 2024-09-16T04:43:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff468d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff468d640 1 -- 172.21.15.46:0/4037072279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafec138ee0 con 0x7fafec13be90 2024-09-16T04:43:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff0c26640 4 set_mon_vals no callback set 2024-09-16T04:43:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff468d640 10 monclient: _renew_subs 2024-09-16T04:43:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff468d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:43:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.642+0000 7faff468d640 1 -- 172.21.15.46:0/4037072279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafec139dd0 con 0x7fafec13be90 2024-09-16T04:43:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.643+0000 7fafeaffd640 1 -- 172.21.15.46:0/4037072279 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fafdc0149c0 con 0x7fafec13be90 2024-09-16T04:43:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.643+0000 7faff468d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.643+0000 7faff468d640 1 --2- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafec1339a0 0x7fafec133d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.643+0000 7faff468d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.643+0000 7faff468d640 1 -- 172.21.15.46:0/4037072279 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fafec09eb40 con 0x7fafec1339a0 2024-09-16T04:43:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.643+0000 7fafeaffd640 1 --2- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fafd80383b0 0x7fafd803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.643+0000 7fafeaffd640 1 -- 172.21.15.46:0/4037072279 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 4738+0+0 (secure 0 0 0) 0x7fafdc04bfe0 con 0x7fafec13be90 2024-09-16T04:43:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.643+0000 7faff2429640 1 --2- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafec1339a0 0x7fafec133d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.643+0000 7faff2429640 10 monclient: get_auth_request con 0x7fafec1339a0 auth_method 0 2024-09-16T04:43:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.643+0000 7faff2429640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.643+0000 7faff2429640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.643+0000 7faff2429640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.643+0000 7faff1c28640 1 --2- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fafd80383b0 0x7fafd803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.643+0000 7faff1c28640 10 monclient: get_auth_request con 0x7fafd80383b0 auth_method 0 2024-09-16T04:43:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.644+0000 7faff2429640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.644+0000 7faff2429640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.644+0000 7faff2429640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.644+0000 7faff2429640 10 monclient(hunting): handle_auth_done global_id 5980 payload 824 2024-09-16T04:43:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.644+0000 7faff2429640 1 -- 172.21.15.46:0/4037072279 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fafec13c810 con 0x7fafec1339a0 2024-09-16T04:43:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.644+0000 7faff2429640 1 --2- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafec1339a0 0x7fafec133d80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fafec0a38d0 tx=0x7fafe40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.644+0000 7faff1c28640 1 --2- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fafd80383b0 0x7fafd803a870 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fafec1cca00 tx=0x7fafe0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.644+0000 7fafeaffd640 1 -- 172.21.15.46:0/4037072279 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe4043070 con 0x7fafec1339a0 2024-09-16T04:43:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.644+0000 7fafeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.644+0000 7fafeaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:43:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.644+0000 7fafeaffd640 10 monclient: dump: 2024-09-16T04:43:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.665 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.644+0000 7fafeaffd640 1 -- 172.21.15.46:0/4037072279 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fafe4032440 con 0x7fafec1339a0 2024-09-16T04:43:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.644+0000 7fafeaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.644+0000 7fafeaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.644+0000 7fafeaffd640 1 -- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafec1339a0 msgr2=0x7fafec133d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.644+0000 7fafeaffd640 1 --2- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafec1339a0 0x7fafec133d80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fafec0a38d0 tx=0x7fafe40046c0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.644+0000 7fafeaffd640 1 -- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafec1339a0 msgr2=0x7fafec133d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:43:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.644+0000 7fafeaffd640 1 --2- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafec1339a0 0x7fafec133d80 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.651+0000 7faff468d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:43:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.651+0000 7faff468d640 1 --2- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafec1342c0 0x7fafec0a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.651+0000 7faff468d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.651+0000 7faff468d640 1 -- 172.21.15.46:0/4037072279 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fafec138c60 con 0x7fafec1342c0 2024-09-16T04:43:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.651+0000 7faff2429640 1 --2- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafec1342c0 0x7fafec0a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.651+0000 7faff2429640 10 monclient: get_auth_request con 0x7fafec1342c0 auth_method 0 2024-09-16T04:43:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.651+0000 7faff2429640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.651+0000 7faff2429640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.651+0000 7faff2429640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.651+0000 7faff2429640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.651+0000 7faff2429640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.651+0000 7faff2429640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.651+0000 7faff2429640 10 monclient(hunting): handle_auth_done global_id 5983 payload 824 2024-09-16T04:43:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.651+0000 7faff2429640 1 -- 172.21.15.46:0/4037072279 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fafec067e20 con 0x7fafec1342c0 2024-09-16T04:43:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.651+0000 7faff2429640 1 --2- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafec1342c0 0x7fafec0a0460 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fafe402a190 tx=0x7fafe4008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.651+0000 7fafeaffd640 1 -- 172.21.15.46:0/4037072279 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe4037040 con 0x7fafec1342c0 2024-09-16T04:43:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.651+0000 7fafeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.651+0000 7fafeaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:43:44.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.651+0000 7fafeaffd640 10 monclient: dump: 2024-09-16T04:43:44.669 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.669 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.669 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.669 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.669 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.669 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.669 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.669 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.669 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.669 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.652+0000 7fafeaffd640 1 -- 172.21.15.46:0/4037072279 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7fafe40039a0 con 0x7fafec1342c0 2024-09-16T04:43:44.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.652+0000 7fafeaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:43:44.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.652+0000 7fafeaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:44.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.652+0000 7fafeaffd640 1 -- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafec1342c0 msgr2=0x7fafec0a0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.652+0000 7fafeaffd640 1 --2- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafec1342c0 0x7fafec0a0460 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fafe402a190 tx=0x7fafe4008030 comp rx=0 tx=0).stop 2024-09-16T04:43:44.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.652+0000 7fafeaffd640 1 -- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafec1342c0 msgr2=0x7fafec0a0460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:43:44.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.652+0000 7fafeaffd640 1 --2- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafec1342c0 0x7fafec0a0460 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.670 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:43:44.670 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:43:44.670 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:43:44.671 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:43:44.671 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 58, 2024-09-16T04:43:44.671 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 310553, 2024-09-16T04:43:44.671 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:43:44.671 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:43:44.671 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:43:44.671 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:43:44.671 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:44.671 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 5, 2024-09-16T04:43:44.671 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:43:44.671 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:43:44.671 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:43:44.672 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:44.672 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:44.672 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:44.672 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:44.672 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:44.672 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:44.672 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:44.672 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:44.672 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:44.672 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:44.672 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:44.672 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:44.672 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:43:44.673 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:43:44.673 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:44.673 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:44.673 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:44.673 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:44.673 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:44.673 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:44.673 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:44.673 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:44.673 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:44.673 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:44.673 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:44.673 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:44.673 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.674 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:43:44.674 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:43:44.674 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:43:44.674 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:43:44.674 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:43:44.674 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:43:44.674 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:43:44.674 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:43:44.674 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:43:44.674 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:43:44.674 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:43:44.674 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:43:44.674 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:43:44.674 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:43:44.675 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:43:44.675 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:43:44.675 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:43:44.675 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:44.675 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:44.675 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:44.675 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:44.675 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:44.675 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:44.675 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:44.675 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:44.675 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:44.675 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:44.675 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:44.675 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:44.676 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:43:44.676 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.676 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:43:44.676 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.676 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:43:44.676 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:43:44.676 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:44.676 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:44.676 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.676 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:44.676 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:43:44.676 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:44.676 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.676 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.677 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:44.677 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:43:44.677 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:44.677 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:44.677 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:44.677 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.677 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:43:44.677 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:43:44.677 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:44.677 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:44.677 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:44.677 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.677 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.678 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:43:44.678 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:43:44.678 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:44.678 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:44.678 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.678 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:44.678 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:43:44.678 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:44.678 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.678 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.678 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:44.678 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:43:44.678 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:44.678 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:44.678 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:44.679 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.679 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:43:44.679 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:43:44.679 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:44.679 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:44.679 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:44.679 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.679 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.679 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:43:44.679 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:43:44.679 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:44.679 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:44.679 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.679 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:44.680 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:43:44.680 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:44.680 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.680 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.680 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:44.680 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:43:44.680 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:44.680 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:44.680 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:44.680 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.681 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:43:44.681 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:43:44.681 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:44.681 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:44.681 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:44.681 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:44.681 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:44.681 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.681 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:43:44.681 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:43:44.681 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.681 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:44.681 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:44.682 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:43:44.682 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:44.682 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:44.682 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:43:44.682 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.682 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:44.682 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:44.682 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4 2024-09-16T04:43:44.682 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:44.682 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:44.682 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:43:44.682 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.682 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:44.682 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:44.683 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 7 2024-09-16T04:43:44.683 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:44.683 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:44.683 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:43:44.683 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:44.683 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:44.683 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:44.683 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:43:44.683 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:44.683 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:44.683 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:44.683 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:43:44.683 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:43:44.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.652+0000 7faff468d640 1 -- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fafd80383b0 msgr2=0x7fafd803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.652+0000 7faff468d640 1 --2- 172.21.15.46:0/4037072279 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fafd80383b0 0x7fafd803a870 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fafec1cca00 tx=0x7fafe0009040 comp rx=0 tx=0).stop 2024-09-16T04:43:44.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.652+0000 7faff468d640 10 monclient: shutdown 2024-09-16T04:43:44.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.652+0000 7faff468d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:44.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.652+0000 7faff468d640 1 -- 172.21.15.46:0/4037072279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafec13be90 msgr2=0x7fafec1cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.652+0000 7faff468d640 1 --2- 172.21.15.46:0/4037072279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafec13be90 0x7fafec1cc3c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fafdc007d90 tx=0x7fafdc00e4b0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.652+0000 7faff2c2a640 1 -- 172.21.15.46:0/4037072279 reap_dead start 2024-09-16T04:43:44.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.652+0000 7faff2c2a640 1 -- 172.21.15.46:0/4037072279 reap_dead start 2024-09-16T04:43:44.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.652+0000 7faff468d640 1 -- 172.21.15.46:0/4037072279 shutdown_connections 2024-09-16T04:43:44.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.652+0000 7faff468d640 1 -- 172.21.15.46:0/4037072279 >> 172.21.15.46:0/4037072279 conn(0x7fafec0a6340 msgr2=0x7fafec13db00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:44.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.652+0000 7faff468d640 1 -- 172.21.15.46:0/4037072279 shutdown_connections 2024-09-16T04:43:44.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.652+0000 7faff468d640 1 -- 172.21.15.46:0/4037072279 wait complete. 2024-09-16T04:43:44.686 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:43:44.686 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T04:43:44.686 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:43:44.686 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T04:43:44.686 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:43:44.687 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T04:43:44.687 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:43:44.698 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T04:43:44.699 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:43:44.699 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:43:44.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.838+0000 7fa4cb00e640 10 monclient: get_monmap_and_config 2024-09-16T04:43:44.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.838+0000 7fa4cb00e640 10 monclient: build_initial_monmap 2024-09-16T04:43:44.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.838+0000 7fa4cb00e640 10 monclient: monmap: 2024-09-16T04:43:44.840 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:44.840 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.840 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:44.839058+0000 2024-09-16T04:43:44.840 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:44.839058+0000 2024-09-16T04:43:44.840 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:44.840 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.840 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:44.840 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:44.840 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:44.840 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:44.840 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:44.840 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.839+0000 7fa4cb00e640 1 Processor -- start 2024-09-16T04:43:44.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.839+0000 7fa4cb00e640 1 -- start start 2024-09-16T04:43:44.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.839+0000 7fa4cb00e640 10 monclient: init 2024-09-16T04:43:44.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.839+0000 7fa4cb00e640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:44.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.839+0000 7fa4cb00e640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-16T04:43:44.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.839+0000 7fa4cb00e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa4c409ed50 0x7fa4c4136df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.839+0000 7fa4cb00e640 10 monclient(hunting): picked mon.noname-c con 0x7fa4c409ed50 addr v2:172.21.15.114:3300/0 2024-09-16T04:43:44.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.839+0000 7fa4cb00e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4c409f130 addr v1:172.21.15.46:6789/0 2024-09-16T04:43:44.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.839+0000 7fa4cb00e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa4c409f560 0x7fa4c40a0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.839+0000 7fa4cb00e640 10 monclient(hunting): picked mon.noname-a con 0x7fa4c409f560 addr v2:172.21.15.46:3300/0 2024-09-16T04:43:44.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.839+0000 7fa4cb00e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4c409cd50 con 0x7fa4c409f130 2024-09-16T04:43:44.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.839+0000 7fa4cb00e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.839+0000 7fa4cb00e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa4c409d680 con 0x7fa4c409f560 2024-09-16T04:43:44.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.839+0000 7fa4cb00e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.839+0000 7fa4cb00e640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fa4c40736d0 con 0x7fa4c409ed50 2024-09-16T04:43:44.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.839+0000 7fa4cb00e640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:44.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.839+0000 7fa4cb00e640 10 monclient(hunting): authenticate will time out at 1023.254886s 2024-09-16T04:43:44.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.839+0000 7fa4bbfff640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7fa4c409f130 legacy=0x7fa4c40a0350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:43:44.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.839+0000 7fa4c95ab640 1 -- >> v2:172.21.15.46:3300/0 conn(0x7fa4c409f560 msgr2=0x7fa4c40a0a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:43:44.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.839+0000 7fa4c95ab640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa4c409f560 0x7fa4c40a0a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.840+0000 7fa4c8daa640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa4c409ed50 0x7fa4c4136df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.840+0000 7fa4c8daa640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa4c409ed50 0x7fa4c4136df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.46:49698/0 (socket says 172.21.15.46:49698) 2024-09-16T04:43:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.840+0000 7fa4c8daa640 1 -- 172.21.15.46:0/2858053852 learned_addr learned my addr 172.21.15.46:0/2858053852 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.840+0000 7fa4c8daa640 10 monclient(hunting): get_auth_request con 0x7fa4c409ed50 auth_method 0 2024-09-16T04:43:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.840+0000 7fa4c8daa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.840+0000 7fa4c8daa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.840+0000 7fa4c8daa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.840+0000 7fa4c8daa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.840+0000 7fa4c8daa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.840+0000 7fa4c8daa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.840+0000 7fa4c8daa640 10 monclient(hunting): handle_auth_done global_id 5986 payload 274 2024-09-16T04:43:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.840+0000 7fa4c8daa640 1 -- 172.21.15.46:0/2858053852 >> v2:172.21.15.46:3300/0 conn(0x7fa4c409f560 msgr2=0x7fa4c40a0a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.840+0000 7fa4c8daa640 1 --2- 172.21.15.46:0/2858053852 >> v2:172.21.15.46:3300/0 conn(0x7fa4c409f560 0x7fa4c40a0a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.840+0000 7fa4c8daa640 1 -- 172.21.15.46:0/2858053852 >> v1:172.21.15.46:6789/0 conn(0x7fa4c409f130 legacy=0x7fa4c40a0350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.840+0000 7fa4c8daa640 10 monclient: _finish_hunting 0 2024-09-16T04:43:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.840+0000 7fa4c8daa640 1 monclient: found mon.noname-c 2024-09-16T04:43:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.840+0000 7fa4c8daa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.840+0000 7fa4c8daa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:43:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.840+0000 7fa4c8daa640 1 -- 172.21.15.46:0/2858053852 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c4137330 con 0x7fa4c409ed50 2024-09-16T04:43:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.841+0000 7fa4c8daa640 1 --2- 172.21.15.46:0/2858053852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c409ed50 0x7fa4c4136df0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fa4c40a6550 tx=0x7fa4c002a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e11bebc2dbba66c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.841+0000 7fa4bb7fe640 1 -- 172.21.15.46:0/2858053852 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0037070 con 0x7fa4c409ed50 2024-09-16T04:43:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.841+0000 7fa4bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.841+0000 7fa4bb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:43:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.841+0000 7fa4bb7fe640 10 monclient: dump: 2024-09-16T04:43:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.852 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.841+0000 7fa4bb7fe640 10 monclient: _finish_auth 0 2024-09-16T04:43:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.841+0000 7fa4bb7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:43:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.841+0000 7fa4bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.841+0000 7fa4bb7fe640 1 -- 172.21.15.46:0/2858053852 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4c0032440 con 0x7fa4c409ed50 2024-09-16T04:43:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.841+0000 7fa4cb00e640 5 monclient: authenticate success, global_id 5986 2024-09-16T04:43:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.841+0000 7fa4cb00e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.841+0000 7fa4bb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.841+0000 7fa4cb00e640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.841+0000 7fa4bb7fe640 1 -- 172.21.15.46:0/2858053852 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0032810 con 0x7fa4c409ed50 2024-09-16T04:43:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.841+0000 7fa4bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.841+0000 7fa4cb00e640 4 set_mon_vals no callback set 2024-09-16T04:43:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.841+0000 7fa4bb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:43:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.841+0000 7fa4bb7fe640 10 monclient: dump: 2024-09-16T04:43:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.854 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.842+0000 7fa4cb00e640 10 monclient: shutdown 2024-09-16T04:43:44.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.842+0000 7fa4cb00e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:44.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.842+0000 7fa4cb00e640 1 -- 172.21.15.46:0/2858053852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c409ed50 msgr2=0x7fa4c4136df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.842+0000 7fa4cb00e640 1 --2- 172.21.15.46:0/2858053852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c409ed50 0x7fa4c4136df0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fa4c40a6550 tx=0x7fa4c002a1c0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.842+0000 7fa4cb00e640 1 -- 172.21.15.46:0/2858053852 shutdown_connections 2024-09-16T04:43:44.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.842+0000 7fa4cb00e640 1 --2- 172.21.15.46:0/2858053852 >> v2:172.21.15.46:3300/0 conn(0x7fa4c409f560 0x7fa4c40a0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.842+0000 7fa4cb00e640 1 --2- 172.21.15.46:0/2858053852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c409ed50 0x7fa4c4136df0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.842+0000 7fa4cb00e640 1 -- 172.21.15.46:0/2858053852 >> 172.21.15.46:0/2858053852 conn(0x7fa4c40a5950 msgr2=0x7fa4c40a5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:44.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.842+0000 7fa4cb00e640 1 -- 172.21.15.46:0/2858053852 shutdown_connections 2024-09-16T04:43:44.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.842+0000 7fa4cb00e640 1 -- 172.21.15.46:0/2858053852 wait complete. 2024-09-16T04:43:44.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.842+0000 7fa4cb00e640 10 monclient: build_initial_monmap 2024-09-16T04:43:44.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.842+0000 7fa4cb00e640 10 monclient: monmap: 2024-09-16T04:43:44.854 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:44.855 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:44.855 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:44.855 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:44.855 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:44.855 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.855 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:44.855 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:44.855 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:44.855 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.842+0000 7fa4cb00e640 1 Processor -- start 2024-09-16T04:43:44.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4cb00e640 1 -- start start 2024-09-16T04:43:44.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4cb00e640 10 monclient: init 2024-09-16T04:43:44.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4cb00e640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:44.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4cb00e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:43:44.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4cb00e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c409ed50 0x7fa4c4131190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4cb00e640 10 monclient(hunting): picked mon.noname-b con 0x7fa4c409ed50 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:44.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4cb00e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4c409f130 0x7fa4c412eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4cb00e640 10 monclient(hunting): picked mon.noname-a con 0x7fa4c409f130 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:44.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4cb00e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa4c409f560 0x7fa4c412f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4cb00e640 10 monclient(hunting): picked mon.noname-c con 0x7fa4c409f560 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:44.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4cb00e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4cb00e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c4067db0 con 0x7fa4c409f130 2024-09-16T04:43:44.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4cb00e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4cb00e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c4067c30 con 0x7fa4c409ed50 2024-09-16T04:43:44.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4cb00e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:44.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4cb00e640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fa4c40a0fa0 con 0x7fa4c409f560 2024-09-16T04:43:44.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4cb00e640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:44.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4cb00e640 10 monclient(hunting): authenticate will time out at 1023.258627s 2024-09-16T04:43:44.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4bbfff640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4c409f130 msgr2=0x7fa4c412eb40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:43:44.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4bbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4c409f130 0x7fa4c412eb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:44.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4c95ab640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa4c409f560 0x7fa4c412f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4c95ab640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa4c409f560 0x7fa4c412f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:52446/0 (socket says 172.21.15.46:52446) 2024-09-16T04:43:44.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4c95ab640 1 -- 172.21.15.46:0/1951300989 learned_addr learned my addr 172.21.15.46:0/1951300989 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:44.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4c95ab640 10 monclient(hunting): get_auth_request con 0x7fa4c409f560 auth_method 0 2024-09-16T04:43:44.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4c95ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4c95ab640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4c95ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4c8daa640 1 --2- 172.21.15.46:0/1951300989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c409ed50 0x7fa4c4131190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4c95ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:44.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4c95ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:44.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4c95ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:44.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4c8daa640 10 monclient(hunting): get_auth_request con 0x7fa4c409ed50 auth_method 0 2024-09-16T04:43:44.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4c8daa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:44.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4c8daa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:44.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4c8daa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:44.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4c8daa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:44.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4c8daa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:44.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.843+0000 7fa4c8daa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:44.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4c95ab640 10 monclient(hunting): handle_auth_done global_id 64142 payload 824 2024-09-16T04:43:44.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4c95ab640 1 -- 172.21.15.46:0/1951300989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c409ed50 msgr2=0x7fa4c4131190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:44.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4c95ab640 1 --2- 172.21.15.46:0/1951300989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c409ed50 0x7fa4c4131190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4c95ab640 1 -- 172.21.15.46:0/1951300989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4c409f130 msgr2=0x7fa4c412eb40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:43:44.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4c95ab640 1 --2- 172.21.15.46:0/1951300989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4c409f130 0x7fa4c412eb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:44.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4c95ab640 10 monclient: _finish_hunting 0 2024-09-16T04:43:44.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4c95ab640 1 monclient: found mon.noname-c 2024-09-16T04:43:44.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4c95ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:44.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4c95ab640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:43:44.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4c95ab640 1 -- 172.21.15.46:0/1951300989 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c40a4000 con 0x7fa4c409f560 2024-09-16T04:43:44.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4c95ab640 1 --2- 172.21.15.46:0/1951300989 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa4c409f560 0x7fa4c412f080 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc00be60 tx=0x7fa4bc00bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:44.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4b97fa640 1 -- 172.21.15.46:0/1951300989 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc015470 con 0x7fa4c409f560 2024-09-16T04:43:44.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4b97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:43:44.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4b97fa640 10 monclient: dump: 2024-09-16T04:43:44.859 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.859 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.859 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.859 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.859 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.859 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.859 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.860 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.860 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.860 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4b97fa640 10 monclient: _finish_auth 0 2024-09-16T04:43:44.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4b97fa640 10 monclient: _check_auth_tickets 2024-09-16T04:43:44.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:44.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4b97fa640 1 -- 172.21.15.46:0/1951300989 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4bc010070 con 0x7fa4c409f560 2024-09-16T04:43:44.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4cb00e640 5 monclient: authenticate success, global_id 64142 2024-09-16T04:43:44.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4cb00e640 10 monclient: _renew_subs 2024-09-16T04:43:44.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4cb00e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:44.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4cb00e640 1 -- 172.21.15.46:0/1951300989 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c412fdf0 con 0x7fa4c409f560 2024-09-16T04:43:44.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4b97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:44.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4b97fa640 1 -- 172.21.15.46:0/1951300989 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc015890 con 0x7fa4c409f560 2024-09-16T04:43:44.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:44.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4b97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:43:44.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4b97fa640 10 monclient: dump: 2024-09-16T04:43:44.861 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:44.861 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:44.861 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.861 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:44.861 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:44.861 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:44.861 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:44.861 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:44.861 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:44.861 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:44.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4baffd640 4 set_mon_vals no callback set 2024-09-16T04:43:44.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4cb00e640 10 monclient: _renew_subs 2024-09-16T04:43:44.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4cb00e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:44.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.844+0000 7fa4cb00e640 1 -- 172.21.15.46:0/1951300989 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4c4132990 con 0x7fa4c409f560 2024-09-16T04:43:44.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.845+0000 7fa4b97fa640 1 -- 172.21.15.46:0/1951300989 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fa4bc01e3f0 con 0x7fa4c409f560 2024-09-16T04:43:44.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.845+0000 7fa4cb00e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:44.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.845+0000 7fa4cb00e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:44.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.845+0000 7fa4cb00e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:44.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.845+0000 7fa4cb00e640 1 -- 172.21.15.46:0/1951300989 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4c4136df0 con 0x7fa4c409f560 2024-09-16T04:43:44.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.845+0000 7fa4b97fa640 1 --2- 172.21.15.46:0/1951300989 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa4b0038360 0x7fa4b003a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:44.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.845+0000 7fa4b97fa640 1 -- 172.21.15.46:0/1951300989 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(195..195 src has 1..195) ==== 4738+0+0 (secure 0 0 0) 0x7fa4bc04bd20 con 0x7fa4c409f560 2024-09-16T04:43:44.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.845+0000 7fa4c8daa640 1 --2- 172.21.15.46:0/1951300989 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa4b0038360 0x7fa4b003a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:44.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.846+0000 7fa4c8daa640 10 monclient: get_auth_request con 0x7fa4b0038360 auth_method 0 2024-09-16T04:43:44.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.846+0000 7fa4c8daa640 1 --2- 172.21.15.46:0/1951300989 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa4b0038360 0x7fa4b003a820 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0034000 tx=0x7fa4c002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:44.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.850+0000 7fa4b97fa640 1 -- 172.21.15.46:0/1951300989 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa4bc050070 con 0x7fa4c409f560 2024-09-16T04:43:44.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.850+0000 7fa4b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:44.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:44.850+0000 7fa4b97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:45.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:45.092+0000 7fa4cb00e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:43:45.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:45.092+0000 7fa4cb00e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:45.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:45.092+0000 7fa4cb00e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:45.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:45.092+0000 7fa4cb00e640 1 -- 172.21.15.46:0/1951300989 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa4c4068840 con 0x7fa4c409f560 2024-09-16T04:43:45.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:45.092+0000 7fa4b97fa640 1 -- 172.21.15.46:0/1951300989 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fa4bc013770 con 0x7fa4c409f560 2024-09-16T04:43:45.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:45.092+0000 7fa4b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:45.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:45.092+0000 7fa4b97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:45.094 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:43:45.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:45.094+0000 7fa4cb00e640 1 -- 172.21.15.46:0/1951300989 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa4b0038360 msgr2=0x7fa4b003a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:45.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:45.094+0000 7fa4cb00e640 1 --2- 172.21.15.46:0/1951300989 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa4b0038360 0x7fa4b003a820 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0034000 tx=0x7fa4c002e000 comp rx=0 tx=0).stop 2024-09-16T04:43:45.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:45.095+0000 7fa4cb00e640 10 monclient: shutdown 2024-09-16T04:43:45.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:45.095+0000 7fa4cb00e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:45.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:45.095+0000 7fa4cb00e640 1 -- 172.21.15.46:0/1951300989 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa4c409f560 msgr2=0x7fa4c412f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:45.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:45.095+0000 7fa4cb00e640 1 --2- 172.21.15.46:0/1951300989 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa4c409f560 0x7fa4c412f080 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc00be60 tx=0x7fa4bc00bf50 comp rx=0 tx=0).stop 2024-09-16T04:43:45.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:45.095+0000 7fa4cb00e640 1 -- 172.21.15.46:0/1951300989 shutdown_connections 2024-09-16T04:43:45.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:45.095+0000 7fa4cb00e640 1 --2- 172.21.15.46:0/1951300989 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa4c409f560 0x7fa4c412f080 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:45.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:45.095+0000 7fa4cb00e640 1 --2- 172.21.15.46:0/1951300989 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa4b0038360 0x7fa4b003a820 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:45.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:45.095+0000 7fa4cb00e640 1 --2- 172.21.15.46:0/1951300989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4c409f130 0x7fa4c412eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:45.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:45.095+0000 7fa4cb00e640 1 --2- 172.21.15.46:0/1951300989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c409ed50 0x7fa4c4131190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:45.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:45.095+0000 7fa4cb00e640 1 -- 172.21.15.46:0/1951300989 >> 172.21.15.46:0/1951300989 conn(0x7fa4c40a5950 msgr2=0x7fa4c4135340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:45.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:45.095+0000 7fa4cb00e640 1 -- 172.21.15.46:0/1951300989 shutdown_connections 2024-09-16T04:43:45.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:45.095+0000 7fa4cb00e640 1 -- 172.21.15.46:0/1951300989 wait complete. 2024-09-16T04:43:48.108 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:43:48.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.220+0000 7fca3fdaf640 10 monclient: get_monmap_and_config 2024-09-16T04:43:48.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.220+0000 7fca3fdaf640 10 monclient: build_initial_monmap 2024-09-16T04:43:48.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.220+0000 7fca3fdaf640 10 monclient: monmap: 2024-09-16T04:43:48.221 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:48.221 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:48.221 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:48.220692+0000 2024-09-16T04:43:48.221 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:48.220692+0000 2024-09-16T04:43:48.222 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:48.222 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:48.222 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:48.222 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:48.222 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:48.222 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:48.222 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:48.222 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:48.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.220+0000 7fca3fdaf640 1 Processor -- start 2024-09-16T04:43:48.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3fdaf640 1 -- start start 2024-09-16T04:43:48.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3fdaf640 10 monclient: init 2024-09-16T04:43:48.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3fdaf640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:48.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3fdaf640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-16T04:43:48.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3fdaf640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fca380a3c10 0x7fca3809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:48.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3fdaf640 10 monclient(hunting): picked mon.noname-c con 0x7fca380a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:43:48.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3fdaf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fca380a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:43:48.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3fdaf640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fca3813be90 0x7fca3813e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:48.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3fdaf640 10 monclient(hunting): picked mon.noname-a con 0x7fca3813be90 addr v2:172.21.15.46:3300/0 2024-09-16T04:43:48.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3fdaf640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca3809b940 con 0x7fca380a3ff0 2024-09-16T04:43:48.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3fdaf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:48.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3fdaf640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fca3809ab00 con 0x7fca3813be90 2024-09-16T04:43:48.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3fdaf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:48.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3fdaf640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fca3809b030 con 0x7fca380a3c10 2024-09-16T04:43:48.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3fdaf640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:48.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3fdaf640 10 monclient(hunting): authenticate will time out at 1026.636618s 2024-09-16T04:43:48.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3e34c640 1 -- >> v2:172.21.15.46:3300/0 conn(0x7fca3813be90 msgr2=0x7fca3813e280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:43:48.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3e34c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fca3813be90 0x7fca3813e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:48.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3d34a640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fca380a3ff0 0x7fca380a43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.46:34930/0 (socket says 172.21.15.46:34930) 2024-09-16T04:43:48.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3db4b640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fca380a3c10 0x7fca3809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:48.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3d34a640 1 -- 172.21.15.46:0/4161499447 learned_addr learned my addr 172.21.15.46:0/4161499447 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:43:48.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.221+0000 7fca3db4b640 10 monclient(hunting): get_auth_request con 0x7fca380a3c10 auth_method 0 2024-09-16T04:43:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.222+0000 7fca3db4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.222+0000 7fca3db4b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.222+0000 7fca3db4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.222+0000 7fca3db4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.222+0000 7fca3db4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.222+0000 7fca3db4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.222+0000 7fca3cb49640 1 -- 172.21.15.46:0/4161499447 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3082829593 0 0) 0x7fca3809b940 con 0x7fca380a3ff0 2024-09-16T04:43:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.222+0000 7fca3cb49640 10 monclient(hunting): _negotiate 2024-09-16T04:43:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.222+0000 7fca3cb49640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.222+0000 7fca3cb49640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.222+0000 7fca3cb49640 10 monclient(hunting): my global_id is 5995 2024-09-16T04:43:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.222+0000 7fca3cb49640 1 -- 172.21.15.46:0/4161499447 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca24003520 con 0x7fca380a3ff0 2024-09-16T04:43:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.222+0000 7fca3db4b640 10 monclient(hunting): handle_auth_done global_id 5992 payload 274 2024-09-16T04:43:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.222+0000 7fca3db4b640 1 -- 172.21.15.46:0/4161499447 >> v2:172.21.15.46:3300/0 conn(0x7fca3813be90 msgr2=0x7fca3813e280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.222+0000 7fca3db4b640 1 --2- 172.21.15.46:0/4161499447 >> v2:172.21.15.46:3300/0 conn(0x7fca3813be90 0x7fca3813e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.222+0000 7fca3db4b640 1 -- 172.21.15.46:0/4161499447 >> v1:172.21.15.114:6789/0 conn(0x7fca380a3ff0 legacy=0x7fca380a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.222+0000 7fca3db4b640 10 monclient: _finish_hunting 0 2024-09-16T04:43:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.222+0000 7fca3db4b640 1 monclient: found mon.noname-c 2024-09-16T04:43:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.222+0000 7fca3db4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.222+0000 7fca3db4b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:43:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.222+0000 7fca3db4b640 1 -- 172.21.15.46:0/4161499447 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca3809b310 con 0x7fca380a3c10 2024-09-16T04:43:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3db4b640 1 --2- 172.21.15.46:0/4161499447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca380a3c10 0x7fca3809bcf0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fca38068240 tx=0x7fca3402a0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f93cb8a61e0e656c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3cb49640 1 -- 172.21.15.46:0/4161499447 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca34037070 con 0x7fca380a3c10 2024-09-16T04:43:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3cb49640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3cb49640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:43:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3cb49640 10 monclient: dump: 2024-09-16T04:43:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:48.234 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3cb49640 10 monclient: _finish_auth 0 2024-09-16T04:43:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3cb49640 10 monclient: _check_auth_tickets 2024-09-16T04:43:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3cb49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3cb49640 1 -- 172.21.15.46:0/4161499447 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca3402adf0 con 0x7fca380a3c10 2024-09-16T04:43:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3fdaf640 5 monclient: authenticate success, global_id 5992 2024-09-16T04:43:48.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3fdaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:48.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3cb49640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:48.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3fdaf640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:48.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3cb49640 1 -- 172.21.15.46:0/4161499447 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca34032640 con 0x7fca380a3c10 2024-09-16T04:43:48.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3fdaf640 4 set_mon_vals no callback set 2024-09-16T04:43:48.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3cb49640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:48.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3cb49640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:43:48.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3cb49640 10 monclient: dump: 2024-09-16T04:43:48.235 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:48.235 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:48.235 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:48.235 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:48.235 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:48.236 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:48.236 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:48.236 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:48.236 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:48.236 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:48.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3fdaf640 10 monclient: shutdown 2024-09-16T04:43:48.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3fdaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:48.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3fdaf640 1 -- 172.21.15.46:0/4161499447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca380a3c10 msgr2=0x7fca3809bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:48.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3fdaf640 1 --2- 172.21.15.46:0/4161499447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca380a3c10 0x7fca3809bcf0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fca38068240 tx=0x7fca3402a0a0 comp rx=0 tx=0).stop 2024-09-16T04:43:48.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3fdaf640 1 -- 172.21.15.46:0/4161499447 shutdown_connections 2024-09-16T04:43:48.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3fdaf640 1 --2- 172.21.15.46:0/4161499447 >> v2:172.21.15.46:3300/0 conn(0x7fca3813be90 0x7fca3813e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:48.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3fdaf640 1 --2- 172.21.15.46:0/4161499447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca380a3c10 0x7fca3809bcf0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:48.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3fdaf640 1 -- 172.21.15.46:0/4161499447 >> 172.21.15.46:0/4161499447 conn(0x7fca380a6340 msgr2=0x7fca3806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:48.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.223+0000 7fca3fdaf640 1 -- 172.21.15.46:0/4161499447 shutdown_connections 2024-09-16T04:43:48.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.224+0000 7fca3fdaf640 1 -- 172.21.15.46:0/4161499447 wait complete. 2024-09-16T04:43:48.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.224+0000 7fca3fdaf640 10 monclient: build_initial_monmap 2024-09-16T04:43:48.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.224+0000 7fca3fdaf640 10 monclient: monmap: 2024-09-16T04:43:48.237 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:48.237 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:48.237 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:48.237 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:48.237 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:48.237 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:48.237 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:48.237 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:48.237 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:48.238 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:48.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.224+0000 7fca3fdaf640 1 Processor -- start 2024-09-16T04:43:48.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3fdaf640 1 -- start start 2024-09-16T04:43:48.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3fdaf640 10 monclient: init 2024-09-16T04:43:48.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3fdaf640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:48.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3fdaf640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:43:48.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3fdaf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca380a3c10 0x7fca38133d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:48.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3fdaf640 10 monclient(hunting): picked mon.noname-a con 0x7fca380a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:48.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3fdaf640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fca380a3ff0 0x7fca381342a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:48.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3fdaf640 10 monclient(hunting): picked mon.noname-c con 0x7fca380a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:48.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3fdaf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca3813be90 0x7fca381cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:48.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3fdaf640 10 monclient(hunting): picked mon.noname-b con 0x7fca3813be90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:48.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3fdaf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:48.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3db4b640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca380a3c10 msgr2=0x7fca38133d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:43:48.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3fdaf640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fca380a6b40 con 0x7fca380a3c10 2024-09-16T04:43:48.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3fdaf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:48.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3fdaf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fca380a69c0 con 0x7fca3813be90 2024-09-16T04:43:48.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3fdaf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:48.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3fdaf640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fca3813e7c0 con 0x7fca380a3ff0 2024-09-16T04:43:48.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3fdaf640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:48.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3db4b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca380a3c10 0x7fca38133d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:48.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3fdaf640 10 monclient(hunting): authenticate will time out at 1026.640542s 2024-09-16T04:43:48.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3d34a640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fca380a3ff0 0x7fca381342a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:48.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3d34a640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fca380a3ff0 0x7fca381342a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:52452/0 (socket says 172.21.15.46:52452) 2024-09-16T04:43:48.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3d34a640 1 -- 172.21.15.46:0/2731938960 learned_addr learned my addr 172.21.15.46:0/2731938960 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:48.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3d34a640 10 monclient(hunting): get_auth_request con 0x7fca380a3ff0 auth_method 0 2024-09-16T04:43:48.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3d34a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:48.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3d34a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:48.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3d34a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:48.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3d34a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:48.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3d34a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:48.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3d34a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:48.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3e34c640 1 --2- 172.21.15.46:0/2731938960 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca3813be90 0x7fca381cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:48.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3e34c640 10 monclient(hunting): get_auth_request con 0x7fca3813be90 auth_method 0 2024-09-16T04:43:48.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3e34c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:48.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3e34c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:48.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.225+0000 7fca3e34c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:48.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca3e34c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:48.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca3e34c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:48.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca3e34c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:48.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca3d34a640 10 monclient(hunting): handle_auth_done global_id 64145 payload 824 2024-09-16T04:43:48.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca3d34a640 1 -- 172.21.15.46:0/2731938960 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca3813be90 msgr2=0x7fca381cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:48.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca3d34a640 1 --2- 172.21.15.46:0/2731938960 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca3813be90 0x7fca381cbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:48.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca3d34a640 1 -- 172.21.15.46:0/2731938960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca380a3c10 msgr2=0x7fca38133d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:43:48.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca3d34a640 1 --2- 172.21.15.46:0/2731938960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca380a3c10 0x7fca38133d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:48.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca3d34a640 10 monclient: _finish_hunting 0 2024-09-16T04:43:48.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca3d34a640 1 monclient: found mon.noname-c 2024-09-16T04:43:48.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca3d34a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:48.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca3d34a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:43:48.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca3d34a640 1 -- 172.21.15.46:0/2731938960 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca38138240 con 0x7fca380a3ff0 2024-09-16T04:43:48.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca3d34a640 1 --2- 172.21.15.46:0/2731938960 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fca380a3ff0 0x7fca381342a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fca2800c460 tx=0x7fca2800c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:48.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca2e7fc640 1 -- 172.21.15.46:0/2731938960 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28017070 con 0x7fca380a3ff0 2024-09-16T04:43:48.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:48.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca2e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:43:48.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca2e7fc640 10 monclient: dump: 2024-09-16T04:43:48.245 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:48.245 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:48.245 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:48.245 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:48.245 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:48.245 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:48.245 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:48.245 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:48.245 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:48.245 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:48.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca2e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:43:48.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca2e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:43:48.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:48.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca2e7fc640 1 -- 172.21.15.46:0/2731938960 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca28005ce0 con 0x7fca380a3ff0 2024-09-16T04:43:48.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca2e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:48.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca2e7fc640 1 -- 172.21.15.46:0/2731938960 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28007290 con 0x7fca380a3ff0 2024-09-16T04:43:48.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca3fdaf640 5 monclient: authenticate success, global_id 64145 2024-09-16T04:43:48.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:48.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca2e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:43:48.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca2e7fc640 10 monclient: dump: 2024-09-16T04:43:48.246 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:48.246 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:48.246 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:48.246 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:48.247 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:48.247 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:48.247 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:48.247 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:48.247 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:48.247 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:48.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca3fdaf640 10 monclient: _renew_subs 2024-09-16T04:43:48.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca3fdaf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:48.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca3fdaf640 1 -- 172.21.15.46:0/2731938960 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca381347e0 con 0x7fca380a3ff0 2024-09-16T04:43:48.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca2ffff640 4 set_mon_vals no callback set 2024-09-16T04:43:48.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca3fdaf640 10 monclient: _renew_subs 2024-09-16T04:43:48.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca3fdaf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:48.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.226+0000 7fca3fdaf640 1 -- 172.21.15.46:0/2731938960 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fca38139ad0 con 0x7fca380a3ff0 2024-09-16T04:43:48.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.227+0000 7fca2e7fc640 1 -- 172.21.15.46:0/2731938960 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fca2801a070 con 0x7fca380a3ff0 2024-09-16T04:43:48.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.227+0000 7fca2e7fc640 1 --2- 172.21.15.46:0/2731938960 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fca1c038360 0x7fca1c03a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:48.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.227+0000 7fca2e7fc640 1 -- 172.21.15.46:0/2731938960 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(195..195 src has 1..195) ==== 4738+0+0 (secure 0 0 0) 0x7fca2804aa80 con 0x7fca380a3ff0 2024-09-16T04:43:48.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.227+0000 7fca3fdaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:48.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.227+0000 7fca3fdaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:48.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.227+0000 7fca3fdaf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:48.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.227+0000 7fca3fdaf640 1 -- 172.21.15.46:0/2731938960 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca3809bd70 con 0x7fca380a3ff0 2024-09-16T04:43:48.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.227+0000 7fca3db4b640 1 --2- 172.21.15.46:0/2731938960 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fca1c038360 0x7fca1c03a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:48.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.228+0000 7fca3db4b640 10 monclient: get_auth_request con 0x7fca1c038360 auth_method 0 2024-09-16T04:43:48.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.228+0000 7fca3db4b640 1 --2- 172.21.15.46:0/2731938960 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fca1c038360 0x7fca1c03a820 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fca34034000 tx=0x7fca3402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:48.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.232+0000 7fca2e7fc640 1 -- 172.21.15.46:0/2731938960 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fca28019e10 con 0x7fca380a3ff0 2024-09-16T04:43:48.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.232+0000 7fca2e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:48.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.232+0000 7fca2e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:48.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.476+0000 7fca3fdaf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:43:48.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.476+0000 7fca3fdaf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:48.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.476+0000 7fca3fdaf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:48.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.476+0000 7fca3fdaf640 1 -- 172.21.15.46:0/2731938960 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fca3813c270 con 0x7fca380a3ff0 2024-09-16T04:43:48.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.476+0000 7fca2e7fc640 1 -- 172.21.15.46:0/2731938960 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fca2801a940 con 0x7fca380a3ff0 2024-09-16T04:43:48.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.476+0000 7fca2e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:48.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.476+0000 7fca2e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:48.478 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:43:48.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.478+0000 7fca3fdaf640 1 -- 172.21.15.46:0/2731938960 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fca1c038360 msgr2=0x7fca1c03a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:48.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.478+0000 7fca3fdaf640 1 --2- 172.21.15.46:0/2731938960 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fca1c038360 0x7fca1c03a820 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fca34034000 tx=0x7fca3402e000 comp rx=0 tx=0).stop 2024-09-16T04:43:48.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.478+0000 7fca3fdaf640 10 monclient: shutdown 2024-09-16T04:43:48.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.478+0000 7fca3fdaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:48.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.478+0000 7fca3fdaf640 1 -- 172.21.15.46:0/2731938960 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fca380a3ff0 msgr2=0x7fca381342a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:48.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.478+0000 7fca3fdaf640 1 --2- 172.21.15.46:0/2731938960 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fca380a3ff0 0x7fca381342a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fca2800c460 tx=0x7fca2800c920 comp rx=0 tx=0).stop 2024-09-16T04:43:48.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.478+0000 7fca3fdaf640 1 -- 172.21.15.46:0/2731938960 shutdown_connections 2024-09-16T04:43:48.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.478+0000 7fca3fdaf640 1 --2- 172.21.15.46:0/2731938960 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca3813be90 0x7fca381cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:48.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.478+0000 7fca3fdaf640 1 --2- 172.21.15.46:0/2731938960 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fca380a3ff0 0x7fca381342a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:48.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.478+0000 7fca3fdaf640 1 --2- 172.21.15.46:0/2731938960 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fca1c038360 0x7fca1c03a820 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:48.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.478+0000 7fca3fdaf640 1 --2- 172.21.15.46:0/2731938960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca380a3c10 0x7fca38133d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:48.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.478+0000 7fca3fdaf640 1 -- 172.21.15.46:0/2731938960 >> 172.21.15.46:0/2731938960 conn(0x7fca380a6340 msgr2=0x7fca3813b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:48.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.478+0000 7fca3fdaf640 1 -- 172.21.15.46:0/2731938960 shutdown_connections 2024-09-16T04:43:48.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:48.478+0000 7fca3fdaf640 1 -- 172.21.15.46:0/2731938960 wait complete. 2024-09-16T04:43:51.491 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:43:51.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.603+0000 7f4be6a17640 10 monclient: get_monmap_and_config 2024-09-16T04:43:51.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.603+0000 7f4be6a17640 10 monclient: build_initial_monmap 2024-09-16T04:43:51.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.603+0000 7f4be6a17640 10 monclient: monmap: 2024-09-16T04:43:51.605 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:51.605 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:51.605 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:51.604292+0000 2024-09-16T04:43:51.605 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:51.604292+0000 2024-09-16T04:43:51.605 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:51.605 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:51.605 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:51.605 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:51.605 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:51.606 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:51.606 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:51.606 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:51.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.604+0000 7f4be6a17640 1 Processor -- start 2024-09-16T04:43:51.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.604+0000 7f4be6a17640 1 -- start start 2024-09-16T04:43:51.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.604+0000 7f4be6a17640 10 monclient: init 2024-09-16T04:43:51.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4be6a17640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:51.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4be6a17640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-16T04:43:51.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4be6a17640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4be00a3c10 0x7f4be009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:51.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4be6a17640 10 monclient(hunting): picked mon.noname-a con 0x7f4be00a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:43:51.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4be6a17640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4be00a3ff0 0x7f4be00a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:51.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4be6a17640 10 monclient(hunting): picked mon.noname-c con 0x7f4be00a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:43:51.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4be6a17640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4be01369d0 addr v1:172.21.15.114:6789/0 2024-09-16T04:43:51.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4be6a17640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4be009b940 con 0x7f4be01369d0 2024-09-16T04:43:51.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4be6a17640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:51.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4be6a17640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4be009ab00 con 0x7f4be00a3c10 2024-09-16T04:43:51.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4be6a17640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:51.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4be6a17640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f4be009b030 con 0x7f4be00a3ff0 2024-09-16T04:43:51.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4be6a17640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:51.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4be6a17640 10 monclient(hunting): authenticate will time out at 1030.020252s 2024-09-16T04:43:51.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4bdffff640 1 -- >> v2:172.21.15.46:3300/0 conn(0x7f4be00a3c10 msgr2=0x7f4be009bcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:43:51.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4bdffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4be00a3c10 0x7f4be009bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:51.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4be4fb4640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4be01369d0 0x7f4be0136e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.46:34936/0 (socket says 172.21.15.46:34936) 2024-09-16T04:43:51.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4bdf7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4be00a3ff0 0x7f4be00a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:51.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4be4fb4640 1 -- 172.21.15.46:0/2589684637 learned_addr learned my addr 172.21.15.46:0/2589684637 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:43:51.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4bdf7fe640 10 monclient(hunting): get_auth_request con 0x7f4be00a3ff0 auth_method 0 2024-09-16T04:43:51.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4bdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:51.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4bdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:51.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4bdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:51.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4bdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:51.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4bdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:51.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4bdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:51.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4bdf7fe640 10 monclient(hunting): handle_auth_done global_id 6001 payload 274 2024-09-16T04:43:51.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4bdf7fe640 1 -- 172.21.15.46:0/2589684637 >> v2:172.21.15.46:3300/0 conn(0x7f4be00a3c10 msgr2=0x7f4be009bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:51.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4bdf7fe640 1 --2- 172.21.15.46:0/2589684637 >> v2:172.21.15.46:3300/0 conn(0x7f4be00a3c10 0x7f4be009bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:51.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4bdf7fe640 1 -- 172.21.15.46:0/2589684637 >> v1:172.21.15.114:6789/0 conn(0x7f4be01369d0 legacy=0x7f4be0136e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:51.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4bdf7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:43:51.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4bdf7fe640 1 monclient: found mon.noname-c 2024-09-16T04:43:51.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4bdf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:51.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4bdf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:43:51.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.605+0000 7f4bdf7fe640 1 -- 172.21.15.46:0/2589684637 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be009b310 con 0x7f4be00a3ff0 2024-09-16T04:43:51.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.606+0000 7f4bdf7fe640 1 --2- 172.21.15.46:0/2589684637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4be00a3ff0 0x7f4be00a43d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f4be00a38d0 tx=0x7f4bd402a0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2db76ddd143e8b13 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:51.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.606+0000 7f4bdeffd640 1 -- 172.21.15.46:0/2589684637 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4037070 con 0x7f4be00a3ff0 2024-09-16T04:43:51.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.606+0000 7f4bdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:51.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.606+0000 7f4bdeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:43:51.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.606+0000 7f4bdeffd640 10 monclient: dump: 2024-09-16T04:43:51.616 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:51.616 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:51.616 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:51.616 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:51.616 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:51.616 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:51.616 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:51.616 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:51.616 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:51.616 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:51.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.606+0000 7f4bdeffd640 10 monclient: _finish_auth 0 2024-09-16T04:43:51.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.606+0000 7f4bdeffd640 10 monclient: _check_auth_tickets 2024-09-16T04:43:51.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.606+0000 7f4bdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:51.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.606+0000 7f4bdeffd640 1 -- 172.21.15.46:0/2589684637 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bd402adf0 con 0x7f4be00a3ff0 2024-09-16T04:43:51.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.607+0000 7f4be6a17640 5 monclient: authenticate success, global_id 6001 2024-09-16T04:43:51.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.607+0000 7f4be6a17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:51.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.607+0000 7f4bdeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:51.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.607+0000 7f4be6a17640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:51.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.607+0000 7f4be6a17640 4 set_mon_vals no callback set 2024-09-16T04:43:51.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.607+0000 7f4bdeffd640 1 -- 172.21.15.46:0/2589684637 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4032660 con 0x7f4be00a3ff0 2024-09-16T04:43:51.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.607+0000 7f4bdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:51.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.607+0000 7f4bdeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:43:51.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.607+0000 7f4bdeffd640 10 monclient: dump: 2024-09-16T04:43:51.618 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:51.618 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:51.618 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:51.618 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:51.618 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:51.618 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:51.618 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:51.618 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:51.618 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:51.618 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:51.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.607+0000 7f4be6a17640 10 monclient: shutdown 2024-09-16T04:43:51.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.607+0000 7f4be6a17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:51.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.607+0000 7f4be6a17640 1 -- 172.21.15.46:0/2589684637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4be00a3ff0 msgr2=0x7f4be00a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:51.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.607+0000 7f4be6a17640 1 --2- 172.21.15.46:0/2589684637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4be00a3ff0 0x7f4be00a43d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f4be00a38d0 tx=0x7f4bd402a0a0 comp rx=0 tx=0).stop 2024-09-16T04:43:51.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.607+0000 7f4be6a17640 1 -- 172.21.15.46:0/2589684637 shutdown_connections 2024-09-16T04:43:51.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.607+0000 7f4be6a17640 1 --2- 172.21.15.46:0/2589684637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4be00a3ff0 0x7f4be00a43d0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:51.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.607+0000 7f4be6a17640 1 --2- 172.21.15.46:0/2589684637 >> v2:172.21.15.46:3300/0 conn(0x7f4be00a3c10 0x7f4be009bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:51.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.607+0000 7f4be6a17640 1 -- 172.21.15.46:0/2589684637 >> 172.21.15.46:0/2589684637 conn(0x7f4be00a6340 msgr2=0x7f4be006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:51.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.607+0000 7f4be6a17640 1 -- 172.21.15.46:0/2589684637 shutdown_connections 2024-09-16T04:43:51.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.607+0000 7f4be6a17640 1 -- 172.21.15.46:0/2589684637 wait complete. 2024-09-16T04:43:51.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.608+0000 7f4be6a17640 10 monclient: build_initial_monmap 2024-09-16T04:43:51.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.608+0000 7f4be6a17640 10 monclient: monmap: 2024-09-16T04:43:51.619 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:51.619 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:51.619 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:51.619 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:51.620 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:51.620 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:51.620 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:51.620 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:51.620 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:51.620 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:51.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.608+0000 7f4be6a17640 1 Processor -- start 2024-09-16T04:43:51.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.608+0000 7f4be6a17640 1 -- start start 2024-09-16T04:43:51.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.608+0000 7f4be6a17640 10 monclient: init 2024-09-16T04:43:51.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be6a17640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:51.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be6a17640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:43:51.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be6a17640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4be00a3c10 0x7f4be01cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:51.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be6a17640 10 monclient(hunting): picked mon.noname-c con 0x7f4be00a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:51.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be6a17640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4be00a3ff0 0x7f4be0123480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:51.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be6a17640 10 monclient(hunting): picked mon.noname-a con 0x7f4be00a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:51.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be6a17640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4be01369d0 0x7f4be01239c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:51.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be6a17640 10 monclient(hunting): picked mon.noname-b con 0x7f4be01369d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:51.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be6a17640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:51.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be6a17640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4be00a4910 con 0x7f4be00a3ff0 2024-09-16T04:43:51.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be6a17640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:51.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be6a17640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4be00a69c0 con 0x7f4be01369d0 2024-09-16T04:43:51.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be6a17640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:51.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be6a17640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f4be0127e10 con 0x7f4be00a3c10 2024-09-16T04:43:51.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be6a17640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:51.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be6a17640 10 monclient(hunting): authenticate will time out at 1030.024245s 2024-09-16T04:43:51.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdf7fe640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4be00a3ff0 msgr2=0x7f4be0123480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:43:51.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4be00a3ff0 0x7f4be0123480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:51.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4be00a3c10 0x7f4be01cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4be00a3c10 0x7f4be01cc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:52464/0 (socket says 172.21.15.46:52464) 2024-09-16T04:43:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdffff640 1 -- 172.21.15.46:0/2126611687 learned_addr learned my addr 172.21.15.46:0/2126611687 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdffff640 10 monclient(hunting): get_auth_request con 0x7f4be00a3c10 auth_method 0 2024-09-16T04:43:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be4fb4640 1 --2- 172.21.15.46:0/2126611687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4be01369d0 0x7f4be01239c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be4fb4640 10 monclient(hunting): get_auth_request con 0x7f4be01369d0 auth_method 0 2024-09-16T04:43:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be4fb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be4fb4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be4fb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdffff640 10 monclient(hunting): handle_auth_done global_id 64148 payload 824 2024-09-16T04:43:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdffff640 1 -- 172.21.15.46:0/2126611687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4be01369d0 msgr2=0x7f4be01239c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdffff640 1 --2- 172.21.15.46:0/2126611687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4be01369d0 0x7f4be01239c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdffff640 1 -- 172.21.15.46:0/2126611687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4be00a3ff0 msgr2=0x7f4be0123480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:43:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdffff640 1 --2- 172.21.15.46:0/2126611687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4be00a3ff0 0x7f4be0123480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdffff640 10 monclient: _finish_hunting 0 2024-09-16T04:43:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdffff640 1 monclient: found mon.noname-c 2024-09-16T04:43:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:43:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdffff640 1 -- 172.21.15.46:0/2126611687 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be0127130 con 0x7f4be00a3c10 2024-09-16T04:43:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4be4fb4640 1 --2- 172.21.15.46:0/2126611687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4be01369d0 0x7f4be01239c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:43:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.609+0000 7f4bdffff640 1 --2- 172.21.15.46:0/2126611687 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4be00a3c10 0x7f4be01cc330 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4bd000b550 tx=0x7f4bd000ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.610+0000 7f4bdcff9640 1 -- 172.21.15.46:0/2126611687 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0013020 con 0x7f4be00a3c10 2024-09-16T04:43:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.610+0000 7f4bdcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.610+0000 7f4bdcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:43:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.610+0000 7f4bdcff9640 10 monclient: dump: 2024-09-16T04:43:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:51.624 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.610+0000 7f4bdcff9640 10 monclient: _finish_auth 0 2024-09-16T04:43:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.610+0000 7f4bdcff9640 10 monclient: _check_auth_tickets 2024-09-16T04:43:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.610+0000 7f4bdcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.610+0000 7f4bdcff9640 1 -- 172.21.15.46:0/2126611687 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bd0003de0 con 0x7f4be00a3c10 2024-09-16T04:43:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.610+0000 7f4be6a17640 5 monclient: authenticate success, global_id 64148 2024-09-16T04:43:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.610+0000 7f4bdcff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.610+0000 7f4bdcff9640 1 -- 172.21.15.46:0/2126611687 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd00052b0 con 0x7f4be00a3c10 2024-09-16T04:43:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.610+0000 7f4be6a17640 10 monclient: _renew_subs 2024-09-16T04:43:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.610+0000 7f4be6a17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.610+0000 7f4be6a17640 1 -- 172.21.15.46:0/2126611687 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4be01255c0 con 0x7f4be00a3c10 2024-09-16T04:43:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.610+0000 7f4bdcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.610+0000 7f4bdcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:43:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.610+0000 7f4bdcff9640 10 monclient: dump: 2024-09-16T04:43:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:51.626 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.610+0000 7f4bde7fc640 4 set_mon_vals no callback set 2024-09-16T04:43:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.610+0000 7f4be6a17640 10 monclient: _renew_subs 2024-09-16T04:43:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.610+0000 7f4be6a17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.610+0000 7f4be6a17640 1 -- 172.21.15.46:0/2126611687 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4be00a2b80 con 0x7f4be00a3c10 2024-09-16T04:43:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.611+0000 7f4bdcff9640 1 -- 172.21.15.46:0/2126611687 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f4bd0016020 con 0x7f4be00a3c10 2024-09-16T04:43:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.611+0000 7f4be6a17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.611+0000 7f4be6a17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.611+0000 7f4be6a17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.611+0000 7f4bdcff9640 1 --2- 172.21.15.46:0/2126611687 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4bcc038360 0x7f4bcc03a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.611+0000 7f4be6a17640 1 -- 172.21.15.46:0/2126611687 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4be00a8e10 con 0x7f4be00a3c10 2024-09-16T04:43:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.611+0000 7f4bdcff9640 1 -- 172.21.15.46:0/2126611687 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(195..195 src has 1..195) ==== 4738+0+0 (secure 0 0 0) 0x7f4bd004b470 con 0x7f4be00a3c10 2024-09-16T04:43:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.611+0000 7f4bdf7fe640 1 --2- 172.21.15.46:0/2126611687 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4bcc038360 0x7f4bcc03a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.611+0000 7f4bdf7fe640 10 monclient: get_auth_request con 0x7f4bcc038360 auth_method 0 2024-09-16T04:43:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.612+0000 7f4bdf7fe640 1 --2- 172.21.15.46:0/2126611687 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4bcc038360 0x7f4bcc03a820 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4034000 tx=0x7f4bd402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:51.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.616+0000 7f4bdcff9640 1 -- 172.21.15.46:0/2126611687 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4be00a8e10 con 0x7f4be00a3c10 2024-09-16T04:43:51.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.616+0000 7f4bdcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:51.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.616+0000 7f4bdcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:51.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.865+0000 7f4be6a17640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:43:51.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.865+0000 7f4be6a17640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:51.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.865+0000 7f4be6a17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:51.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.865+0000 7f4be6a17640 1 -- 172.21.15.46:0/2126611687 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4be009bd70 con 0x7f4be00a3c10 2024-09-16T04:43:51.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.866+0000 7f4bdcff9640 1 -- 172.21.15.46:0/2126611687 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f4bd001d020 con 0x7f4be00a3c10 2024-09-16T04:43:51.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.866+0000 7f4bdcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:51.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.866+0000 7f4bdcff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:51.867 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:43:51.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.868+0000 7f4be6a17640 1 -- 172.21.15.46:0/2126611687 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4bcc038360 msgr2=0x7f4bcc03a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:51.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.868+0000 7f4be6a17640 1 --2- 172.21.15.46:0/2126611687 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4bcc038360 0x7f4bcc03a820 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4034000 tx=0x7f4bd402e000 comp rx=0 tx=0).stop 2024-09-16T04:43:51.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.868+0000 7f4be6a17640 10 monclient: shutdown 2024-09-16T04:43:51.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.868+0000 7f4be6a17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:51.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.868+0000 7f4be6a17640 1 -- 172.21.15.46:0/2126611687 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4be00a3c10 msgr2=0x7f4be01cc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:51.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.868+0000 7f4be6a17640 1 --2- 172.21.15.46:0/2126611687 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4be00a3c10 0x7f4be01cc330 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4bd000b550 tx=0x7f4bd000ba10 comp rx=0 tx=0).stop 2024-09-16T04:43:51.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.868+0000 7f4be6a17640 1 -- 172.21.15.46:0/2126611687 shutdown_connections 2024-09-16T04:43:51.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.868+0000 7f4be6a17640 1 --2- 172.21.15.46:0/2126611687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4be01369d0 0x7f4be01239c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:51.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.868+0000 7f4be6a17640 1 --2- 172.21.15.46:0/2126611687 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4bcc038360 0x7f4bcc03a820 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:51.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.868+0000 7f4be6a17640 1 --2- 172.21.15.46:0/2126611687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4be00a3ff0 0x7f4be0123480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:51.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.868+0000 7f4be6a17640 1 --2- 172.21.15.46:0/2126611687 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4be00a3c10 0x7f4be01cc330 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:51.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.868+0000 7f4be6a17640 1 -- 172.21.15.46:0/2126611687 >> 172.21.15.46:0/2126611687 conn(0x7f4be00a6340 msgr2=0x7f4be009f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:51.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.868+0000 7f4be6a17640 1 -- 172.21.15.46:0/2126611687 shutdown_connections 2024-09-16T04:43:51.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:51.868+0000 7f4be6a17640 1 -- 172.21.15.46:0/2126611687 wait complete. 2024-09-16T04:43:54.881 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:43:54.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.993+0000 7f3e34952640 10 monclient: get_monmap_and_config 2024-09-16T04:43:54.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.993+0000 7f3e34952640 10 monclient: build_initial_monmap 2024-09-16T04:43:54.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.993+0000 7f3e34952640 10 monclient: monmap: 2024-09-16T04:43:54.994 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:54.994 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:54.995 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:54.993945+0000 2024-09-16T04:43:54.995 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:54.993945+0000 2024-09-16T04:43:54.995 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:54.995 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:54.995 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:54.995 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:54.995 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:54.995 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:54.995 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:54.995 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:54.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.994+0000 7f3e34952640 1 Processor -- start 2024-09-16T04:43:54.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.994+0000 7f3e34952640 1 -- start start 2024-09-16T04:43:54.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.994+0000 7f3e34952640 10 monclient: init 2024-09-16T04:43:54.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.994+0000 7f3e34952640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:54.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.994+0000 7f3e34952640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-16T04:43:54.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.994+0000 7f3e34952640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 0x7f3e300a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:54.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.994+0000 7f3e34952640 10 monclient(hunting): picked mon.noname-b con 0x7f3e3006e160 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:54.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.994+0000 7f3e34952640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3e300a2ee0 0x7f3e30136940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:54.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.994+0000 7f3e34952640 10 monclient(hunting): picked mon.noname-a con 0x7f3e300a2ee0 addr v2:172.21.15.46:3300/0 2024-09-16T04:43:54.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.994+0000 7f3e34952640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 0x7f3e30137330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:54.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.994+0000 7f3e34952640 10 monclient(hunting): picked mon.noname-c con 0x7f3e30136e80 addr v2:172.21.15.114:3300/0 2024-09-16T04:43:54.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.994+0000 7f3e34952640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:54.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.994+0000 7f3e34952640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f3e300a3300 con 0x7f3e300a2ee0 2024-09-16T04:43:54.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.994+0000 7f3e34952640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:54.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.994+0000 7f3e34952640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f3e3006e540 con 0x7f3e30136e80 2024-09-16T04:43:54.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.994+0000 7f3e34952640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:54.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.994+0000 7f3e34952640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f3e300a3fd0 con 0x7f3e3006e160 2024-09-16T04:43:54.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.994+0000 7f3e34952640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:54.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.994+0000 7f3e34952640 10 monclient(hunting): authenticate will time out at 1033.409852s 2024-09-16T04:43:54.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.994+0000 7f3e2dd9b640 1 -- >> v2:172.21.15.46:3300/0 conn(0x7f3e300a2ee0 msgr2=0x7f3e30136940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:43:54.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.994+0000 7f3e2dd9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3e300a2ee0 0x7f3e30136940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:54.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.995+0000 7f3e2e59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 0x7f3e300a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:54.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.995+0000 7f3e2e59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 0x7f3e300a29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39166/0 (socket says 172.21.15.46:39166) 2024-09-16T04:43:54.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.995+0000 7f3e2e59c640 1 -- 172.21.15.46:0/3310459982 learned_addr learned my addr 172.21.15.46:0/3310459982 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:54.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.995+0000 7f3e2e59c640 10 monclient(hunting): get_auth_request con 0x7f3e3006e160 auth_method 0 2024-09-16T04:43:54.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.995+0000 7f3e2e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:54.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.995+0000 7f3e2e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.995+0000 7f3e2e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.995+0000 7f3e2ed9d640 1 --2- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 0x7f3e30137330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.995+0000 7f3e2ed9d640 10 monclient(hunting): get_auth_request con 0x7f3e30136e80 auth_method 0 2024-09-16T04:43:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.995+0000 7f3e2ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.995+0000 7f3e2ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.995+0000 7f3e2ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.995+0000 7f3e2e59c640 1 -- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 msgr2=0x7f3e300a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:43:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.995+0000 7f3e2e59c640 1 -- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 msgr2=0x7f3e300a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.995+0000 7f3e2e59c640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 0x7f3e300a29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.995+0000 7f3e2e59c640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 0x7f3e300a29a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.995+0000 7f3e2ed9d640 1 -- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 msgr2=0x7f3e30137330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:43:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.995+0000 7f3e2ed9d640 1 -- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 msgr2=0x7f3e30137330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.995+0000 7f3e2ed9d640 1 --2- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 0x7f3e30137330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:54.995+0000 7f3e2ed9d640 1 --2- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 0x7f3e30137330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:55.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.195+0000 7f3e2dd9b640 1 -- 172.21.15.46:0/3310459982 >> v2:172.21.15.46:3300/0 conn(0x7f3e300a2ee0 msgr2=0x7f3e30136940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:43:55.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.195+0000 7f3e2dd9b640 1 --2- 172.21.15.46:0/3310459982 >> v2:172.21.15.46:3300/0 conn(0x7f3e300a2ee0 0x7f3e30136940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:43:55.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.196+0000 7f3e2e59c640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 0x7f3e300a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:55.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.196+0000 7f3e2e59c640 10 monclient(hunting): get_auth_request con 0x7f3e3006e160 auth_method 0 2024-09-16T04:43:55.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.196+0000 7f3e2e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:55.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.196+0000 7f3e2e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:55.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.196+0000 7f3e2e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:55.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.196+0000 7f3e2e59c640 1 -- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 msgr2=0x7f3e300a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:43:55.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.196+0000 7f3e2e59c640 1 -- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 msgr2=0x7f3e300a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:55.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.196+0000 7f3e2e59c640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 0x7f3e300a29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:55.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.196+0000 7f3e2e59c640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 0x7f3e300a29a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:43:55.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.196+0000 7f3e2ed9d640 1 --2- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 0x7f3e30137330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:55.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.196+0000 7f3e2ed9d640 10 monclient(hunting): get_auth_request con 0x7f3e30136e80 auth_method 0 2024-09-16T04:43:55.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.196+0000 7f3e2ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:55.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.196+0000 7f3e2ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:55.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.196+0000 7f3e2ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:55.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.196+0000 7f3e2ed9d640 1 -- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 msgr2=0x7f3e30137330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:43:55.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.196+0000 7f3e2ed9d640 1 -- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 msgr2=0x7f3e30137330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:55.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.196+0000 7f3e2ed9d640 1 --2- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 0x7f3e30137330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:55.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.196+0000 7f3e2ed9d640 1 --2- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 0x7f3e30137330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:43:55.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.596+0000 7f3e2dd9b640 1 -- 172.21.15.46:0/3310459982 >> v2:172.21.15.46:3300/0 conn(0x7f3e300a2ee0 msgr2=0x7f3e30136940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:43:55.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.596+0000 7f3e2dd9b640 1 --2- 172.21.15.46:0/3310459982 >> v2:172.21.15.46:3300/0 conn(0x7f3e300a2ee0 0x7f3e30136940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:43:55.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.597+0000 7f3e2e59c640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 0x7f3e300a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:55.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.597+0000 7f3e2e59c640 10 monclient(hunting): get_auth_request con 0x7f3e3006e160 auth_method 0 2024-09-16T04:43:55.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.597+0000 7f3e2e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:55.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.597+0000 7f3e2e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:55.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.597+0000 7f3e2e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:55.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.597+0000 7f3e2e59c640 1 -- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 msgr2=0x7f3e300a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:43:55.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.597+0000 7f3e2e59c640 1 -- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 msgr2=0x7f3e300a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:55.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.597+0000 7f3e2e59c640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 0x7f3e300a29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:55.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.597+0000 7f3e2e59c640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 0x7f3e300a29a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:43:55.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.597+0000 7f3e2ed9d640 1 --2- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 0x7f3e30137330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:55.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.597+0000 7f3e2ed9d640 10 monclient(hunting): get_auth_request con 0x7f3e30136e80 auth_method 0 2024-09-16T04:43:55.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.597+0000 7f3e2ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:55.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.597+0000 7f3e2ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:55.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.597+0000 7f3e2ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:55.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.597+0000 7f3e2ed9d640 1 -- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 msgr2=0x7f3e30137330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:43:55.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.597+0000 7f3e2ed9d640 1 -- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 msgr2=0x7f3e30137330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:55.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.597+0000 7f3e2ed9d640 1 --2- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 0x7f3e30137330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:55.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:55.597+0000 7f3e2ed9d640 1 --2- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 0x7f3e30137330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:43:56.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:56.397+0000 7f3e2dd9b640 1 -- 172.21.15.46:0/3310459982 >> v2:172.21.15.46:3300/0 conn(0x7f3e300a2ee0 msgr2=0x7f3e30136940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:43:56.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:56.397+0000 7f3e2dd9b640 1 --2- 172.21.15.46:0/3310459982 >> v2:172.21.15.46:3300/0 conn(0x7f3e300a2ee0 0x7f3e30136940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:43:56.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:56.398+0000 7f3e2e59c640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 0x7f3e300a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:56.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:56.398+0000 7f3e2e59c640 10 monclient(hunting): get_auth_request con 0x7f3e3006e160 auth_method 0 2024-09-16T04:43:56.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:56.398+0000 7f3e2e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:56.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:56.398+0000 7f3e2e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:56.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:56.398+0000 7f3e2e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:56.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:56.398+0000 7f3e2e59c640 1 -- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 msgr2=0x7f3e300a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:43:56.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:56.398+0000 7f3e2e59c640 1 -- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 msgr2=0x7f3e300a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:56.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:56.398+0000 7f3e2e59c640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 0x7f3e300a29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:56.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:56.398+0000 7f3e2e59c640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 0x7f3e300a29a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:43:56.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:56.398+0000 7f3e2ed9d640 1 --2- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 0x7f3e30137330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:56.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:56.398+0000 7f3e2ed9d640 10 monclient(hunting): get_auth_request con 0x7f3e30136e80 auth_method 0 2024-09-16T04:43:56.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:56.398+0000 7f3e2ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:56.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:56.398+0000 7f3e2ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:56.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:56.398+0000 7f3e2ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:56.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:56.399+0000 7f3e2ed9d640 1 -- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 msgr2=0x7f3e30137330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:43:56.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:56.399+0000 7f3e2ed9d640 1 -- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 msgr2=0x7f3e30137330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:43:56.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:56.399+0000 7f3e2ed9d640 1 --2- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 0x7f3e30137330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:43:56.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:56.399+0000 7f3e2ed9d640 1 --2- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 0x7f3e30137330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:43:57.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:57.995+0000 7f3e1ffff640 10 monclient(hunting): tick 2024-09-16T04:43:57.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:57.995+0000 7f3e1ffff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:43:57.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:57.995+0000 7f3e1ffff640 1 monclient(hunting): continuing hunt 2024-09-16T04:43:57.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:57.995+0000 7f3e1ffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:43:57.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:57.995+0000 7f3e1ffff640 1 -- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 msgr2=0x7f3e300a29a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:57.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:57.995+0000 7f3e1ffff640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e3006e160 0x7f3e300a29a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:57.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:57.995+0000 7f3e1ffff640 1 -- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 msgr2=0x7f3e30137330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:57.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:57.995+0000 7f3e1ffff640 1 --2- 172.21.15.46:0/3310459982 >> v2:172.21.15.114:3300/0 conn(0x7f3e30136e80 0x7f3e30137330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:57.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:57.995+0000 7f3e1ffff640 1 -- 172.21.15.46:0/3310459982 >> v2:172.21.15.46:3300/0 conn(0x7f3e300a2ee0 msgr2=0x7f3e30136940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:57.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:57.995+0000 7f3e1ffff640 1 --2- 172.21.15.46:0/3310459982 >> v2:172.21.15.46:3300/0 conn(0x7f3e300a2ee0 0x7f3e30136940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:57.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:57.995+0000 7f3e1ffff640 10 monclient: _add_conns ranks=[0,1] 2024-09-16T04:43:57.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:57.995+0000 7f3e1ffff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e100036e0 addr v1:172.21.15.46:6789/0 2024-09-16T04:43:57.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:57.995+0000 7f3e1ffff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3e10007430 addr v1:172.21.15.114:6789/0 2024-09-16T04:43:57.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:57.995+0000 7f3e1ffff640 1 -- 172.21.15.46:0/3310459982 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e300a0bd0 con 0x7f3e100036e0 2024-09-16T04:43:57.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:57.995+0000 7f3e1ffff640 1 -- 172.21.15.46:0/3310459982 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e1000c130 con 0x7f3e10007430 2024-09-16T04:43:57.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:57.995+0000 7f3e1ffff640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:57.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:57.995+0000 7f3e2e59c640 1 -- 172.21.15.46:0/3310459982 >> v1:172.21.15.46:6789/0 conn(0x7f3e100036e0 legacy=0x7f3e10005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:43:58.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:58.196+0000 7f3e2e59c640 1 -- 172.21.15.46:0/3310459982 >> v1:172.21.15.46:6789/0 conn(0x7f3e100036e0 legacy=0x7f3e10005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:43:58.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:58.597+0000 7f3e2e59c640 1 -- 172.21.15.46:0/3310459982 >> v1:172.21.15.46:6789/0 conn(0x7f3e100036e0 legacy=0x7f3e10005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:43:59.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.306+0000 7f3e2d59a640 1 -- 172.21.15.46:0/3310459982 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2035348193 0 0) 0x7f3e1000c130 con 0x7f3e10007430 2024-09-16T04:43:59.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.306+0000 7f3e2d59a640 10 monclient(hunting): _negotiate 2024-09-16T04:43:59.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.306+0000 7f3e2d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:59.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.306+0000 7f3e2d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:59.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.306+0000 7f3e2d59a640 10 monclient(hunting): my global_id is 6007 2024-09-16T04:43:59.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.306+0000 7f3e2d59a640 1 -- 172.21.15.46:0/3310459982 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e18003520 con 0x7f3e10007430 2024-09-16T04:43:59.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 1 -- 172.21.15.46:0/3310459982 <== mon.1 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1860789172 0 0) 0x7f3e18003520 con 0x7f3e10007430 2024-09-16T04:43:59.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 1 -- 172.21.15.46:0/3310459982 >> v1:172.21.15.46:6789/0 conn(0x7f3e100036e0 legacy=0x7f3e10005b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:59.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 10 monclient: _finish_hunting 0 2024-09-16T04:43:59.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 1 monclient: found mon.noname-c-legacy 2024-09-16T04:43:59.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:59.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.114:6789/0 2024-09-16T04:43:59.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 1 -- 172.21.15.46:0/3310459982 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1000d370 con 0x7f3e10007430 2024-09-16T04:43:59.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 10 monclient: _finish_auth 0 2024-09-16T04:43:59.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 10 monclient: _check_auth_tickets 2024-09-16T04:43:59.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:59.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 1 -- 172.21.15.46:0/3310459982 <== mon.1 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f3e240045b0 con 0x7f3e10007430 2024-09-16T04:43:59.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:59.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T04:43:59.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 10 monclient: dump: 2024-09-16T04:43:59.316 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:59.316 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:59.317 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.317 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.317 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:59.317 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.317 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:59.318 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:59.318 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:59.318 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T04:43:59.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:59.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 1 -- 172.21.15.46:0/3310459982 >> v1:172.21.15.114:6789/0 conn(0x7f3e10007430 legacy=0x7f3e100098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:59.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2ed9d640 1 -- 172.21.15.46:0/3310459982 reap_dead start 2024-09-16T04:43:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:43:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e180039b0 0x7f3e18003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 10 monclient(hunting): picked mon.a con 0x7f3e180039b0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e18004800 0x7f3e18004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 10 monclient(hunting): picked mon.c con 0x7f3e18004800 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e180052e0 0x7f3e1800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 10 monclient(hunting): picked mon.b con 0x7f3e180052e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 1 -- 172.21.15.46:0/3310459982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1800c1d0 con 0x7f3e180039b0 2024-09-16T04:43:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 1 -- 172.21.15.46:0/3310459982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1800c380 con 0x7f3e180052e0 2024-09-16T04:43:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 1 -- 172.21.15.46:0/3310459982 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f3e1800c580 con 0x7f3e18004800 2024-09-16T04:43:59.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:59.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2d59a640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:43:59.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2e59c640 1 -- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e180039b0 msgr2=0x7f3e18003d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:43:59.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2e59c640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e180039b0 0x7f3e18003d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:59.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2dd9b640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e18004800 0x7f3e18004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:59.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2dd9b640 10 monclient(hunting): get_auth_request con 0x7f3e18004800 auth_method 0 2024-09-16T04:43:59.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:59.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:59.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:59.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2ed9d640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e180052e0 0x7f3e1800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:59.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2ed9d640 10 monclient(hunting): get_auth_request con 0x7f3e180052e0 auth_method 0 2024-09-16T04:43:59.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:59.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:59.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:43:59.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:59.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:59.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:43:59.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:59.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:59.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:43:59.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.310+0000 7f3e2dd9b640 10 monclient(hunting): handle_auth_done global_id 6007 payload 293 2024-09-16T04:43:59.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.311+0000 7f3e2dd9b640 1 -- 172.21.15.46:0/3310459982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e180052e0 msgr2=0x7f3e1800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:43:59.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.311+0000 7f3e2dd9b640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e180052e0 0x7f3e1800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.311+0000 7f3e2dd9b640 1 -- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e180039b0 msgr2=0x7f3e18003d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:59.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.311+0000 7f3e2dd9b640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e180039b0 0x7f3e18003d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.311+0000 7f3e2dd9b640 10 monclient: _finish_hunting 0 2024-09-16T04:43:59.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.311+0000 7f3e2dd9b640 1 monclient: found mon.c 2024-09-16T04:43:59.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.311+0000 7f3e2dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:59.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.311+0000 7f3e2dd9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:59.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.311+0000 7f3e2dd9b640 1 -- 172.21.15.46:0/3310459982 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e18003520 con 0x7f3e18004800 2024-09-16T04:43:59.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.311+0000 7f3e2dd9b640 10 monclient: _finish_auth 0 2024-09-16T04:43:59.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.311+0000 7f3e2dd9b640 10 monclient: _check_auth_tickets 2024-09-16T04:43:59.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.311+0000 7f3e2dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:59.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.311+0000 7f3e34952640 5 monclient: authenticate success, global_id 6007 2024-09-16T04:43:59.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.311+0000 7f3e34952640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:59.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.311+0000 7f3e2dd9b640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e18004800 0x7f3e18004c80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3e3009fec0 tx=0x7f3e18005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91306faf412977b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:59.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.312+0000 7f3e2d59a640 1 -- 172.21.15.46:0/3310459982 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2400d300 con 0x7f3e18004800 2024-09-16T04:43:59.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.312+0000 7f3e2d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:59.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.312+0000 7f3e2d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:43:59.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.312+0000 7f3e2d59a640 10 monclient: dump: 2024-09-16T04:43:59.328 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:59.328 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:59.329 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.329 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.329 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:59.329 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.329 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:59.329 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:59.329 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:59.329 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.312+0000 7f3e34952640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:59.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.312+0000 7f3e2d59a640 1 -- 172.21.15.46:0/3310459982 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e2402f070 con 0x7f3e18004800 2024-09-16T04:43:59.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.312+0000 7f3e2d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:59.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.312+0000 7f3e34952640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:59.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.312+0000 7f3e2d59a640 1 -- 172.21.15.46:0/3310459982 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e24033410 con 0x7f3e18004800 2024-09-16T04:43:59.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.312+0000 7f3e2d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:59.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.312+0000 7f3e2d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:43:59.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.312+0000 7f3e34952640 4 set_mon_vals no callback set 2024-09-16T04:43:59.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.312+0000 7f3e2d59a640 10 monclient: dump: 2024-09-16T04:43:59.330 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:59.331 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:59.331 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.331 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.331 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:59.331 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.331 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:59.331 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:59.331 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:59.331 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.312+0000 7f3e34952640 10 monclient: shutdown 2024-09-16T04:43:59.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.312+0000 7f3e34952640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:59.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.312+0000 7f3e34952640 1 -- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e18004800 msgr2=0x7f3e18004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:59.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.312+0000 7f3e34952640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e18004800 0x7f3e18004c80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3e3009fec0 tx=0x7f3e18005830 comp rx=0 tx=0).stop 2024-09-16T04:43:59.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.313+0000 7f3e34952640 1 -- 172.21.15.46:0/3310459982 shutdown_connections 2024-09-16T04:43:59.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.313+0000 7f3e34952640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e180052e0 0x7f3e1800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.313+0000 7f3e34952640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e18004800 0x7f3e18004c80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.313+0000 7f3e34952640 1 --2- 172.21.15.46:0/3310459982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e180039b0 0x7f3e18003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.313+0000 7f3e34952640 1 -- 172.21.15.46:0/3310459982 >> 172.21.15.46:0/3310459982 conn(0x7f3e300a5c10 msgr2=0x7f3e300a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:59.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.313+0000 7f3e34952640 1 -- 172.21.15.46:0/3310459982 shutdown_connections 2024-09-16T04:43:59.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.313+0000 7f3e34952640 1 -- 172.21.15.46:0/3310459982 wait complete. 2024-09-16T04:43:59.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.314+0000 7f3e34952640 10 monclient: build_initial_monmap 2024-09-16T04:43:59.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.314+0000 7f3e34952640 10 monclient: monmap: 2024-09-16T04:43:59.338 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:59.339 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:59.339 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:59.339 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:59.339 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:59.339 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.339 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:59.339 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:59.339 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:59.339 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.314+0000 7f3e34952640 1 Processor -- start 2024-09-16T04:43:59.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.314+0000 7f3e34952640 1 -- start start 2024-09-16T04:43:59.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.314+0000 7f3e34952640 10 monclient: init 2024-09-16T04:43:59.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e34952640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:59.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e34952640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:43:59.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e34952640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e180039b0 0x7f3e301c7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e34952640 10 monclient(hunting): picked mon.noname-a con 0x7f3e180039b0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:59.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e34952640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e18004800 0x7f3e30136050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e34952640 10 monclient(hunting): picked mon.noname-b con 0x7f3e18004800 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:59.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e34952640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e180052e0 0x7f3e301cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e34952640 10 monclient(hunting): picked mon.noname-c con 0x7f3e180052e0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:59.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e34952640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:59.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e34952640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3e300a5130 con 0x7f3e180039b0 2024-09-16T04:43:59.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e34952640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:59.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e34952640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3e300a1190 con 0x7f3e18004800 2024-09-16T04:43:59.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e34952640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:59.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e34952640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f3e300a34a0 con 0x7f3e180052e0 2024-09-16T04:43:59.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e34952640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:59.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e2e59c640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e180039b0 msgr2=0x7f3e301c7c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:43:59.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e34952640 10 monclient(hunting): authenticate will time out at 1037.730526s 2024-09-16T04:43:59.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e2e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e180039b0 0x7f3e301c7c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:59.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e2ed9d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e180052e0 0x7f3e301cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:59.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e2ed9d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e180052e0 0x7f3e301cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39208/0 (socket says 172.21.15.46:39208) 2024-09-16T04:43:59.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e2ed9d640 1 -- 172.21.15.46:0/3820893026 learned_addr learned my addr 172.21.15.46:0/3820893026 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:59.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e2ed9d640 10 monclient(hunting): get_auth_request con 0x7f3e180052e0 auth_method 0 2024-09-16T04:43:59.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e2ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:59.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e2ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:59.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e2ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:59.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e2dd9b640 1 --2- 172.21.15.46:0/3820893026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e18004800 0x7f3e30136050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:59.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e2dd9b640 10 monclient(hunting): get_auth_request con 0x7f3e18004800 auth_method 0 2024-09-16T04:43:59.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e2dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:59.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e2dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:59.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e2dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:59.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e2ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:59.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e2ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:59.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e2ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:59.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e2dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:59.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e2dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:59.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.315+0000 7f3e2dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:59.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e2ed9d640 10 monclient(hunting): handle_auth_done global_id 64151 payload 824 2024-09-16T04:43:59.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e2ed9d640 1 -- 172.21.15.46:0/3820893026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e18004800 msgr2=0x7f3e30136050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:59.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e2ed9d640 1 --2- 172.21.15.46:0/3820893026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e18004800 0x7f3e30136050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e2ed9d640 1 -- 172.21.15.46:0/3820893026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e180039b0 msgr2=0x7f3e301c7c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:43:59.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e2ed9d640 1 --2- 172.21.15.46:0/3820893026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e180039b0 0x7f3e301c7c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e2ed9d640 10 monclient: _finish_hunting 0 2024-09-16T04:43:59.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e2ed9d640 1 monclient: found mon.noname-c 2024-09-16T04:43:59.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e2ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:59.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e2ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:43:59.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e2ed9d640 1 -- 172.21.15.46:0/3820893026 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e30121720 con 0x7f3e180052e0 2024-09-16T04:43:59.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e2ed9d640 1 --2- 172.21.15.46:0/3820893026 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e180052e0 0x7f3e301cb1b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3e100073a0 tx=0x7f3e28011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:59.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e1effd640 1 -- 172.21.15.46:0/3820893026 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e28017650 con 0x7f3e180052e0 2024-09-16T04:43:59.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:59.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e1effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:43:59.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e1effd640 10 monclient: dump: 2024-09-16T04:43:59.345 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:59.345 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:59.345 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.345 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.345 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:59.345 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.345 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:59.346 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:59.346 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:59.346 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e1effd640 10 monclient: _finish_auth 0 2024-09-16T04:43:59.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e1effd640 10 monclient: _check_auth_tickets 2024-09-16T04:43:59.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:59.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e34952640 5 monclient: authenticate success, global_id 64151 2024-09-16T04:43:59.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e1effd640 1 -- 172.21.15.46:0/3820893026 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e2801b070 con 0x7f3e180052e0 2024-09-16T04:43:59.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e34952640 10 monclient: _renew_subs 2024-09-16T04:43:59.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e34952640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:59.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e34952640 1 -- 172.21.15.46:0/3820893026 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e30123a10 con 0x7f3e180052e0 2024-09-16T04:43:59.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e1effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:59.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e1effd640 1 -- 172.21.15.46:0/3820893026 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e28017a50 con 0x7f3e180052e0 2024-09-16T04:43:59.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:59.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e1effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:43:59.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e1effd640 10 monclient: dump: 2024-09-16T04:43:59.347 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:59.347 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:59.347 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.347 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.347 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:59.347 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.347 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:59.347 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:59.347 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:59.347 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e1ffff640 4 set_mon_vals no callback set 2024-09-16T04:43:59.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e34952640 10 monclient: _renew_subs 2024-09-16T04:43:59.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e34952640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:59.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.316+0000 7f3e34952640 1 -- 172.21.15.46:0/3820893026 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e30123730 con 0x7f3e180052e0 2024-09-16T04:43:59.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.317+0000 7f3e1effd640 1 -- 172.21.15.46:0/3820893026 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f3e28017bf0 con 0x7f3e180052e0 2024-09-16T04:43:59.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.317+0000 7f3e34952640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:59.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.317+0000 7f3e34952640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:59.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.317+0000 7f3e34952640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:59.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.317+0000 7f3e34952640 1 -- 172.21.15.46:0/3820893026 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e300a4d90 con 0x7f3e180052e0 2024-09-16T04:43:59.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.317+0000 7f3e1effd640 1 --2- 172.21.15.46:0/3820893026 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3e10038bf0 0x7f3e1003b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.317+0000 7f3e1effd640 1 -- 172.21.15.46:0/3820893026 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(195..195 src has 1..195) ==== 4738+0+0 (secure 0 0 0) 0x7f3e2804c6e0 con 0x7f3e180052e0 2024-09-16T04:43:59.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.318+0000 7f3e2e59c640 1 --2- 172.21.15.46:0/3820893026 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3e10038bf0 0x7f3e1003b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:59.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.318+0000 7f3e2e59c640 10 monclient: get_auth_request con 0x7f3e10038bf0 auth_method 0 2024-09-16T04:43:59.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.318+0000 7f3e2e59c640 1 --2- 172.21.15.46:0/3820893026 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3e10038bf0 0x7f3e1003b0b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3e20009100 tx=0x7f3e20008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:59.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.323+0000 7f3e1effd640 1 -- 172.21.15.46:0/3820893026 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3e28025060 con 0x7f3e180052e0 2024-09-16T04:43:59.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.323+0000 7f3e1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:59.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.323+0000 7f3e1effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:59.454 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38356]: starting. 2024-09-16T04:43:59.455 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38356]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-38355 2024-09-16T04:43:59.455 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38356]: created object 0... 2024-09-16T04:43:59.455 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38356]: created object 25... 2024-09-16T04:43:59.455 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38356]: created object 49... 2024-09-16T04:43:59.456 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38356]: finishing. 2024-09-16T04:43:59.456 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38356]: shutting down. 2024-09-16T04:43:59.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.584+0000 7f3e34952640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:43:59.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.584+0000 7f3e34952640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:59.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.584+0000 7f3e34952640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:59.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.584+0000 7f3e34952640 1 -- 172.21.15.46:0/3820893026 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3e30067e00 con 0x7f3e180052e0 2024-09-16T04:43:59.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.585+0000 7f3e1effd640 1 -- 172.21.15.46:0/3820893026 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f3e2801e370 con 0x7f3e180052e0 2024-09-16T04:43:59.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.585+0000 7f3e1effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:43:59.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.585+0000 7f3e1effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:59.586 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":60,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:43:59.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.587+0000 7f3e34952640 1 -- 172.21.15.46:0/3820893026 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3e10038bf0 msgr2=0x7f3e1003b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:59.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.587+0000 7f3e34952640 1 --2- 172.21.15.46:0/3820893026 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3e10038bf0 0x7f3e1003b0b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3e20009100 tx=0x7f3e20008040 comp rx=0 tx=0).stop 2024-09-16T04:43:59.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.587+0000 7f3e34952640 10 monclient: shutdown 2024-09-16T04:43:59.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.587+0000 7f3e34952640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:59.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.587+0000 7f3e34952640 1 -- 172.21.15.46:0/3820893026 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e180052e0 msgr2=0x7f3e301cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:59.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.587+0000 7f3e34952640 1 --2- 172.21.15.46:0/3820893026 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e180052e0 0x7f3e301cb1b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3e100073a0 tx=0x7f3e28011900 comp rx=0 tx=0).stop 2024-09-16T04:43:59.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.587+0000 7f3e34952640 1 -- 172.21.15.46:0/3820893026 shutdown_connections 2024-09-16T04:43:59.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.587+0000 7f3e34952640 1 --2- 172.21.15.46:0/3820893026 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f3e180052e0 0x7f3e301cb1b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.587+0000 7f3e34952640 1 --2- 172.21.15.46:0/3820893026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e18004800 0x7f3e30136050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.587+0000 7f3e34952640 1 --2- 172.21.15.46:0/3820893026 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f3e10038bf0 0x7f3e1003b0b0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.587+0000 7f3e34952640 1 --2- 172.21.15.46:0/3820893026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e180039b0 0x7f3e301c7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.587+0000 7f3e34952640 1 -- 172.21.15.46:0/3820893026 >> 172.21.15.46:0/3820893026 conn(0x7f3e300a5c10 msgr2=0x7f3e301be7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:59.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.587+0000 7f3e34952640 1 -- 172.21.15.46:0/3820893026 shutdown_connections 2024-09-16T04:43:59.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.587+0000 7f3e34952640 1 -- 172.21.15.46:0/3820893026 wait complete. 2024-09-16T04:43:59.599 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:43:59.599 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T04:43:59.599 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:43:59.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.701+0000 7f3992c82640 10 monclient: get_monmap_and_config 2024-09-16T04:43:59.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.701+0000 7f3992c82640 10 monclient: build_initial_monmap 2024-09-16T04:43:59.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.701+0000 7f3992c82640 10 monclient: monmap: 2024-09-16T04:43:59.702 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:59.702 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:59.702 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:59.701518+0000 2024-09-16T04:43:59.703 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:59.701518+0000 2024-09-16T04:43:59.703 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:59.703 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.703 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:59.703 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:59.703 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:59.703 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:59.703 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:59.703 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.701+0000 7f3992c82640 1 Processor -- start 2024-09-16T04:43:59.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.701+0000 7f3992c82640 1 -- start start 2024-09-16T04:43:59.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.701+0000 7f3992c82640 10 monclient: init 2024-09-16T04:43:59.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3992c82640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:59.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3992c82640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-16T04:43:59.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3992c82640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f398c0a3c10 0x7f398c09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3992c82640 10 monclient(hunting): picked mon.noname-c con 0x7f398c0a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:43:59.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3992c82640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f398c0a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:43:59.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3992c82640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c13be90 0x7f398c13e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3992c82640 10 monclient(hunting): picked mon.noname-b con 0x7f398c13be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:59.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3992c82640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f398c09ab70 con 0x7f398c0a3ff0 2024-09-16T04:43:59.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3992c82640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:59.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3992c82640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f398c0a4d60 con 0x7f398c0a3c10 2024-09-16T04:43:59.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3992c82640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:59.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3992c82640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f398c09b030 con 0x7f398c13be90 2024-09-16T04:43:59.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3992c82640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:59.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3992c82640 10 monclient(hunting): authenticate will time out at 1038.117399s 2024-09-16T04:43:59.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f398bfff640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f398c0a3ff0 legacy=0x7f398c0a4400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:43:59.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f399121f640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c13be90 0x7f398c13e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f399121f640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c13be90 0x7f398c13e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39216/0 (socket says 172.21.15.46:39216) 2024-09-16T04:43:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f399121f640 1 -- 172.21.15.46:0/3123315363 learned_addr learned my addr 172.21.15.46:0/3123315363 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f399121f640 10 monclient(hunting): get_auth_request con 0x7f398c13be90 auth_method 0 2024-09-16T04:43:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f399121f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f399121f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f399121f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3990a1e640 1 --2- 172.21.15.46:0/3123315363 >> v2:172.21.15.114:3300/0 conn(0x7f398c0a3c10 0x7f398c09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3990a1e640 10 monclient(hunting): get_auth_request con 0x7f398c0a3c10 auth_method 0 2024-09-16T04:43:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3990a1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3990a1e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3990a1e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f399121f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f399121f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:59.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f399121f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:59.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3990a1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:59.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3990a1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:59.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f3990a1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:59.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f399121f640 10 monclient(hunting): handle_auth_done global_id 64154 payload 274 2024-09-16T04:43:59.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f399121f640 1 -- 172.21.15.46:0/3123315363 >> v2:172.21.15.114:3300/0 conn(0x7f398c0a3c10 msgr2=0x7f398c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:43:59.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f399121f640 1 --2- 172.21.15.46:0/3123315363 >> v2:172.21.15.114:3300/0 conn(0x7f398c0a3c10 0x7f398c09bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f399121f640 1 -- 172.21.15.46:0/3123315363 >> v1:172.21.15.46:6789/0 conn(0x7f398c0a3ff0 legacy=0x7f398c0a4400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:59.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f399121f640 10 monclient: _finish_hunting 0 2024-09-16T04:43:59.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f399121f640 1 monclient: found mon.noname-b 2024-09-16T04:43:59.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f399121f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:59.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f399121f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:43:59.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.702+0000 7f399121f640 1 -- 172.21.15.46:0/3123315363 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f398c09b310 con 0x7f398c13be90 2024-09-16T04:43:59.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.703+0000 7f399121f640 1 --2- 172.21.15.46:0/3123315363 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c13be90 0x7f398c13e280 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3974009870 tx=0x7f397402a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d77c5b6a6cbd59cf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:59.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.703+0000 7f398b7fe640 1 -- 172.21.15.46:0/3123315363 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3974037070 con 0x7f398c13be90 2024-09-16T04:43:59.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.703+0000 7f398b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:59.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.703+0000 7f398b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:43:59.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.703+0000 7f398b7fe640 10 monclient: dump: 2024-09-16T04:43:59.716 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:59.716 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:59.716 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.716 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.716 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:59.717 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.717 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:59.717 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:59.717 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:59.717 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.703+0000 7f398b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:43:59.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.703+0000 7f398b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:43:59.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.703+0000 7f398b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:59.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.703+0000 7f3992c82640 5 monclient: authenticate success, global_id 64154 2024-09-16T04:43:59.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.703+0000 7f3992c82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:59.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.703+0000 7f398b7fe640 1 -- 172.21.15.46:0/3123315363 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f397402ae10 con 0x7f398c13be90 2024-09-16T04:43:59.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.703+0000 7f398b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:59.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.703+0000 7f3992c82640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:59.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.703+0000 7f3992c82640 4 set_mon_vals no callback set 2024-09-16T04:43:59.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.703+0000 7f398b7fe640 1 -- 172.21.15.46:0/3123315363 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3974032660 con 0x7f398c13be90 2024-09-16T04:43:59.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.703+0000 7f398b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:59.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.703+0000 7f398b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:43:59.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.703+0000 7f398b7fe640 10 monclient: dump: 2024-09-16T04:43:59.718 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:59.718 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:59.718 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.718 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.718 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:59.718 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.719 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:59.719 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:59.719 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:59.719 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.704+0000 7f3992c82640 10 monclient: shutdown 2024-09-16T04:43:59.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.704+0000 7f3992c82640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:59.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.704+0000 7f3992c82640 1 -- 172.21.15.46:0/3123315363 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c13be90 msgr2=0x7f398c13e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:59.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.704+0000 7f3992c82640 1 --2- 172.21.15.46:0/3123315363 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c13be90 0x7f398c13e280 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3974009870 tx=0x7f397402a100 comp rx=0 tx=0).stop 2024-09-16T04:43:59.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.704+0000 7f3992c82640 1 -- 172.21.15.46:0/3123315363 shutdown_connections 2024-09-16T04:43:59.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.704+0000 7f3992c82640 1 --2- 172.21.15.46:0/3123315363 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c13be90 0x7f398c13e280 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.704+0000 7f3992c82640 1 --2- 172.21.15.46:0/3123315363 >> v2:172.21.15.114:3300/0 conn(0x7f398c0a3c10 0x7f398c09bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.704+0000 7f3992c82640 1 -- 172.21.15.46:0/3123315363 >> 172.21.15.46:0/3123315363 conn(0x7f398c0a5d10 msgr2=0x7f398c0a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:59.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.704+0000 7f3992c82640 1 -- 172.21.15.46:0/3123315363 shutdown_connections 2024-09-16T04:43:59.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.704+0000 7f3992c82640 1 -- 172.21.15.46:0/3123315363 wait complete. 2024-09-16T04:43:59.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.704+0000 7f3992c82640 10 monclient: build_initial_monmap 2024-09-16T04:43:59.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.704+0000 7f3992c82640 10 monclient: monmap: 2024-09-16T04:43:59.720 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:59.720 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:59.720 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:59.720 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:59.720 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:59.720 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.720 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:59.720 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:59.720 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:59.721 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.705+0000 7f3992c82640 1 Processor -- start 2024-09-16T04:43:59.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.705+0000 7f3992c82640 1 -- start start 2024-09-16T04:43:59.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.705+0000 7f3992c82640 10 monclient: init 2024-09-16T04:43:59.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.705+0000 7f3992c82640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:59.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.705+0000 7f3992c82640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:43:59.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.705+0000 7f3992c82640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c0a3c10 0x7f398c1cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.705+0000 7f3992c82640 10 monclient(hunting): picked mon.noname-c con 0x7f398c0a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:59.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.705+0000 7f3992c82640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f398c0a3ff0 0x7f398c136eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.705+0000 7f3992c82640 10 monclient(hunting): picked mon.noname-a con 0x7f398c0a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:59.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.705+0000 7f3992c82640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f398c13be90 0x7f398c139970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.705+0000 7f3992c82640 10 monclient(hunting): picked mon.noname-b con 0x7f398c13be90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:59.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.705+0000 7f3992c82640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:59.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.705+0000 7f3992c82640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f398c06e550 con 0x7f398c0a3ff0 2024-09-16T04:43:59.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.705+0000 7f3992c82640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:59.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.705+0000 7f3992c82640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f398c06e3d0 con 0x7f398c13be90 2024-09-16T04:43:59.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.705+0000 7f3992c82640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:59.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.705+0000 7f3992c82640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f398c13e7c0 con 0x7f398c0a3c10 2024-09-16T04:43:59.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3992c82640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:59.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3992c82640 10 monclient(hunting): authenticate will time out at 1038.121012s 2024-09-16T04:43:59.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f398bfff640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f398c0a3ff0 msgr2=0x7f398c136eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:43:59.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f398bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f398c0a3ff0 0x7f398c136eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:59.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3990a1e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c0a3c10 0x7f398c1cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:59.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3990a1e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c0a3c10 0x7f398c1cb9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39220/0 (socket says 172.21.15.46:39220) 2024-09-16T04:43:59.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3990a1e640 1 -- 172.21.15.46:0/1904978866 learned_addr learned my addr 172.21.15.46:0/1904978866 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:59.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3990a1e640 10 monclient(hunting): get_auth_request con 0x7f398c0a3c10 auth_method 0 2024-09-16T04:43:59.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3990a1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:59.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3990a1e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:59.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3990a1e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:59.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3990a1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:59.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3990a1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:59.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3990a1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:59.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f399121f640 1 --2- 172.21.15.46:0/1904978866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f398c13be90 0x7f398c139970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:59.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f399121f640 10 monclient(hunting): get_auth_request con 0x7f398c13be90 auth_method 0 2024-09-16T04:43:59.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f399121f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:59.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f399121f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:59.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f399121f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:59.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3990a1e640 10 monclient(hunting): handle_auth_done global_id 64157 payload 824 2024-09-16T04:43:59.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3990a1e640 1 -- 172.21.15.46:0/1904978866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f398c13be90 msgr2=0x7f398c139970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:59.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3990a1e640 1 --2- 172.21.15.46:0/1904978866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f398c13be90 0x7f398c139970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3990a1e640 1 -- 172.21.15.46:0/1904978866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f398c0a3ff0 msgr2=0x7f398c136eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:43:59.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3990a1e640 1 --2- 172.21.15.46:0/1904978866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f398c0a3ff0 0x7f398c136eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3990a1e640 10 monclient: _finish_hunting 0 2024-09-16T04:43:59.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3990a1e640 1 monclient: found mon.noname-c 2024-09-16T04:43:59.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3990a1e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:59.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3990a1e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:43:59.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3990a1e640 1 -- 172.21.15.46:0/1904978866 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f398c121840 con 0x7f398c0a3c10 2024-09-16T04:43:59.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f399121f640 1 --2- 172.21.15.46:0/1904978866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f398c13be90 0x7f398c139970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:43:59.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3990a1e640 1 --2- 172.21.15.46:0/1904978866 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c0a3c10 0x7f398c1cb9d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f398400e9e0 tx=0x7f398400eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:59.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f39897fa640 1 -- 172.21.15.46:0/1904978866 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3984017070 con 0x7f398c0a3c10 2024-09-16T04:43:59.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f39897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:59.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f39897fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:43:59.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f39897fa640 10 monclient: dump: 2024-09-16T04:43:59.724 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:59.724 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:59.725 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.725 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.725 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:59.725 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.725 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:59.725 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:59.725 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:59.725 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f39897fa640 10 monclient: _finish_auth 0 2024-09-16T04:43:59.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f39897fa640 10 monclient: _check_auth_tickets 2024-09-16T04:43:59.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f39897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:59.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f39897fa640 1 -- 172.21.15.46:0/1904978866 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f398400c9d0 con 0x7f398c0a3c10 2024-09-16T04:43:59.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3992c82640 5 monclient: authenticate success, global_id 64157 2024-09-16T04:43:59.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3992c82640 10 monclient: _renew_subs 2024-09-16T04:43:59.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3992c82640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:59.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3992c82640 1 -- 172.21.15.46:0/1904978866 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f398c13c950 con 0x7f398c0a3c10 2024-09-16T04:43:59.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f39897fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:59.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f39897fa640 1 -- 172.21.15.46:0/1904978866 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398400cdc0 con 0x7f398c0a3c10 2024-09-16T04:43:59.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f39897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:59.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f39897fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:43:59.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f39897fa640 10 monclient: dump: 2024-09-16T04:43:59.726 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:59.726 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:59.726 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.726 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.726 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:59.726 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.726 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:59.727 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:59.727 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:59.727 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f398a7fc640 4 set_mon_vals no callback set 2024-09-16T04:43:59.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3992c82640 10 monclient: _renew_subs 2024-09-16T04:43:59.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3992c82640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:59.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.706+0000 7f3992c82640 1 -- 172.21.15.46:0/1904978866 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f398c0a0400 con 0x7f398c0a3c10 2024-09-16T04:43:59.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.707+0000 7f3992c82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:59.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.707+0000 7f39897fa640 1 -- 172.21.15.46:0/1904978866 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f398401a070 con 0x7f398c0a3c10 2024-09-16T04:43:59.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.707+0000 7f3992c82640 1 --2- 172.21.15.46:0/1904978866 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c1373f0 0x7f398c139eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.707+0000 7f3992c82640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:59.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.707+0000 7f3992c82640 1 -- 172.21.15.46:0/1904978866 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f398c09bcf0 con 0x7f398c1373f0 2024-09-16T04:43:59.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.707+0000 7f398bfff640 1 --2- 172.21.15.46:0/1904978866 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c1373f0 0x7f398c139eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:59.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.707+0000 7f398bfff640 10 monclient: get_auth_request con 0x7f398c1373f0 auth_method 0 2024-09-16T04:43:59.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.707+0000 7f398bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:59.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.707+0000 7f398bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:59.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.707+0000 7f398bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:59.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.707+0000 7f398bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:59.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.707+0000 7f398bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:59.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.707+0000 7f398bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:59.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.707+0000 7f39897fa640 1 --2- 172.21.15.46:0/1904978866 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f396c038390 0x7f396c03a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.707+0000 7f39897fa640 1 -- 172.21.15.46:0/1904978866 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(196..196 src has 1..196) ==== 4738+0+0 (secure 0 0 0) 0x7f398404b6f0 con 0x7f398c0a3c10 2024-09-16T04:43:59.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.707+0000 7f398bfff640 10 monclient(hunting): handle_auth_done global_id 64160 payload 824 2024-09-16T04:43:59.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.707+0000 7f398bfff640 1 -- 172.21.15.46:0/1904978866 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f398c121130 con 0x7f398c1373f0 2024-09-16T04:43:59.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.708+0000 7f398bfff640 1 --2- 172.21.15.46:0/1904978866 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c1373f0 0x7f398c139eb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f397c003d00 tx=0x7f397c009f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:59.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.708+0000 7f399121f640 1 --2- 172.21.15.46:0/1904978866 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f396c038390 0x7f396c03a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.708+0000 7f39897fa640 1 -- 172.21.15.46:0/1904978866 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397c014630 con 0x7f398c1373f0 2024-09-16T04:43:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.708+0000 7f39897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.708+0000 7f39897fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:43:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.708+0000 7f39897fa640 10 monclient: dump: 2024-09-16T04:43:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:59.729 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.708+0000 7f399121f640 10 monclient: get_auth_request con 0x7f396c038390 auth_method 0 2024-09-16T04:43:59.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.708+0000 7f39897fa640 1 -- 172.21.15.46:0/1904978866 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f397c014630 con 0x7f398c1373f0 2024-09-16T04:43:59.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.708+0000 7f39897fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:59.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.708+0000 7f39897fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:59.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.708+0000 7f39897fa640 1 -- 172.21.15.46:0/1904978866 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c1373f0 msgr2=0x7f398c139eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:59.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.708+0000 7f39897fa640 1 --2- 172.21.15.46:0/1904978866 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c1373f0 0x7f398c139eb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f397c003d00 tx=0x7f397c009f30 comp rx=0 tx=0).stop 2024-09-16T04:43:59.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.708+0000 7f39897fa640 1 -- 172.21.15.46:0/1904978866 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c1373f0 msgr2=0x7f398c139eb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:43:59.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.708+0000 7f39897fa640 1 --2- 172.21.15.46:0/1904978866 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c1373f0 0x7f398c139eb0 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f397c003d00 tx=0x7f397c009f30 comp rx=0 tx=0).stop 2024-09-16T04:43:59.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.708+0000 7f399121f640 1 --2- 172.21.15.46:0/1904978866 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f396c038390 0x7f396c03a850 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3974034000 tx=0x7f397402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:59.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.714+0000 7f3992c82640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:43:59.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.714+0000 7f3992c82640 1 --2- 172.21.15.46:0/1904978866 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c13a3f0 0x7f398c0a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.714+0000 7f3992c82640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:59.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.714+0000 7f3992c82640 1 -- 172.21.15.46:0/1904978866 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f398c13a800 con 0x7f398c13a3f0 2024-09-16T04:43:59.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.714+0000 7f398bfff640 1 --2- 172.21.15.46:0/1904978866 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c13a3f0 0x7f398c0a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:59.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.714+0000 7f398bfff640 10 monclient: get_auth_request con 0x7f398c13a3f0 auth_method 0 2024-09-16T04:43:59.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.714+0000 7f398bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:59.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.714+0000 7f398bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:59.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.714+0000 7f398bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:59.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f398bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:59.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f398bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:59.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f398bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:59.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f398bfff640 10 monclient(hunting): handle_auth_done global_id 64163 payload 824 2024-09-16T04:43:59.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f398bfff640 1 -- 172.21.15.46:0/1904978866 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f398c067e40 con 0x7f398c13a3f0 2024-09-16T04:43:59.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f398bfff640 1 --2- 172.21.15.46:0/1904978866 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c13a3f0 0x7f398c0a05d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f397c003d00 tx=0x7f397c005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:59.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f39897fa640 1 -- 172.21.15.46:0/1904978866 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397c01a9c0 con 0x7f398c13a3f0 2024-09-16T04:43:59.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f39897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:59.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f39897fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:43:59.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f39897fa640 10 monclient: dump: 2024-09-16T04:43:59.734 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:59.734 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:59.734 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.734 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.734 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:59.734 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.734 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:59.734 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:59.734 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:59.734 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f39897fa640 1 -- 172.21.15.46:0/1904978866 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4700 (secure 0 0 0) 0x7f397c023450 con 0x7f398c13a3f0 2024-09-16T04:43:59.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f39897fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:43:59.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f39897fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:59.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f39897fa640 1 -- 172.21.15.46:0/1904978866 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c13a3f0 msgr2=0x7f398c0a05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:59.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f39897fa640 1 --2- 172.21.15.46:0/1904978866 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c13a3f0 0x7f398c0a05d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f397c003d00 tx=0x7f397c005d50 comp rx=0 tx=0).stop 2024-09-16T04:43:59.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f39897fa640 1 -- 172.21.15.46:0/1904978866 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c13a3f0 msgr2=0x7f398c0a05d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:43:59.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f39897fa640 1 --2- 172.21.15.46:0/1904978866 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c13a3f0 0x7f398c0a05d0 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f397c003d00 tx=0x7f397c005d50 comp rx=0 tx=0).stop 2024-09-16T04:43:59.735 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:43:59.735 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:43:59.735 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:43:59.736 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:43:59.736 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 60, 2024-09-16T04:43:59.736 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 27651, 2024-09-16T04:43:59.736 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:43:59.736 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:43:59.736 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:43:59.736 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:59.736 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:43:59.736 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:43:59.736 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:43:59.736 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:43:59.736 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:59.736 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:59.737 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:59.737 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:59.737 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:59.737 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:59.737 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:59.737 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:59.737 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:59.737 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:59.737 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:59.737 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:59.737 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:43:59.737 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:43:59.737 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:59.738 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:59.738 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:59.738 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:59.738 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:59.738 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:59.738 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:59.738 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:59.738 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:59.738 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:59.738 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:59.738 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:59.738 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.738 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:43:59.738 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:43:59.739 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:43:59.739 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:43:59.739 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:43:59.739 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:43:59.739 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:43:59.739 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:43:59.739 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:43:59.739 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:43:59.739 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:43:59.739 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:43:59.739 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:43:59.739 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:43:59.739 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:43:59.739 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:43:59.740 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:43:59.740 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:59.740 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:59.740 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:59.740 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:59.740 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:59.740 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:59.740 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:59.740 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:59.740 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:59.740 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:59.740 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:59.740 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:59.741 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:43:59.741 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.741 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:43:59.741 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.741 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:43:59.741 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:43:59.741 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:59.741 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:59.741 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.745 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:59.745 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:43:59.745 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:59.745 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.745 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.745 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:59.745 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:43:59.745 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:59.745 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:59.746 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:59.746 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.746 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:43:59.746 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:43:59.746 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:59.746 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:59.746 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:59.746 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.746 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.746 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:43:59.746 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:43:59.746 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:59.746 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:59.746 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.747 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:59.747 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:43:59.747 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:59.747 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.747 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.747 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:59.747 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:43:59.747 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:59.747 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:59.747 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:59.747 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.747 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:43:59.747 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:43:59.747 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:59.748 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:59.748 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:59.748 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.748 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.748 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:43:59.748 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:43:59.748 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:59.748 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:59.748 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.748 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:59.748 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:43:59.748 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:59.748 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.748 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.749 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:59.749 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:43:59.749 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:59.749 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:59.749 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:59.749 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.749 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:43:59.749 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:43:59.749 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:59.749 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:59.749 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:59.749 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:59.749 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:59.749 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.750 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:43:59.750 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:43:59.750 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.750 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:59.750 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:59.750 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:43:59.750 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:59.750 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:59.750 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:43:59.750 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.750 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:59.750 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:59.750 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:43:59.750 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:59.751 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:59.751 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:43:59.751 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.751 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:59.751 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:59.751 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:43:59.751 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:59.751 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:59.751 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.751 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:43:59.751 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:43:59.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f3992c82640 1 -- 172.21.15.46:0/1904978866 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f396c038390 msgr2=0x7f396c03a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:59.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f3992c82640 1 --2- 172.21.15.46:0/1904978866 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f396c038390 0x7f396c03a850 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3974034000 tx=0x7f397402e000 comp rx=0 tx=0).stop 2024-09-16T04:43:59.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f3992c82640 10 monclient: shutdown 2024-09-16T04:43:59.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f3992c82640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:59.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f3992c82640 1 -- 172.21.15.46:0/1904978866 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c0a3c10 msgr2=0x7f398c1cb9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:59.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f399121f640 1 -- 172.21.15.46:0/1904978866 reap_dead start 2024-09-16T04:43:59.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.715+0000 7f3992c82640 1 --2- 172.21.15.46:0/1904978866 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c0a3c10 0x7f398c1cb9d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f398400e9e0 tx=0x7f398400eea0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.716+0000 7f3992c82640 1 -- 172.21.15.46:0/1904978866 shutdown_connections 2024-09-16T04:43:59.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.716+0000 7f3992c82640 1 --2- 172.21.15.46:0/1904978866 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f398c0a3c10 0x7f398c1cb9d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.716+0000 7f3992c82640 1 -- 172.21.15.46:0/1904978866 >> 172.21.15.46:0/1904978866 conn(0x7f398c0a5d10 msgr2=0x7f398c0a7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:59.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.716+0000 7f3992c82640 1 -- 172.21.15.46:0/1904978866 shutdown_connections 2024-09-16T04:43:59.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.716+0000 7f3992c82640 1 -- 172.21.15.46:0/1904978866 wait complete. 2024-09-16T04:43:59.754 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:43:59.754 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:43:59.754 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:43:59.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.855+0000 7f5429711640 10 monclient: get_monmap_and_config 2024-09-16T04:43:59.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.855+0000 7f5429711640 10 monclient: build_initial_monmap 2024-09-16T04:43:59.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.855+0000 7f5429711640 10 monclient: monmap: 2024-09-16T04:43:59.857 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:59.857 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:59.857 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:43:59.855798+0000 2024-09-16T04:43:59.857 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:43:59.855798+0000 2024-09-16T04:43:59.857 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:59.857 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.857 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:43:59.857 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:43:59.857 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:43:59.857 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:43:59.857 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:43:59.858 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.855+0000 7f5429711640 1 Processor -- start 2024-09-16T04:43:59.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.855+0000 7f5429711640 1 -- start start 2024-09-16T04:43:59.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.855+0000 7f5429711640 10 monclient: init 2024-09-16T04:43:59.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f5429711640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:59.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f5429711640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-16T04:43:59.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f5429711640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54240a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:43:59.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f5429711640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f54240a3ff0 0x7f54240a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f5429711640 10 monclient(hunting): picked mon.noname-c con 0x7f54240a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:43:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f5429711640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f542413be90 0x7f542413e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f5429711640 10 monclient(hunting): picked mon.noname-a con 0x7f542413be90 addr v2:172.21.15.46:3300/0 2024-09-16T04:43:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f5429711640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f542409b940 con 0x7f54240a3c10 2024-09-16T04:43:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f5429711640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f5429711640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f542409ab00 con 0x7f542413be90 2024-09-16T04:43:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f5429711640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f5429711640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f542409b030 con 0x7f54240a3ff0 2024-09-16T04:43:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f5429711640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f5429711640 10 monclient(hunting): authenticate will time out at 1038.271485s 2024-09-16T04:43:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f54237fe640 1 -- >> v2:172.21.15.46:3300/0 conn(0x7f542413be90 msgr2=0x7f542413e280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:43:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f54237fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f542413be90 0x7f542413e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f54227fc640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f54240a3ff0 0x7f54240a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:59.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f54227fc640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f54240a3ff0 0x7f54240a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.46:56784/0 (socket says 172.21.15.46:56784) 2024-09-16T04:43:59.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f54227fc640 1 -- 172.21.15.46:0/3790222392 learned_addr learned my addr 172.21.15.46:0/3790222392 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:59.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f54227fc640 10 monclient(hunting): get_auth_request con 0x7f54240a3ff0 auth_method 0 2024-09-16T04:43:59.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f54227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:59.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f54227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:59.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f54227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:59.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f54227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:59.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f54227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:59.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.856+0000 7f54227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:59.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f54227fc640 10 monclient(hunting): handle_auth_done global_id 6019 payload 274 2024-09-16T04:43:59.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f54227fc640 1 -- 172.21.15.46:0/3790222392 >> v2:172.21.15.46:3300/0 conn(0x7f542413be90 msgr2=0x7f542413e280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:43:59.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f54227fc640 1 --2- 172.21.15.46:0/3790222392 >> v2:172.21.15.46:3300/0 conn(0x7f542413be90 0x7f542413e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f54227fc640 1 -- 172.21.15.46:0/3790222392 >> v1:172.21.15.114:6789/0 conn(0x7f54240a3c10 legacy=0x7f542409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:59.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f54227fc640 10 monclient: _finish_hunting 0 2024-09-16T04:43:59.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f54227fc640 1 monclient: found mon.noname-c 2024-09-16T04:43:59.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f54227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:59.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f54227fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:43:59.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f54227fc640 1 -- 172.21.15.46:0/3790222392 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542409b310 con 0x7f54240a3ff0 2024-09-16T04:43:59.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f54227fc640 1 --2- 172.21.15.46:0/3790222392 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54240a3ff0 0x7f54240a43d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f54240a38d0 tx=0x7f541802a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=872f9569ed38c4da server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:59.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f5421ffb640 1 -- 172.21.15.46:0/3790222392 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418037070 con 0x7f54240a3ff0 2024-09-16T04:43:59.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f5421ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:59.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f5421ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:43:59.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f5421ffb640 10 monclient: dump: 2024-09-16T04:43:59.869 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:59.869 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:59.869 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.870 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.870 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:59.870 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.870 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:59.870 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:59.870 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:59.870 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f5421ffb640 10 monclient: _finish_auth 0 2024-09-16T04:43:59.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f5421ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:43:59.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f5421ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:59.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f5421ffb640 1 -- 172.21.15.46:0/3790222392 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5418032440 con 0x7f54240a3ff0 2024-09-16T04:43:59.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f5429711640 5 monclient: authenticate success, global_id 6019 2024-09-16T04:43:59.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f5429711640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:43:59.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f5421ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:59.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f5429711640 10 monclient: get_monmap_and_config success 2024-09-16T04:43:59.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f5421ffb640 1 -- 172.21.15.46:0/3790222392 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418032810 con 0x7f54240a3ff0 2024-09-16T04:43:59.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f5421ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:59.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f5429711640 4 set_mon_vals no callback set 2024-09-16T04:43:59.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f5421ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:43:59.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.857+0000 7f5421ffb640 10 monclient: dump: 2024-09-16T04:43:59.871 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:59.871 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:59.871 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.871 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.871 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:59.871 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.871 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:59.871 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:59.872 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:59.872 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.858+0000 7f5429711640 10 monclient: shutdown 2024-09-16T04:43:59.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.858+0000 7f5429711640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:59.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.858+0000 7f5429711640 1 -- 172.21.15.46:0/3790222392 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54240a3ff0 msgr2=0x7f54240a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:59.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.858+0000 7f5429711640 1 --2- 172.21.15.46:0/3790222392 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54240a3ff0 0x7f54240a43d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f54240a38d0 tx=0x7f541802a210 comp rx=0 tx=0).stop 2024-09-16T04:43:59.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.858+0000 7f5429711640 1 -- 172.21.15.46:0/3790222392 shutdown_connections 2024-09-16T04:43:59.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.858+0000 7f5429711640 1 --2- 172.21.15.46:0/3790222392 >> v2:172.21.15.46:3300/0 conn(0x7f542413be90 0x7f542413e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.858+0000 7f5429711640 1 --2- 172.21.15.46:0/3790222392 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54240a3ff0 0x7f54240a43d0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.858+0000 7f5429711640 1 -- 172.21.15.46:0/3790222392 >> 172.21.15.46:0/3790222392 conn(0x7f54240a6340 msgr2=0x7f542406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.858+0000 7f5429711640 1 -- 172.21.15.46:0/3790222392 shutdown_connections 2024-09-16T04:43:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.858+0000 7f5429711640 1 -- 172.21.15.46:0/3790222392 wait complete. 2024-09-16T04:43:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.859+0000 7f5429711640 10 monclient: build_initial_monmap 2024-09-16T04:43:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.859+0000 7f5429711640 10 monclient: monmap: 2024-09-16T04:43:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:43:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:43:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:43:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:43:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:43:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:43:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:43:59.874 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:43:59.874 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.859+0000 7f5429711640 1 Processor -- start 2024-09-16T04:43:59.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.859+0000 7f5429711640 1 -- start start 2024-09-16T04:43:59.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.859+0000 7f5429711640 10 monclient: init 2024-09-16T04:43:59.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.859+0000 7f5429711640 10 monclient: _reopen_session rank -1 2024-09-16T04:43:59.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f5429711640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:43:59.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f5429711640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54240a3c10 0x7f54240a3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f5429711640 10 monclient(hunting): picked mon.noname-a con 0x7f54240a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:43:59.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f5429711640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54240a3ff0 0x7f54241cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f5429711640 10 monclient(hunting): picked mon.noname-c con 0x7f54240a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:43:59.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f5429711640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f542413be90 0x7f54241cc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f5429711640 10 monclient(hunting): picked mon.noname-b con 0x7f542413be90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:43:59.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f5429711640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:59.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f5429711640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f54240a6b40 con 0x7f54240a3c10 2024-09-16T04:43:59.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f5429711640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:59.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f5422ffd640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54240a3c10 msgr2=0x7f54240a3070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:43:59.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f5429711640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f54240a69c0 con 0x7f542413be90 2024-09-16T04:43:59.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f5429711640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:59.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f5429711640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f542413e7c0 con 0x7f54240a3ff0 2024-09-16T04:43:59.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f5429711640 10 monclient(hunting): _renew_subs 2024-09-16T04:43:59.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f5429711640 10 monclient(hunting): authenticate will time out at 1038.275112s 2024-09-16T04:43:59.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f5422ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54240a3c10 0x7f54240a3070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:43:59.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54227fc640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54240a3ff0 0x7f54241cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:59.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54227fc640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54240a3ff0 0x7f54241cbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39256/0 (socket says 172.21.15.46:39256) 2024-09-16T04:43:59.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54227fc640 1 -- 172.21.15.46:0/2651404003 learned_addr learned my addr 172.21.15.46:0/2651404003 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:43:59.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54227fc640 10 monclient(hunting): get_auth_request con 0x7f54240a3ff0 auth_method 0 2024-09-16T04:43:59.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:59.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:59.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:59.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:59.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:59.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:59.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54237fe640 1 --2- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f542413be90 0x7f54241cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:59.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54237fe640 10 monclient(hunting): get_auth_request con 0x7f542413be90 auth_method 0 2024-09-16T04:43:59.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:59.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:59.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:59.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:59.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:59.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54227fc640 10 monclient(hunting): handle_auth_done global_id 64166 payload 824 2024-09-16T04:43:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54227fc640 1 -- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f542413be90 msgr2=0x7f54241cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54227fc640 1 --2- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f542413be90 0x7f54241cc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54227fc640 1 -- 172.21.15.46:0/2651404003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54240a3c10 msgr2=0x7f54240a3070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:43:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54227fc640 1 --2- 172.21.15.46:0/2651404003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54240a3c10 0x7f54240a3070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54227fc640 10 monclient: _finish_hunting 0 2024-09-16T04:43:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54227fc640 1 monclient: found mon.noname-c 2024-09-16T04:43:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:43:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54227fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:43:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54227fc640 1 -- 172.21.15.46:0/2651404003 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54240a05b0 con 0x7f54240a3ff0 2024-09-16T04:43:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f54227fc640 1 --2- 172.21.15.46:0/2651404003 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54240a3ff0 0x7f54241cbc30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5418000c00 tx=0x7f54180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f540b7fe640 1 -- 172.21.15.46:0/2651404003 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418041070 con 0x7f54240a3ff0 2024-09-16T04:43:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f540b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f540b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:43:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f540b7fe640 10 monclient: dump: 2024-09-16T04:43:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:59.878 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f540b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:43:59.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f540b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:43:59.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f540b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:43:59.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f5429711640 5 monclient: authenticate success, global_id 64166 2024-09-16T04:43:59.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f540b7fe640 1 -- 172.21.15.46:0/2651404003 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f541803c870 con 0x7f54240a3ff0 2024-09-16T04:43:59.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f5429711640 10 monclient: _renew_subs 2024-09-16T04:43:59.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f5429711640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:59.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f5429711640 1 -- 172.21.15.46:0/2651404003 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54240a2040 con 0x7f54240a3ff0 2024-09-16T04:43:59.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.860+0000 7f540b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:43:59.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.861+0000 7f540b7fe640 1 -- 172.21.15.46:0/2651404003 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418037040 con 0x7f54240a3ff0 2024-09-16T04:43:59.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.861+0000 7f540b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:59.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.861+0000 7f540b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:43:59.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.861+0000 7f540b7fe640 10 monclient: dump: 2024-09-16T04:43:59.879 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:59.880 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:59.880 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.880 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.880 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:59.880 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.880 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:59.880 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:59.880 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:59.880 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.861+0000 7f5429711640 10 monclient: _renew_subs 2024-09-16T04:43:59.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.861+0000 7f5429711640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:43:59.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.861+0000 7f5429711640 1 -- 172.21.15.46:0/2651404003 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f54240a1150 con 0x7f54240a3ff0 2024-09-16T04:43:59.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.861+0000 7f5420ff9640 4 set_mon_vals no callback set 2024-09-16T04:43:59.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.861+0000 7f5429711640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:59.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.861+0000 7f540b7fe640 1 -- 172.21.15.46:0/2651404003 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f5418044070 con 0x7f54240a3ff0 2024-09-16T04:43:59.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.861+0000 7f5429711640 1 --2- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54241cc6b0 0x7f54241339a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.861+0000 7f5429711640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:59.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.861+0000 7f5429711640 1 -- 172.21.15.46:0/2651404003 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f54241cca90 con 0x7f54241cc6b0 2024-09-16T04:43:59.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.862+0000 7f540b7fe640 1 --2- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5410038360 0x7f541003a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.862+0000 7f540b7fe640 1 -- 172.21.15.46:0/2651404003 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(196..196 src has 1..196) ==== 4738+0+0 (secure 0 0 0) 0x7f541806c390 con 0x7f54240a3ff0 2024-09-16T04:43:59.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.862+0000 7f5422ffd640 1 --2- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54241cc6b0 0x7f54241339a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:59.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.862+0000 7f5422ffd640 10 monclient: get_auth_request con 0x7f54241cc6b0 auth_method 0 2024-09-16T04:43:59.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.862+0000 7f5422ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:59.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.862+0000 7f5422ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:59.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.862+0000 7f5422ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:59.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.862+0000 7f5422ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:59.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.862+0000 7f5422ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:59.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.862+0000 7f5422ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:59.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.862+0000 7f54237fe640 1 --2- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5410038360 0x7f541003a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:59.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.862+0000 7f54237fe640 10 monclient: get_auth_request con 0x7f5410038360 auth_method 0 2024-09-16T04:43:59.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.862+0000 7f5422ffd640 10 monclient(hunting): handle_auth_done global_id 6028 payload 824 2024-09-16T04:43:59.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.862+0000 7f5422ffd640 1 -- 172.21.15.46:0/2651404003 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f54240a4450 con 0x7f54241cc6b0 2024-09-16T04:43:59.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.862+0000 7f5422ffd640 1 --2- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54241cc6b0 0x7f54241339a0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f5414001030 tx=0x7f541400c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:59.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.862+0000 7f54237fe640 1 --2- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5410038360 0x7f541003a820 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f54240a4d80 tx=0x7f541c009540 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:59.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.862+0000 7f540b7fe640 1 -- 172.21.15.46:0/2651404003 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54140074e0 con 0x7f54241cc6b0 2024-09-16T04:43:59.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.862+0000 7f540b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:59.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.862+0000 7f540b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:43:59.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.862+0000 7f540b7fe640 10 monclient: dump: 2024-09-16T04:43:59.883 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:59.883 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:59.883 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.883 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.883 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:59.883 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.883 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:59.883 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:59.883 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:59.883 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.863+0000 7f540b7fe640 1 -- 172.21.15.46:0/2651404003 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f54140074e0 con 0x7f54241cc6b0 2024-09-16T04:43:59.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.863+0000 7f540b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:43:59.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.863+0000 7f540b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:43:59.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.863+0000 7f540b7fe640 1 -- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54241cc6b0 msgr2=0x7f54241339a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:59.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.863+0000 7f540b7fe640 1 --2- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54241cc6b0 0x7f54241339a0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f5414001030 tx=0x7f541400c810 comp rx=0 tx=0).stop 2024-09-16T04:43:59.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.863+0000 7f540b7fe640 1 -- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54241cc6b0 msgr2=0x7f54241339a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:43:59.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.863+0000 7f540b7fe640 1 --2- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54241cc6b0 0x7f54241339a0 secure :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f5414001030 tx=0x7f541400c810 comp rx=0 tx=0).stop 2024-09-16T04:43:59.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.869+0000 7f5429711640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:43:59.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.869+0000 7f5429711640 1 --2- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5424133ee0 0x7f54241342f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:43:59.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.869+0000 7f5429711640 10 monclient(hunting): start opening mon connection 2024-09-16T04:43:59.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.869+0000 7f5429711640 1 -- 172.21.15.46:0/2651404003 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5424134830 con 0x7f5424133ee0 2024-09-16T04:43:59.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.869+0000 7f5422ffd640 1 --2- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5424133ee0 0x7f54241342f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:43:59.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.869+0000 7f5422ffd640 10 monclient: get_auth_request con 0x7f5424133ee0 auth_method 0 2024-09-16T04:43:59.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.869+0000 7f5422ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:43:59.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.869+0000 7f5422ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:43:59.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.869+0000 7f5422ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:43:59.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.869+0000 7f5422ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:43:59.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.869+0000 7f5422ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:43:59.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.869+0000 7f5422ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:43:59.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.869+0000 7f5422ffd640 10 monclient(hunting): handle_auth_done global_id 6031 payload 824 2024-09-16T04:43:59.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.869+0000 7f5422ffd640 1 -- 172.21.15.46:0/2651404003 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5424067e20 con 0x7f5424133ee0 2024-09-16T04:43:59.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f5422ffd640 1 --2- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5424133ee0 0x7f54241342f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f5414001eb0 tx=0x7f5414000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:43:59.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f540b7fe640 1 -- 172.21.15.46:0/2651404003 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54140192b0 con 0x7f5424133ee0 2024-09-16T04:43:59.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f540b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:43:59.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f540b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:43:59.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f540b7fe640 10 monclient: dump: 2024-09-16T04:43:59.887 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:43:59.887 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:43:59.887 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.887 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:43:59.887 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:43:59.887 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:43:59.887 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:43:59.887 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:43:59.887 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:43:59.887 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:43:59.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f540b7fe640 1 -- 172.21.15.46:0/2651404003 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4870 (secure 0 0 0) 0x7f54140192b0 con 0x7f5424133ee0 2024-09-16T04:43:59.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f540b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:43:59.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f540b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:43:59.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f540b7fe640 1 -- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5424133ee0 msgr2=0x7f54241342f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:59.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f540b7fe640 1 --2- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5424133ee0 0x7f54241342f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f5414001eb0 tx=0x7f5414000ed0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f540b7fe640 1 -- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5424133ee0 msgr2=0x7f54241342f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:43:59.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f540b7fe640 1 --2- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5424133ee0 0x7f54241342f0 secure :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f5414001eb0 tx=0x7f5414000ed0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.888 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:43:59.888 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:43:59.889 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:43:59.889 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:43:59.889 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 60, 2024-09-16T04:43:59.889 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 325772, 2024-09-16T04:43:59.889 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:43:59.889 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:43:59.889 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:43:59.889 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:59.889 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:43:59.889 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:43:59.889 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:43:59.889 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:43:59.889 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:59.889 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:59.890 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:59.890 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:59.890 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:59.890 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:59.890 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:59.890 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:59.890 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:59.890 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:59.890 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:59.890 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:59.890 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:43:59.890 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:43:59.890 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:59.891 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:59.891 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:59.891 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:59.891 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:59.891 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:59.891 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:59.891 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:59.891 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:59.891 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:59.891 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:59.891 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:59.891 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.891 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:43:59.891 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:43:59.892 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:43:59.892 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:43:59.892 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:43:59.892 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:43:59.892 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:43:59.892 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:43:59.892 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:43:59.892 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:43:59.892 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:43:59.892 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:43:59.892 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:43:59.892 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:43:59.892 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:43:59.893 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:43:59.893 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:43:59.893 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:43:59.893 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:43:59.893 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:43:59.893 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:43:59.893 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:43:59.893 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:43:59.893 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:43:59.893 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:43:59.893 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:43:59.893 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:43:59.893 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:43:59.894 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:59.894 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:43:59.894 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.894 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:43:59.894 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.894 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:43:59.894 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:43:59.894 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:59.894 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:59.894 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.894 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:59.894 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:43:59.894 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:59.895 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.895 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.895 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:59.895 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:43:59.895 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:59.895 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:59.895 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:59.895 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.895 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:43:59.895 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:43:59.895 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:59.895 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:59.895 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:59.896 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.896 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.896 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:43:59.896 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:43:59.896 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:59.896 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:59.896 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.896 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:59.896 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:43:59.896 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:59.896 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.896 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.896 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:59.897 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:43:59.897 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:59.897 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:59.897 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:59.897 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.897 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:43:59.897 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:43:59.897 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:59.897 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:59.897 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:59.897 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.898 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.898 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:43:59.898 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:43:59.898 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:43:59.898 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:43:59.898 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.898 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:43:59.898 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:43:59.898 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:59.898 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.898 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.898 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:43:59.898 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:43:59.898 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:43:59.899 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:59.899 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:59.899 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.899 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:43:59.899 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:43:59.899 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:43:59.899 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:43:59.899 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:43:59.899 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:59.899 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:59.899 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.899 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:43:59.899 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:43:59.900 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.900 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:59.900 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:59.900 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:43:59.900 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:59.900 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:59.900 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:43:59.900 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.900 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:59.900 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:59.900 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5 2024-09-16T04:43:59.900 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:59.901 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:59.901 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:43:59.901 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.901 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:59.901 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:59.901 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6 2024-09-16T04:43:59.901 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:59.901 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:43:59.901 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:43:59.902 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:43:59.902 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:43:59.902 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:43:59.902 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:43:59.902 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:43:59.902 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:43:59.902 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:43:59.902 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:43:59.902 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:43:59.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f5429711640 1 -- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5410038360 msgr2=0x7f541003a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:59.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f5429711640 1 --2- 172.21.15.46:0/2651404003 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5410038360 0x7f541003a820 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f54240a4d80 tx=0x7f541c009540 comp rx=0 tx=0).stop 2024-09-16T04:43:59.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f5429711640 10 monclient: shutdown 2024-09-16T04:43:59.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f5429711640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:43:59.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f5429711640 1 -- 172.21.15.46:0/2651404003 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54240a3ff0 msgr2=0x7f54241cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:43:59.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f5429711640 1 --2- 172.21.15.46:0/2651404003 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54240a3ff0 0x7f54241cbc30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5418000c00 tx=0x7f54180046c0 comp rx=0 tx=0).stop 2024-09-16T04:43:59.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f54237fe640 1 -- 172.21.15.46:0/2651404003 reap_dead start 2024-09-16T04:43:59.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f54237fe640 1 -- 172.21.15.46:0/2651404003 reap_dead start 2024-09-16T04:43:59.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f5429711640 1 -- 172.21.15.46:0/2651404003 shutdown_connections 2024-09-16T04:43:59.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.870+0000 7f5429711640 1 -- 172.21.15.46:0/2651404003 >> 172.21.15.46:0/2651404003 conn(0x7f54240a6340 msgr2=0x7f542413b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:43:59.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.871+0000 7f5429711640 1 -- 172.21.15.46:0/2651404003 shutdown_connections 2024-09-16T04:43:59.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:43:59.871+0000 7f5429711640 1 -- 172.21.15.46:0/2651404003 wait complete. 2024-09-16T04:43:59.905 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:43:59.905 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:44:00.371 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38357]: starting. 2024-09-16T04:44:00.371 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38357]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-38355 2024-09-16T04:44:00.371 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38357]: shutting down. 2024-09-16T04:44:02.473 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:02.474 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38390]: starting. 2024-09-16T04:44:02.474 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38390]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-38355 2024-09-16T04:44:02.474 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38390]: created object 0... 2024-09-16T04:44:02.474 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38390]: created object 25... 2024-09-16T04:44:02.474 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38390]: created object 49... 2024-09-16T04:44:02.474 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38390]: finishing. 2024-09-16T04:44:02.474 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38390]: shutting down. 2024-09-16T04:44:02.485 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:02.485 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38392]: starting. 2024-09-16T04:44:02.486 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38392]: listing objects. 2024-09-16T04:44:02.486 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38392]: listed object 0... 2024-09-16T04:44:02.486 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38392]: listed object 25... 2024-09-16T04:44:02.486 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38392]: saw 50 objects 2024-09-16T04:44:02.486 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38392]: shutting down. 2024-09-16T04:44:03.382 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:03.383 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38391]: starting. 2024-09-16T04:44:03.383 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38391]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-38355 2024-09-16T04:44:03.383 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38391]: shutting down. 2024-09-16T04:44:03.384 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:03.384 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:03.384 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:44:03.386 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:44:05.228 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38461]: starting. 2024-09-16T04:44:05.229 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38461]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-38460 2024-09-16T04:44:05.229 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38461]: created object 0... 2024-09-16T04:44:05.229 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38461]: created object 25... 2024-09-16T04:44:05.229 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38461]: created object 49... 2024-09-16T04:44:05.229 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38461]: finishing. 2024-09-16T04:44:05.229 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38461]: shutting down. 2024-09-16T04:44:06.147 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38462]: starting. 2024-09-16T04:44:06.147 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38462]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-38460 2024-09-16T04:44:06.147 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38462]: shutting down. 2024-09-16T04:44:08.249 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38498]: starting. 2024-09-16T04:44:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38498]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-38460 2024-09-16T04:44:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38498]: created object 0... 2024-09-16T04:44:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38498]: created object 25... 2024-09-16T04:44:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38498]: created object 49... 2024-09-16T04:44:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38498]: finishing. 2024-09-16T04:44:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38498]: shutting down. 2024-09-16T04:44:08.262 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:08.262 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38500]: starting. 2024-09-16T04:44:08.262 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38500]: listing objects. 2024-09-16T04:44:08.263 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38500]: listed object 0... 2024-09-16T04:44:08.263 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38500]: listed object 25... 2024-09-16T04:44:08.263 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38500]: saw 50 objects 2024-09-16T04:44:08.263 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38500]: shutting down. 2024-09-16T04:44:09.163 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:09.164 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38499]: starting. 2024-09-16T04:44:09.164 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38499]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-38460 2024-09-16T04:44:09.164 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38499]: shutting down. 2024-09-16T04:44:09.165 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:09.165 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:09.165 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:44:09.168 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:44:11.266 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38570]: starting. 2024-09-16T04:44:11.266 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38570]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-38569 2024-09-16T04:44:11.266 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38570]: created object 0... 2024-09-16T04:44:11.266 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38570]: created object 25... 2024-09-16T04:44:11.266 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38570]: created object 49... 2024-09-16T04:44:11.267 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38570]: finishing. 2024-09-16T04:44:11.267 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38570]: shutting down. 2024-09-16T04:44:12.190 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38571]: starting. 2024-09-16T04:44:12.191 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38571]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-38569 2024-09-16T04:44:12.191 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38571]: shutting down. 2024-09-16T04:44:14.230 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:14.230 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38606]: starting. 2024-09-16T04:44:14.230 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38606]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-38569 2024-09-16T04:44:14.230 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38606]: created object 0... 2024-09-16T04:44:14.230 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38606]: created object 25... 2024-09-16T04:44:14.230 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38606]: created object 49... 2024-09-16T04:44:14.230 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38606]: finishing. 2024-09-16T04:44:14.231 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38606]: shutting down. 2024-09-16T04:44:14.243 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:14.243 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38608]: starting. 2024-09-16T04:44:14.243 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38608]: listing objects. 2024-09-16T04:44:14.243 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38608]: listed object 0... 2024-09-16T04:44:14.243 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38608]: listed object 25... 2024-09-16T04:44:14.243 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38608]: saw 50 objects 2024-09-16T04:44:14.243 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38608]: shutting down. 2024-09-16T04:44:15.147 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:15.147 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38607]: starting. 2024-09-16T04:44:15.147 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38607]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-38569 2024-09-16T04:44:15.147 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38607]: shutting down. 2024-09-16T04:44:15.148 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:15.148 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:15.149 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:44:15.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:44:17.304 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38677]: starting. 2024-09-16T04:44:17.304 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38677]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-38676 2024-09-16T04:44:17.304 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38677]: created object 0... 2024-09-16T04:44:17.304 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38677]: created object 25... 2024-09-16T04:44:17.304 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38677]: created object 49... 2024-09-16T04:44:17.305 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38677]: finishing. 2024-09-16T04:44:17.305 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38677]: shutting down. 2024-09-16T04:44:18.213 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38678]: starting. 2024-09-16T04:44:18.213 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38678]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-38676 2024-09-16T04:44:18.213 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38678]: shutting down. 2024-09-16T04:44:19.909 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:44:19.910 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T04:44:19.910 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T04:44:19.910 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T04:44:19.913 INFO:tasks.ceph.mon.a:Started 2024-09-16T04:44:19.913 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:44:19.913 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:44:20.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.024+0000 7efd62b03640 10 monclient: get_monmap_and_config 2024-09-16T04:44:20.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.024+0000 7efd62b03640 10 monclient: build_initial_monmap 2024-09-16T04:44:20.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.024+0000 7efd62b03640 10 monclient: monmap: 2024-09-16T04:44:20.027 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:20.027 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:20.027 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:44:20.026161+0000 2024-09-16T04:44:20.027 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:44:20.026161+0000 2024-09-16T04:44:20.027 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:20.027 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:20.027 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:44:20.027 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:44:20.027 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:44:20.027 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:44:20.027 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:44:20.027 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.025+0000 7efd62b03640 1 Processor -- start 2024-09-16T04:44:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.025+0000 7efd62b03640 1 -- start start 2024-09-16T04:44:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.025+0000 7efd62b03640 10 monclient: init 2024-09-16T04:44:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.025+0000 7efd62b03640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.025+0000 7efd62b03640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-16T04:44:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.025+0000 7efd62b03640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efd5c0a3c10 0x7efd5c09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.025+0000 7efd62b03640 10 monclient(hunting): picked mon.noname-c con 0x7efd5c0a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:44:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.025+0000 7efd62b03640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efd5c0a3ff0 0x7efd5c0a4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.025+0000 7efd62b03640 10 monclient(hunting): picked mon.noname-a con 0x7efd5c0a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:44:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.025+0000 7efd62b03640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd5c1369d0 0x7efd5c136e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.025+0000 7efd62b03640 10 monclient(hunting): picked mon.noname-b con 0x7efd5c1369d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.025+0000 7efd62b03640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.025+0000 7efd62b03640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7efd5c09ab70 con 0x7efd5c0a3ff0 2024-09-16T04:44:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd62b03640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd62b03640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7efd5c0a4d60 con 0x7efd5c0a3c10 2024-09-16T04:44:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd62b03640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd62b03640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7efd5c09b030 con 0x7efd5c1369d0 2024-09-16T04:44:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd62b03640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd62b03640 10 monclient(hunting): authenticate will time out at 1058.442017s 2024-09-16T04:44:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd5bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efd5c0a3ff0 0x7efd5c0a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd610a0640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd5c1369d0 0x7efd5c136e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd5bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efd5c0a3ff0 0x7efd5c0a4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50150/0 (socket says 172.21.15.46:50150) 2024-09-16T04:44:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd5bfff640 1 -- 172.21.15.46:0/1952403252 learned_addr learned my addr 172.21.15.46:0/1952403252 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:20.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd5bfff640 10 monclient(hunting): get_auth_request con 0x7efd5c0a3ff0 auth_method 0 2024-09-16T04:44:20.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:20.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:20.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:20.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd610a0640 10 monclient(hunting): get_auth_request con 0x7efd5c1369d0 auth_method 0 2024-09-16T04:44:20.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd610a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:20.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd610a0640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:20.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd610a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:20.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd610a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:20.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd610a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd5bfff640 1 -- 172.21.15.46:0/1952403252 >> v2:172.21.15.46:3300/0 conn(0x7efd5c0a3ff0 msgr2=0x7efd5c0a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:44:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd5bfff640 1 -- 172.21.15.46:0/1952403252 >> v2:172.21.15.46:3300/0 conn(0x7efd5c0a3ff0 msgr2=0x7efd5c0a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd610a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd5bfff640 1 --2- 172.21.15.46:0/1952403252 >> v2:172.21.15.46:3300/0 conn(0x7efd5c0a3ff0 0x7efd5c0a4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd6089f640 1 --2- 172.21.15.46:0/1952403252 >> v2:172.21.15.114:3300/0 conn(0x7efd5c0a3c10 0x7efd5c09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd5bfff640 1 --2- 172.21.15.46:0/1952403252 >> v2:172.21.15.46:3300/0 conn(0x7efd5c0a3ff0 0x7efd5c0a4400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd6089f640 10 monclient(hunting): get_auth_request con 0x7efd5c0a3c10 auth_method 0 2024-09-16T04:44:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd6089f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd6089f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd6089f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd610a0640 10 monclient(hunting): handle_auth_done global_id 64244 payload 274 2024-09-16T04:44:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd610a0640 1 -- 172.21.15.46:0/1952403252 >> v2:172.21.15.114:3300/0 conn(0x7efd5c0a3c10 msgr2=0x7efd5c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:44:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd610a0640 1 --2- 172.21.15.46:0/1952403252 >> v2:172.21.15.114:3300/0 conn(0x7efd5c0a3c10 0x7efd5c09bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd610a0640 1 -- 172.21.15.46:0/1952403252 >> v2:172.21.15.46:3300/0 conn(0x7efd5c0a3ff0 msgr2=0x7efd5c0a4400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:44:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd610a0640 1 --2- 172.21.15.46:0/1952403252 >> v2:172.21.15.46:3300/0 conn(0x7efd5c0a3ff0 0x7efd5c0a4400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd610a0640 10 monclient: _finish_hunting 0 2024-09-16T04:44:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd610a0640 1 monclient: found mon.noname-b 2024-09-16T04:44:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd610a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd610a0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:44:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.026+0000 7efd610a0640 1 -- 172.21.15.46:0/1952403252 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd5c09b310 con 0x7efd5c1369d0 2024-09-16T04:44:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd610a0640 1 --2- 172.21.15.46:0/1952403252 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd5c1369d0 0x7efd5c136e20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efd5c06be80 tx=0x7efd5402a4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=478f78db9a5291f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd5b7fe640 1 -- 172.21.15.46:0/1952403252 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd54007740 con 0x7efd5c1369d0 2024-09-16T04:44:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd5b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:44:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd5b7fe640 10 monclient: dump: 2024-09-16T04:44:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:20.040 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:20.040 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:20.040 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:20.040 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:20.040 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:20.040 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:20.040 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:20.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd5b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:44:20.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd5b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:44:20.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:20.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd62b03640 5 monclient: authenticate success, global_id 64244 2024-09-16T04:44:20.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd62b03640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:44:20.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd5b7fe640 1 -- 172.21.15.46:0/1952403252 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd540078e0 con 0x7efd5c1369d0 2024-09-16T04:44:20.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd5b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:20.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd62b03640 10 monclient: get_monmap_and_config success 2024-09-16T04:44:20.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd62b03640 4 set_mon_vals no callback set 2024-09-16T04:44:20.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd5b7fe640 1 -- 172.21.15.46:0/1952403252 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd54007cb0 con 0x7efd5c1369d0 2024-09-16T04:44:20.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:20.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd5b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:44:20.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd5b7fe640 10 monclient: dump: 2024-09-16T04:44:20.041 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:20.041 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:20.041 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:20.041 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:20.041 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:20.041 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:20.041 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:20.041 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:20.041 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:20.042 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:20.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd62b03640 10 monclient: shutdown 2024-09-16T04:44:20.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd62b03640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:20.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd62b03640 1 -- 172.21.15.46:0/1952403252 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd5c1369d0 msgr2=0x7efd5c136e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:20.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd62b03640 1 --2- 172.21.15.46:0/1952403252 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd5c1369d0 0x7efd5c136e20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efd5c06be80 tx=0x7efd5402a4b0 comp rx=0 tx=0).stop 2024-09-16T04:44:20.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd62b03640 1 -- 172.21.15.46:0/1952403252 shutdown_connections 2024-09-16T04:44:20.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd62b03640 1 --2- 172.21.15.46:0/1952403252 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd5c1369d0 0x7efd5c136e20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:20.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd62b03640 1 --2- 172.21.15.46:0/1952403252 >> v2:172.21.15.46:3300/0 conn(0x7efd5c0a3ff0 0x7efd5c0a4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:20.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd62b03640 1 --2- 172.21.15.46:0/1952403252 >> v2:172.21.15.114:3300/0 conn(0x7efd5c0a3c10 0x7efd5c09bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:20.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.027+0000 7efd62b03640 1 -- 172.21.15.46:0/1952403252 >> 172.21.15.46:0/1952403252 conn(0x7efd5c0a5d10 msgr2=0x7efd5c0a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:20.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.028+0000 7efd62b03640 1 -- 172.21.15.46:0/1952403252 shutdown_connections 2024-09-16T04:44:20.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.028+0000 7efd62b03640 1 -- 172.21.15.46:0/1952403252 wait complete. 2024-09-16T04:44:20.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.028+0000 7efd62b03640 10 monclient: build_initial_monmap 2024-09-16T04:44:20.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.028+0000 7efd62b03640 10 monclient: monmap: 2024-09-16T04:44:20.043 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:20.043 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:44:20.043 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:44:20.043 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:44:20.043 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:20.043 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:20.043 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:44:20.043 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:44:20.043 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:44:20.044 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.028+0000 7efd62b03640 1 Processor -- start 2024-09-16T04:44:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd62b03640 1 -- start start 2024-09-16T04:44:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd62b03640 10 monclient: init 2024-09-16T04:44:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd62b03640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd62b03640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:44:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd62b03640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd5c0a3c10 0x7efd5c1cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd62b03640 10 monclient(hunting): picked mon.noname-b con 0x7efd5c0a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:44:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd62b03640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd5c0a3ff0 0x7efd5c1cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd62b03640 10 monclient(hunting): picked mon.noname-c con 0x7efd5c0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd62b03640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd5c1369d0 0x7efd5c123530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd62b03640 10 monclient(hunting): picked mon.noname-a con 0x7efd5c1369d0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:44:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd62b03640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd62b03640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efd5c06e3d0 con 0x7efd5c1369d0 2024-09-16T04:44:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd62b03640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd62b03640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efd5c0a4940 con 0x7efd5c0a3c10 2024-09-16T04:44:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd62b03640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd62b03640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7efd5c06e550 con 0x7efd5c0a3ff0 2024-09-16T04:44:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd62b03640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd62b03640 10 monclient(hunting): authenticate will time out at 1058.445666s 2024-09-16T04:44:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd5bfff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd5c0a3ff0 0x7efd5c1cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd610a0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd5c1369d0 0x7efd5c123530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd5bfff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd5c0a3ff0 0x7efd5c1cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:36058/0 (socket says 172.21.15.46:36058) 2024-09-16T04:44:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd5bfff640 1 -- 172.21.15.46:0/32802145 learned_addr learned my addr 172.21.15.46:0/32802145 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:20.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd5bfff640 10 monclient(hunting): get_auth_request con 0x7efd5c0a3ff0 auth_method 0 2024-09-16T04:44:20.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:20.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:20.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd610a0640 10 monclient(hunting): get_auth_request con 0x7efd5c1369d0 auth_method 0 2024-09-16T04:44:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd610a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd610a0640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd610a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd610a0640 1 -- 172.21.15.46:0/32802145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd5c1369d0 msgr2=0x7efd5c123530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:44:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd610a0640 1 -- 172.21.15.46:0/32802145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd5c1369d0 msgr2=0x7efd5c123530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:44:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd610a0640 1 --2- 172.21.15.46:0/32802145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd5c1369d0 0x7efd5c123530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.029+0000 7efd610a0640 1 --2- 172.21.15.46:0/32802145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd5c1369d0 0x7efd5c123530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd6089f640 1 --2- 172.21.15.46:0/32802145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd5c0a3c10 0x7efd5c1cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd5bfff640 10 monclient(hunting): handle_auth_done global_id 64247 payload 824 2024-09-16T04:44:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd5bfff640 1 -- 172.21.15.46:0/32802145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd5c0a3c10 msgr2=0x7efd5c1cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd5bfff640 1 --2- 172.21.15.46:0/32802145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd5c0a3c10 0x7efd5c1cbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd5bfff640 1 -- 172.21.15.46:0/32802145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd5c1369d0 msgr2=0x7efd5c123530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:44:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd5bfff640 1 --2- 172.21.15.46:0/32802145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd5c1369d0 0x7efd5c123530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd5bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:44:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd5bfff640 1 monclient: found mon.noname-c 2024-09-16T04:44:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd5bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:44:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd5bfff640 1 -- 172.21.15.46:0/32802145 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd5c127160 con 0x7efd5c0a3ff0 2024-09-16T04:44:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd6089f640 10 monclient: get_auth_request con 0x7efd5c0a3c10 auth_method 0 2024-09-16T04:44:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd6089f640 1 --2- 172.21.15.46:0/32802145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd5c0a3c10 0x7efd5c1cbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:44:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd5bfff640 1 --2- 172.21.15.46:0/32802145 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd5c0a3ff0 0x7efd5c1cc4b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efd4800e810 tx=0x7efd4800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd597fa640 1 -- 172.21.15.46:0/32802145 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd48017070 con 0x7efd5c0a3ff0 2024-09-16T04:44:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd597fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:44:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd597fa640 10 monclient: dump: 2024-09-16T04:44:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:20.049 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd597fa640 10 monclient: _finish_auth 0 2024-09-16T04:44:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd597fa640 10 monclient: _check_auth_tickets 2024-09-16T04:44:20.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:20.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd62b03640 5 monclient: authenticate success, global_id 64247 2024-09-16T04:44:20.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd597fa640 1 -- 172.21.15.46:0/32802145 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd48004020 con 0x7efd5c0a3ff0 2024-09-16T04:44:20.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd62b03640 10 monclient: _renew_subs 2024-09-16T04:44:20.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd62b03640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:44:20.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd62b03640 1 -- 172.21.15.46:0/32802145 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd5c1263b0 con 0x7efd5c0a3ff0 2024-09-16T04:44:20.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd597fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:20.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd597fa640 1 -- 172.21.15.46:0/32802145 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd4800f040 con 0x7efd5c0a3ff0 2024-09-16T04:44:20.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:20.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd597fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:44:20.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd597fa640 10 monclient: dump: 2024-09-16T04:44:20.050 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:20.050 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:20.050 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:20.050 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:20.051 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd62b03640 10 monclient: _renew_subs 2024-09-16T04:44:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd62b03640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:44:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd5affd640 4 set_mon_vals no callback set 2024-09-16T04:44:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.030+0000 7efd62b03640 1 -- 172.21.15.46:0/32802145 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efd5c0a8ea0 con 0x7efd5c0a3ff0 2024-09-16T04:44:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.031+0000 7efd597fa640 1 -- 172.21.15.46:0/32802145 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7efd4800c670 con 0x7efd5c0a3ff0 2024-09-16T04:44:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.031+0000 7efd62b03640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.031+0000 7efd62b03640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.031+0000 7efd62b03640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:44:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.031+0000 7efd597fa640 1 --2- 172.21.15.46:0/32802145 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7efd400383e0 0x7efd4003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.031+0000 7efd62b03640 1 -- 172.21.15.46:0/32802145 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd5c09bcf0 con 0x7efd5c0a3ff0 2024-09-16T04:44:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.031+0000 7efd597fa640 1 -- 172.21.15.46:0/32802145 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(216..216 src has 1..216) ==== 4738+0+0 (secure 0 0 0) 0x7efd4804b2b0 con 0x7efd5c0a3ff0 2024-09-16T04:44:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.032+0000 7efd6089f640 1 --2- 172.21.15.46:0/32802145 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7efd400383e0 0x7efd4003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.032+0000 7efd6089f640 10 monclient: get_auth_request con 0x7efd400383e0 auth_method 0 2024-09-16T04:44:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.032+0000 7efd6089f640 1 --2- 172.21.15.46:0/32802145 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7efd400383e0 0x7efd4003a8a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7efd50006fd0 tx=0x7efd50008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.037+0000 7efd597fa640 1 -- 172.21.15.46:0/32802145 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7efd48004410 con 0x7efd5c0a3ff0 2024-09-16T04:44:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.037+0000 7efd597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:20.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.037+0000 7efd597fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:44:20.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.296+0000 7efd62b03640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:44:20.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.296+0000 7efd62b03640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:44:20.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.296+0000 7efd62b03640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:44:20.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.296+0000 7efd62b03640 1 -- 172.21.15.46:0/32802145 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7efd5c126f40 con 0x7efd5c0a3ff0 2024-09-16T04:44:20.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.296+0000 7efd597fa640 1 -- 172.21.15.46:0/32802145 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7efd4804b080 con 0x7efd5c0a3ff0 2024-09-16T04:44:20.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.296+0000 7efd597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:44:20.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.296+0000 7efd597fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:44:20.299 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":60,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:44:20.299 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:20.299 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38713]: starting. 2024-09-16T04:44:20.299 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38713]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-38676 2024-09-16T04:44:20.300 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38713]: created object 0... 2024-09-16T04:44:20.300 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38713]: created object 25... 2024-09-16T04:44:20.300 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38713]: created object 49... 2024-09-16T04:44:20.300 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38713]: finishing. 2024-09-16T04:44:20.300 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38713]: shutting down. 2024-09-16T04:44:20.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.298+0000 7efd62b03640 1 -- 172.21.15.46:0/32802145 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7efd400383e0 msgr2=0x7efd4003a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:20.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.298+0000 7efd62b03640 1 --2- 172.21.15.46:0/32802145 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7efd400383e0 0x7efd4003a8a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7efd50006fd0 tx=0x7efd50008040 comp rx=0 tx=0).stop 2024-09-16T04:44:20.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.298+0000 7efd62b03640 10 monclient: shutdown 2024-09-16T04:44:20.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.298+0000 7efd62b03640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:20.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.298+0000 7efd62b03640 1 -- 172.21.15.46:0/32802145 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd5c0a3ff0 msgr2=0x7efd5c1cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:20.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.298+0000 7efd62b03640 1 --2- 172.21.15.46:0/32802145 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd5c0a3ff0 0x7efd5c1cc4b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efd4800e810 tx=0x7efd4800ecd0 comp rx=0 tx=0).stop 2024-09-16T04:44:20.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.298+0000 7efd62b03640 1 -- 172.21.15.46:0/32802145 shutdown_connections 2024-09-16T04:44:20.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.298+0000 7efd62b03640 1 --2- 172.21.15.46:0/32802145 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7efd400383e0 0x7efd4003a8a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:20.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.298+0000 7efd62b03640 1 --2- 172.21.15.46:0/32802145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd5c1369d0 0x7efd5c123530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:20.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.298+0000 7efd62b03640 1 --2- 172.21.15.46:0/32802145 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd5c0a3ff0 0x7efd5c1cc4b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:20.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.298+0000 7efd62b03640 1 --2- 172.21.15.46:0/32802145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd5c0a3c10 0x7efd5c1cbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:20.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.298+0000 7efd62b03640 1 -- 172.21.15.46:0/32802145 >> 172.21.15.46:0/32802145 conn(0x7efd5c0a5d10 msgr2=0x7efd5c0a24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:20.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.298+0000 7efd62b03640 1 -- 172.21.15.46:0/32802145 shutdown_connections 2024-09-16T04:44:20.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:20.298+0000 7efd62b03640 1 -- 172.21.15.46:0/32802145 wait complete. 2024-09-16T04:44:20.310 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:20.311 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38715]: starting. 2024-09-16T04:44:20.311 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38715]: listing objects. 2024-09-16T04:44:20.311 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38715]: listed object 0... 2024-09-16T04:44:20.311 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38715]: listed object 25... 2024-09-16T04:44:20.311 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38715]: saw 50 objects 2024-09-16T04:44:20.311 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38715]: shutting down. 2024-09-16T04:44:21.224 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:21.224 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38714]: starting. 2024-09-16T04:44:21.224 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38714]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-38676 2024-09-16T04:44:21.224 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38714]: shutting down. 2024-09-16T04:44:21.226 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:21.226 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:21.226 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:44:21.228 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:44:23.313 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:44:23.326 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38784]: starting. 2024-09-16T04:44:23.327 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38784]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-38783 2024-09-16T04:44:23.327 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38784]: created object 0... 2024-09-16T04:44:23.327 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38784]: created object 25... 2024-09-16T04:44:23.327 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38784]: created object 49... 2024-09-16T04:44:23.327 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38784]: finishing. 2024-09-16T04:44:23.327 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38784]: shutting down. 2024-09-16T04:44:23.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.415+0000 7fb07b991640 10 monclient: get_monmap_and_config 2024-09-16T04:44:23.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.416+0000 7fb07b991640 10 monclient: build_initial_monmap 2024-09-16T04:44:23.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.416+0000 7fb07b991640 10 monclient: monmap: 2024-09-16T04:44:23.430 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:23.430 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:23.430 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:44:23.417498+0000 2024-09-16T04:44:23.430 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:44:23.417498+0000 2024-09-16T04:44:23.430 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:23.430 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:23.430 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:44:23.430 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:44:23.430 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:44:23.430 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:44:23.431 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:44:23.431 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:23.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.416+0000 7fb07b991640 1 Processor -- start 2024-09-16T04:44:23.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.416+0000 7fb07b991640 1 -- start start 2024-09-16T04:44:23.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.416+0000 7fb07b991640 10 monclient: init 2024-09-16T04:44:23.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07b991640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:23.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07b991640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-16T04:44:23.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07b991640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb07406e160 0x7fb0740a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:23.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07b991640 10 monclient(hunting): picked mon.noname-a con 0x7fb07406e160 addr v2:172.21.15.46:3300/0 2024-09-16T04:44:23.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07b991640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb0740a2ee0 addr v1:172.21.15.114:6789/0 2024-09-16T04:44:23.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07b991640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb074137050 0x7fb07413ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:23.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07b991640 10 monclient(hunting): picked mon.noname-b con 0x7fb074137050 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:23.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07b991640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0740a3fd0 con 0x7fb0740a2ee0 2024-09-16T04:44:23.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07b991640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:23.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07b991640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb0740a3300 con 0x7fb07406e160 2024-09-16T04:44:23.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07b991640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:23.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07b991640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fb07406e540 con 0x7fb074137050 2024-09-16T04:44:23.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07b991640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:23.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07b991640 10 monclient(hunting): authenticate will time out at 1061.833366s 2024-09-16T04:44:23.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07972d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb07406e160 0x7fb0740a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:23.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb079f2e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb074137050 0x7fb07413ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:23.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07972d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb07406e160 0x7fb0740a29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50168/0 (socket says 172.21.15.46:50168) 2024-09-16T04:44:23.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07972d640 1 -- 172.21.15.46:0/1439760016 learned_addr learned my addr 172.21.15.46:0/1439760016 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:23.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07972d640 10 monclient(hunting): get_auth_request con 0x7fb07406e160 auth_method 0 2024-09-16T04:44:23.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07972d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:23.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07972d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:23.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07972d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:23.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb079f2e640 10 monclient(hunting): get_auth_request con 0x7fb074137050 auth_method 0 2024-09-16T04:44:23.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb079f2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb079f2e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb079f2e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb079f2e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb079f2e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07972d640 1 -- 172.21.15.46:0/1439760016 >> v2:172.21.15.46:3300/0 conn(0x7fb07406e160 msgr2=0x7fb0740a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:44:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07972d640 1 -- 172.21.15.46:0/1439760016 >> v2:172.21.15.46:3300/0 conn(0x7fb07406e160 msgr2=0x7fb0740a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07972d640 1 --2- 172.21.15.46:0/1439760016 >> v2:172.21.15.46:3300/0 conn(0x7fb07406e160 0x7fb0740a29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb079f2e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb07972d640 1 --2- 172.21.15.46:0/1439760016 >> v2:172.21.15.46:3300/0 conn(0x7fb07406e160 0x7fb0740a29a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb079f2e640 10 monclient(hunting): handle_auth_done global_id 64259 payload 274 2024-09-16T04:44:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb079f2e640 1 -- 172.21.15.46:0/1439760016 >> v2:172.21.15.46:3300/0 conn(0x7fb07406e160 msgr2=0x7fb0740a29a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:44:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb079f2e640 1 --2- 172.21.15.46:0/1439760016 >> v2:172.21.15.46:3300/0 conn(0x7fb07406e160 0x7fb0740a29a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb079f2e640 1 -- 172.21.15.46:0/1439760016 >> v1:172.21.15.114:6789/0 conn(0x7fb0740a2ee0 legacy=0x7fb074136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:23.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb079f2e640 10 monclient: _finish_hunting 0 2024-09-16T04:44:23.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb079f2e640 1 monclient: found mon.noname-b 2024-09-16T04:44:23.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb079f2e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:23.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb079f2e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:44:23.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.417+0000 7fb079f2e640 1 -- 172.21.15.46:0/1439760016 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0740a0bd0 con 0x7fb074137050 2024-09-16T04:44:23.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.418+0000 7fb079f2e640 1 --2- 172.21.15.46:0/1439760016 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb074137050 0x7fb07413ec80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb06c009d90 tx=0x7fb06c02a530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c0b1d728b48db7c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:23.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.418+0000 7fb06bfff640 1 -- 172.21.15.46:0/1439760016 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb06c007740 con 0x7fb074137050 2024-09-16T04:44:23.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.418+0000 7fb06bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:23.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.418+0000 7fb06bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:44:23.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.418+0000 7fb06bfff640 10 monclient: dump: 2024-09-16T04:44:23.435 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:23.435 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:23.435 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:23.436 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:23.436 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:23.436 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:23.436 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:23.436 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:23.436 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:23.436 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:23.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.418+0000 7fb06bfff640 10 monclient: _finish_auth 0 2024-09-16T04:44:23.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.418+0000 7fb06bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:44:23.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.418+0000 7fb06bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:23.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.418+0000 7fb06bfff640 1 -- 172.21.15.46:0/1439760016 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb06c0078e0 con 0x7fb074137050 2024-09-16T04:44:23.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.418+0000 7fb07b991640 5 monclient: authenticate success, global_id 64259 2024-09-16T04:44:23.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.418+0000 7fb07b991640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:44:23.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.418+0000 7fb06bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:23.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.418+0000 7fb07b991640 10 monclient: get_monmap_and_config success 2024-09-16T04:44:23.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.418+0000 7fb06bfff640 1 -- 172.21.15.46:0/1439760016 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb06c007cb0 con 0x7fb074137050 2024-09-16T04:44:23.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.418+0000 7fb06bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:23.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.418+0000 7fb07b991640 4 set_mon_vals no callback set 2024-09-16T04:44:23.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.418+0000 7fb06bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:44:23.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.418+0000 7fb06bfff640 10 monclient: dump: 2024-09-16T04:44:23.437 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:23.438 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.419+0000 7fb07b991640 10 monclient: shutdown 2024-09-16T04:44:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.419+0000 7fb07b991640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.419+0000 7fb07b991640 1 -- 172.21.15.46:0/1439760016 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb074137050 msgr2=0x7fb07413ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.419+0000 7fb07b991640 1 --2- 172.21.15.46:0/1439760016 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb074137050 0x7fb07413ec80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb06c009d90 tx=0x7fb06c02a530 comp rx=0 tx=0).stop 2024-09-16T04:44:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.419+0000 7fb07b991640 1 -- 172.21.15.46:0/1439760016 shutdown_connections 2024-09-16T04:44:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.419+0000 7fb07b991640 1 --2- 172.21.15.46:0/1439760016 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb074137050 0x7fb07413ec80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.419+0000 7fb07b991640 1 --2- 172.21.15.46:0/1439760016 >> v2:172.21.15.46:3300/0 conn(0x7fb07406e160 0x7fb0740a29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.419+0000 7fb07b991640 1 -- 172.21.15.46:0/1439760016 >> 172.21.15.46:0/1439760016 conn(0x7fb0740a5c10 msgr2=0x7fb0740a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.419+0000 7fb07b991640 1 -- 172.21.15.46:0/1439760016 shutdown_connections 2024-09-16T04:44:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.419+0000 7fb07b991640 1 -- 172.21.15.46:0/1439760016 wait complete. 2024-09-16T04:44:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.419+0000 7fb07b991640 10 monclient: build_initial_monmap 2024-09-16T04:44:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.420+0000 7fb07b991640 10 monclient: monmap: 2024-09-16T04:44:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:44:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:44:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:44:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:23.440 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:44:23.440 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:44:23.440 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:44:23.440 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:23.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.420+0000 7fb07b991640 1 Processor -- start 2024-09-16T04:44:23.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.420+0000 7fb07b991640 1 -- start start 2024-09-16T04:44:23.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.420+0000 7fb07b991640 10 monclient: init 2024-09-16T04:44:23.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.420+0000 7fb07b991640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:23.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.420+0000 7fb07b991640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:44:23.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.420+0000 7fb07b991640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb07406e160 0x7fb0741cb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.420+0000 7fb07b991640 10 monclient(hunting): picked mon.noname-a con 0x7fb07406e160 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:44:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.420+0000 7fb07b991640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb0740a2ee0 0x7fb074135bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.420+0000 7fb07b991640 10 monclient(hunting): picked mon.noname-c con 0x7fb0740a2ee0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.420+0000 7fb07b991640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb074137050 0x7fb074136100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.420+0000 7fb07b991640 10 monclient(hunting): picked mon.noname-b con 0x7fb074137050 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:44:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.420+0000 7fb07b991640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.420+0000 7fb07b991640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb0740a34a0 con 0x7fb07406e160 2024-09-16T04:44:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.420+0000 7fb07b991640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.420+0000 7fb07b991640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb0740a5130 con 0x7fb074137050 2024-09-16T04:44:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.420+0000 7fb07b991640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.420+0000 7fb07b991640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fb0740a8500 con 0x7fb0740a2ee0 2024-09-16T04:44:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.420+0000 7fb07b991640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.420+0000 7fb07b991640 10 monclient(hunting): authenticate will time out at 1061.836962s 2024-09-16T04:44:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb07972d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb07406e160 0x7fb0741cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb078f2c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb0740a2ee0 0x7fb074135bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb07972d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb07406e160 0x7fb0741cb910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50184/0 (socket says 172.21.15.46:50184) 2024-09-16T04:44:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb07972d640 1 -- 172.21.15.46:0/340288807 learned_addr learned my addr 172.21.15.46:0/340288807 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb07972d640 10 monclient(hunting): get_auth_request con 0x7fb07406e160 auth_method 0 2024-09-16T04:44:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb07972d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb07972d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb07972d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb078f2c640 10 monclient(hunting): get_auth_request con 0x7fb0740a2ee0 auth_method 0 2024-09-16T04:44:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb078f2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb078f2c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb078f2c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb079f2e640 1 --2- 172.21.15.46:0/340288807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb074137050 0x7fb074136100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb079f2e640 10 monclient(hunting): get_auth_request con 0x7fb074137050 auth_method 0 2024-09-16T04:44:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb07972d640 1 -- 172.21.15.46:0/340288807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb07406e160 msgr2=0x7fb0741cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:44:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb079f2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb079f2e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb079f2e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb07972d640 1 -- 172.21.15.46:0/340288807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb07406e160 msgr2=0x7fb0741cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:44:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb07972d640 1 --2- 172.21.15.46:0/340288807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb07406e160 0x7fb0741cb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb078f2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb078f2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb078f2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb07972d640 1 --2- 172.21.15.46:0/340288807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb07406e160 0x7fb0741cb910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb079f2e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb079f2e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb079f2e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb078f2c640 10 monclient(hunting): handle_auth_done global_id 64262 payload 824 2024-09-16T04:44:23.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb078f2c640 1 -- 172.21.15.46:0/340288807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb074137050 msgr2=0x7fb074136100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:23.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb078f2c640 1 --2- 172.21.15.46:0/340288807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb074137050 0x7fb074136100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:23.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb078f2c640 1 -- 172.21.15.46:0/340288807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb07406e160 msgr2=0x7fb0741cb910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:44:23.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb078f2c640 1 --2- 172.21.15.46:0/340288807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb07406e160 0x7fb0741cb910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:23.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb078f2c640 10 monclient: _finish_hunting 0 2024-09-16T04:44:23.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb078f2c640 1 monclient: found mon.noname-c 2024-09-16T04:44:23.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb078f2c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:23.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb078f2c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:44:23.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb078f2c640 1 -- 172.21.15.46:0/340288807 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0741272e0 con 0x7fb0740a2ee0 2024-09-16T04:44:23.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb078f2c640 1 --2- 172.21.15.46:0/340288807 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb0740a2ee0 0x7fb074135bc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb06400c340 tx=0x7fb06400c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:23.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb069ffb640 1 -- 172.21.15.46:0/340288807 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0640052d0 con 0x7fb0740a2ee0 2024-09-16T04:44:23.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb069ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:23.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb069ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:44:23.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb069ffb640 10 monclient: dump: 2024-09-16T04:44:23.445 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:23.445 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:23.445 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:23.445 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:23.445 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:23.445 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:23.445 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:23.445 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:23.445 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:23.445 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:23.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb069ffb640 10 monclient: _finish_auth 0 2024-09-16T04:44:23.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb069ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:44:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb069ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb069ffb640 1 -- 172.21.15.46:0/340288807 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb064005490 con 0x7fb0740a2ee0 2024-09-16T04:44:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb07b991640 5 monclient: authenticate success, global_id 64262 2024-09-16T04:44:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb07b991640 10 monclient: _renew_subs 2024-09-16T04:44:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb07b991640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:44:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb07b991640 1 -- 172.21.15.46:0/340288807 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0741275c0 con 0x7fb0740a2ee0 2024-09-16T04:44:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb069ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb069ffb640 1 -- 172.21.15.46:0/340288807 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb064007260 con 0x7fb0740a2ee0 2024-09-16T04:44:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb069ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb069ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:44:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb069ffb640 10 monclient: dump: 2024-09-16T04:44:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:23.447 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb07b991640 10 monclient: _renew_subs 2024-09-16T04:44:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb06b7fe640 4 set_mon_vals no callback set 2024-09-16T04:44:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb07b991640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:44:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.421+0000 7fb07b991640 1 -- 172.21.15.46:0/340288807 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb074125770 con 0x7fb0740a2ee0 2024-09-16T04:44:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.422+0000 7fb069ffb640 1 -- 172.21.15.46:0/340288807 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fb064007400 con 0x7fb0740a2ee0 2024-09-16T04:44:23.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.422+0000 7fb07b991640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:23.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.422+0000 7fb07b991640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:23.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.422+0000 7fb07b991640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:44:23.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.422+0000 7fb07b991640 1 -- 172.21.15.46:0/340288807 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0740a7fb0 con 0x7fb0740a2ee0 2024-09-16T04:44:23.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.423+0000 7fb069ffb640 1 --2- 172.21.15.46:0/340288807 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb0580383b0 0x7fb05803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:23.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.423+0000 7fb069ffb640 1 -- 172.21.15.46:0/340288807 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(220..220 src has 1..220) ==== 4738+0+0 (secure 0 0 0) 0x7fb06404ad10 con 0x7fb0740a2ee0 2024-09-16T04:44:23.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.423+0000 7fb07972d640 1 --2- 172.21.15.46:0/340288807 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb0580383b0 0x7fb05803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:23.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.423+0000 7fb07972d640 10 monclient: get_auth_request con 0x7fb0580383b0 auth_method 0 2024-09-16T04:44:23.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.423+0000 7fb07972d640 1 --2- 172.21.15.46:0/340288807 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb0580383b0 0x7fb05803a870 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb0700091d0 tx=0x7fb070008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:23.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.428+0000 7fb069ffb640 1 -- 172.21.15.46:0/340288807 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb0740a3fd0 con 0x7fb0740a2ee0 2024-09-16T04:44:23.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.428+0000 7fb069ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:23.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.428+0000 7fb069ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:44:23.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.669+0000 7fb07b991640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:44:23.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.669+0000 7fb07b991640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:44:23.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.669+0000 7fb07b991640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:44:23.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.669+0000 7fb07b991640 1 -- 172.21.15.46:0/340288807 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb0741374c0 con 0x7fb0740a2ee0 2024-09-16T04:44:23.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.670+0000 7fb069ffb640 1 -- 172.21.15.46:0/340288807 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fb064012820 con 0x7fb0740a2ee0 2024-09-16T04:44:23.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.670+0000 7fb069ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:44:23.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.670+0000 7fb069ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:44:23.673 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":60,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:44:23.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.672+0000 7fb07b991640 1 -- 172.21.15.46:0/340288807 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb0580383b0 msgr2=0x7fb05803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:23.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.672+0000 7fb07b991640 1 --2- 172.21.15.46:0/340288807 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb0580383b0 0x7fb05803a870 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb0700091d0 tx=0x7fb070008040 comp rx=0 tx=0).stop 2024-09-16T04:44:23.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.672+0000 7fb07b991640 10 monclient: shutdown 2024-09-16T04:44:23.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.672+0000 7fb07b991640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:23.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.672+0000 7fb07b991640 1 -- 172.21.15.46:0/340288807 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb0740a2ee0 msgr2=0x7fb074135bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:23.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.672+0000 7fb07b991640 1 --2- 172.21.15.46:0/340288807 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb0740a2ee0 0x7fb074135bc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb06400c340 tx=0x7fb06400c800 comp rx=0 tx=0).stop 2024-09-16T04:44:23.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.672+0000 7fb07b991640 1 -- 172.21.15.46:0/340288807 shutdown_connections 2024-09-16T04:44:23.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.672+0000 7fb07b991640 1 --2- 172.21.15.46:0/340288807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb074137050 0x7fb074136100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:23.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.672+0000 7fb07b991640 1 --2- 172.21.15.46:0/340288807 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb0740a2ee0 0x7fb074135bc0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:23.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.672+0000 7fb07b991640 1 --2- 172.21.15.46:0/340288807 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb0580383b0 0x7fb05803a870 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:23.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.672+0000 7fb07b991640 1 --2- 172.21.15.46:0/340288807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb07406e160 0x7fb0741cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:23.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.672+0000 7fb07b991640 1 -- 172.21.15.46:0/340288807 >> 172.21.15.46:0/340288807 conn(0x7fb0740a5c10 msgr2=0x7fb0740a8710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:23.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.672+0000 7fb07b991640 1 -- 172.21.15.46:0/340288807 shutdown_connections 2024-09-16T04:44:23.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:23.672+0000 7fb07b991640 1 -- 172.21.15.46:0/340288807 wait complete. 2024-09-16T04:44:24.148 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38785]: starting. 2024-09-16T04:44:24.148 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38785]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-38783 2024-09-16T04:44:24.149 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38785]: shutting down. 2024-09-16T04:44:26.252 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:26.253 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38820]: starting. 2024-09-16T04:44:26.253 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38820]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-38783 2024-09-16T04:44:26.253 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38820]: created object 0... 2024-09-16T04:44:26.253 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38820]: created object 25... 2024-09-16T04:44:26.253 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38820]: created object 49... 2024-09-16T04:44:26.253 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38820]: finishing. 2024-09-16T04:44:26.253 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38820]: shutting down. 2024-09-16T04:44:26.685 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:44:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.796+0000 7f6a4b93b640 10 monclient: get_monmap_and_config 2024-09-16T04:44:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.796+0000 7f6a4b93b640 10 monclient: build_initial_monmap 2024-09-16T04:44:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.796+0000 7f6a4b93b640 10 monclient: monmap: 2024-09-16T04:44:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:44:26.797783+0000 2024-09-16T04:44:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:44:26.797783+0000 2024-09-16T04:44:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:44:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:44:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:44:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:44:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:44:26.799 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.796+0000 7f6a4b93b640 1 Processor -- start 2024-09-16T04:44:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.797+0000 7f6a4b93b640 1 -- start start 2024-09-16T04:44:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.797+0000 7f6a4b93b640 10 monclient: init 2024-09-16T04:44:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.797+0000 7f6a4b93b640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.797+0000 7f6a4b93b640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-16T04:44:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.797+0000 7f6a4b93b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 0x7f6a4409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.797+0000 7f6a4b93b640 10 monclient(hunting): picked mon.noname-a con 0x7f6a440a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:44:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.797+0000 7f6a4b93b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6a440a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:44:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.797+0000 7f6a4b93b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a4413be90 addr v1:172.21.15.114:6789/0 2024-09-16T04:44:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.797+0000 7f6a4b93b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a4409b940 con 0x7f6a440a3ff0 2024-09-16T04:44:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.797+0000 7f6a4b93b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a4409ab00 con 0x7f6a4413be90 2024-09-16T04:44:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.797+0000 7f6a4b93b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.797+0000 7f6a4b93b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f6a4409b030 con 0x7f6a440a3c10 2024-09-16T04:44:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.797+0000 7f6a4b93b640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.797+0000 7f6a4b93b640 10 monclient(hunting): authenticate will time out at 1065.213746s 2024-09-16T04:44:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.797+0000 7f6a496d7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 0x7f6a4409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.797+0000 7f6a48ed6640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6a440a3ff0 0x7f6a440a43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45328/0 (socket says 172.21.15.46:45328) 2024-09-16T04:44:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.797+0000 7f6a48ed6640 1 -- 172.21.15.46:0/3159420079 learned_addr learned my addr 172.21.15.46:0/3159420079 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:44:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.797+0000 7f6a496d7640 10 monclient(hunting): get_auth_request con 0x7f6a440a3c10 auth_method 0 2024-09-16T04:44:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.797+0000 7f6a496d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.797+0000 7f6a496d7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.797+0000 7f6a496d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.798+0000 7f6a496d7640 1 -- 172.21.15.46:0/3159420079 >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 msgr2=0x7f6a4409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:44:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.798+0000 7f6a496d7640 1 -- 172.21.15.46:0/3159420079 >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 msgr2=0x7f6a4409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.798+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 0x7f6a4409bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.798+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 0x7f6a4409bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:27.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.999+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 0x7f6a4409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:27.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.999+0000 7f6a496d7640 10 monclient(hunting): get_auth_request con 0x7f6a440a3c10 auth_method 0 2024-09-16T04:44:27.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.999+0000 7f6a496d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:27.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.999+0000 7f6a496d7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:27.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.999+0000 7f6a496d7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:44:27.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.999+0000 7f6a496d7640 1 -- 172.21.15.46:0/3159420079 >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 msgr2=0x7f6a4409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:44:27.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.999+0000 7f6a496d7640 1 -- 172.21.15.46:0/3159420079 >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 msgr2=0x7f6a4409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:27.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.999+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 0x7f6a4409bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:27.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:26.999+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 0x7f6a4409bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:44:27.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:27.400+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 0x7f6a4409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:27.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:27.400+0000 7f6a496d7640 10 monclient(hunting): get_auth_request con 0x7f6a440a3c10 auth_method 0 2024-09-16T04:44:27.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:27.400+0000 7f6a496d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:27.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:27.400+0000 7f6a496d7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:27.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:27.400+0000 7f6a496d7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:44:27.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:27.400+0000 7f6a496d7640 1 -- 172.21.15.46:0/3159420079 >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 msgr2=0x7f6a4409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:44:27.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:27.400+0000 7f6a496d7640 1 -- 172.21.15.46:0/3159420079 >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 msgr2=0x7f6a4409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:27.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:27.400+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 0x7f6a4409bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:27.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:27.400+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 0x7f6a4409bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:44:28.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:28.201+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 0x7f6a4409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:28.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:28.201+0000 7f6a496d7640 10 monclient(hunting): get_auth_request con 0x7f6a440a3c10 auth_method 0 2024-09-16T04:44:28.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:28.201+0000 7f6a496d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:28.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:28.201+0000 7f6a496d7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:28.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:28.201+0000 7f6a496d7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:44:28.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:28.201+0000 7f6a496d7640 1 -- 172.21.15.46:0/3159420079 >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 msgr2=0x7f6a4409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:44:28.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:28.201+0000 7f6a496d7640 1 -- 172.21.15.46:0/3159420079 >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 msgr2=0x7f6a4409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:28.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:28.201+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 0x7f6a4409bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:28.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:28.201+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 0x7f6a4409bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:44:29.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a3affd640 10 monclient(hunting): tick 2024-09-16T04:44:29.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a3affd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:44:29.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a3affd640 1 monclient(hunting): continuing hunt 2024-09-16T04:44:29.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a3affd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:44:29.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a3affd640 1 -- 172.21.15.46:0/3159420079 >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 msgr2=0x7f6a4409bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:44:29.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a3affd640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.46:3300/0 conn(0x7f6a440a3c10 0x7f6a4409bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:29.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a3affd640 1 -- 172.21.15.46:0/3159420079 >> v1:172.21.15.114:6789/0 conn(0x7f6a4413be90 legacy=0x7f6a4413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:29.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a3affd640 1 -- 172.21.15.46:0/3159420079 >> v1:172.21.15.46:6789/0 conn(0x7f6a440a3ff0 legacy=0x7f6a440a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:29.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a3affd640 10 monclient: _add_conns ranks=[2,4] 2024-09-16T04:44:29.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a3affd640 1 --2- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 0x7f6a28005b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:29.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a3affd640 10 monclient(hunting): picked mon.noname-b con 0x7f6a28003730 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:29.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a3affd640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 0x7f6a28008730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:29.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a3affd640 10 monclient(hunting): picked mon.noname-c con 0x7f6a28006290 addr v2:172.21.15.114:3300/0 2024-09-16T04:44:29.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a3affd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:29.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a3affd640 1 -- 172.21.15.46:0/3159420079 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f6a4409b030 con 0x7f6a28006290 2024-09-16T04:44:29.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a3affd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:29.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a3affd640 1 -- 172.21.15.46:0/3159420079 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f6a28008dd0 con 0x7f6a28003730 2024-09-16T04:44:29.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a3affd640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:29.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 0x7f6a28005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:29.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a496d7640 10 monclient(hunting): get_auth_request con 0x7f6a28003730 auth_method 0 2024-09-16T04:44:29.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a496d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:29.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a496d7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:29.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a496d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:29.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a496d7640 1 -- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 msgr2=0x7f6a28005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:44:29.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a496d7640 1 -- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 msgr2=0x7f6a28005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:29.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 0x7f6a28005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:29.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a48ed6640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 0x7f6a28008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:29.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 0x7f6a28005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:29.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a48ed6640 10 monclient(hunting): get_auth_request con 0x7f6a28006290 auth_method 0 2024-09-16T04:44:29.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a48ed6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:29.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a48ed6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:29.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a48ed6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:29.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a48ed6640 1 -- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 msgr2=0x7f6a28008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:44:29.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a48ed6640 1 -- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 msgr2=0x7f6a28008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:29.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a48ed6640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 0x7f6a28008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:29.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.798+0000 7f6a48ed6640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 0x7f6a28008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:30.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.999+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 0x7f6a28005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:30.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.999+0000 7f6a496d7640 10 monclient(hunting): get_auth_request con 0x7f6a28003730 auth_method 0 2024-09-16T04:44:30.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.999+0000 7f6a496d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:30.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.999+0000 7f6a496d7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:30.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.999+0000 7f6a496d7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:44:30.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.999+0000 7f6a496d7640 1 -- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 msgr2=0x7f6a28005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:44:30.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.999+0000 7f6a496d7640 1 -- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 msgr2=0x7f6a28005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:30.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.999+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 0x7f6a28005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:30.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.999+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 0x7f6a28005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:44:30.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.999+0000 7f6a48ed6640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 0x7f6a28008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:30.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.999+0000 7f6a48ed6640 10 monclient(hunting): get_auth_request con 0x7f6a28006290 auth_method 0 2024-09-16T04:44:30.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.999+0000 7f6a48ed6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:30.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.999+0000 7f6a48ed6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:30.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.999+0000 7f6a48ed6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:44:30.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.999+0000 7f6a48ed6640 1 -- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 msgr2=0x7f6a28008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:44:30.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.999+0000 7f6a48ed6640 1 -- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 msgr2=0x7f6a28008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:30.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.999+0000 7f6a48ed6640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 0x7f6a28008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:30.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:29.999+0000 7f6a48ed6640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 0x7f6a28008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:44:30.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:30.400+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 0x7f6a28005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:30.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:30.400+0000 7f6a496d7640 10 monclient(hunting): get_auth_request con 0x7f6a28003730 auth_method 0 2024-09-16T04:44:30.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:30.400+0000 7f6a496d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:30.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:30.400+0000 7f6a496d7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:30.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:30.400+0000 7f6a496d7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:44:30.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:30.400+0000 7f6a496d7640 1 -- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 msgr2=0x7f6a28005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:44:30.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:30.400+0000 7f6a496d7640 1 -- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 msgr2=0x7f6a28005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:30.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:30.400+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 0x7f6a28005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:30.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:30.400+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 0x7f6a28005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:44:30.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:30.400+0000 7f6a48ed6640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 0x7f6a28008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:30.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:30.400+0000 7f6a48ed6640 10 monclient(hunting): get_auth_request con 0x7f6a28006290 auth_method 0 2024-09-16T04:44:30.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:30.400+0000 7f6a48ed6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:30.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:30.400+0000 7f6a48ed6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:30.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:30.400+0000 7f6a48ed6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:44:30.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:30.400+0000 7f6a48ed6640 1 -- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 msgr2=0x7f6a28008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:44:30.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:30.400+0000 7f6a48ed6640 1 -- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 msgr2=0x7f6a28008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:30.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:30.400+0000 7f6a48ed6640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 0x7f6a28008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:30.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:30.400+0000 7f6a48ed6640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 0x7f6a28008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:44:31.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:31.202+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 0x7f6a28005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:31.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:31.202+0000 7f6a496d7640 10 monclient(hunting): get_auth_request con 0x7f6a28003730 auth_method 0 2024-09-16T04:44:31.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:31.202+0000 7f6a496d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:31.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:31.202+0000 7f6a496d7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:31.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:31.202+0000 7f6a496d7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:44:31.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:31.202+0000 7f6a496d7640 1 -- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 msgr2=0x7f6a28005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:44:31.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:31.202+0000 7f6a496d7640 1 -- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 msgr2=0x7f6a28005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:31.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:31.202+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 0x7f6a28005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:31.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:31.202+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 0x7f6a28005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:44:31.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:31.202+0000 7f6a48ed6640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 0x7f6a28008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:31.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:31.202+0000 7f6a48ed6640 10 monclient(hunting): get_auth_request con 0x7f6a28006290 auth_method 0 2024-09-16T04:44:31.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:31.202+0000 7f6a48ed6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:31.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:31.202+0000 7f6a48ed6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:31.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:31.202+0000 7f6a48ed6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:44:31.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:31.202+0000 7f6a48ed6640 1 -- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 msgr2=0x7f6a28008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:44:31.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:31.202+0000 7f6a48ed6640 1 -- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 msgr2=0x7f6a28008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:31.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:31.202+0000 7f6a48ed6640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 0x7f6a28008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:31.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:31.202+0000 7f6a48ed6640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 0x7f6a28008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:44:32.650 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:32.650 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38822]: starting. 2024-09-16T04:44:32.650 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38822]: listing objects. 2024-09-16T04:44:32.650 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38822]: listed object 0... 2024-09-16T04:44:32.650 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38822]: listed object 25... 2024-09-16T04:44:32.650 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38822]: saw 50 objects 2024-09-16T04:44:32.650 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38822]: shutting down. 2024-09-16T04:44:32.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3affd640 10 monclient(hunting): tick 2024-09-16T04:44:32.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3affd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:44:32.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3affd640 1 monclient(hunting): continuing hunt 2024-09-16T04:44:32.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3affd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:44:32.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3affd640 1 -- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 msgr2=0x7f6a28005b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:44:32.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3affd640 1 --2- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003730 0x7f6a28005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:32.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3affd640 1 -- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 msgr2=0x7f6a28008730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:44:32.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3affd640 1 --2- 172.21.15.46:0/3159420079 >> v2:172.21.15.114:3300/0 conn(0x7f6a28006290 0x7f6a28008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:32.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3affd640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-16T04:44:32.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a49ed8640 1 -- 172.21.15.46:0/3159420079 reap_dead start 2024-09-16T04:44:32.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3affd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6a440a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:44:32.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3affd640 1 --2- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003b50 0x7f6a28006670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:32.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3affd640 10 monclient(hunting): picked mon.noname-b con 0x7f6a28003b50 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:32.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3affd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a28006bb0 addr v1:172.21.15.114:6789/0 2024-09-16T04:44:32.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3affd640 1 -- 172.21.15.46:0/3159420079 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a4409b310 con 0x7f6a440a3ff0 2024-09-16T04:44:32.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3affd640 1 -- 172.21.15.46:0/3159420079 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a28008900 con 0x7f6a28006bb0 2024-09-16T04:44:32.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3affd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:32.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3affd640 1 -- 172.21.15.46:0/3159420079 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f6a4409b030 con 0x7f6a28003b50 2024-09-16T04:44:32.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3affd640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:32.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a48ed6640 1 --2- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003b50 0x7f6a28006670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:32.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a48ed6640 10 monclient(hunting): get_auth_request con 0x7f6a28003b50 auth_method 0 2024-09-16T04:44:32.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a48ed6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:32.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a48ed6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:32.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a48ed6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:32.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a48ed6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:32.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a48ed6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:32.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a48ed6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:32.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3bfff640 1 -- 172.21.15.46:0/3159420079 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3287353174 0 0) 0x7f6a4409b310 con 0x7f6a440a3ff0 2024-09-16T04:44:32.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:44:32.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:32.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:32.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3bfff640 10 monclient(hunting): my global_id is 74097 2024-09-16T04:44:32.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.798+0000 7f6a3bfff640 1 -- 172.21.15.46:0/3159420079 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a30003520 con 0x7f6a440a3ff0 2024-09-16T04:44:32.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.799+0000 7f6a48ed6640 10 monclient(hunting): handle_auth_done global_id 64277 payload 274 2024-09-16T04:44:32.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.799+0000 7f6a48ed6640 1 -- 172.21.15.46:0/3159420079 >> v1:172.21.15.114:6789/0 conn(0x7f6a28006bb0 legacy=0x7f6a28007060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:32.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.799+0000 7f6a48ed6640 1 -- 172.21.15.46:0/3159420079 >> v1:172.21.15.46:6789/0 conn(0x7f6a440a3ff0 legacy=0x7f6a2800b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:32.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.799+0000 7f6a48ed6640 10 monclient: _finish_hunting 0 2024-09-16T04:44:32.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.799+0000 7f6a48ed6640 1 monclient: found mon.noname-b 2024-09-16T04:44:32.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.799+0000 7f6a48ed6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:32.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.799+0000 7f6a48ed6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:44:32.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.799+0000 7f6a48ed6640 1 -- 172.21.15.46:0/3159420079 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a28010970 con 0x7f6a28003b50 2024-09-16T04:44:32.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.799+0000 7f6a48ed6640 1 --2- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003b50 0x7f6a28006670 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6a34005d50 tx=0x7f6a3402ad90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f3a9e0065ce4c1b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:32.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.800+0000 7f6a3bfff640 1 -- 172.21.15.46:0/3159420079 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a34031070 con 0x7f6a28003b50 2024-09-16T04:44:32.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.800+0000 7f6a3bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:32.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.800+0000 7f6a3bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:44:32.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.800+0000 7f6a3bfff640 10 monclient: dump: 2024-09-16T04:44:32.807 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:32.807 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:32.807 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:32.807 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:32.807 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:32.807 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:32.807 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:32.807 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:32.807 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:32.807 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:32.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.800+0000 7f6a3bfff640 10 monclient: _finish_auth 0 2024-09-16T04:44:32.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.800+0000 7f6a3bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:44:32.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.800+0000 7f6a3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:32.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.800+0000 7f6a3bfff640 1 -- 172.21.15.46:0/3159420079 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a3402b8b0 con 0x7f6a28003b50 2024-09-16T04:44:32.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.800+0000 7f6a4b93b640 5 monclient: authenticate success, global_id 64277 2024-09-16T04:44:32.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.800+0000 7f6a4b93b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:44:32.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.800+0000 7f6a3bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:32.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.800+0000 7f6a4b93b640 10 monclient: get_monmap_and_config success 2024-09-16T04:44:32.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.800+0000 7f6a4b93b640 4 set_mon_vals no callback set 2024-09-16T04:44:32.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.800+0000 7f6a3bfff640 1 -- 172.21.15.46:0/3159420079 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3402bca0 con 0x7f6a28003b50 2024-09-16T04:44:32.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.800+0000 7f6a3bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:32.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.800+0000 7f6a3bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:44:32.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.800+0000 7f6a3bfff640 10 monclient: dump: 2024-09-16T04:44:32.808 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:32.808 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:32.808 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:32.809 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:32.809 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:32.809 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:32.809 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:32.809 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:32.809 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:32.809 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:32.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.800+0000 7f6a4b93b640 10 monclient: shutdown 2024-09-16T04:44:32.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.800+0000 7f6a4b93b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:32.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.800+0000 7f6a4b93b640 1 -- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003b50 msgr2=0x7f6a28006670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:32.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.801+0000 7f6a4b93b640 1 --2- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003b50 0x7f6a28006670 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6a34005d50 tx=0x7f6a3402ad90 comp rx=0 tx=0).stop 2024-09-16T04:44:32.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.801+0000 7f6a4b93b640 1 -- 172.21.15.46:0/3159420079 shutdown_connections 2024-09-16T04:44:32.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.801+0000 7f6a4b93b640 1 --2- 172.21.15.46:0/3159420079 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28003b50 0x7f6a28006670 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:32.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.801+0000 7f6a4b93b640 1 -- 172.21.15.46:0/3159420079 >> 172.21.15.46:0/3159420079 conn(0x7f6a440a6340 msgr2=0x7f6a4406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:32.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.801+0000 7f6a4b93b640 1 -- 172.21.15.46:0/3159420079 shutdown_connections 2024-09-16T04:44:32.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.801+0000 7f6a4b93b640 1 -- 172.21.15.46:0/3159420079 wait complete. 2024-09-16T04:44:32.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.802+0000 7f6a4b93b640 10 monclient: build_initial_monmap 2024-09-16T04:44:32.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.802+0000 7f6a4b93b640 10 monclient: monmap: 2024-09-16T04:44:32.814 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:32.814 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:44:32.814 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:44:32.814 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:44:32.814 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:32.814 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:32.815 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:44:32.815 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:44:32.815 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:44:32.815 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:32.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.802+0000 7f6a4b93b640 1 Processor -- start 2024-09-16T04:44:32.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.802+0000 7f6a4b93b640 1 -- start start 2024-09-16T04:44:32.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.802+0000 7f6a4b93b640 10 monclient: init 2024-09-16T04:44:32.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a4b93b640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:32.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a4b93b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:44:32.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a4b93b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a28003b50 0x7f6a441cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:32.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a4b93b640 10 monclient(hunting): picked mon.noname-b con 0x7f6a28003b50 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:44:32.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a4b93b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28006bb0 0x7f6a441cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:32.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a4b93b640 10 monclient(hunting): picked mon.noname-c con 0x7f6a28006bb0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:32.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a4b93b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a440a3ff0 0x7f6a4411fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:32.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a4b93b640 10 monclient(hunting): picked mon.noname-a con 0x7f6a440a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:44:32.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a4b93b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:32.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a4b93b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4409be70 con 0x7f6a440a3ff0 2024-09-16T04:44:32.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a4b93b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:32.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a4b93b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4409bcf0 con 0x7f6a28003b50 2024-09-16T04:44:32.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a4b93b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:32.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a4b93b640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f6a4409bff0 con 0x7f6a28006bb0 2024-09-16T04:44:32.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a4b93b640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:32.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a4b93b640 10 monclient(hunting): authenticate will time out at 1071.219179s 2024-09-16T04:44:32.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a49ed8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a440a3ff0 0x7f6a4411fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:32.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a48ed6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28006bb0 0x7f6a441cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:32.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a48ed6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28006bb0 0x7f6a441cc7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:53474/0 (socket says 172.21.15.46:53474) 2024-09-16T04:44:32.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a49ed8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a440a3ff0 0x7f6a4411fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41236/0 (socket says 172.21.15.46:41236) 2024-09-16T04:44:32.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a48ed6640 1 -- 172.21.15.46:0/3645159700 learned_addr learned my addr 172.21.15.46:0/3645159700 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:32.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a48ed6640 10 monclient(hunting): get_auth_request con 0x7f6a28006bb0 auth_method 0 2024-09-16T04:44:32.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3645159700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a28003b50 0x7f6a441cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:32.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a48ed6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:32.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a48ed6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:32.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a48ed6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:32.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a49ed8640 10 monclient(hunting): get_auth_request con 0x7f6a440a3ff0 auth_method 0 2024-09-16T04:44:32.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a49ed8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:32.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a49ed8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:32.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a49ed8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:32.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a496d7640 10 monclient(hunting): get_auth_request con 0x7f6a28003b50 auth_method 0 2024-09-16T04:44:32.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a496d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a496d7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a496d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a48ed6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a48ed6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a48ed6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a49ed8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a49ed8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a49ed8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a496d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a496d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a496d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a48ed6640 10 monclient(hunting): handle_auth_done global_id 64280 payload 824 2024-09-16T04:44:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a48ed6640 1 -- 172.21.15.46:0/3645159700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a28003b50 msgr2=0x7f6a441cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a48ed6640 1 --2- 172.21.15.46:0/3645159700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a28003b50 0x7f6a441cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a48ed6640 1 -- 172.21.15.46:0/3645159700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a440a3ff0 msgr2=0x7f6a4411fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a48ed6640 1 --2- 172.21.15.46:0/3645159700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a440a3ff0 0x7f6a4411fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a48ed6640 10 monclient: _finish_hunting 0 2024-09-16T04:44:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a48ed6640 1 monclient: found mon.noname-c 2024-09-16T04:44:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a48ed6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a48ed6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:44:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a48ed6640 1 -- 172.21.15.46:0/3645159700 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a44122470 con 0x7f6a28006bb0 2024-09-16T04:44:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.803+0000 7f6a49ed8640 1 --2- 172.21.15.46:0/3645159700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a440a3ff0 0x7f6a4411fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:44:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a48ed6640 1 --2- 172.21.15.46:0/3645159700 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28006bb0 0x7f6a441cc7e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6a34005e70 tx=0x7f6a34001120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a39ffb640 1 -- 172.21.15.46:0/3645159700 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a34044030 con 0x7f6a28006bb0 2024-09-16T04:44:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a39ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a39ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:44:32.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a39ffb640 10 monclient: dump: 2024-09-16T04:44:32.820 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:32.820 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:32.820 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:32.820 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:32.820 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:32.820 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:32.820 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:32.820 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:32.820 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:32.820 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:32.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a39ffb640 10 monclient: _finish_auth 0 2024-09-16T04:44:32.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a39ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:44:32.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:32.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a39ffb640 1 -- 172.21.15.46:0/3645159700 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a34004e20 con 0x7f6a28006bb0 2024-09-16T04:44:32.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a39ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:32.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a39ffb640 1 -- 172.21.15.46:0/3645159700 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a34005240 con 0x7f6a28006bb0 2024-09-16T04:44:32.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a4b93b640 5 monclient: authenticate success, global_id 64280 2024-09-16T04:44:32.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a39ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:32.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a39ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:44:32.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a39ffb640 10 monclient: dump: 2024-09-16T04:44:32.821 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:32.822 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:32.822 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:32.822 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:32.822 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:32.822 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:32.822 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:32.822 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:32.822 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:32.822 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:32.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a3b7fe640 4 set_mon_vals no callback set 2024-09-16T04:44:32.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a4b93b640 10 monclient: _renew_subs 2024-09-16T04:44:32.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a4b93b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:44:32.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a4b93b640 1 -- 172.21.15.46:0/3645159700 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a44123e80 con 0x7f6a28006bb0 2024-09-16T04:44:32.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a4b93b640 10 monclient: _renew_subs 2024-09-16T04:44:32.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a4b93b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:44:32.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.804+0000 7f6a4b93b640 1 -- 172.21.15.46:0/3645159700 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a440a0c50 con 0x7f6a28006bb0 2024-09-16T04:44:32.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.805+0000 7f6a39ffb640 1 -- 172.21.15.46:0/3645159700 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f6a34047070 con 0x7f6a28006bb0 2024-09-16T04:44:32.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.805+0000 7f6a4b93b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:32.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.805+0000 7f6a4b93b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:32.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.805+0000 7f6a4b93b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:44:32.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.805+0000 7f6a4b93b640 1 -- 172.21.15.46:0/3645159700 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a440a43d0 con 0x7f6a28006bb0 2024-09-16T04:44:32.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.805+0000 7f6a39ffb640 1 --2- 172.21.15.46:0/3645159700 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6a28039d00 0x7f6a2803c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:32.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.806+0000 7f6a39ffb640 1 -- 172.21.15.46:0/3645159700 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(223..223 src has 1..223) ==== 4738+0+0 (secure 0 0 0) 0x7f6a3406c890 con 0x7f6a28006bb0 2024-09-16T04:44:32.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.806+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3645159700 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6a28039d00 0x7f6a2803c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:32.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.806+0000 7f6a496d7640 10 monclient: get_auth_request con 0x7f6a28039d00 auth_method 0 2024-09-16T04:44:32.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.806+0000 7f6a496d7640 1 --2- 172.21.15.46:0/3645159700 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6a28039d00 0x7f6a2803c1c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6a40006fd0 tx=0x7f6a40008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:32.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.811+0000 7f6a39ffb640 1 -- 172.21.15.46:0/3645159700 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6a340407f0 con 0x7f6a28006bb0 2024-09-16T04:44:32.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.811+0000 7f6a39ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:32.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:32.811+0000 7f6a39ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:44:33.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.057+0000 7f6a4b93b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:44:33.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.057+0000 7f6a4b93b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:44:33.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.057+0000 7f6a4b93b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:44:33.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.057+0000 7f6a4b93b640 1 -- 172.21.15.46:0/3645159700 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6a440a43d0 con 0x7f6a28006bb0 2024-09-16T04:44:33.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.058+0000 7f6a39ffb640 1 -- 172.21.15.46:0/3645159700 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f6a3403f3d0 con 0x7f6a28006bb0 2024-09-16T04:44:33.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.058+0000 7f6a39ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:44:33.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.058+0000 7f6a39ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:44:33.060 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:44:33.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.060+0000 7f6a4b93b640 1 -- 172.21.15.46:0/3645159700 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6a28039d00 msgr2=0x7f6a2803c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.060+0000 7f6a4b93b640 1 --2- 172.21.15.46:0/3645159700 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6a28039d00 0x7f6a2803c1c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6a40006fd0 tx=0x7f6a40008040 comp rx=0 tx=0).stop 2024-09-16T04:44:33.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.060+0000 7f6a4b93b640 10 monclient: shutdown 2024-09-16T04:44:33.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.060+0000 7f6a4b93b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.060+0000 7f6a4b93b640 1 -- 172.21.15.46:0/3645159700 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28006bb0 msgr2=0x7f6a441cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.060+0000 7f6a4b93b640 1 --2- 172.21.15.46:0/3645159700 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28006bb0 0x7f6a441cc7e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6a34005e70 tx=0x7f6a34001120 comp rx=0 tx=0).stop 2024-09-16T04:44:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.060+0000 7f6a4b93b640 1 -- 172.21.15.46:0/3645159700 shutdown_connections 2024-09-16T04:44:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.060+0000 7f6a4b93b640 1 --2- 172.21.15.46:0/3645159700 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6a28039d00 0x7f6a2803c1c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.060+0000 7f6a4b93b640 1 --2- 172.21.15.46:0/3645159700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a440a3ff0 0x7f6a4411fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.060+0000 7f6a4b93b640 1 --2- 172.21.15.46:0/3645159700 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6a28006bb0 0x7f6a441cc7e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.060+0000 7f6a4b93b640 1 --2- 172.21.15.46:0/3645159700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a28003b50 0x7f6a441cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.060+0000 7f6a4b93b640 1 -- 172.21.15.46:0/3645159700 >> 172.21.15.46:0/3645159700 conn(0x7f6a440a6340 msgr2=0x7f6a4413b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.060+0000 7f6a4b93b640 1 -- 172.21.15.46:0/3645159700 shutdown_connections 2024-09-16T04:44:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.060+0000 7f6a4b93b640 1 -- 172.21.15.46:0/3645159700 wait complete. 2024-09-16T04:44:33.073 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:44:33.073 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:44:33.073 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:44:33.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.173+0000 7f430d3fd640 10 monclient: get_monmap_and_config 2024-09-16T04:44:33.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.173+0000 7f430d3fd640 10 monclient: build_initial_monmap 2024-09-16T04:44:33.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.173+0000 7f430d3fd640 10 monclient: monmap: 2024-09-16T04:44:33.176 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:33.176 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.176 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:44:33.175151+0000 2024-09-16T04:44:33.176 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:44:33.175151+0000 2024-09-16T04:44:33.176 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:33.176 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.176 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:44:33.176 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:44:33.176 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:44:33.176 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:44:33.176 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:44:33.177 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.174+0000 7f430d3fd640 1 Processor -- start 2024-09-16T04:44:33.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.174+0000 7f430d3fd640 1 -- start start 2024-09-16T04:44:33.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.174+0000 7f430d3fd640 10 monclient: init 2024-09-16T04:44:33.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.174+0000 7f430d3fd640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:33.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.174+0000 7f430d3fd640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-16T04:44:33.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.174+0000 7f430d3fd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f430806e160 0x7f43080a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.174+0000 7f430d3fd640 10 monclient(hunting): picked mon.noname-a con 0x7f430806e160 addr v2:172.21.15.46:3300/0 2024-09-16T04:44:33.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.174+0000 7f430d3fd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f43080a2ee0 0x7f4308136940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.174+0000 7f430d3fd640 10 monclient(hunting): picked mon.noname-c con 0x7f43080a2ee0 addr v2:172.21.15.114:3300/0 2024-09-16T04:44:33.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.174+0000 7f430d3fd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4308136e80 0x7f43081372f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.174+0000 7f430d3fd640 10 monclient(hunting): picked mon.noname-b con 0x7f4308136e80 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:33.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.174+0000 7f430d3fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.174+0000 7f430d3fd640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f43080a3300 con 0x7f430806e160 2024-09-16T04:44:33.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.174+0000 7f430d3fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.174+0000 7f430d3fd640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f430806e540 con 0x7f43080a2ee0 2024-09-16T04:44:33.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.174+0000 7f430d3fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.174+0000 7f430d3fd640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f43080a3fd0 con 0x7f4308136e80 2024-09-16T04:44:33.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.174+0000 7f430d3fd640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.174+0000 7f430d3fd640 10 monclient(hunting): authenticate will time out at 1071.590872s 2024-09-16T04:44:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.174+0000 7f4306ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f430806e160 0x7f43080a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f4306ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f430806e160 0x7f43080a29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41248/0 (socket says 172.21.15.46:41248) 2024-09-16T04:44:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f4306ffd640 1 -- 172.21.15.46:0/3693853877 learned_addr learned my addr 172.21.15.46:0/3693853877 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f4306ffd640 10 monclient(hunting): get_auth_request con 0x7f430806e160 auth_method 0 2024-09-16T04:44:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f4306ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f4306ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f4306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.174+0000 7f43077fe640 1 --2- 172.21.15.46:0/3693853877 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4308136e80 0x7f43081372f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f43077fe640 10 monclient(hunting): get_auth_request con 0x7f4308136e80 auth_method 0 2024-09-16T04:44:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f43077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f43077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f43077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f4306ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f4306ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f4306ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f43077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f43077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f43077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f4306ffd640 10 monclient(hunting): handle_auth_done global_id 74103 payload 274 2024-09-16T04:44:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f43067fc640 1 --2- 172.21.15.46:0/3693853877 >> v2:172.21.15.114:3300/0 conn(0x7f43080a2ee0 0x7f4308136940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f4306ffd640 1 -- 172.21.15.46:0/3693853877 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4308136e80 msgr2=0x7f43081372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:44:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f4306ffd640 1 --2- 172.21.15.46:0/3693853877 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4308136e80 0x7f43081372f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f4306ffd640 1 -- 172.21.15.46:0/3693853877 >> v2:172.21.15.114:3300/0 conn(0x7f43080a2ee0 msgr2=0x7f4308136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:44:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f4306ffd640 1 --2- 172.21.15.46:0/3693853877 >> v2:172.21.15.114:3300/0 conn(0x7f43080a2ee0 0x7f4308136940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f4306ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:44:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f4306ffd640 1 monclient: found mon.noname-a 2024-09-16T04:44:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f4306ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f4306ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:44:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f4306ffd640 1 -- 172.21.15.46:0/3693853877 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43080a0bd0 con 0x7f430806e160 2024-09-16T04:44:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.175+0000 7f4306ffd640 1 --2- 172.21.15.46:0/3693853877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f430806e160 0x7f43080a29a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f430809fec0 tx=0x7f42f802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f53639e1a14f70b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f4305ffb640 1 -- 172.21.15.46:0/3693853877 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f8037070 con 0x7f430806e160 2024-09-16T04:44:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f4305ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f4305ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:44:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f4305ffb640 10 monclient: dump: 2024-09-16T04:44:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.193 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f4305ffb640 10 monclient: _finish_auth 0 2024-09-16T04:44:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f4305ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:44:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f4305ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:33.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f430d3fd640 5 monclient: authenticate success, global_id 74103 2024-09-16T04:44:33.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f4305ffb640 1 -- 172.21.15.46:0/3693853877 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f802ae10 con 0x7f430806e160 2024-09-16T04:44:33.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f430d3fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:44:33.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f4305ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:33.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f430d3fd640 10 monclient: get_monmap_and_config success 2024-09-16T04:44:33.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f430d3fd640 4 set_mon_vals no callback set 2024-09-16T04:44:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f4305ffb640 1 -- 172.21.15.46:0/3693853877 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f8032660 con 0x7f430806e160 2024-09-16T04:44:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f4305ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f4305ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f4305ffb640 10 monclient: dump: 2024-09-16T04:44:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.196 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.196 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.196 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f430d3fd640 10 monclient: shutdown 2024-09-16T04:44:33.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f430d3fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:33.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f430d3fd640 1 -- 172.21.15.46:0/3693853877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f430806e160 msgr2=0x7f43080a29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f430d3fd640 1 --2- 172.21.15.46:0/3693853877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f430806e160 0x7f43080a29a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f430809fec0 tx=0x7f42f802a100 comp rx=0 tx=0).stop 2024-09-16T04:44:33.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f430d3fd640 1 -- 172.21.15.46:0/3693853877 shutdown_connections 2024-09-16T04:44:33.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f430d3fd640 1 --2- 172.21.15.46:0/3693853877 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4308136e80 0x7f43081372f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f430d3fd640 1 --2- 172.21.15.46:0/3693853877 >> v2:172.21.15.114:3300/0 conn(0x7f43080a2ee0 0x7f4308136940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f430d3fd640 1 --2- 172.21.15.46:0/3693853877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f430806e160 0x7f43080a29a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f430d3fd640 1 -- 172.21.15.46:0/3693853877 >> 172.21.15.46:0/3693853877 conn(0x7f43080a5c10 msgr2=0x7f43080a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f430d3fd640 1 -- 172.21.15.46:0/3693853877 shutdown_connections 2024-09-16T04:44:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.176+0000 7f430d3fd640 1 -- 172.21.15.46:0/3693853877 wait complete. 2024-09-16T04:44:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.177+0000 7f430d3fd640 10 monclient: build_initial_monmap 2024-09-16T04:44:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.177+0000 7f430d3fd640 10 monclient: monmap: 2024-09-16T04:44:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:44:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:44:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:44:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:44:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:44:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:44:33.197 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.177+0000 7f430d3fd640 1 Processor -- start 2024-09-16T04:44:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.177+0000 7f430d3fd640 1 -- start start 2024-09-16T04:44:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.177+0000 7f430d3fd640 10 monclient: init 2024-09-16T04:44:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f430d3fd640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f430d3fd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:44:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f430d3fd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f430806e160 0x7f43081cbe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f430d3fd640 10 monclient(hunting): picked mon.noname-b con 0x7f430806e160 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:44:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f430d3fd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43080a2ee0 0x7f4308135e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f430d3fd640 10 monclient(hunting): picked mon.noname-a con 0x7f43080a2ee0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:44:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f430d3fd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4308136e80 0x7f4308136390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f430d3fd640 10 monclient(hunting): picked mon.noname-c con 0x7f4308136e80 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f430d3fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f430d3fd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f43080a5130 con 0x7f43080a2ee0 2024-09-16T04:44:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f430d3fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f430d3fd640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f43080a1190 con 0x7f430806e160 2024-09-16T04:44:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f430d3fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f430d3fd640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f43080a34a0 con 0x7f4308136e80 2024-09-16T04:44:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f430d3fd640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f430d3fd640 10 monclient(hunting): authenticate will time out at 1071.594348s 2024-09-16T04:44:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43067fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43080a2ee0 0x7f4308135e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43077fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4308136e80 0x7f4308136390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43067fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43080a2ee0 0x7f4308135e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41262/0 (socket says 172.21.15.46:41262) 2024-09-16T04:44:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43067fc640 1 -- 172.21.15.46:0/4283756153 learned_addr learned my addr 172.21.15.46:0/4283756153 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43067fc640 10 monclient(hunting): get_auth_request con 0x7f43080a2ee0 auth_method 0 2024-09-16T04:44:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43077fe640 10 monclient(hunting): get_auth_request con 0x7f4308136e80 auth_method 0 2024-09-16T04:44:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f4306ffd640 1 --2- 172.21.15.46:0/4283756153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f430806e160 0x7f43081cbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f4306ffd640 10 monclient(hunting): get_auth_request con 0x7f430806e160 auth_method 0 2024-09-16T04:44:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f4306ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f4306ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f4306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f4306ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f4306ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f4306ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43067fc640 10 monclient(hunting): handle_auth_done global_id 74106 payload 824 2024-09-16T04:44:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43067fc640 1 -- 172.21.15.46:0/4283756153 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4308136e80 msgr2=0x7f4308136390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43067fc640 1 --2- 172.21.15.46:0/4283756153 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4308136e80 0x7f4308136390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43067fc640 1 -- 172.21.15.46:0/4283756153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f430806e160 msgr2=0x7f43081cbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43067fc640 1 --2- 172.21.15.46:0/4283756153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f430806e160 0x7f43081cbe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43067fc640 10 monclient: _finish_hunting 0 2024-09-16T04:44:33.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43067fc640 1 monclient: found mon.noname-a 2024-09-16T04:44:33.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43067fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:44:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43067fc640 1 -- 172.21.15.46:0/4283756153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4308123530 con 0x7f43080a2ee0 2024-09-16T04:44:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43077fe640 1 --2- 172.21.15.46:0/4283756153 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4308136e80 0x7f4308136390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:44:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.178+0000 7f43067fc640 1 --2- 172.21.15.46:0/4283756153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43080a2ee0 0x7f4308135e50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f42fc00c7f0 tx=0x7f42fc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f42ef7fe640 1 -- 172.21.15.46:0/4283756153 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42fc015070 con 0x7f43080a2ee0 2024-09-16T04:44:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f42ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f42ef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:44:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f42ef7fe640 10 monclient: dump: 2024-09-16T04:44:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.203 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.204 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f42ef7fe640 10 monclient: _finish_auth 0 2024-09-16T04:44:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f42ef7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:44:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f42ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f430d3fd640 5 monclient: authenticate success, global_id 74106 2024-09-16T04:44:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f42ef7fe640 1 -- 172.21.15.46:0/4283756153 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42fc004240 con 0x7f43080a2ee0 2024-09-16T04:44:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f42ef7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f42ef7fe640 1 -- 172.21.15.46:0/4283756153 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42fc007620 con 0x7f43080a2ee0 2024-09-16T04:44:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f430d3fd640 10 monclient: _renew_subs 2024-09-16T04:44:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f430d3fd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:33.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f430d3fd640 1 -- 172.21.15.46:0/4283756153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43081cc370 con 0x7f43080a2ee0 2024-09-16T04:44:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f42ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f42ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f43057fa640 4 set_mon_vals no callback set 2024-09-16T04:44:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f42ef7fe640 10 monclient: dump: 2024-09-16T04:44:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.206 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f430d3fd640 10 monclient: _renew_subs 2024-09-16T04:44:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f430d3fd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f430d3fd640 1 -- 172.21.15.46:0/4283756153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43080a84e0 con 0x7f43080a2ee0 2024-09-16T04:44:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f430d3fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f430d3fd640 1 --2- 172.21.15.46:0/4283756153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f430811fa30 0x7f430811fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f430d3fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f430d3fd640 1 -- 172.21.15.46:0/4283756153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f43080a5ff0 con 0x7f430811fa30 2024-09-16T04:44:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.179+0000 7f42ef7fe640 1 -- 172.21.15.46:0/4283756153 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f42fc0079d0 con 0x7f43080a2ee0 2024-09-16T04:44:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f4306ffd640 1 --2- 172.21.15.46:0/4283756153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f430811fa30 0x7f430811fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f4306ffd640 10 monclient: get_auth_request con 0x7f430811fa30 auth_method 0 2024-09-16T04:44:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f4306ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f4306ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f4306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f4306ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f4306ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f4306ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f42ef7fe640 1 --2- 172.21.15.46:0/4283756153 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f42f40383b0 0x7f42f403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f42ef7fe640 1 -- 172.21.15.46:0/4283756153 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(223..223 src has 1..223) ==== 4738+0+0 (secure 0 0 0) 0x7f42fc04bb40 con 0x7f43080a2ee0 2024-09-16T04:44:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f4306ffd640 10 monclient(hunting): handle_auth_done global_id 74109 payload 824 2024-09-16T04:44:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f4306ffd640 1 -- 172.21.15.46:0/4283756153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f43080a4d90 con 0x7f430811fa30 2024-09-16T04:44:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f43077fe640 1 --2- 172.21.15.46:0/4283756153 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f42f40383b0 0x7f42f403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f43077fe640 10 monclient: get_auth_request con 0x7f42f40383b0 auth_method 0 2024-09-16T04:44:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f4306ffd640 1 --2- 172.21.15.46:0/4283756153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f430811fa30 0x7f430811fe10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4308068700 tx=0x7f42f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f42ef7fe640 1 -- 172.21.15.46:0/4283756153 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f8037040 con 0x7f430811fa30 2024-09-16T04:44:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f42ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f42ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f42ef7fe640 10 monclient: dump: 2024-09-16T04:44:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.208 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f43077fe640 1 --2- 172.21.15.46:0/4283756153 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f42f40383b0 0x7f42f403a870 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4308121230 tx=0x7f430000b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f42ef7fe640 1 -- 172.21.15.46:0/4283756153 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f42f803c640 con 0x7f430811fa30 2024-09-16T04:44:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f42ef7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f42ef7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:44:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f42ef7fe640 1 -- 172.21.15.46:0/4283756153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f430811fa30 msgr2=0x7f430811fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f42ef7fe640 1 --2- 172.21.15.46:0/4283756153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f430811fa30 0x7f430811fe10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4308068700 tx=0x7f42f80046c0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f42ef7fe640 1 -- 172.21.15.46:0/4283756153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f430811fa30 msgr2=0x7f430811fe10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:44:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.180+0000 7f42ef7fe640 1 --2- 172.21.15.46:0/4283756153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f430811fa30 0x7f430811fe10 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4308068700 tx=0x7f42f80046c0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f430d3fd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:44:33.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f430d3fd640 1 --2- 172.21.15.46:0/4283756153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4308120350 0x7f430809ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f430d3fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f430d3fd640 1 -- 172.21.15.46:0/4283756153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f43080008d0 con 0x7f4308120350 2024-09-16T04:44:33.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f4306ffd640 1 --2- 172.21.15.46:0/4283756153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4308120350 0x7f430809ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f4306ffd640 10 monclient: get_auth_request con 0x7f4308120350 auth_method 0 2024-09-16T04:44:33.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f4306ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f4306ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f4306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f4306ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f4306ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f4306ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f4306ffd640 10 monclient(hunting): handle_auth_done global_id 74112 payload 824 2024-09-16T04:44:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f4306ffd640 1 -- 172.21.15.46:0/4283756153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f43080a7e40 con 0x7f4308120350 2024-09-16T04:44:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f4306ffd640 1 --2- 172.21.15.46:0/4283756153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4308120350 0x7f430809ae50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f42f80090b0 tx=0x7f42f8008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f42ef7fe640 1 -- 172.21.15.46:0/4283756153 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f8037030 con 0x7f4308120350 2024-09-16T04:44:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f42ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f42ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f42ef7fe640 10 monclient: dump: 2024-09-16T04:44:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.213 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f42ef7fe640 1 -- 172.21.15.46:0/4283756153 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f42f8047b60 con 0x7f4308120350 2024-09-16T04:44:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f42ef7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:44:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f42ef7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:44:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f42ef7fe640 1 -- 172.21.15.46:0/4283756153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4308120350 msgr2=0x7f430809ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f42ef7fe640 1 --2- 172.21.15.46:0/4283756153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4308120350 0x7f430809ae50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f42f80090b0 tx=0x7f42f8008030 comp rx=0 tx=0).stop 2024-09-16T04:44:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f42ef7fe640 1 -- 172.21.15.46:0/4283756153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4308120350 msgr2=0x7f430809ae50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:44:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.187+0000 7f42ef7fe640 1 --2- 172.21.15.46:0/4283756153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4308120350 0x7f430809ae50 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f42f80090b0 tx=0x7f42f8008030 comp rx=0 tx=0).stop 2024-09-16T04:44:33.214 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:44:33.214 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:44:33.214 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:44:33.214 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:44:33.214 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 68, 2024-09-16T04:44:33.215 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 13183, 2024-09-16T04:44:33.215 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:44:33.215 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:44:33.215 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:44:33.215 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:44:33.215 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:33.215 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:44:33.216 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:44:33.216 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:44:33.216 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:44:33.216 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:33.216 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:33.216 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:33.216 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:33.216 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:33.217 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:33.217 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:33.217 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:33.217 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:33.217 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:44:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:44:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:44:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:44:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:44:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:44:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:44:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:44:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:44:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:44:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:44:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:44:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:44:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:44:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:44:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:44:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:44:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:44:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:44:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:33.221 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:44:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:44:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:44:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:44:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:33.222 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:44:33.223 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:33.223 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.223 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.223 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:33.223 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:44:33.223 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:33.223 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:33.223 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:44:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:44:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:44:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:44:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:33.224 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:44:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:44:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:44:33.225 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:44:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:44:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:44:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:44:33.226 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:33.227 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.227 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.227 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:33.227 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:44:33.227 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:33.227 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:33.227 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:33.227 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.227 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:44:33.227 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:44:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:44:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:44:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:44:33.228 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:44:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:44:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.229 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:44:33.229 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:44:33.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.188+0000 7f430d3fd640 1 -- 172.21.15.46:0/4283756153 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f42f40383b0 msgr2=0x7f42f403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.188+0000 7f430d3fd640 1 --2- 172.21.15.46:0/4283756153 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f42f40383b0 0x7f42f403a870 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4308121230 tx=0x7f430000b040 comp rx=0 tx=0).stop 2024-09-16T04:44:33.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.188+0000 7f430d3fd640 10 monclient: shutdown 2024-09-16T04:44:33.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.188+0000 7f430d3fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:33.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.188+0000 7f430d3fd640 1 -- 172.21.15.46:0/4283756153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43080a2ee0 msgr2=0x7f4308135e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.188+0000 7f430d3fd640 1 --2- 172.21.15.46:0/4283756153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43080a2ee0 0x7f4308135e50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f42fc00c7f0 tx=0x7f42fc00ccb0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.188+0000 7f43077fe640 1 -- 172.21.15.46:0/4283756153 reap_dead start 2024-09-16T04:44:33.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.188+0000 7f430d3fd640 1 -- 172.21.15.46:0/4283756153 shutdown_connections 2024-09-16T04:44:33.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.188+0000 7f430d3fd640 1 --2- 172.21.15.46:0/4283756153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43080a2ee0 0x7f4308135e50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.188+0000 7f430d3fd640 1 -- 172.21.15.46:0/4283756153 >> 172.21.15.46:0/4283756153 conn(0x7f43080a5c10 msgr2=0x7f43080a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:33.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.188+0000 7f430d3fd640 1 -- 172.21.15.46:0/4283756153 shutdown_connections 2024-09-16T04:44:33.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.188+0000 7f430d3fd640 1 -- 172.21.15.46:0/4283756153 wait complete. 2024-09-16T04:44:33.232 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:44:33.232 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:44:33.232 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:44:33.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.333+0000 7f347e5cf640 10 monclient: get_monmap_and_config 2024-09-16T04:44:33.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.333+0000 7f347e5cf640 10 monclient: build_initial_monmap 2024-09-16T04:44:33.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.333+0000 7f347e5cf640 10 monclient: monmap: 2024-09-16T04:44:33.335 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:33.335 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.335 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:44:33.334529+0000 2024-09-16T04:44:33.335 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:44:33.334529+0000 2024-09-16T04:44:33.335 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:33.336 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.336 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:44:33.336 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:44:33.336 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:44:33.336 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:44:33.336 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:44:33.336 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.333+0000 7f347e5cf640 1 Processor -- start 2024-09-16T04:44:33.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.333+0000 7f347e5cf640 1 -- start start 2024-09-16T04:44:33.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.333+0000 7f347e5cf640 10 monclient: init 2024-09-16T04:44:33.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347e5cf640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:33.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347e5cf640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-16T04:44:33.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347e5cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f34780a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:44:33.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347e5cf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f34780a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:44:33.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347e5cf640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f347813cea0 0x7f347813f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347e5cf640 10 monclient(hunting): picked mon.noname-a con 0x7f347813cea0 addr v2:172.21.15.46:3300/0 2024-09-16T04:44:33.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347e5cf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f347809b940 con 0x7f34780a3c10 2024-09-16T04:44:33.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347e5cf640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f347809ab00 con 0x7f34780a3ff0 2024-09-16T04:44:33.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347e5cf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347e5cf640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f347809b030 con 0x7f347813cea0 2024-09-16T04:44:33.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347e5cf640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:33.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347e5cf640 10 monclient(hunting): authenticate will time out at 1071.750393s 2024-09-16T04:44:33.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347cb6c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f347813cea0 0x7f347813f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f3477fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f34780a3c10 0x7f347809bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45352/0 (socket says 172.21.15.46:45352) 2024-09-16T04:44:33.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f3477fff640 1 -- 172.21.15.46:0/4193232036 learned_addr learned my addr 172.21.15.46:0/4193232036 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:44:33.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347cb6c640 10 monclient(hunting): get_auth_request con 0x7f347813cea0 auth_method 0 2024-09-16T04:44:33.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347cb6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347cb6c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347cb6c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347cb6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347cb6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347cb6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347cb6c640 10 monclient(hunting): handle_auth_done global_id 74115 payload 274 2024-09-16T04:44:33.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347cb6c640 1 -- 172.21.15.46:0/4193232036 >> v1:172.21.15.114:6789/0 conn(0x7f34780a3ff0 legacy=0x7f34780a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347cb6c640 1 -- 172.21.15.46:0/4193232036 >> v1:172.21.15.46:6789/0 conn(0x7f34780a3c10 legacy=0x7f347809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347cb6c640 10 monclient: _finish_hunting 0 2024-09-16T04:44:33.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347cb6c640 1 monclient: found mon.noname-a 2024-09-16T04:44:33.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347cb6c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:33.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347cb6c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:44:33.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.334+0000 7f347cb6c640 1 -- 172.21.15.46:0/4193232036 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347809b310 con 0x7f347813cea0 2024-09-16T04:44:33.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.335+0000 7f347cb6c640 1 --2- 172.21.15.46:0/4193232036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f347813cea0 0x7f347813f290 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3470009d90 tx=0x7f347002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d645793f7d3c53c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:33.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.335+0000 7f3476ffd640 1 -- 172.21.15.46:0/4193232036 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3470007740 con 0x7f347813cea0 2024-09-16T04:44:33.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.335+0000 7f3476ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.335+0000 7f3476ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:44:33.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.335+0000 7f3476ffd640 10 monclient: dump: 2024-09-16T04:44:33.348 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.348 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.348 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.348 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.348 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.348 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.348 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.348 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.349 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.349 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.335+0000 7f3476ffd640 10 monclient: _finish_auth 0 2024-09-16T04:44:33.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.335+0000 7f3476ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:44:33.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.335+0000 7f3476ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:33.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.335+0000 7f3476ffd640 1 -- 172.21.15.46:0/4193232036 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34700078e0 con 0x7f347813cea0 2024-09-16T04:44:33.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.335+0000 7f3476ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:33.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.335+0000 7f3476ffd640 1 -- 172.21.15.46:0/4193232036 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3470007cb0 con 0x7f347813cea0 2024-09-16T04:44:33.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.335+0000 7f347e5cf640 5 monclient: authenticate success, global_id 74115 2024-09-16T04:44:33.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.335+0000 7f347e5cf640 10 monclient: get_monmap_and_config success 2024-09-16T04:44:33.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.335+0000 7f347e5cf640 4 set_mon_vals no callback set 2024-09-16T04:44:33.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.335+0000 7f3476ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.335+0000 7f3476ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:33.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.335+0000 7f3476ffd640 10 monclient: dump: 2024-09-16T04:44:33.350 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.351 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.351 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.351 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.351 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.351 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.352 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.352 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.352 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.352 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.336+0000 7f347e5cf640 10 monclient: shutdown 2024-09-16T04:44:33.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.336+0000 7f347e5cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:33.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.336+0000 7f347e5cf640 1 -- 172.21.15.46:0/4193232036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f347813cea0 msgr2=0x7f347813f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.336+0000 7f347e5cf640 1 --2- 172.21.15.46:0/4193232036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f347813cea0 0x7f347813f290 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3470009d90 tx=0x7f347002a660 comp rx=0 tx=0).stop 2024-09-16T04:44:33.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.336+0000 7f347e5cf640 1 -- 172.21.15.46:0/4193232036 shutdown_connections 2024-09-16T04:44:33.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.336+0000 7f347e5cf640 1 --2- 172.21.15.46:0/4193232036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f347813cea0 0x7f347813f290 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.336+0000 7f347e5cf640 1 -- 172.21.15.46:0/4193232036 >> 172.21.15.46:0/4193232036 conn(0x7f34780a6340 msgr2=0x7f347806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:33.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.336+0000 7f347e5cf640 1 -- 172.21.15.46:0/4193232036 shutdown_connections 2024-09-16T04:44:33.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.336+0000 7f347e5cf640 1 -- 172.21.15.46:0/4193232036 wait complete. 2024-09-16T04:44:33.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.336+0000 7f347e5cf640 10 monclient: build_initial_monmap 2024-09-16T04:44:33.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.336+0000 7f347e5cf640 10 monclient: monmap: 2024-09-16T04:44:33.354 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:33.354 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:44:33.355 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:44:33.355 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:44:33.355 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:33.355 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.355 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:44:33.355 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:44:33.356 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:44:33.356 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.337+0000 7f347e5cf640 1 Processor -- start 2024-09-16T04:44:33.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.337+0000 7f347e5cf640 1 -- start start 2024-09-16T04:44:33.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.337+0000 7f347e5cf640 10 monclient: init 2024-09-16T04:44:33.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.337+0000 7f347e5cf640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:33.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.337+0000 7f347e5cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:44:33.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.337+0000 7f347e5cf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34780a3c10 0x7f34781cbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.337+0000 7f347e5cf640 10 monclient(hunting): picked mon.noname-b con 0x7f34780a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:44:33.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.337+0000 7f347e5cf640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f34780a3ff0 0x7f3478136e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.337+0000 7f347e5cf640 10 monclient(hunting): picked mon.noname-c con 0x7f34780a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:33.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.337+0000 7f347e5cf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f347813cea0 0x7f3478139e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.337+0000 7f347e5cf640 10 monclient(hunting): picked mon.noname-a con 0x7f347813cea0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:44:33.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.337+0000 7f347e5cf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.337+0000 7f347e5cf640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f347813f950 con 0x7f347813cea0 2024-09-16T04:44:33.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.337+0000 7f347e5cf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.337+0000 7f347e5cf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f347813f7d0 con 0x7f34780a3c10 2024-09-16T04:44:33.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.337+0000 7f347e5cf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.337+0000 7f347e5cf640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f347813fad0 con 0x7f34780a3ff0 2024-09-16T04:44:33.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.337+0000 7f347e5cf640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:33.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.337+0000 7f347e5cf640 10 monclient(hunting): authenticate will time out at 1071.753904s 2024-09-16T04:44:33.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.337+0000 7f347cb6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f347813cea0 0x7f3478139e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347cb6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f347813cea0 0x7f3478139e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41296/0 (socket says 172.21.15.46:41296) 2024-09-16T04:44:33.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347cb6c640 1 -- 172.21.15.46:0/716184088 learned_addr learned my addr 172.21.15.46:0/716184088 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:33.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347cb6c640 10 monclient(hunting): get_auth_request con 0x7f347813cea0 auth_method 0 2024-09-16T04:44:33.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f34777fe640 1 --2- 172.21.15.46:0/716184088 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f34780a3ff0 0x7f3478136e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347cb6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347cb6c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347cb6c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f34777fe640 10 monclient(hunting): get_auth_request con 0x7f34780a3ff0 auth_method 0 2024-09-16T04:44:33.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f34777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f34777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f34777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f3477fff640 1 --2- 172.21.15.46:0/716184088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34780a3c10 0x7f34781cbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347cb6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347cb6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347cb6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f3477fff640 10 monclient(hunting): get_auth_request con 0x7f34780a3c10 auth_method 0 2024-09-16T04:44:33.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f3477fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f3477fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f3477fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f34777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f34777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f34777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f3477fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f3477fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f3477fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347cb6c640 10 monclient(hunting): handle_auth_done global_id 74121 payload 824 2024-09-16T04:44:33.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347cb6c640 1 -- 172.21.15.46:0/716184088 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f34780a3ff0 msgr2=0x7f3478136e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347cb6c640 1 --2- 172.21.15.46:0/716184088 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f34780a3ff0 0x7f3478136e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347cb6c640 1 -- 172.21.15.46:0/716184088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34780a3c10 msgr2=0x7f34781cbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347cb6c640 1 --2- 172.21.15.46:0/716184088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34780a3c10 0x7f34781cbbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347cb6c640 10 monclient: _finish_hunting 0 2024-09-16T04:44:33.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347cb6c640 1 monclient: found mon.noname-a 2024-09-16T04:44:33.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347cb6c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:33.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347cb6c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:44:33.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347cb6c640 1 -- 172.21.15.46:0/716184088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347813c670 con 0x7f347813cea0 2024-09-16T04:44:33.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347cb6c640 1 --2- 172.21.15.46:0/716184088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f347813cea0 0x7f3478139e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3470009eb0 tx=0x7f3470004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:33.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f3474ff9640 1 -- 172.21.15.46:0/716184088 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3470040070 con 0x7f347813cea0 2024-09-16T04:44:33.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f3474ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f3474ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:44:33.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f3474ff9640 10 monclient: dump: 2024-09-16T04:44:33.365 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.365 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.365 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.365 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.365 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.365 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.365 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.365 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.365 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.366 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f3474ff9640 10 monclient: _finish_auth 0 2024-09-16T04:44:33.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f3474ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:44:33.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f3474ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:33.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f3474ff9640 1 -- 172.21.15.46:0/716184088 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3470031bc0 con 0x7f347813cea0 2024-09-16T04:44:33.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347e5cf640 5 monclient: authenticate success, global_id 74121 2024-09-16T04:44:33.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347e5cf640 10 monclient: _renew_subs 2024-09-16T04:44:33.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347e5cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:33.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347e5cf640 1 -- 172.21.15.46:0/716184088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3478120510 con 0x7f347813cea0 2024-09-16T04:44:33.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f3474ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:33.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f3474ff9640 1 -- 172.21.15.46:0/716184088 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3470045460 con 0x7f347813cea0 2024-09-16T04:44:33.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f3474ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f3474ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:33.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f3474ff9640 10 monclient: dump: 2024-09-16T04:44:33.367 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.367 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.367 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.367 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.367 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.367 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.368 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.368 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.368 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.368 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f34767fc640 4 set_mon_vals no callback set 2024-09-16T04:44:33.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347e5cf640 10 monclient: _renew_subs 2024-09-16T04:44:33.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347e5cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:33.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.338+0000 7f347e5cf640 1 -- 172.21.15.46:0/716184088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f347813c3f0 con 0x7f347813cea0 2024-09-16T04:44:33.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.339+0000 7f3474ff9640 1 -- 172.21.15.46:0/716184088 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f3470039040 con 0x7f347813cea0 2024-09-16T04:44:33.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.339+0000 7f347e5cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:33.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.339+0000 7f347e5cf640 1 --2- 172.21.15.46:0/716184088 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f347802c640 0x7f347802ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.339+0000 7f347e5cf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.339+0000 7f347e5cf640 1 -- 172.21.15.46:0/716184088 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f347813c890 con 0x7f347802c640 2024-09-16T04:44:33.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.339+0000 7f3474ff9640 1 --2- 172.21.15.46:0/716184088 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f34580383b0 0x7f345803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.339+0000 7f3477fff640 1 --2- 172.21.15.46:0/716184088 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f347802c640 0x7f347802ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.339+0000 7f3477fff640 10 monclient: get_auth_request con 0x7f347802c640 auth_method 0 2024-09-16T04:44:33.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.339+0000 7f3477fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.339+0000 7f3477fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.339+0000 7f3477fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.339+0000 7f3474ff9640 1 -- 172.21.15.46:0/716184088 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(223..223 src has 1..223) ==== 4738+0+0 (secure 0 0 0) 0x7f347006d100 con 0x7f347813cea0 2024-09-16T04:44:33.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.339+0000 7f3477fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.339+0000 7f3477fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.339+0000 7f3477fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.340+0000 7f34777fe640 1 --2- 172.21.15.46:0/716184088 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f34580383b0 0x7f345803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.340+0000 7f34777fe640 10 monclient: get_auth_request con 0x7f34580383b0 auth_method 0 2024-09-16T04:44:33.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.340+0000 7f3477fff640 10 monclient(hunting): handle_auth_done global_id 64292 payload 824 2024-09-16T04:44:33.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.340+0000 7f3477fff640 1 -- 172.21.15.46:0/716184088 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f347811ff50 con 0x7f347802c640 2024-09-16T04:44:33.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.340+0000 7f3477fff640 1 --2- 172.21.15.46:0/716184088 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f347802c640 0x7f347802ca20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3468000f60 tx=0x7f346800fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:33.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.340+0000 7f3474ff9640 1 -- 172.21.15.46:0/716184088 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34680058b0 con 0x7f347802c640 2024-09-16T04:44:33.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.340+0000 7f3474ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.340+0000 7f3474ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:44:33.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.340+0000 7f3474ff9640 10 monclient: dump: 2024-09-16T04:44:33.371 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.371 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.371 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.371 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.371 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.371 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.372 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.372 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.372 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.372 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.340+0000 7f34777fe640 1 --2- 172.21.15.46:0/716184088 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f34580383b0 0x7f345803a870 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f347813ce70 tx=0x7f3460009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:33.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.340+0000 7f3474ff9640 1 -- 172.21.15.46:0/716184088 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3468018070 con 0x7f347802c640 2024-09-16T04:44:33.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.340+0000 7f3474ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:33.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.340+0000 7f3474ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:44:33.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.340+0000 7f3474ff9640 1 -- 172.21.15.46:0/716184088 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f347802c640 msgr2=0x7f347802ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.340+0000 7f3474ff9640 1 --2- 172.21.15.46:0/716184088 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f347802c640 0x7f347802ca20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3468000f60 tx=0x7f346800fc00 comp rx=0 tx=0).stop 2024-09-16T04:44:33.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.340+0000 7f3474ff9640 1 -- 172.21.15.46:0/716184088 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f347802c640 msgr2=0x7f347802ca20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:44:33.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.340+0000 7f3474ff9640 1 --2- 172.21.15.46:0/716184088 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f347802c640 0x7f347802ca20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.346+0000 7f347e5cf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:44:33.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.346+0000 7f347e5cf640 1 --2- 172.21.15.46:0/716184088 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f347802cf60 0x7f347802d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.346+0000 7f347e5cf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.346+0000 7f347e5cf640 1 -- 172.21.15.46:0/716184088 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f347809bcf0 con 0x7f347802cf60 2024-09-16T04:44:33.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.346+0000 7f3477fff640 1 --2- 172.21.15.46:0/716184088 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f347802cf60 0x7f347802d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.346+0000 7f3477fff640 10 monclient: get_auth_request con 0x7f347802cf60 auth_method 0 2024-09-16T04:44:33.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.346+0000 7f3477fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.346+0000 7f3477fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.346+0000 7f3477fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.347+0000 7f3477fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.347+0000 7f3477fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.347+0000 7f3477fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.347+0000 7f3477fff640 10 monclient(hunting): handle_auth_done global_id 64295 payload 824 2024-09-16T04:44:33.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.347+0000 7f3477fff640 1 -- 172.21.15.46:0/716184088 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f347813d330 con 0x7f347802cf60 2024-09-16T04:44:33.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.347+0000 7f3477fff640 1 --2- 172.21.15.46:0/716184088 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f347802cf60 0x7f347802d340 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3468000f60 tx=0x7f34680057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:33.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.347+0000 7f3474ff9640 1 -- 172.21.15.46:0/716184088 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3468007740 con 0x7f347802cf60 2024-09-16T04:44:33.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.347+0000 7f3474ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.347+0000 7f3474ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:44:33.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.347+0000 7f3474ff9640 10 monclient: dump: 2024-09-16T04:44:33.377 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.377 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.378 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.378 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.378 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.378 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.378 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.378 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.378 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.378 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.347+0000 7f3474ff9640 1 -- 172.21.15.46:0/716184088 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f346801d9d0 con 0x7f347802cf60 2024-09-16T04:44:33.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.347+0000 7f3474ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:44:33.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.347+0000 7f3474ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:44:33.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.347+0000 7f3474ff9640 1 -- 172.21.15.46:0/716184088 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f347802cf60 msgr2=0x7f347802d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.347+0000 7f3474ff9640 1 --2- 172.21.15.46:0/716184088 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f347802cf60 0x7f347802d340 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3468000f60 tx=0x7f34680057f0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.347+0000 7f3474ff9640 1 -- 172.21.15.46:0/716184088 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f347802cf60 msgr2=0x7f347802d340 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:44:33.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.347+0000 7f3474ff9640 1 --2- 172.21.15.46:0/716184088 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f347802cf60 0x7f347802d340 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3468000f60 tx=0x7f34680057f0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.379 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:44:33.379 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:44:33.379 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:44:33.379 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:44:33.379 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 68, 2024-09-16T04:44:33.379 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 61284, 2024-09-16T04:44:33.379 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:44:33.379 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:44:33.379 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:44:33.379 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:44:33.380 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:33.380 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:44:33.380 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:44:33.380 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:44:33.380 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:44:33.380 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:33.380 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:33.380 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:33.380 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:33.380 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:33.380 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:33.380 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:33.381 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:33.381 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:33.381 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:33.381 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:33.381 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:33.381 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:44:33.381 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:44:33.381 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:33.381 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:33.381 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:33.382 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:33.382 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:33.382 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:33.382 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:33.382 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:33.382 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:33.382 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:33.382 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:33.382 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:33.382 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.382 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:44:33.382 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:44:33.382 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:44:33.382 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:44:33.383 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:44:33.383 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:44:33.383 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:44:33.383 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:44:33.383 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:44:33.383 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:44:33.383 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:44:33.383 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:44:33.383 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:44:33.383 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:44:33.383 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:44:33.383 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:44:33.383 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:44:33.383 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:33.384 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:33.384 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:33.384 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:33.384 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:33.384 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:33.384 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:33.384 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:33.384 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:33.384 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:33.384 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:33.385 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:33.385 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:44:33.385 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.385 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:44:33.385 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.385 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:44:33.385 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:44:33.385 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:33.385 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:33.385 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.385 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:33.385 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:44:33.386 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:33.386 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.386 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.386 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:33.386 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:44:33.386 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:33.386 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:33.386 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:33.386 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.386 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:44:33.386 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:44:33.386 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:33.386 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:33.386 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:33.387 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.387 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.387 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:44:33.387 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:44:33.387 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:33.387 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:33.387 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.387 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:33.387 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:44:33.387 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:33.387 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.387 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.387 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:33.387 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:44:33.388 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:33.388 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:33.388 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:33.388 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.388 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:44:33.388 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:44:33.388 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:33.388 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:33.388 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:33.388 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.388 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.388 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:44:33.389 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:44:33.389 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:33.389 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:33.389 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.389 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:33.389 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:44:33.389 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:33.389 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.389 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.390 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:33.390 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:44:33.390 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:33.390 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:33.390 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:33.390 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.390 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:44:33.390 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:44:33.390 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:33.390 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:33.390 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:33.390 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:33.390 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:33.390 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.390 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:44:33.390 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:44:33.391 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.391 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:33.391 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:33.391 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:44:33.391 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:33.391 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:33.391 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:44:33.391 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.391 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:33.391 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:33.391 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:44:33.391 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:33.391 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:33.391 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:44:33.391 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.391 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:33.392 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:33.392 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:44:33.392 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:33.392 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:33.392 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.392 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:44:33.392 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:44:33.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.348+0000 7f347e5cf640 1 -- 172.21.15.46:0/716184088 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f34580383b0 msgr2=0x7f345803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.348+0000 7f347e5cf640 1 --2- 172.21.15.46:0/716184088 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f34580383b0 0x7f345803a870 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f347813ce70 tx=0x7f3460009040 comp rx=0 tx=0).stop 2024-09-16T04:44:33.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.348+0000 7f347e5cf640 10 monclient: shutdown 2024-09-16T04:44:33.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.348+0000 7f347e5cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:33.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.348+0000 7f347e5cf640 1 -- 172.21.15.46:0/716184088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f347813cea0 msgr2=0x7f3478139e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.348+0000 7f347cb6c640 1 -- 172.21.15.46:0/716184088 reap_dead start 2024-09-16T04:44:33.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.348+0000 7f347e5cf640 1 --2- 172.21.15.46:0/716184088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f347813cea0 0x7f3478139e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3470009eb0 tx=0x7f3470004a40 comp rx=0 tx=0).stop 2024-09-16T04:44:33.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.348+0000 7f347e5cf640 1 -- 172.21.15.46:0/716184088 shutdown_connections 2024-09-16T04:44:33.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.348+0000 7f347e5cf640 1 --2- 172.21.15.46:0/716184088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f347813cea0 0x7f3478139e70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.348+0000 7f347e5cf640 1 -- 172.21.15.46:0/716184088 >> 172.21.15.46:0/716184088 conn(0x7f34780a6340 msgr2=0x7f347809f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:33.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.348+0000 7f347e5cf640 1 -- 172.21.15.46:0/716184088 shutdown_connections 2024-09-16T04:44:33.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.348+0000 7f347e5cf640 1 -- 172.21.15.46:0/716184088 wait complete. 2024-09-16T04:44:33.394 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:44:33.394 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:44:33.394 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:44:33.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.495+0000 7fdec159d640 10 monclient: get_monmap_and_config 2024-09-16T04:44:33.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.495+0000 7fdec159d640 10 monclient: build_initial_monmap 2024-09-16T04:44:33.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.495+0000 7fdec159d640 10 monclient: monmap: 2024-09-16T04:44:33.497 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:33.498 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.498 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:44:33.496755+0000 2024-09-16T04:44:33.498 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:44:33.496755+0000 2024-09-16T04:44:33.498 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:33.498 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.498 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:44:33.498 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:44:33.498 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:44:33.498 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:44:33.498 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:44:33.498 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.495+0000 7fdec159d640 1 Processor -- start 2024-09-16T04:44:33.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec159d640 1 -- start start 2024-09-16T04:44:33.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec159d640 10 monclient: init 2024-09-16T04:44:33.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec159d640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:33.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec159d640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-16T04:44:33.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec159d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdebc136960 0x7fdebc136d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec159d640 10 monclient(hunting): picked mon.noname-c con 0x7fdebc136960 addr v2:172.21.15.114:3300/0 2024-09-16T04:44:33.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec159d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdebc0a32b0 0x7fdebc137280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec159d640 10 monclient(hunting): picked mon.noname-b con 0x7fdebc0a32b0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:33.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec159d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdebc1337c0 0x7fdebc133c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec159d640 10 monclient(hunting): picked mon.noname-a con 0x7fdebc1337c0 addr v2:172.21.15.46:3300/0 2024-09-16T04:44:33.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec159d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec159d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fdebc1377c0 con 0x7fdebc1337c0 2024-09-16T04:44:33.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec159d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec159d640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fdebc0a1480 con 0x7fdebc136960 2024-09-16T04:44:33.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec159d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec159d640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fdebc0a4e40 con 0x7fdebc0a32b0 2024-09-16T04:44:33.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec159d640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:33.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec159d640 10 monclient(hunting): authenticate will time out at 1071.912617s 2024-09-16T04:44:33.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdebb7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdebc0a32b0 0x7fdebc137280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec0d9c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdebc1337c0 0x7fdebc133c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdebb7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdebc0a32b0 0x7fdebc137280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:53520/0 (socket says 172.21.15.46:53520) 2024-09-16T04:44:33.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec0d9c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdebc1337c0 0x7fdebc133c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41298/0 (socket says 172.21.15.46:41298) 2024-09-16T04:44:33.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdebb7fe640 1 -- 172.21.15.46:0/724999905 learned_addr learned my addr 172.21.15.46:0/724999905 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:33.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdebb7fe640 10 monclient(hunting): get_auth_request con 0x7fdebc0a32b0 auth_method 0 2024-09-16T04:44:33.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdebb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdebb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdebb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec0d9c640 10 monclient(hunting): get_auth_request con 0x7fdebc1337c0 auth_method 0 2024-09-16T04:44:33.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec0d9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec0d9c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec0d9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec0d9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec0d9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdec0d9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdebb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdebb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdebb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdebbfff640 1 --2- 172.21.15.46:0/724999905 >> v2:172.21.15.114:3300/0 conn(0x7fdebc136960 0x7fdebc136d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdebbfff640 10 monclient(hunting): get_auth_request con 0x7fdebc136960 auth_method 0 2024-09-16T04:44:33.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdebbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdebbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.496+0000 7fdebbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdebb7fe640 10 monclient(hunting): handle_auth_done global_id 64298 payload 274 2024-09-16T04:44:33.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdebb7fe640 1 -- 172.21.15.46:0/724999905 >> v2:172.21.15.114:3300/0 conn(0x7fdebc136960 msgr2=0x7fdebc136d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:44:33.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdebb7fe640 1 --2- 172.21.15.46:0/724999905 >> v2:172.21.15.114:3300/0 conn(0x7fdebc136960 0x7fdebc136d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdebb7fe640 1 -- 172.21.15.46:0/724999905 >> v2:172.21.15.46:3300/0 conn(0x7fdebc1337c0 msgr2=0x7fdebc133c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:44:33.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdebb7fe640 1 --2- 172.21.15.46:0/724999905 >> v2:172.21.15.46:3300/0 conn(0x7fdebc1337c0 0x7fdebc133c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdebb7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:44:33.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdebb7fe640 1 monclient: found mon.noname-b 2024-09-16T04:44:33.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdebb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:33.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdebb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:44:33.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdebb7fe640 1 -- 172.21.15.46:0/724999905 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdebc0a0b70 con 0x7fdebc0a32b0 2024-09-16T04:44:33.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdebbfff640 1 --2- 172.21.15.46:0/724999905 >> v2:172.21.15.114:3300/0 conn(0x7fdebc136960 0x7fdebc136d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:44:33.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdec0d9c640 1 --2- 172.21.15.46:0/724999905 >> v2:172.21.15.46:3300/0 conn(0x7fdebc1337c0 0x7fdebc133c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:44:33.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdebb7fe640 1 --2- 172.21.15.46:0/724999905 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdebc0a32b0 0x7fdebc137280 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdebc0aa0c0 tx=0x7fdeb002a0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=416a1c86b5465c13 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:33.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdebaffd640 1 -- 172.21.15.46:0/724999905 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0037070 con 0x7fdebc0a32b0 2024-09-16T04:44:33.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdebaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdebaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:44:33.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdebaffd640 10 monclient: dump: 2024-09-16T04:44:33.512 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.512 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.512 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.512 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.512 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.513 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.513 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.513 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.513 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.513 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdebaffd640 10 monclient: _finish_auth 0 2024-09-16T04:44:33.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdebaffd640 10 monclient: _check_auth_tickets 2024-09-16T04:44:33.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdebaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:33.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdebaffd640 1 -- 172.21.15.46:0/724999905 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeb002ae00 con 0x7fdebc0a32b0 2024-09-16T04:44:33.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdec159d640 5 monclient: authenticate success, global_id 64298 2024-09-16T04:44:33.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdec159d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:44:33.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.497+0000 7fdebaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:33.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.498+0000 7fdebaffd640 1 -- 172.21.15.46:0/724999905 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0032660 con 0x7fdebc0a32b0 2024-09-16T04:44:33.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.498+0000 7fdebaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.498+0000 7fdebaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:44:33.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.498+0000 7fdebaffd640 10 monclient: dump: 2024-09-16T04:44:33.514 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.514 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.514 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.514 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.514 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.514 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.514 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.514 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.514 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.514 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.498+0000 7fdec159d640 10 monclient: get_monmap_and_config success 2024-09-16T04:44:33.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.498+0000 7fdec159d640 4 set_mon_vals no callback set 2024-09-16T04:44:33.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.498+0000 7fdec159d640 10 monclient: shutdown 2024-09-16T04:44:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.498+0000 7fdec159d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.498+0000 7fdec159d640 1 -- 172.21.15.46:0/724999905 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdebc0a32b0 msgr2=0x7fdebc137280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.498+0000 7fdec159d640 1 --2- 172.21.15.46:0/724999905 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdebc0a32b0 0x7fdebc137280 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdebc0aa0c0 tx=0x7fdeb002a0b0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.498+0000 7fdec159d640 1 -- 172.21.15.46:0/724999905 shutdown_connections 2024-09-16T04:44:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.498+0000 7fdec159d640 1 --2- 172.21.15.46:0/724999905 >> v2:172.21.15.46:3300/0 conn(0x7fdebc1337c0 0x7fdebc133c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.498+0000 7fdec159d640 1 --2- 172.21.15.46:0/724999905 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdebc0a32b0 0x7fdebc137280 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.498+0000 7fdec159d640 1 --2- 172.21.15.46:0/724999905 >> v2:172.21.15.114:3300/0 conn(0x7fdebc136960 0x7fdebc136d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.498+0000 7fdec159d640 1 -- 172.21.15.46:0/724999905 >> 172.21.15.46:0/724999905 conn(0x7fdebc0a9b80 msgr2=0x7fdebc0a5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.498+0000 7fdec159d640 1 -- 172.21.15.46:0/724999905 shutdown_connections 2024-09-16T04:44:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.498+0000 7fdec159d640 1 -- 172.21.15.46:0/724999905 wait complete. 2024-09-16T04:44:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.499+0000 7fdec159d640 10 monclient: build_initial_monmap 2024-09-16T04:44:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.499+0000 7fdec159d640 10 monclient: monmap: 2024-09-16T04:44:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:44:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:44:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:44:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:44:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:44:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:44:33.516 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.499+0000 7fdec159d640 1 Processor -- start 2024-09-16T04:44:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.499+0000 7fdec159d640 1 -- start start 2024-09-16T04:44:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.499+0000 7fdec159d640 10 monclient: init 2024-09-16T04:44:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdec159d640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdec159d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:44:33.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdec159d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdebc0a32b0 0x7fdebc1cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdec159d640 10 monclient(hunting): picked mon.noname-a con 0x7fdebc0a32b0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:44:33.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdec159d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdebc1337c0 0x7fdebc1cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdec159d640 10 monclient(hunting): picked mon.noname-c con 0x7fdebc1337c0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:33.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdec159d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdebc136960 0x7fdebc120170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdec159d640 10 monclient(hunting): picked mon.noname-b con 0x7fdebc136960 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:44:33.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdec159d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdec159d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc06e3c0 con 0x7fdebc0a32b0 2024-09-16T04:44:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdec159d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdec159d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc06e240 con 0x7fdebc136960 2024-09-16T04:44:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdec159d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdec159d640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fdebc0a3a30 con 0x7fdebc1337c0 2024-09-16T04:44:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdec159d640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdec159d640 10 monclient(hunting): authenticate will time out at 1071.916314s 2024-09-16T04:44:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdebc0a32b0 0x7fdebc1cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebb7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdebc1337c0 0x7fdebc1cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdebc0a32b0 0x7fdebc1cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41308/0 (socket says 172.21.15.46:41308) 2024-09-16T04:44:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebbfff640 1 -- 172.21.15.46:0/1115953258 learned_addr learned my addr 172.21.15.46:0/1115953258 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebbfff640 10 monclient(hunting): get_auth_request con 0x7fdebc0a32b0 auth_method 0 2024-09-16T04:44:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebb7fe640 10 monclient(hunting): get_auth_request con 0x7fdebc1337c0 auth_method 0 2024-09-16T04:44:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdec0d9c640 1 --2- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdebc136960 0x7fdebc120170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdec0d9c640 10 monclient(hunting): get_auth_request con 0x7fdebc136960 auth_method 0 2024-09-16T04:44:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdec0d9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdec0d9c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdec0d9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebb7fe640 10 monclient(hunting): handle_auth_done global_id 64301 payload 824 2024-09-16T04:44:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebb7fe640 1 -- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdebc136960 msgr2=0x7fdebc120170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebb7fe640 1 --2- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdebc136960 0x7fdebc120170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebb7fe640 1 -- 172.21.15.46:0/1115953258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdebc0a32b0 msgr2=0x7fdebc1cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebb7fe640 1 --2- 172.21.15.46:0/1115953258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdebc0a32b0 0x7fdebc1cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebb7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:44:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebb7fe640 1 monclient: found mon.noname-c 2024-09-16T04:44:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:44:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebb7fe640 1 -- 172.21.15.46:0/1115953258 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdebc1221e0 con 0x7fdebc1337c0 2024-09-16T04:44:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebbfff640 1 --2- 172.21.15.46:0/1115953258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdebc0a32b0 0x7fdebc1cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:44:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdebb7fe640 1 --2- 172.21.15.46:0/1115953258 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdebc1337c0 0x7fdebc1cc420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0031000 tx=0x7fdeb00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.500+0000 7fdeb8ff9640 1 -- 172.21.15.46:0/1115953258 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0041070 con 0x7fdebc1337c0 2024-09-16T04:44:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdeb8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdeb8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:44:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdeb8ff9640 10 monclient: dump: 2024-09-16T04:44:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.522 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdeb8ff9640 10 monclient: _finish_auth 0 2024-09-16T04:44:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdeb8ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:44:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdeb8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdeb8ff9640 1 -- 172.21.15.46:0/1115953258 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeb003c460 con 0x7fdebc1337c0 2024-09-16T04:44:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdec159d640 5 monclient: authenticate success, global_id 64301 2024-09-16T04:44:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdec159d640 10 monclient: _renew_subs 2024-09-16T04:44:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdec159d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:44:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdec159d640 1 -- 172.21.15.46:0/1115953258 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdebc1228a0 con 0x7fdebc1337c0 2024-09-16T04:44:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdeb8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdeb8ff9640 1 -- 172.21.15.46:0/1115953258 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0037040 con 0x7fdebc1337c0 2024-09-16T04:44:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdeb8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdeb8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:44:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdeb8ff9640 10 monclient: dump: 2024-09-16T04:44:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.524 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdec159d640 10 monclient: _renew_subs 2024-09-16T04:44:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdec159d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:44:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdec159d640 1 -- 172.21.15.46:0/1115953258 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdebc123a70 con 0x7fdebc1337c0 2024-09-16T04:44:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdeba7fc640 4 set_mon_vals no callback set 2024-09-16T04:44:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdec159d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdec159d640 1 --2- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdebc123cb0 0x7fdebc1206b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdec159d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdec159d640 1 -- 172.21.15.46:0/1115953258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc136d40 con 0x7fdebc123cb0 2024-09-16T04:44:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.501+0000 7fdeb8ff9640 1 -- 172.21.15.46:0/1115953258 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fdeb003a070 con 0x7fdebc1337c0 2024-09-16T04:44:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.502+0000 7fdebbfff640 1 --2- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdebc123cb0 0x7fdebc1206b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.502+0000 7fdeb8ff9640 1 --2- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdea80383b0 0x7fdea803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.502+0000 7fdebbfff640 10 monclient: get_auth_request con 0x7fdebc123cb0 auth_method 0 2024-09-16T04:44:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.502+0000 7fdebbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.502+0000 7fdebbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.502+0000 7fdebbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.502+0000 7fdeb8ff9640 1 -- 172.21.15.46:0/1115953258 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(223..223 src has 1..223) ==== 4738+0+0 (secure 0 0 0) 0x7fdeb006bcd0 con 0x7fdebc1337c0 2024-09-16T04:44:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.502+0000 7fdebbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.502+0000 7fdebbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.502+0000 7fdebbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.502+0000 7fdec0d9c640 1 --2- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdea80383b0 0x7fdea803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.502+0000 7fdec0d9c640 10 monclient: get_auth_request con 0x7fdea80383b0 auth_method 0 2024-09-16T04:44:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.502+0000 7fdebbfff640 10 monclient(hunting): handle_auth_done global_id 6220 payload 824 2024-09-16T04:44:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.502+0000 7fdebbfff640 1 -- 172.21.15.46:0/1115953258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdebc124090 con 0x7fdebc123cb0 2024-09-16T04:44:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.502+0000 7fdebbfff640 1 --2- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdebc123cb0 0x7fdebc1206b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fdeac009fd0 tx=0x7fdeac011f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.502+0000 7fdeb8ff9640 1 -- 172.21.15.46:0/1115953258 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeac00f040 con 0x7fdebc123cb0 2024-09-16T04:44:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.502+0000 7fdeb8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.502+0000 7fdec0d9c640 1 --2- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdea80383b0 0x7fdea803a870 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4007520 tx=0x7fdeb400d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.502+0000 7fdeb8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:44:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.502+0000 7fdeb8ff9640 10 monclient: dump: 2024-09-16T04:44:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.527 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.503+0000 7fdeb8ff9640 1 -- 172.21.15.46:0/1115953258 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdeac00fe60 con 0x7fdebc123cb0 2024-09-16T04:44:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.503+0000 7fdeb8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.503+0000 7fdeb8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:44:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.503+0000 7fdeb8ff9640 1 -- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdebc123cb0 msgr2=0x7fdebc1206b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.503+0000 7fdeb8ff9640 1 --2- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdebc123cb0 0x7fdebc1206b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fdeac009fd0 tx=0x7fdeac011f10 comp rx=0 tx=0).stop 2024-09-16T04:44:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.503+0000 7fdeb8ff9640 1 -- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdebc123cb0 msgr2=0x7fdebc1206b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:44:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.503+0000 7fdeb8ff9640 1 --2- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdebc123cb0 0x7fdebc1206b0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.509+0000 7fdec159d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:44:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.509+0000 7fdec159d640 1 --2- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdebc120bf0 0x7fdebc121000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.509+0000 7fdec159d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.509+0000 7fdec159d640 1 -- 172.21.15.46:0/1115953258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc121540 con 0x7fdebc120bf0 2024-09-16T04:44:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.509+0000 7fdebbfff640 1 --2- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdebc120bf0 0x7fdebc121000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.509+0000 7fdebbfff640 10 monclient: get_auth_request con 0x7fdebc120bf0 auth_method 0 2024-09-16T04:44:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.509+0000 7fdebbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.509+0000 7fdebbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.509+0000 7fdebbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.509+0000 7fdebbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.509+0000 7fdebbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.509+0000 7fdebbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.510+0000 7fdebbfff640 10 monclient(hunting): handle_auth_done global_id 6223 payload 824 2024-09-16T04:44:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.510+0000 7fdebbfff640 1 -- 172.21.15.46:0/1115953258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdebc067c60 con 0x7fdebc120bf0 2024-09-16T04:44:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.510+0000 7fdebbfff640 1 --2- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdebc120bf0 0x7fdebc121000 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fdeac009fd0 tx=0x7fdeac004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.510+0000 7fdeb8ff9640 1 -- 172.21.15.46:0/1115953258 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeac0158f0 con 0x7fdebc120bf0 2024-09-16T04:44:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.510+0000 7fdeb8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.510+0000 7fdeb8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:44:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.510+0000 7fdeb8ff9640 10 monclient: dump: 2024-09-16T04:44:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.531 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.510+0000 7fdeb8ff9640 1 -- 172.21.15.46:0/1115953258 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7fdeac0158f0 con 0x7fdebc120bf0 2024-09-16T04:44:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.510+0000 7fdeb8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:44:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.510+0000 7fdeb8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:44:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.510+0000 7fdeb8ff9640 1 -- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdebc120bf0 msgr2=0x7fdebc121000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.510+0000 7fdeb8ff9640 1 --2- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdebc120bf0 0x7fdebc121000 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fdeac009fd0 tx=0x7fdeac004730 comp rx=0 tx=0).stop 2024-09-16T04:44:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.510+0000 7fdeb8ff9640 1 -- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdebc120bf0 msgr2=0x7fdebc121000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:44:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.510+0000 7fdeb8ff9640 1 --2- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdebc120bf0 0x7fdebc121000 secure :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fdeac009fd0 tx=0x7fdeac004730 comp rx=0 tx=0).stop 2024-09-16T04:44:33.532 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:44:33.532 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:44:33.532 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:44:33.532 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:44:33.532 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 68, 2024-09-16T04:44:33.533 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 359413, 2024-09-16T04:44:33.533 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:44:33.533 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:44:33.533 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:44:33.533 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:44:33.533 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:33.533 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:44:33.533 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:44:33.533 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:44:33.533 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:44:33.533 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:33.533 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:33.533 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:33.533 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:33.534 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:33.534 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:33.534 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:33.534 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:33.534 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:33.534 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:33.534 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:33.534 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:33.534 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:44:33.535 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:44:33.535 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:33.535 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:33.535 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:33.535 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:33.535 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:33.536 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:33.536 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:33.536 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:33.536 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:33.536 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:33.536 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:33.537 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:33.537 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.537 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:44:33.537 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:44:33.537 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:44:33.537 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:44:33.538 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:44:33.538 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:44:33.538 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:44:33.538 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:44:33.538 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:44:33.538 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:44:33.538 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:44:33.538 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:44:33.539 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:44:33.539 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:44:33.539 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:44:33.539 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:44:33.539 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:44:33.539 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:33.539 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:33.539 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:33.540 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:33.540 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:33.540 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:33.540 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:33.540 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:33.540 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:33.541 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:33.541 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:33.541 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:33.541 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:44:33.541 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.541 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:44:33.542 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.542 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:44:33.542 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:44:33.542 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:33.542 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:33.542 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.543 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:33.543 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:44:33.543 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:33.543 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.543 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.543 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:33.543 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:44:33.543 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:33.543 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:33.544 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:33.544 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.544 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:44:33.544 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:44:33.544 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:33.544 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:33.544 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:33.544 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.544 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.544 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:44:33.544 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:44:33.544 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:33.545 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:33.545 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.545 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:33.545 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:44:33.545 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:33.545 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.545 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.545 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:33.545 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:44:33.545 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:33.545 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:33.545 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:33.545 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.546 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:44:33.546 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:44:33.546 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:33.546 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:33.546 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:33.546 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.546 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.546 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:44:33.546 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:44:33.546 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:33.546 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:33.546 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.546 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:33.546 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:44:33.546 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:33.547 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.547 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.547 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:33.547 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:44:33.547 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:33.547 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:33.547 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:33.547 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.547 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:44:33.547 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:44:33.547 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:33.547 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:33.547 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:33.547 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:33.547 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:33.548 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.548 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:44:33.548 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:44:33.548 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.548 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:33.548 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:33.548 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:44:33.548 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:33.548 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:33.548 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:44:33.548 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.548 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:33.548 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:33.548 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5 2024-09-16T04:44:33.548 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:33.549 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:33.552 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:44:33.552 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.553 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:33.553 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:33.553 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6 2024-09-16T04:44:33.553 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:33.554 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:33.554 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:44:33.554 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:33.554 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:33.554 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:33.554 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:44:33.555 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:33.555 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:33.555 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:33.555 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:44:33.555 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:44:33.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.510+0000 7fdec159d640 1 -- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdea80383b0 msgr2=0x7fdea803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.510+0000 7fdec159d640 1 --2- 172.21.15.46:0/1115953258 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdea80383b0 0x7fdea803a870 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4007520 tx=0x7fdeb400d040 comp rx=0 tx=0).stop 2024-09-16T04:44:33.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.510+0000 7fdec159d640 10 monclient: shutdown 2024-09-16T04:44:33.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.510+0000 7fdec159d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:33.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.510+0000 7fdec159d640 1 -- 172.21.15.46:0/1115953258 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdebc1337c0 msgr2=0x7fdebc1cc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.510+0000 7fdec159d640 1 --2- 172.21.15.46:0/1115953258 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdebc1337c0 0x7fdebc1cc420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0031000 tx=0x7fdeb00046c0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.510+0000 7fdec0d9c640 1 -- 172.21.15.46:0/1115953258 reap_dead start 2024-09-16T04:44:33.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.511+0000 7fdec0d9c640 1 -- 172.21.15.46:0/1115953258 reap_dead start 2024-09-16T04:44:33.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.511+0000 7fdec159d640 1 -- 172.21.15.46:0/1115953258 shutdown_connections 2024-09-16T04:44:33.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.511+0000 7fdec159d640 1 -- 172.21.15.46:0/1115953258 >> 172.21.15.46:0/1115953258 conn(0x7fdebc0a9b80 msgr2=0x7fdebc0a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:33.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.511+0000 7fdec159d640 1 -- 172.21.15.46:0/1115953258 shutdown_connections 2024-09-16T04:44:33.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.511+0000 7fdec159d640 1 -- 172.21.15.46:0/1115953258 wait complete. 2024-09-16T04:44:33.560 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:44:33.561 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:44:33.561 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:44:33.646 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:33.646 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38821]: starting. 2024-09-16T04:44:33.647 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38821]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-38783 2024-09-16T04:44:33.647 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38821]: shutting down. 2024-09-16T04:44:33.648 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:33.648 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:33.649 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:44:33.651 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:44:33.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.663+0000 7f760961e640 10 monclient: get_monmap_and_config 2024-09-16T04:44:33.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.663+0000 7f760961e640 10 monclient: build_initial_monmap 2024-09-16T04:44:33.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.663+0000 7f760961e640 10 monclient: monmap: 2024-09-16T04:44:33.665 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:33.665 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.665 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:44:33.664588+0000 2024-09-16T04:44:33.665 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:44:33.664588+0000 2024-09-16T04:44:33.665 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:33.665 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.666 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:44:33.666 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:44:33.666 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:44:33.666 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:44:33.666 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:44:33.666 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.663+0000 7f760961e640 1 Processor -- start 2024-09-16T04:44:33.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.663+0000 7f760961e640 1 -- start start 2024-09-16T04:44:33.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.663+0000 7f760961e640 10 monclient: init 2024-09-16T04:44:33.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f760961e640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:33.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f760961e640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-16T04:44:33.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f760961e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f76040a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:44:33.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f760961e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f76040a3ff0 0x7f76040a4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f760961e640 10 monclient(hunting): picked mon.noname-c con 0x7f76040a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:44:33.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f760961e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f760413be90 addr v1:172.21.15.46:6789/0 2024-09-16T04:44:33.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f760961e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f760409ab70 con 0x7f760413be90 2024-09-16T04:44:33.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f760961e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76040a4d60 con 0x7f76040a3c10 2024-09-16T04:44:33.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f760961e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f760961e640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f760409b030 con 0x7f76040a3ff0 2024-09-16T04:44:33.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f760961e640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:33.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f760961e640 10 monclient(hunting): authenticate will time out at 1072.080500s 2024-09-16T04:44:33.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f76037fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f760413be90 0x7f760413e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45368/0 (socket says 172.21.15.46:45368) 2024-09-16T04:44:33.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f76037fe640 1 -- 172.21.15.46:0/172868445 learned_addr learned my addr 172.21.15.46:0/172868445 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:44:33.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f75fbfff640 1 --2- 172.21.15.46:0/172868445 >> v2:172.21.15.114:3300/0 conn(0x7f76040a3ff0 0x7f76040a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f75fbfff640 10 monclient(hunting): get_auth_request con 0x7f76040a3ff0 auth_method 0 2024-09-16T04:44:33.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f75fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f75fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f75fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f76027fc640 1 -- 172.21.15.46:0/172868445 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4134222574 0 0) 0x7f760409ab70 con 0x7f760413be90 2024-09-16T04:44:33.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f75fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f75fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f75fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f76027fc640 10 monclient(hunting): _negotiate 2024-09-16T04:44:33.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f76027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f76027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f76027fc640 10 monclient(hunting): my global_id is 74130 2024-09-16T04:44:33.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f76027fc640 1 -- 172.21.15.46:0/172868445 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75f0003520 con 0x7f760413be90 2024-09-16T04:44:33.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.664+0000 7f76027fc640 1 -- 172.21.15.46:0/172868445 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3857203672 0 0) 0x7f76040a4d60 con 0x7f76040a3c10 2024-09-16T04:44:33.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.665+0000 7f76027fc640 10 monclient(hunting): _negotiate 2024-09-16T04:44:33.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.665+0000 7f76027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.665+0000 7f76027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.665+0000 7f76027fc640 10 monclient(hunting): my global_id is 6229 2024-09-16T04:44:33.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.665+0000 7f76027fc640 1 -- 172.21.15.46:0/172868445 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f760409ab70 con 0x7f76040a3c10 2024-09-16T04:44:33.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.665+0000 7f75fbfff640 10 monclient(hunting): handle_auth_done global_id 6226 payload 274 2024-09-16T04:44:33.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.665+0000 7f75fbfff640 1 -- 172.21.15.46:0/172868445 >> v1:172.21.15.114:6789/0 conn(0x7f76040a3c10 legacy=0x7f760409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.665+0000 7f75fbfff640 1 -- 172.21.15.46:0/172868445 >> v1:172.21.15.46:6789/0 conn(0x7f760413be90 legacy=0x7f760413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.665+0000 7f76027fc640 1 -- 172.21.15.46:0/172868445 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2714752133 0 0) 0x7f75f0003520 con 0x7f760413be90 2024-09-16T04:44:33.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.665+0000 7f75fbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:44:33.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.665+0000 7f75fbfff640 1 monclient: found mon.noname-c 2024-09-16T04:44:33.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.665+0000 7f75fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:33.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.665+0000 7f75fbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:44:33.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.665+0000 7f75fbfff640 1 -- 172.21.15.46:0/172868445 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f760409b310 con 0x7f76040a3ff0 2024-09-16T04:44:33.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.665+0000 7f76027fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:44:33.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.665+0000 7f75fbfff640 1 --2- 172.21.15.46:0/172868445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76040a3ff0 0x7f76040a4400 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f75ec0097a0 tx=0x7f75ec02a0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46db9d83dd1721ca server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:33.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f76027fc640 1 -- 172.21.15.46:0/172868445 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ec037070 con 0x7f76040a3ff0 2024-09-16T04:44:33.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f76027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f76027fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:44:33.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f76027fc640 10 monclient: dump: 2024-09-16T04:44:33.678 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.678 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.678 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.678 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.678 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.678 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.678 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.678 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.678 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.679 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f76027fc640 10 monclient: _finish_auth 0 2024-09-16T04:44:33.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f76027fc640 10 monclient: _check_auth_tickets 2024-09-16T04:44:33.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f76027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:33.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f76027fc640 1 -- 172.21.15.46:0/172868445 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75ec02ab30 con 0x7f76040a3ff0 2024-09-16T04:44:33.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f760961e640 5 monclient: authenticate success, global_id 6226 2024-09-16T04:44:33.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f760961e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:44:33.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f76027fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:33.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f760961e640 10 monclient: get_monmap_and_config success 2024-09-16T04:44:33.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f760961e640 4 set_mon_vals no callback set 2024-09-16T04:44:33.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f76027fc640 1 -- 172.21.15.46:0/172868445 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75fc0019f0 con 0x7f76040a3ff0 2024-09-16T04:44:33.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f76027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f76027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:44:33.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f76027fc640 10 monclient: dump: 2024-09-16T04:44:33.680 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.680 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.680 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.680 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.680 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.680 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.680 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.680 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.680 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.680 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f760961e640 10 monclient: shutdown 2024-09-16T04:44:33.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f760961e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:33.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f760961e640 1 -- 172.21.15.46:0/172868445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76040a3ff0 msgr2=0x7f76040a4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f760961e640 1 --2- 172.21.15.46:0/172868445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76040a3ff0 0x7f76040a4400 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f75ec0097a0 tx=0x7f75ec02a0a0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f760961e640 1 -- 172.21.15.46:0/172868445 shutdown_connections 2024-09-16T04:44:33.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f760961e640 1 --2- 172.21.15.46:0/172868445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76040a3ff0 0x7f76040a4400 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f760961e640 1 -- 172.21.15.46:0/172868445 >> 172.21.15.46:0/172868445 conn(0x7f76040a5d10 msgr2=0x7f76040a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:33.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f760961e640 1 -- 172.21.15.46:0/172868445 shutdown_connections 2024-09-16T04:44:33.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.666+0000 7f760961e640 1 -- 172.21.15.46:0/172868445 wait complete. 2024-09-16T04:44:33.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.667+0000 7f760961e640 10 monclient: build_initial_monmap 2024-09-16T04:44:33.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.667+0000 7f760961e640 10 monclient: monmap: 2024-09-16T04:44:33.681 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:33.681 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:44:33.681 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:44:33.681 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:44:33.681 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:33.682 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.682 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:44:33.682 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:44:33.682 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:44:33.682 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.667+0000 7f760961e640 1 Processor -- start 2024-09-16T04:44:33.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.667+0000 7f760961e640 1 -- start start 2024-09-16T04:44:33.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.667+0000 7f760961e640 10 monclient: init 2024-09-16T04:44:33.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:33.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:44:33.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f76040a3c10 0x7f7604136b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 10 monclient(hunting): picked mon.noname-c con 0x7f76040a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:33.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76040a3ff0 0x7f7604138c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 10 monclient(hunting): picked mon.noname-b con 0x7f76040a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:44:33.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f760413be90 0x7f76041391a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 10 monclient(hunting): picked mon.noname-a con 0x7f760413be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:44:33.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f760406e550 con 0x7f760413be90 2024-09-16T04:44:33.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f760406e3d0 con 0x7f76040a3ff0 2024-09-16T04:44:33.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:33.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f760413f9a0 con 0x7f76040a3c10 2024-09-16T04:44:33.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:33.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 10 monclient(hunting): authenticate will time out at 1072.084111s 2024-09-16T04:44:33.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f76037fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f760413be90 0x7f76041391a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f7602ffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f76040a3c10 0x7f7604136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f76037fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f760413be90 0x7f76041391a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41316/0 (socket says 172.21.15.46:41316) 2024-09-16T04:44:33.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f76037fe640 1 -- 172.21.15.46:0/2799358267 learned_addr learned my addr 172.21.15.46:0/2799358267 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:33.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f76037fe640 10 monclient(hunting): get_auth_request con 0x7f760413be90 auth_method 0 2024-09-16T04:44:33.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f76037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f76037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f76037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f7602ffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f76040a3c10 0x7f7604136b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:53542/0 (socket says 172.21.15.46:53542) 2024-09-16T04:44:33.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f7602ffd640 10 monclient(hunting): get_auth_request con 0x7f76040a3c10 auth_method 0 2024-09-16T04:44:33.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f7602ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f7602ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f7602ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f75fbfff640 1 --2- 172.21.15.46:0/2799358267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76040a3ff0 0x7f7604138c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f75fbfff640 10 monclient(hunting): get_auth_request con 0x7f76040a3ff0 auth_method 0 2024-09-16T04:44:33.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f75fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:33.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f75fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f75fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f76037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f76037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f76037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f7602ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f7602ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f7602ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f75fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f75fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f75fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f76037fe640 10 monclient(hunting): handle_auth_done global_id 74133 payload 824 2024-09-16T04:44:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f76037fe640 1 -- 172.21.15.46:0/2799358267 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f76040a3c10 msgr2=0x7f7604136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f76037fe640 1 --2- 172.21.15.46:0/2799358267 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f76040a3c10 0x7f7604136b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f76037fe640 1 -- 172.21.15.46:0/2799358267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76040a3ff0 msgr2=0x7f7604138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f76037fe640 1 --2- 172.21.15.46:0/2799358267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76040a3ff0 0x7f7604138c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f76037fe640 10 monclient: _finish_hunting 0 2024-09-16T04:44:33.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f76037fe640 1 monclient: found mon.noname-a 2024-09-16T04:44:33.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f76037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:33.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f76037fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:44:33.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f76037fe640 1 -- 172.21.15.46:0/2799358267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76041202e0 con 0x7f760413be90 2024-09-16T04:44:33.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f7602ffd640 1 --2- 172.21.15.46:0/2799358267 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f76040a3c10 0x7f7604136b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:44:33.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f76037fe640 1 --2- 172.21.15.46:0/2799358267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f760413be90 0x7f76041391a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f75fc00cbe0 tx=0x7f75fc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:33.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f75fb7fe640 1 -- 172.21.15.46:0/2799358267 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75fc017070 con 0x7f760413be90 2024-09-16T04:44:33.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f75fb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f75fb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:44:33.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f75fb7fe640 10 monclient: dump: 2024-09-16T04:44:33.689 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.689 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.689 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.689 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.689 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.689 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.689 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.690 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.690 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.690 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f75fb7fe640 10 monclient: _finish_auth 0 2024-09-16T04:44:33.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f75fb7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:44:33.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f75fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:33.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f75fb7fe640 1 -- 172.21.15.46:0/2799358267 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75fc00ad40 con 0x7f760413be90 2024-09-16T04:44:33.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 5 monclient: authenticate success, global_id 74133 2024-09-16T04:44:33.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 10 monclient: _renew_subs 2024-09-16T04:44:33.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:33.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 1 -- 172.21.15.46:0/2799358267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76041205c0 con 0x7f760413be90 2024-09-16T04:44:33.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f75fb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:33.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f75fb7fe640 1 -- 172.21.15.46:0/2799358267 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75fc012640 con 0x7f760413be90 2024-09-16T04:44:33.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f75fb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:33.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f75fb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:33.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f75fb7fe640 10 monclient: dump: 2024-09-16T04:44:33.691 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:33.691 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:33.691 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.691 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:33.691 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:33.691 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:33.691 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:33.692 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:33.692 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:33.692 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:33.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 10 monclient: _renew_subs 2024-09-16T04:44:33.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f7601ffb640 4 set_mon_vals no callback set 2024-09-16T04:44:33.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:33.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.668+0000 7f760961e640 1 -- 172.21.15.46:0/2799358267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f760409d330 con 0x7f760413be90 2024-09-16T04:44:33.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.669+0000 7f75fb7fe640 1 -- 172.21.15.46:0/2799358267 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f75fc01a070 con 0x7f760413be90 2024-09-16T04:44:33.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.669+0000 7f760961e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:33.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.669+0000 7f760961e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:33.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.669+0000 7f760961e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:33.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.669+0000 7f760961e640 1 -- 172.21.15.46:0/2799358267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76040a4400 con 0x7f760413be90 2024-09-16T04:44:33.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.670+0000 7f75fb7fe640 1 --2- 172.21.15.46:0/2799358267 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f75e40383b0 0x7f75e403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:33.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.670+0000 7f75fb7fe640 1 -- 172.21.15.46:0/2799358267 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(224..224 src has 1..224) ==== 4346+0+0 (secure 0 0 0) 0x7f75fc04b520 con 0x7f760413be90 2024-09-16T04:44:33.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.670+0000 7f7602ffd640 1 --2- 172.21.15.46:0/2799358267 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f75e40383b0 0x7f75e403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.670+0000 7f7602ffd640 1 -- 172.21.15.46:0/2799358267 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f75e40383b0 msgr2=0x7f75e403a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:44:33.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.670+0000 7f7602ffd640 1 -- 172.21.15.46:0/2799358267 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f75e40383b0 msgr2=0x7f75e403a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:44:33.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.670+0000 7f7602ffd640 1 --2- 172.21.15.46:0/2799358267 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f75e40383b0 0x7f75e403a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:33.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.670+0000 7f7602ffd640 1 --2- 172.21.15.46:0/2799358267 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f75e40383b0 0x7f75e403a870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:33.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.674+0000 7f75fb7fe640 1 -- 172.21.15.46:0/2799358267 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f75fc007a50 con 0x7f760413be90 2024-09-16T04:44:33.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.674+0000 7f75fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:33.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.674+0000 7f75fb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:44:33.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.871+0000 7f7602ffd640 1 --2- 172.21.15.46:0/2799358267 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f75e40383b0 0x7f75e403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:33.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.871+0000 7f7602ffd640 10 monclient: get_auth_request con 0x7f75e40383b0 auth_method 0 2024-09-16T04:44:33.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.871+0000 7f7602ffd640 1 --2- 172.21.15.46:0/2799358267 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f75e40383b0 0x7f75e403a870 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f75f40011e0 tx=0x7f75f4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:33.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.919+0000 7f760961e640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:44:33.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.919+0000 7f760961e640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:44:33.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.919+0000 7f760961e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:33.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.919+0000 7f760961e640 1 -- 172.21.15.46:0/2799358267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f7604069150 con 0x7f760413be90 2024-09-16T04:44:33.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.920+0000 7f75fb7fe640 1 -- 172.21.15.46:0/2799358267 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f75fc04b1e0 con 0x7f760413be90 2024-09-16T04:44:33.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.920+0000 7f75fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:44:33.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.920+0000 7f75fb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:44:33.923 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.920+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.923 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.920+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=11,kv=10,logm=64} crc {auth=2095741464,config=3541143443,health=1596738747,kv=2534035223,logm=20302848}) 2024-09-16T04:44:33.923 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.920+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=1,health=10,kv=12,logm=63} crc {auth=576579414,config=1228700967,health=633749879,kv=4017288519,logm=1121244381}) 2024-09-16T04:44:33.923 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.920+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.923 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.920+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=11,kv=10,logm=64} crc {auth=2095741464,config=3541143443,health=1596738747,kv=2534035223,logm=20302848}) 2024-09-16T04:44:33.923 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.920+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=11,kv=11,logm=64} crc {auth=2959794092,config=3541143443,health=3614319468,kv=1889325247,logm=2831609069}) 2024-09-16T04:44:33.924 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.921+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.924 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.921+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1625705998}) 2024-09-16T04:44:33.924 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.921+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=388227562}) 2024-09-16T04:44:33.924 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.921+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.924 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.921+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1625705998}) 2024-09-16T04:44:33.925 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.921+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2412788103}) 2024-09-16T04:44:33.925 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.922+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.925 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.922+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1913918160}) 2024-09-16T04:44:33.925 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.922+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2969599018}) 2024-09-16T04:44:33.925 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.922+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.925 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.922+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1913918160}) 2024-09-16T04:44:33.925 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.922+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1758816686}) 2024-09-16T04:44:33.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.922+0000 7f760961e640 1 -- 172.21.15.46:0/2799358267 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f75e40383b0 msgr2=0x7f75e403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.922+0000 7f760961e640 1 --2- 172.21.15.46:0/2799358267 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f75e40383b0 0x7f75e403a870 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f75f40011e0 tx=0x7f75f4009040 comp rx=0 tx=0).stop 2024-09-16T04:44:33.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.922+0000 7f760961e640 10 monclient: shutdown 2024-09-16T04:44:33.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.922+0000 7f760961e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:33.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.922+0000 7f760961e640 1 -- 172.21.15.46:0/2799358267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f760413be90 msgr2=0x7f76041391a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:33.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.922+0000 7f760961e640 1 --2- 172.21.15.46:0/2799358267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f760413be90 0x7f76041391a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f75fc00cbe0 tx=0x7f75fc00a430 comp rx=0 tx=0).stop 2024-09-16T04:44:33.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.922+0000 7f760961e640 1 -- 172.21.15.46:0/2799358267 shutdown_connections 2024-09-16T04:44:33.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.922+0000 7f760961e640 1 --2- 172.21.15.46:0/2799358267 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f75e40383b0 0x7f75e403a870 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.922+0000 7f760961e640 1 --2- 172.21.15.46:0/2799358267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f760413be90 0x7f76041391a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.922+0000 7f760961e640 1 --2- 172.21.15.46:0/2799358267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76040a3ff0 0x7f7604138c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.922+0000 7f760961e640 1 --2- 172.21.15.46:0/2799358267 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f76040a3c10 0x7f7604136b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:33.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.922+0000 7f760961e640 1 -- 172.21.15.46:0/2799358267 >> 172.21.15.46:0/2799358267 conn(0x7f76040a5d10 msgr2=0x7f760409f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:33.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.922+0000 7f760961e640 1 -- 172.21.15.46:0/2799358267 shutdown_connections 2024-09-16T04:44:33.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:33.922+0000 7f760961e640 1 -- 172.21.15.46:0/2799358267 wait complete. 2024-09-16T04:44:33.930 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.923+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.930 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.923+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445456932}) 2024-09-16T04:44:33.930 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.923+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=808981860}) 2024-09-16T04:44:33.930 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.923+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.930 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.923+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445456932}) 2024-09-16T04:44:33.930 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.923+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2406819971}) 2024-09-16T04:44:33.930 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.923+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.930 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.923+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217491064}) 2024-09-16T04:44:33.930 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.923+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3276968148}) 2024-09-16T04:44:33.930 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.923+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.930 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.924+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217491064}) 2024-09-16T04:44:33.930 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.924+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2320605385}) 2024-09-16T04:44:33.930 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.924+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.931 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.924+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2707354825}) 2024-09-16T04:44:33.931 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.924+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2585030469}) 2024-09-16T04:44:33.931 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.924+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.931 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.924+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2707354825}) 2024-09-16T04:44:33.931 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.924+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=52950756}) 2024-09-16T04:44:33.931 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.925+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.931 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.925+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1081339506}) 2024-09-16T04:44:33.931 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.925+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3894949258}) 2024-09-16T04:44:33.931 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.925+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.931 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.925+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1081339506}) 2024-09-16T04:44:33.931 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.925+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=148771010}) 2024-09-16T04:44:33.931 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.926+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.931 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.926+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660749485}) 2024-09-16T04:44:33.932 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.926+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4115051148}) 2024-09-16T04:44:33.932 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.926+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.932 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.926+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660749485}) 2024-09-16T04:44:33.932 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.926+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4168194517}) 2024-09-16T04:44:33.932 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.928+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.932 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.928+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1871561291}) 2024-09-16T04:44:33.932 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.928+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=79236117}) 2024-09-16T04:44:33.932 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.928+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.932 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.928+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1871561291}) 2024-09-16T04:44:33.932 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.928+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=327926165}) 2024-09-16T04:44:33.934 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.929+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.934 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.929+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=782360386}) 2024-09-16T04:44:33.934 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.929+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1070214775}) 2024-09-16T04:44:33.934 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.929+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.934 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.929+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=782360386}) 2024-09-16T04:44:33.934 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.929+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2226601942}) 2024-09-16T04:44:33.934 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.930+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.934 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.930+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=852679315}) 2024-09-16T04:44:33.934 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.930+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=201233231}) 2024-09-16T04:44:33.934 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.930+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.935 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.930+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=852679315}) 2024-09-16T04:44:33.935 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.930+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=474691819}) 2024-09-16T04:44:33.935 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.931+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.935 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.931+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3277953673}) 2024-09-16T04:44:33.935 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.931+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=554971554}) 2024-09-16T04:44:33.935 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.931+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.935 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.931+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3277953673}) 2024-09-16T04:44:33.935 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.931+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2367440139}) 2024-09-16T04:44:33.935 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.932+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.935 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.932+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1687831528}) 2024-09-16T04:44:33.935 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.932+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3680345905}) 2024-09-16T04:44:33.935 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.932+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.935 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.932+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1687831528}) 2024-09-16T04:44:33.935 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.932+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1403723749}) 2024-09-16T04:44:33.936 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.933+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.936 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.933+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1755456937}) 2024-09-16T04:44:33.936 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.933+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1973773366}) 2024-09-16T04:44:33.936 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.933+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.937 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.933+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1755456937}) 2024-09-16T04:44:33.937 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.933+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2397254477}) 2024-09-16T04:44:33.937 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:44:33.939 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.934+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.939 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.934+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1906678027}) 2024-09-16T04:44:33.939 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.934+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1189267633}) 2024-09-16T04:44:33.939 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.934+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.939 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.934+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1906678027}) 2024-09-16T04:44:33.940 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.934+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2061460527}) 2024-09-16T04:44:33.940 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.935+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.940 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.935+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426164952}) 2024-09-16T04:44:33.940 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.935+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2322856948}) 2024-09-16T04:44:33.940 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.935+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.941 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.935+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426164952}) 2024-09-16T04:44:33.941 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.935+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=372293380}) 2024-09-16T04:44:33.941 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.936+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.941 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.936+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=13} crc {logm=3214552001,mdsmap=3956149183,mgr=1081907435,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=3243720963,mon_config_key=373967839}) 2024-09-16T04:44:33.941 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.936+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=13} crc {logm=1465942426,mdsmap=3956149183,mgr=1081907435,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=3243720963,mon_config_key=1708457055}) 2024-09-16T04:44:33.942 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.936+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.942 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.936+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=13} crc {logm=3214552001,mdsmap=3956149183,mgr=1081907435,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=3243720963,mon_config_key=373967839}) 2024-09-16T04:44:33.942 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.936+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=61,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=14,monmap=3,osd_metadata=1} crc {logm=2957384377,mdsmap=3956149183,mgr=2343726890,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=2376787405,mon_config_key=1385660123,monmap=3088247691,osd_metadata=576798153}) 2024-09-16T04:44:33.944 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.937+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.944 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.937+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=1,monmap=3,osd_metadata=6,osd_snap=2,osdmap=88} crc {mon_config_key=329078487,monmap=3088247691,osd_metadata=1999947163,osd_snap=723727393,osdmap=4282991580}) 2024-09-16T04:44:33.945 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.937+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=1,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=88} crc {mon_config_key=329078487,monmap=1203950418,osd_metadata=777518657,osd_pg_creating=1278323181,osd_snap=723727393,osdmap=1307860282}) 2024-09-16T04:44:33.945 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.937+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.945 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.937+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=1,monmap=3,osd_metadata=6,osd_snap=2,osdmap=88} crc {mon_config_key=329078487,monmap=3088247691,osd_metadata=1999947163,osd_snap=723727393,osdmap=4282991580}) 2024-09-16T04:44:33.945 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.937+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=1,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=87} crc {mon_config_key=329078487,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=1278323181,osd_snap=723727393,osdmap=2960764032}) 2024-09-16T04:44:33.945 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.938+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.945 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.938+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3517694796}) 2024-09-16T04:44:33.946 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.938+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1850638306}) 2024-09-16T04:44:33.946 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.938+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.946 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.938+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3517694796}) 2024-09-16T04:44:33.946 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.938+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2013033440}) 2024-09-16T04:44:33.946 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.940+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.946 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.940+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2570531720}) 2024-09-16T04:44:33.947 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.940+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2478069097}) 2024-09-16T04:44:33.947 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.940+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.947 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.940+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2570531720}) 2024-09-16T04:44:33.947 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.940+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1436108690}) 2024-09-16T04:44:33.949 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.947+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.949 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.947+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=52} crc {osdmap=301747202}) 2024-09-16T04:44:33.949 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.947+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=47} crc {osdmap=2610339521}) 2024-09-16T04:44:33.949 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.947+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:44:33.949 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.947+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=52} crc {osdmap=301747202}) 2024-09-16T04:44:33.950 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:44:33.947+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=48} crc {osdmap=3568699385}) 2024-09-16T04:44:34.938 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:44:34.939 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:44:35.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.046+0000 7f9d0eaaf640 10 monclient: get_monmap_and_config 2024-09-16T04:44:35.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.046+0000 7f9d0eaaf640 10 monclient: build_initial_monmap 2024-09-16T04:44:35.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.046+0000 7f9d0eaaf640 10 monclient: monmap: 2024-09-16T04:44:35.048 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:35.048 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.048 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:44:35.047655+0000 2024-09-16T04:44:35.048 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:44:35.047655+0000 2024-09-16T04:44:35.048 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:35.048 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.049 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:44:35.049 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:44:35.049 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:44:35.049 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:44:35.049 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:44:35.049 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.046+0000 7f9d0eaaf640 1 Processor -- start 2024-09-16T04:44:35.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0eaaf640 1 -- start start 2024-09-16T04:44:35.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0eaaf640 10 monclient: init 2024-09-16T04:44:35.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0eaaf640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:35.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0eaaf640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-16T04:44:35.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0eaaf640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9d080a3c10 0x7f9d0809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0eaaf640 10 monclient(hunting): picked mon.noname-a con 0x7f9d080a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:44:35.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0eaaf640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9d080a3ff0 0x7f9d080a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0eaaf640 10 monclient(hunting): picked mon.noname-b con 0x7f9d080a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:35.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0eaaf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d081369d0 addr v1:172.21.15.114:6789/0 2024-09-16T04:44:35.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0eaaf640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d0809ab70 con 0x7f9d081369d0 2024-09-16T04:44:35.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0eaaf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0eaaf640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f9d080a4d60 con 0x7f9d080a3c10 2024-09-16T04:44:35.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0eaaf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0eaaf640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f9d0809b030 con 0x7f9d080a3ff0 2024-09-16T04:44:35.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0eaaf640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:35.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0eaaf640 10 monclient(hunting): authenticate will time out at 1073.463642s 2024-09-16T04:44:35.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d07fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9d080a3ff0 0x7f9d080a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0c84b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9d080a3c10 0x7f9d0809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d07fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9d080a3ff0 0x7f9d080a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:60860/0 (socket says 172.21.15.46:60860) 2024-09-16T04:44:35.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d07fff640 1 -- 172.21.15.46:0/1100802378 learned_addr learned my addr 172.21.15.46:0/1100802378 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:35.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d07fff640 10 monclient(hunting): get_auth_request con 0x7f9d080a3ff0 auth_method 0 2024-09-16T04:44:35.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d07fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0c84b640 1 --2- 172.21.15.46:0/1100802378 >> v2:172.21.15.46:3300/0 conn(0x7f9d080a3c10 0x7f9d0809bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43204/0 (socket says 172.21.15.46:43204) 2024-09-16T04:44:35.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0c84b640 10 monclient(hunting): get_auth_request con 0x7f9d080a3c10 auth_method 0 2024-09-16T04:44:35.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0c84b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0c84b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0c84b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0c84b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0c84b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.047+0000 7f9d0c84b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.048+0000 7f9d07fff640 10 monclient(hunting): handle_auth_done global_id 64316 payload 274 2024-09-16T04:44:35.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.048+0000 7f9d07fff640 1 -- 172.21.15.46:0/1100802378 >> v2:172.21.15.46:3300/0 conn(0x7f9d080a3c10 msgr2=0x7f9d0809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:44:35.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.048+0000 7f9d07fff640 1 --2- 172.21.15.46:0/1100802378 >> v2:172.21.15.46:3300/0 conn(0x7f9d080a3c10 0x7f9d0809bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.048+0000 7f9d07fff640 1 -- 172.21.15.46:0/1100802378 >> v1:172.21.15.114:6789/0 conn(0x7f9d081369d0 legacy=0x7f9d08136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:44:35.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.048+0000 7f9d07fff640 10 monclient: _finish_hunting 0 2024-09-16T04:44:35.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.048+0000 7f9d07fff640 1 monclient: found mon.noname-b 2024-09-16T04:44:35.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.048+0000 7f9d07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:35.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.048+0000 7f9d07fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:44:35.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.048+0000 7f9d07fff640 1 -- 172.21.15.46:0/1100802378 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d0809b310 con 0x7f9d080a3ff0 2024-09-16T04:44:35.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.048+0000 7f9d07fff640 1 --2- 172.21.15.46:0/1100802378 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9d080a3ff0 0x7f9d080a43d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4009870 tx=0x7f9cf402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82b5b8d8c62296f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:35.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.048+0000 7f9d077fe640 1 -- 172.21.15.46:0/1100802378 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4037070 con 0x7f9d080a3ff0 2024-09-16T04:44:35.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.048+0000 7f9d077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.048+0000 7f9d077fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:44:35.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.048+0000 7f9d077fe640 10 monclient: dump: 2024-09-16T04:44:35.060 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.060 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.060 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.060 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.061 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.061 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.061 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.061 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.061 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.061 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.048+0000 7f9d077fe640 10 monclient: _finish_auth 0 2024-09-16T04:44:35.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.048+0000 7f9d077fe640 10 monclient: _check_auth_tickets 2024-09-16T04:44:35.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.048+0000 7f9d077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:35.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.049+0000 7f9d0eaaf640 5 monclient: authenticate success, global_id 64316 2024-09-16T04:44:35.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.049+0000 7f9d0eaaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:44:35.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.048+0000 7f9d077fe640 1 -- 172.21.15.46:0/1100802378 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cf4032440 con 0x7f9d080a3ff0 2024-09-16T04:44:35.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.049+0000 7f9d077fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:35.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.049+0000 7f9d0eaaf640 10 monclient: get_monmap_and_config success 2024-09-16T04:44:35.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.049+0000 7f9d0eaaf640 4 set_mon_vals no callback set 2024-09-16T04:44:35.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.049+0000 7f9d077fe640 1 -- 172.21.15.46:0/1100802378 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4032810 con 0x7f9d080a3ff0 2024-09-16T04:44:35.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.049+0000 7f9d077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.049+0000 7f9d077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:44:35.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.049+0000 7f9d077fe640 10 monclient: dump: 2024-09-16T04:44:35.062 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.062 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.062 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.062 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.062 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.062 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.062 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.062 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.063 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.063 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.049+0000 7f9d0eaaf640 10 monclient: shutdown 2024-09-16T04:44:35.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.049+0000 7f9d0eaaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:35.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.049+0000 7f9d0eaaf640 1 -- 172.21.15.46:0/1100802378 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9d080a3ff0 msgr2=0x7f9d080a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.049+0000 7f9d0eaaf640 1 --2- 172.21.15.46:0/1100802378 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9d080a3ff0 0x7f9d080a43d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4009870 tx=0x7f9cf402a180 comp rx=0 tx=0).stop 2024-09-16T04:44:35.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.049+0000 7f9d0eaaf640 1 -- 172.21.15.46:0/1100802378 shutdown_connections 2024-09-16T04:44:35.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.049+0000 7f9d0eaaf640 1 --2- 172.21.15.46:0/1100802378 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9d080a3ff0 0x7f9d080a43d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.049+0000 7f9d0eaaf640 1 --2- 172.21.15.46:0/1100802378 >> v2:172.21.15.46:3300/0 conn(0x7f9d080a3c10 0x7f9d0809bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.049+0000 7f9d0eaaf640 1 -- 172.21.15.46:0/1100802378 >> 172.21.15.46:0/1100802378 conn(0x7f9d080a5d10 msgr2=0x7f9d080a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:35.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.049+0000 7f9d0eaaf640 1 -- 172.21.15.46:0/1100802378 shutdown_connections 2024-09-16T04:44:35.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.049+0000 7f9d0eaaf640 1 -- 172.21.15.46:0/1100802378 wait complete. 2024-09-16T04:44:35.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.050+0000 7f9d0eaaf640 10 monclient: build_initial_monmap 2024-09-16T04:44:35.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.050+0000 7f9d0eaaf640 10 monclient: monmap: 2024-09-16T04:44:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:44:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:44:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:44:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:44:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:44:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:44:35.064 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.050+0000 7f9d0eaaf640 1 Processor -- start 2024-09-16T04:44:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.050+0000 7f9d0eaaf640 1 -- start start 2024-09-16T04:44:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.050+0000 7f9d0eaaf640 10 monclient: init 2024-09-16T04:44:35.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0eaaf640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:35.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0eaaf640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:44:35.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0eaaf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d080a3c10 0x7f9d081cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0eaaf640 10 monclient(hunting): picked mon.noname-b con 0x7f9d080a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:44:35.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0eaaf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d080a3ff0 0x7f9d081234c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0eaaf640 10 monclient(hunting): picked mon.noname-a con 0x7f9d080a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:44:35.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0eaaf640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9d081369d0 0x7f9d08123a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0eaaf640 10 monclient(hunting): picked mon.noname-c con 0x7f9d081369d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:35.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0eaaf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0eaaf640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9d080a4910 con 0x7f9d080a3ff0 2024-09-16T04:44:35.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0eaaf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0eaaf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0806e3d0 con 0x7f9d080a3c10 2024-09-16T04:44:35.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0eaaf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0eaaf640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f9d08127e00 con 0x7f9d081369d0 2024-09-16T04:44:35.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0eaaf640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:35.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0eaaf640 10 monclient(hunting): authenticate will time out at 1073.467210s 2024-09-16T04:44:35.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0d04c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9d081369d0 0x7f9d08123a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d07fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d080a3ff0 0x7f9d081234c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0d04c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9d081369d0 0x7f9d08123a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:60876/0 (socket says 172.21.15.46:60876) 2024-09-16T04:44:35.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0d04c640 1 -- 172.21.15.46:0/2127164403 learned_addr learned my addr 172.21.15.46:0/2127164403 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:35.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0d04c640 10 monclient(hunting): get_auth_request con 0x7f9d081369d0 auth_method 0 2024-09-16T04:44:35.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0d04c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0d04c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0d04c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d07fff640 10 monclient(hunting): get_auth_request con 0x7f9d080a3ff0 auth_method 0 2024-09-16T04:44:35.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d07fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0d04c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0d04c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0d04c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0c84b640 1 --2- 172.21.15.46:0/2127164403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d080a3c10 0x7f9d081cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0c84b640 10 monclient(hunting): get_auth_request con 0x7f9d080a3c10 auth_method 0 2024-09-16T04:44:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0c84b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0c84b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0c84b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0d04c640 10 monclient(hunting): handle_auth_done global_id 64319 payload 824 2024-09-16T04:44:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0d04c640 1 -- 172.21.15.46:0/2127164403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d080a3c10 msgr2=0x7f9d081cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0d04c640 1 --2- 172.21.15.46:0/2127164403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d080a3c10 0x7f9d081cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0d04c640 1 -- 172.21.15.46:0/2127164403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d080a3ff0 msgr2=0x7f9d081234c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0d04c640 1 --2- 172.21.15.46:0/2127164403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d080a3ff0 0x7f9d081234c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0d04c640 10 monclient: _finish_hunting 0 2024-09-16T04:44:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0d04c640 1 monclient: found mon.noname-c 2024-09-16T04:44:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0d04c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0d04c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:44:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0d04c640 1 -- 172.21.15.46:0/2127164403 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d081255b0 con 0x7f9d081369d0 2024-09-16T04:44:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d07fff640 1 --2- 172.21.15.46:0/2127164403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d080a3ff0 0x7f9d081234c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:44:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.051+0000 7f9d0d04c640 1 --2- 172.21.15.46:0/2127164403 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9d081369d0 0x7f9d08123a00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9d0000c780 tx=0x7f9d0000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d057fa640 1 -- 172.21.15.46:0/2127164403 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d00004b10 con 0x7f9d081369d0 2024-09-16T04:44:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d057fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:44:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d057fa640 10 monclient: dump: 2024-09-16T04:44:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.070 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.070 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.070 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d057fa640 10 monclient: _finish_auth 0 2024-09-16T04:44:35.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d057fa640 10 monclient: _check_auth_tickets 2024-09-16T04:44:35.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:35.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d057fa640 1 -- 172.21.15.46:0/2127164403 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d00004cb0 con 0x7f9d081369d0 2024-09-16T04:44:35.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d0eaaf640 5 monclient: authenticate success, global_id 64319 2024-09-16T04:44:35.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d0eaaf640 10 monclient: _renew_subs 2024-09-16T04:44:35.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d0eaaf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:44:35.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d0eaaf640 1 -- 172.21.15.46:0/2127164403 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d080a8a70 con 0x7f9d081369d0 2024-09-16T04:44:35.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d057fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:35.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d057fa640 1 -- 172.21.15.46:0/2127164403 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0000aa90 con 0x7f9d081369d0 2024-09-16T04:44:35.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d057fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:44:35.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d057fa640 10 monclient: dump: 2024-09-16T04:44:35.071 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.071 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.071 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.071 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.071 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.071 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.071 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.072 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.072 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.072 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d0eaaf640 10 monclient: _renew_subs 2024-09-16T04:44:35.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d0eaaf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:44:35.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d06ffd640 4 set_mon_vals no callback set 2024-09-16T04:44:35.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d0eaaf640 1 -- 172.21.15.46:0/2127164403 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d08126e40 con 0x7f9d081369d0 2024-09-16T04:44:35.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.052+0000 7f9d057fa640 1 -- 172.21.15.46:0/2127164403 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f9d0000ac30 con 0x7f9d081369d0 2024-09-16T04:44:35.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.053+0000 7f9d057fa640 1 --2- 172.21.15.46:0/2127164403 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9cec0383b0 0x7f9cec03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.053+0000 7f9d057fa640 1 -- 172.21.15.46:0/2127164403 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(225..225 src has 1..225) ==== 4738+0+0 (secure 0 0 0) 0x7f9d0004bb60 con 0x7f9d081369d0 2024-09-16T04:44:35.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.053+0000 7f9d0eaaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:35.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.053+0000 7f9d0eaaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:35.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.053+0000 7f9d0eaaf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:44:35.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.053+0000 7f9d0eaaf640 1 -- 172.21.15.46:0/2127164403 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d08137580 con 0x7f9d081369d0 2024-09-16T04:44:35.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.053+0000 7f9d0c84b640 1 --2- 172.21.15.46:0/2127164403 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9cec0383b0 0x7f9cec03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.053+0000 7f9d0c84b640 10 monclient: get_auth_request con 0x7f9cec0383b0 auth_method 0 2024-09-16T04:44:35.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.053+0000 7f9d0c84b640 1 --2- 172.21.15.46:0/2127164403 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9cec0383b0 0x7f9cec03a870 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc009ad0 tx=0x7f9cfc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:35.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.058+0000 7f9d057fa640 1 -- 172.21.15.46:0/2127164403 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9d00011780 con 0x7f9d081369d0 2024-09-16T04:44:35.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.058+0000 7f9d057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:35.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.058+0000 7f9d057fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:44:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.302+0000 7f9d0eaaf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:44:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.302+0000 7f9d0eaaf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:44:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.302+0000 7f9d0eaaf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:44:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.302+0000 7f9d0eaaf640 1 -- 172.21.15.46:0/2127164403 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9d08069150 con 0x7f9d081369d0 2024-09-16T04:44:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.303+0000 7f9d057fa640 1 -- 172.21.15.46:0/2127164403 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f9d00012420 con 0x7f9d081369d0 2024-09-16T04:44:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.303+0000 7f9d057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:44:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.303+0000 7f9d057fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:44:35.306 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:44:35.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.305+0000 7f9d0eaaf640 1 -- 172.21.15.46:0/2127164403 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9cec0383b0 msgr2=0x7f9cec03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.305+0000 7f9d0eaaf640 1 --2- 172.21.15.46:0/2127164403 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9cec0383b0 0x7f9cec03a870 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc009ad0 tx=0x7f9cfc008040 comp rx=0 tx=0).stop 2024-09-16T04:44:35.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.305+0000 7f9d0eaaf640 10 monclient: shutdown 2024-09-16T04:44:35.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.305+0000 7f9d0eaaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:35.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.305+0000 7f9d0eaaf640 1 -- 172.21.15.46:0/2127164403 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9d081369d0 msgr2=0x7f9d08123a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.305+0000 7f9d0eaaf640 1 --2- 172.21.15.46:0/2127164403 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9d081369d0 0x7f9d08123a00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9d0000c780 tx=0x7f9d0000cc40 comp rx=0 tx=0).stop 2024-09-16T04:44:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.305+0000 7f9d0eaaf640 1 -- 172.21.15.46:0/2127164403 shutdown_connections 2024-09-16T04:44:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.305+0000 7f9d0eaaf640 1 --2- 172.21.15.46:0/2127164403 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9d081369d0 0x7f9d08123a00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.305+0000 7f9d0eaaf640 1 --2- 172.21.15.46:0/2127164403 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9cec0383b0 0x7f9cec03a870 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.305+0000 7f9d0eaaf640 1 --2- 172.21.15.46:0/2127164403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d080a3ff0 0x7f9d081234c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.305+0000 7f9d0eaaf640 1 --2- 172.21.15.46:0/2127164403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d080a3c10 0x7f9d081cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.305+0000 7f9d0eaaf640 1 -- 172.21.15.46:0/2127164403 >> 172.21.15.46:0/2127164403 conn(0x7f9d080a5d10 msgr2=0x7f9d0809e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.305+0000 7f9d0eaaf640 1 -- 172.21.15.46:0/2127164403 shutdown_connections 2024-09-16T04:44:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.305+0000 7f9d0eaaf640 1 -- 172.21.15.46:0/2127164403 wait complete. 2024-09-16T04:44:35.317 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:44:35.317 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:44:35.318 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:44:35.318 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:44:35.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.417+0000 7fd7c5533640 10 monclient: get_monmap_and_config 2024-09-16T04:44:35.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.417+0000 7fd7c5533640 10 monclient: build_initial_monmap 2024-09-16T04:44:35.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.417+0000 7fd7c5533640 10 monclient: monmap: 2024-09-16T04:44:35.420 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:35.420 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.420 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:44:35.419267+0000 2024-09-16T04:44:35.420 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:44:35.419267+0000 2024-09-16T04:44:35.421 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:35.421 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.421 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:44:35.421 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:44:35.421 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:44:35.421 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:44:35.421 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:44:35.421 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.418+0000 7fd7c5533640 1 Processor -- start 2024-09-16T04:44:35.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.418+0000 7fd7c5533640 1 -- start start 2024-09-16T04:44:35.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.418+0000 7fd7c5533640 10 monclient: init 2024-09-16T04:44:35.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.418+0000 7fd7c5533640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:35.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.418+0000 7fd7c5533640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-16T04:44:35.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.418+0000 7fd7c5533640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd7c00a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:44:35.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.418+0000 7fd7c5533640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd7c00a3ff0 0x7fd7c00a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.418+0000 7fd7c5533640 10 monclient(hunting): picked mon.noname-a con 0x7fd7c00a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:44:35.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7c5533640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c013be90 0x7fd7c013e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7c5533640 10 monclient(hunting): picked mon.noname-b con 0x7fd7c013be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:35.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7c5533640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7c009b940 con 0x7fd7c00a3c10 2024-09-16T04:44:35.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7c5533640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7c5533640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd7c009ab00 con 0x7fd7c00a3ff0 2024-09-16T04:44:35.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7c5533640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7c5533640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd7c009b030 con 0x7fd7c013be90 2024-09-16T04:44:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7c5533640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7c5533640 10 monclient(hunting): authenticate will time out at 1073.835064s 2024-09-16T04:44:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7be7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd7c00a3ff0 0x7fd7c00a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7bf7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c013be90 0x7fd7c013e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7be7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd7c00a3ff0 0x7fd7c00a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43222/0 (socket says 172.21.15.46:43222) 2024-09-16T04:44:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7be7fc640 1 -- 172.21.15.46:0/1954741792 learned_addr learned my addr 172.21.15.46:0/1954741792 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7bf7fe640 10 monclient(hunting): get_auth_request con 0x7fd7c013be90 auth_method 0 2024-09-16T04:44:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7be7fc640 10 monclient(hunting): get_auth_request con 0x7fd7c00a3ff0 auth_method 0 2024-09-16T04:44:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7be7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7be7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7be7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7be7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7be7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7be7fc640 10 monclient(hunting): handle_auth_done global_id 74151 payload 274 2024-09-16T04:44:35.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7be7fc640 1 -- 172.21.15.46:0/1954741792 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c013be90 msgr2=0x7fd7c013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:44:35.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7be7fc640 1 --2- 172.21.15.46:0/1954741792 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c013be90 0x7fd7c013e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7be7fc640 1 -- 172.21.15.46:0/1954741792 >> v1:172.21.15.114:6789/0 conn(0x7fd7c00a3c10 legacy=0x7fd7c009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7be7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:44:35.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7be7fc640 1 monclient: found mon.noname-a 2024-09-16T04:44:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7be7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:44:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7be7fc640 1 -- 172.21.15.46:0/1954741792 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c009b310 con 0x7fd7c00a3ff0 2024-09-16T04:44:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.419+0000 7fd7bf7fe640 1 --2- 172.21.15.46:0/1954741792 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c013be90 0x7fd7c013e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:44:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7be7fc640 1 --2- 172.21.15.46:0/1954741792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c00a3ff0 0x7fd7c00a43d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4009e50 tx=0x7fd7b402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a97c64aba59cabd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7bdffb640 1 -- 172.21.15.46:0/1954741792 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4036070 con 0x7fd7c00a3ff0 2024-09-16T04:44:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7bdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:44:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7bdffb640 10 monclient: dump: 2024-09-16T04:44:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.435 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.435 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.435 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.435 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.435 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.435 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.435 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.435 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.435 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7bdffb640 10 monclient: _finish_auth 0 2024-09-16T04:44:35.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7bdffb640 10 monclient: _check_auth_tickets 2024-09-16T04:44:35.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:35.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7bdffb640 1 -- 172.21.15.46:0/1954741792 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7b4031600 con 0x7fd7c00a3ff0 2024-09-16T04:44:35.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7bdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:35.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7bdffb640 1 -- 172.21.15.46:0/1954741792 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b40319d0 con 0x7fd7c00a3ff0 2024-09-16T04:44:35.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7c5533640 5 monclient: authenticate success, global_id 74151 2024-09-16T04:44:35.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7c5533640 10 monclient: get_monmap_and_config success 2024-09-16T04:44:35.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7c5533640 4 set_mon_vals no callback set 2024-09-16T04:44:35.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7bdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:35.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7bdffb640 10 monclient: dump: 2024-09-16T04:44:35.436 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.436 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.436 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.436 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.436 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.436 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.437 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7c5533640 10 monclient: shutdown 2024-09-16T04:44:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7c5533640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7c5533640 1 -- 172.21.15.46:0/1954741792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c00a3ff0 msgr2=0x7fd7c00a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7c5533640 1 --2- 172.21.15.46:0/1954741792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c00a3ff0 0x7fd7c00a43d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4009e50 tx=0x7fd7b402a680 comp rx=0 tx=0).stop 2024-09-16T04:44:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7c5533640 1 -- 172.21.15.46:0/1954741792 shutdown_connections 2024-09-16T04:44:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7c5533640 1 --2- 172.21.15.46:0/1954741792 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c013be90 0x7fd7c013e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7c5533640 1 --2- 172.21.15.46:0/1954741792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c00a3ff0 0x7fd7c00a43d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.420+0000 7fd7c5533640 1 -- 172.21.15.46:0/1954741792 >> 172.21.15.46:0/1954741792 conn(0x7fd7c00a6340 msgr2=0x7fd7c006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.421+0000 7fd7c5533640 1 -- 172.21.15.46:0/1954741792 shutdown_connections 2024-09-16T04:44:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.421+0000 7fd7c5533640 1 -- 172.21.15.46:0/1954741792 wait complete. 2024-09-16T04:44:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.421+0000 7fd7c5533640 10 monclient: build_initial_monmap 2024-09-16T04:44:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.421+0000 7fd7c5533640 10 monclient: monmap: 2024-09-16T04:44:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:44:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:44:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:44:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:44:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:44:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:44:35.438 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.421+0000 7fd7c5533640 1 Processor -- start 2024-09-16T04:44:35.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7c5533640 1 -- start start 2024-09-16T04:44:35.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7c5533640 10 monclient: init 2024-09-16T04:44:35.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7c5533640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:35.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7c5533640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:44:35.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7c5533640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c00a3c10 0x7fd7c0133d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7c5533640 10 monclient(hunting): picked mon.noname-c con 0x7fd7c00a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:35.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7c5533640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c00a3ff0 0x7fd7c01342c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7c5533640 10 monclient(hunting): picked mon.noname-a con 0x7fd7c00a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:44:35.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7c5533640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c013be90 0x7fd7c01cb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7c5533640 10 monclient(hunting): picked mon.noname-b con 0x7fd7c013be90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:44:35.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7c5533640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7c5533640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c00a6b40 con 0x7fd7c00a3ff0 2024-09-16T04:44:35.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7c5533640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7c5533640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c00a69c0 con 0x7fd7c013be90 2024-09-16T04:44:35.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7c5533640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7c5533640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd7c013e7c0 con 0x7fd7c00a3c10 2024-09-16T04:44:35.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7c5533640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:35.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7c5533640 10 monclient(hunting): authenticate will time out at 1073.838501s 2024-09-16T04:44:35.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7beffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c00a3c10 0x7fd7c0133d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7be7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c00a3ff0 0x7fd7c01342c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7beffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c00a3c10 0x7fd7c0133d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:60890/0 (socket says 172.21.15.46:60890) 2024-09-16T04:44:35.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7beffd640 1 -- 172.21.15.46:0/1497628517 learned_addr learned my addr 172.21.15.46:0/1497628517 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:35.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7beffd640 10 monclient(hunting): get_auth_request con 0x7fd7c00a3c10 auth_method 0 2024-09-16T04:44:35.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7beffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7be7fc640 1 --2- 172.21.15.46:0/1497628517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c00a3ff0 0x7fd7c01342c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43228/0 (socket says 172.21.15.46:43228) 2024-09-16T04:44:35.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7be7fc640 10 monclient(hunting): get_auth_request con 0x7fd7c00a3ff0 auth_method 0 2024-09-16T04:44:35.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7be7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7be7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7bf7fe640 1 --2- 172.21.15.46:0/1497628517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c013be90 0x7fd7c01cb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7bf7fe640 10 monclient(hunting): get_auth_request con 0x7fd7c013be90 auth_method 0 2024-09-16T04:44:35.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7be7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7be7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7be7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7beffd640 10 monclient(hunting): handle_auth_done global_id 64325 payload 824 2024-09-16T04:44:35.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7beffd640 1 -- 172.21.15.46:0/1497628517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c013be90 msgr2=0x7fd7c01cb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7beffd640 1 --2- 172.21.15.46:0/1497628517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c013be90 0x7fd7c01cb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7beffd640 1 -- 172.21.15.46:0/1497628517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c00a3ff0 msgr2=0x7fd7c01342c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7beffd640 1 --2- 172.21.15.46:0/1497628517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c00a3ff0 0x7fd7c01342c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7beffd640 10 monclient: _finish_hunting 0 2024-09-16T04:44:35.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7beffd640 1 monclient: found mon.noname-c 2024-09-16T04:44:35.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:35.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7beffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:44:35.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7beffd640 1 -- 172.21.15.46:0/1497628517 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c013c680 con 0x7fd7c00a3c10 2024-09-16T04:44:35.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.422+0000 7fd7be7fc640 1 --2- 172.21.15.46:0/1497628517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c00a3ff0 0x7fd7c01342c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:44:35.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7beffd640 1 --2- 172.21.15.46:0/1497628517 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c00a3c10 0x7fd7c0133d80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd7b000c480 tx=0x7fd7b000c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:35.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7a77fe640 1 -- 172.21.15.46:0/1497628517 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b00052d0 con 0x7fd7c00a3c10 2024-09-16T04:44:35.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7a77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:44:35.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7a77fe640 10 monclient: dump: 2024-09-16T04:44:35.443 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.443 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.443 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.443 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.444 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.444 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.444 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.444 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.444 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.444 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7a77fe640 10 monclient: _finish_auth 0 2024-09-16T04:44:35.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7a77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:44:35.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:35.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7a77fe640 1 -- 172.21.15.46:0/1497628517 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7b0005470 con 0x7fd7c00a3c10 2024-09-16T04:44:35.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7c5533640 5 monclient: authenticate success, global_id 64325 2024-09-16T04:44:35.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7c5533640 10 monclient: _renew_subs 2024-09-16T04:44:35.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7c5533640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:44:35.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7c5533640 1 -- 172.21.15.46:0/1497628517 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7c013c270 con 0x7fd7c00a3c10 2024-09-16T04:44:35.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7a77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:35.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7a77fe640 1 -- 172.21.15.46:0/1497628517 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b0012870 con 0x7fd7c00a3c10 2024-09-16T04:44:35.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:44:35.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7a77fe640 10 monclient: dump: 2024-09-16T04:44:35.445 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.445 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.445 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.445 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.445 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.445 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.445 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.446 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.446 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.446 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7c5533640 10 monclient: _renew_subs 2024-09-16T04:44:35.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7c5533640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:44:35.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7c5533640 1 -- 172.21.15.46:0/1497628517 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7c013a150 con 0x7fd7c00a3c10 2024-09-16T04:44:35.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.423+0000 7fd7bd7fa640 4 set_mon_vals no callback set 2024-09-16T04:44:35.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7c5533640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:35.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7c5533640 1 --2- 172.21.15.46:0/1497628517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c013a390 0x7fd7c01cbe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7c5533640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7c5533640 1 -- 172.21.15.46:0/1497628517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c0134800 con 0x7fd7c013a390 2024-09-16T04:44:35.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7a77fe640 1 -- 172.21.15.46:0/1497628517 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fd7b0012a10 con 0x7fd7c00a3c10 2024-09-16T04:44:35.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7be7fc640 1 --2- 172.21.15.46:0/1497628517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c013a390 0x7fd7c01cbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7be7fc640 10 monclient: get_auth_request con 0x7fd7c013a390 auth_method 0 2024-09-16T04:44:35.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7be7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7be7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7be7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7be7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7be7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7a77fe640 1 --2- 172.21.15.46:0/1497628517 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd7ac0383b0 0x7fd7ac03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7be7fc640 10 monclient(hunting): handle_auth_done global_id 74157 payload 824 2024-09-16T04:44:35.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7be7fc640 1 -- 172.21.15.46:0/1497628517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7c00a43d0 con 0x7fd7c013a390 2024-09-16T04:44:35.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7a77fe640 1 -- 172.21.15.46:0/1497628517 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(225..225 src has 1..225) ==== 4738+0+0 (secure 0 0 0) 0x7fd7b004b3b0 con 0x7fd7c00a3c10 2024-09-16T04:44:35.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7be7fc640 1 --2- 172.21.15.46:0/1497628517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c013a390 0x7fd7c01cbe10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0068240 tx=0x7fd7b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:35.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7bf7fe640 1 --2- 172.21.15.46:0/1497628517 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd7ac0383b0 0x7fd7ac03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7bf7fe640 10 monclient: get_auth_request con 0x7fd7ac0383b0 auth_method 0 2024-09-16T04:44:35.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7a77fe640 1 -- 172.21.15.46:0/1497628517 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4036040 con 0x7fd7c013a390 2024-09-16T04:44:35.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:35.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7a77fe640 10 monclient: dump: 2024-09-16T04:44:35.448 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.448 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.448 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.448 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.448 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.448 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.448 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.448 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.449 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.449 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.424+0000 7fd7a77fe640 1 -- 172.21.15.46:0/1497628517 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd7b4037040 con 0x7fd7c013a390 2024-09-16T04:44:35.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.425+0000 7fd7a77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:35.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.425+0000 7fd7a77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:44:35.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.425+0000 7fd7a77fe640 1 -- 172.21.15.46:0/1497628517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c013a390 msgr2=0x7fd7c01cbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.425+0000 7fd7a77fe640 1 --2- 172.21.15.46:0/1497628517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c013a390 0x7fd7c01cbe10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0068240 tx=0x7fd7b4004a40 comp rx=0 tx=0).stop 2024-09-16T04:44:35.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.425+0000 7fd7a77fe640 1 -- 172.21.15.46:0/1497628517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c013a390 msgr2=0x7fd7c01cbe10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:44:35.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.425+0000 7fd7a77fe640 1 --2- 172.21.15.46:0/1497628517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c013a390 0x7fd7c01cbe10 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0068240 tx=0x7fd7b4004a40 comp rx=0 tx=0).stop 2024-09-16T04:44:35.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.425+0000 7fd7bf7fe640 1 --2- 172.21.15.46:0/1497628517 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd7ac0383b0 0x7fd7ac03a870 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0138a90 tx=0x7fd7b8009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:35.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.431+0000 7fd7c5533640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:44:35.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.431+0000 7fd7c5533640 1 --2- 172.21.15.46:0/1497628517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c01cc350 0x7fd7c00a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.431+0000 7fd7c5533640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.431+0000 7fd7c5533640 1 -- 172.21.15.46:0/1497628517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c01cc730 con 0x7fd7c01cc350 2024-09-16T04:44:35.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.431+0000 7fd7be7fc640 1 --2- 172.21.15.46:0/1497628517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c01cc350 0x7fd7c00a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.431+0000 7fd7be7fc640 10 monclient: get_auth_request con 0x7fd7c01cc350 auth_method 0 2024-09-16T04:44:35.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.431+0000 7fd7be7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.431+0000 7fd7be7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.431+0000 7fd7be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.431+0000 7fd7be7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.431+0000 7fd7be7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.431+0000 7fd7be7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.431+0000 7fd7be7fc640 10 monclient(hunting): handle_auth_done global_id 74160 payload 824 2024-09-16T04:44:35.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.431+0000 7fd7be7fc640 1 -- 172.21.15.46:0/1497628517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd7c0138d00 con 0x7fd7c01cc350 2024-09-16T04:44:35.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.431+0000 7fd7be7fc640 1 --2- 172.21.15.46:0/1497628517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c01cc350 0x7fd7c00a05e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd7b402a6c0 tx=0x7fd7b402f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:35.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.431+0000 7fd7a77fe640 1 -- 172.21.15.46:0/1497628517 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b403c820 con 0x7fd7c01cc350 2024-09-16T04:44:35.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.431+0000 7fd7a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.431+0000 7fd7a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:35.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.431+0000 7fd7a77fe640 10 monclient: dump: 2024-09-16T04:44:35.453 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.453 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.453 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.453 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.453 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.453 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.453 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.453 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.453 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.453 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.432+0000 7fd7a77fe640 1 -- 172.21.15.46:0/1497628517 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7fd7b403c800 con 0x7fd7c01cc350 2024-09-16T04:44:35.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.432+0000 7fd7a77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:44:35.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.432+0000 7fd7a77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:44:35.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.432+0000 7fd7a77fe640 1 -- 172.21.15.46:0/1497628517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c01cc350 msgr2=0x7fd7c00a05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.432+0000 7fd7a77fe640 1 --2- 172.21.15.46:0/1497628517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c01cc350 0x7fd7c00a05e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd7b402a6c0 tx=0x7fd7b402f000 comp rx=0 tx=0).stop 2024-09-16T04:44:35.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.432+0000 7fd7a77fe640 1 -- 172.21.15.46:0/1497628517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c01cc350 msgr2=0x7fd7c00a05e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:44:35.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.432+0000 7fd7a77fe640 1 --2- 172.21.15.46:0/1497628517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c01cc350 0x7fd7c00a05e0 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd7b402a6c0 tx=0x7fd7b402f000 comp rx=0 tx=0).stop 2024-09-16T04:44:35.454 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:44:35.454 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:44:35.454 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:44:35.454 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:44:35.454 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 68, 2024-09-16T04:44:35.455 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 15427, 2024-09-16T04:44:35.455 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:44:35.455 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:44:35.455 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:44:35.455 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:44:35.455 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:35.455 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 2, 2024-09-16T04:44:35.455 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:44:35.455 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:44:35.455 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:44:35.455 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:35.455 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:35.455 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:35.455 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:35.456 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:35.456 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:35.456 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:35.456 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:35.456 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:35.456 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:35.456 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:35.456 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:35.456 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:44:35.456 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:44:35.456 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:35.456 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:35.456 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:35.457 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:35.457 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:35.457 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:35.457 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:35.457 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:35.457 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:35.457 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:35.457 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:35.457 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:35.457 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.457 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:44:35.457 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:44:35.457 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:44:35.457 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:44:35.458 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:44:35.458 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:44:35.458 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:44:35.458 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:44:35.458 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:44:35.458 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:44:35.458 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:44:35.458 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:44:35.458 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:44:35.458 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:44:35.458 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:44:35.458 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:44:35.458 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:44:35.459 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:35.459 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:35.459 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:35.459 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:35.459 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:35.459 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:35.459 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:35.459 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:35.459 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:35.459 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:35.459 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:35.459 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:35.459 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:44:35.459 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.460 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:44:35.460 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.460 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:44:35.460 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:44:35.460 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:35.460 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:35.460 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.460 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:35.460 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:44:35.460 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:35.460 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.460 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.460 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:35.461 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:44:35.461 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:35.461 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:35.461 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:35.461 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.461 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:44:35.461 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:44:35.461 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:35.461 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:35.461 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:35.461 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.461 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.461 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:44:35.462 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:44:35.462 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:35.462 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:35.462 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.462 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:35.462 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:44:35.462 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:35.462 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.462 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.462 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:35.462 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:44:35.462 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:35.462 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:35.462 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:35.463 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.463 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:44:35.463 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:44:35.463 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:35.463 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:35.463 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:35.463 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.463 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.463 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:44:35.463 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:44:35.463 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:35.463 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:35.463 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.463 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:35.464 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:44:35.464 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:35.464 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.464 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.464 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:35.464 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:44:35.464 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:35.464 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:35.464 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:35.464 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.464 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:44:35.464 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:44:35.464 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:35.465 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:35.465 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:35.465 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:35.465 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:35.465 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.465 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:44:35.465 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:44:35.465 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.465 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:35.465 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:35.465 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:44:35.465 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:35.465 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:35.465 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:44:35.466 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.466 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:35.466 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:35.466 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:44:35.466 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:35.466 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:35.466 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.466 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:44:35.466 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:44:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.432+0000 7fd7c5533640 1 -- 172.21.15.46:0/1497628517 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd7ac0383b0 msgr2=0x7fd7ac03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.432+0000 7fd7c5533640 1 --2- 172.21.15.46:0/1497628517 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd7ac0383b0 0x7fd7ac03a870 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0138a90 tx=0x7fd7b8009190 comp rx=0 tx=0).stop 2024-09-16T04:44:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.432+0000 7fd7c5533640 10 monclient: shutdown 2024-09-16T04:44:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.432+0000 7fd7c5533640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.432+0000 7fd7c5533640 1 -- 172.21.15.46:0/1497628517 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c00a3c10 msgr2=0x7fd7c0133d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.432+0000 7fd7c5533640 1 --2- 172.21.15.46:0/1497628517 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c00a3c10 0x7fd7c0133d80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd7b000c480 tx=0x7fd7b000c940 comp rx=0 tx=0).stop 2024-09-16T04:44:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.432+0000 7fd7bf7fe640 1 -- 172.21.15.46:0/1497628517 reap_dead start 2024-09-16T04:44:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.432+0000 7fd7bf7fe640 1 -- 172.21.15.46:0/1497628517 reap_dead start 2024-09-16T04:44:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.432+0000 7fd7c5533640 1 -- 172.21.15.46:0/1497628517 shutdown_connections 2024-09-16T04:44:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.432+0000 7fd7c5533640 1 -- 172.21.15.46:0/1497628517 >> 172.21.15.46:0/1497628517 conn(0x7fd7c00a6340 msgr2=0x7fd7c013da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.432+0000 7fd7c5533640 1 -- 172.21.15.46:0/1497628517 shutdown_connections 2024-09-16T04:44:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.432+0000 7fd7c5533640 1 -- 172.21.15.46:0/1497628517 wait complete. 2024-09-16T04:44:35.469 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:44:35.469 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:44:35.469 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:44:35.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.569+0000 7f77e7e5e640 10 monclient: get_monmap_and_config 2024-09-16T04:44:35.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.569+0000 7f77e7e5e640 10 monclient: build_initial_monmap 2024-09-16T04:44:35.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.569+0000 7f77e7e5e640 10 monclient: monmap: 2024-09-16T04:44:35.572 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:35.572 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.572 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:44:35.571151+0000 2024-09-16T04:44:35.572 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:44:35.571151+0000 2024-09-16T04:44:35.572 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:35.572 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.572 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:44:35.572 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:44:35.573 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:44:35.573 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:44:35.573 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:44:35.573 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.570+0000 7f77e7e5e640 1 Processor -- start 2024-09-16T04:44:35.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.570+0000 7f77e7e5e640 1 -- start start 2024-09-16T04:44:35.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.570+0000 7f77e7e5e640 10 monclient: init 2024-09-16T04:44:35.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.570+0000 7f77e7e5e640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:35.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.570+0000 7f77e7e5e640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-16T04:44:35.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.570+0000 7f77e7e5e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f77e006e160 0x7f77e00a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.570+0000 7f77e7e5e640 10 monclient(hunting): picked mon.noname-a con 0x7f77e006e160 addr v2:172.21.15.46:3300/0 2024-09-16T04:44:35.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e7e5e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f77e00a2ee0 addr v1:172.21.15.114:6789/0 2024-09-16T04:44:35.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e7e5e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f77e0137050 0x7f77e013ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e7e5e640 10 monclient(hunting): picked mon.noname-c con 0x7f77e0137050 addr v2:172.21.15.114:3300/0 2024-09-16T04:44:35.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e7e5e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77e00a3fd0 con 0x7f77e00a2ee0 2024-09-16T04:44:35.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e7e5e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e7e5e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f77e00a3300 con 0x7f77e006e160 2024-09-16T04:44:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e7e5e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e7e5e640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f77e006e540 con 0x7f77e0137050 2024-09-16T04:44:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e7e5e640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e7e5e640 10 monclient(hunting): authenticate will time out at 1073.987058s 2024-09-16T04:44:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e5bfa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f77e006e160 0x7f77e00a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e5bfa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f77e006e160 0x7f77e00a29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43248/0 (socket says 172.21.15.46:43248) 2024-09-16T04:44:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e5bfa640 1 -- 172.21.15.46:0/769674141 learned_addr learned my addr 172.21.15.46:0/769674141 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e5bfa640 10 monclient(hunting): get_auth_request con 0x7f77e006e160 auth_method 0 2024-09-16T04:44:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e5bfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e5bfa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e5bfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e5bfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e5bfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e5bfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e63fb640 1 --2- 172.21.15.46:0/769674141 >> v2:172.21.15.114:3300/0 conn(0x7f77e0137050 0x7f77e013ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e5bfa640 10 monclient(hunting): handle_auth_done global_id 74163 payload 274 2024-09-16T04:44:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e5bfa640 1 -- 172.21.15.46:0/769674141 >> v2:172.21.15.114:3300/0 conn(0x7f77e0137050 msgr2=0x7f77e013ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:44:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e5bfa640 1 --2- 172.21.15.46:0/769674141 >> v2:172.21.15.114:3300/0 conn(0x7f77e0137050 0x7f77e013ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e5bfa640 1 -- 172.21.15.46:0/769674141 >> v1:172.21.15.114:6789/0 conn(0x7f77e00a2ee0 legacy=0x7f77e0136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:44:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e5bfa640 10 monclient: _finish_hunting 0 2024-09-16T04:44:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e5bfa640 1 monclient: found mon.noname-a 2024-09-16T04:44:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e5bfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e5bfa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:44:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e5bfa640 1 -- 172.21.15.46:0/769674141 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77e00a0bd0 con 0x7f77e006e160 2024-09-16T04:44:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e63fb640 10 monclient: get_auth_request con 0x7f77e0137050 auth_method 0 2024-09-16T04:44:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.571+0000 7f77e63fb640 1 --2- 172.21.15.46:0/769674141 >> v2:172.21.15.114:3300/0 conn(0x7f77e0137050 0x7f77e013ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:44:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e5bfa640 1 --2- 172.21.15.46:0/769674141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77e006e160 0x7f77e00a29a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f77e0067c50 tx=0x7f77dc02a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0679ddea3f465a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e4bf8640 1 -- 172.21.15.46:0/769674141 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77dc037070 con 0x7f77e006e160 2024-09-16T04:44:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e4bf8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e4bf8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:44:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e4bf8640 10 monclient: dump: 2024-09-16T04:44:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.586 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e4bf8640 10 monclient: _finish_auth 0 2024-09-16T04:44:35.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e4bf8640 10 monclient: _check_auth_tickets 2024-09-16T04:44:35.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e4bf8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:35.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e7e5e640 5 monclient: authenticate success, global_id 74163 2024-09-16T04:44:35.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e7e5e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:44:35.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e4bf8640 1 -- 172.21.15.46:0/769674141 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77dc02ae40 con 0x7f77e006e160 2024-09-16T04:44:35.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e4bf8640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:35.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e7e5e640 10 monclient: get_monmap_and_config success 2024-09-16T04:44:35.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e7e5e640 4 set_mon_vals no callback set 2024-09-16T04:44:35.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e4bf8640 1 -- 172.21.15.46:0/769674141 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77dc032660 con 0x7f77e006e160 2024-09-16T04:44:35.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e4bf8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e4bf8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:35.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e4bf8640 10 monclient: dump: 2024-09-16T04:44:35.586 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.587 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.587 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.587 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.587 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.587 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.587 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.587 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.587 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.587 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e7e5e640 10 monclient: shutdown 2024-09-16T04:44:35.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e7e5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:35.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e7e5e640 1 -- 172.21.15.46:0/769674141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77e006e160 msgr2=0x7f77e00a29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e7e5e640 1 --2- 172.21.15.46:0/769674141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77e006e160 0x7f77e00a29a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f77e0067c50 tx=0x7f77dc02a130 comp rx=0 tx=0).stop 2024-09-16T04:44:35.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e7e5e640 1 -- 172.21.15.46:0/769674141 shutdown_connections 2024-09-16T04:44:35.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e7e5e640 1 --2- 172.21.15.46:0/769674141 >> v2:172.21.15.114:3300/0 conn(0x7f77e0137050 0x7f77e013ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e7e5e640 1 --2- 172.21.15.46:0/769674141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77e006e160 0x7f77e00a29a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.572+0000 7f77e7e5e640 1 -- 172.21.15.46:0/769674141 >> 172.21.15.46:0/769674141 conn(0x7f77e00a5c10 msgr2=0x7f77e00a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:35.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.573+0000 7f77e7e5e640 1 -- 172.21.15.46:0/769674141 shutdown_connections 2024-09-16T04:44:35.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.573+0000 7f77e7e5e640 1 -- 172.21.15.46:0/769674141 wait complete. 2024-09-16T04:44:35.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.573+0000 7f77e7e5e640 10 monclient: build_initial_monmap 2024-09-16T04:44:35.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.573+0000 7f77e7e5e640 10 monclient: monmap: 2024-09-16T04:44:35.588 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:35.588 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:44:35.588 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:44:35.588 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:44:35.588 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:35.589 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.589 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:44:35.589 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:44:35.589 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:44:35.589 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.573+0000 7f77e7e5e640 1 Processor -- start 2024-09-16T04:44:35.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.573+0000 7f77e7e5e640 1 -- start start 2024-09-16T04:44:35.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e7e5e640 10 monclient: init 2024-09-16T04:44:35.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e7e5e640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:35.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e7e5e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:44:35.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e7e5e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77e006e160 0x7f77e0135b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e7e5e640 10 monclient(hunting): picked mon.noname-b con 0x7f77e006e160 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:44:35.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e7e5e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77e00a2ee0 0x7f77e01360d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e7e5e640 10 monclient(hunting): picked mon.noname-a con 0x7f77e00a2ee0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:44:35.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e7e5e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77e0137050 0x7f77e0123600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e7e5e640 10 monclient(hunting): picked mon.noname-c con 0x7f77e0137050 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:35.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e7e5e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e7e5e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f77e00a34a0 con 0x7f77e00a2ee0 2024-09-16T04:44:35.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e7e5e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e7e5e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f77e00a5130 con 0x7f77e006e160 2024-09-16T04:44:35.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e7e5e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e7e5e640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f77e0125740 con 0x7f77e0137050 2024-09-16T04:44:35.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e7e5e640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:35.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e7e5e640 10 monclient(hunting): authenticate will time out at 1073.990499s 2024-09-16T04:44:35.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e53f9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77e00a2ee0 0x7f77e01360d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e63fb640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77e0137050 0x7f77e0123600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e53f9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77e00a2ee0 0x7f77e01360d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43250/0 (socket says 172.21.15.46:43250) 2024-09-16T04:44:35.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e53f9640 1 -- 172.21.15.46:0/1684228036 learned_addr learned my addr 172.21.15.46:0/1684228036 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:35.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e5bfa640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77e006e160 0x7f77e0135b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e53f9640 10 monclient(hunting): get_auth_request con 0x7f77e00a2ee0 auth_method 0 2024-09-16T04:44:35.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e53f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e53f9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e53f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e63fb640 10 monclient(hunting): get_auth_request con 0x7f77e0137050 auth_method 0 2024-09-16T04:44:35.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e63fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e63fb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e63fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e5bfa640 10 monclient(hunting): get_auth_request con 0x7f77e006e160 auth_method 0 2024-09-16T04:44:35.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e5bfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e5bfa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e5bfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e53f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e53f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e53f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e63fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e63fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e63fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e5bfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e5bfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e5bfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e53f9640 10 monclient(hunting): handle_auth_done global_id 74166 payload 824 2024-09-16T04:44:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e53f9640 1 -- 172.21.15.46:0/1684228036 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77e0137050 msgr2=0x7f77e0123600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e53f9640 1 --2- 172.21.15.46:0/1684228036 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77e0137050 0x7f77e0123600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e53f9640 1 -- 172.21.15.46:0/1684228036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77e006e160 msgr2=0x7f77e0135b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e53f9640 1 --2- 172.21.15.46:0/1684228036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77e006e160 0x7f77e0135b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e53f9640 10 monclient: _finish_hunting 0 2024-09-16T04:44:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e53f9640 1 monclient: found mon.noname-a 2024-09-16T04:44:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e53f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e53f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:44:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e53f9640 1 -- 172.21.15.46:0/1684228036 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77e00a83d0 con 0x7f77e00a2ee0 2024-09-16T04:44:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.574+0000 7f77e63fb640 1 --2- 172.21.15.46:0/1684228036 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77e0137050 0x7f77e0123600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:44:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77e53f9640 1 --2- 172.21.15.46:0/1684228036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77e00a2ee0 0x7f77e01360d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f77d000c430 tx=0x7f77d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77d67fc640 1 -- 172.21.15.46:0/1684228036 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d0011790 con 0x7f77e00a2ee0 2024-09-16T04:44:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77d67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:44:35.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77d67fc640 10 monclient: dump: 2024-09-16T04:44:35.594 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.596 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.596 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.596 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.596 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.597 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.597 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.597 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.597 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.597 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77d67fc640 10 monclient: _finish_auth 0 2024-09-16T04:44:35.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77d67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:44:35.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:35.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77d67fc640 1 -- 172.21.15.46:0/1684228036 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77d0017070 con 0x7f77e00a2ee0 2024-09-16T04:44:35.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77d67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:35.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77d67fc640 1 -- 172.21.15.46:0/1684228036 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d0011bb0 con 0x7f77e00a2ee0 2024-09-16T04:44:35.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:35.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77d67fc640 10 monclient: dump: 2024-09-16T04:44:35.598 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.598 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.598 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.598 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.598 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.598 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.598 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.598 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.598 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.598 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77e7e5e640 5 monclient: authenticate success, global_id 74166 2024-09-16T04:44:35.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77e7e5e640 10 monclient: _renew_subs 2024-09-16T04:44:35.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77e7e5e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:35.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77e7e5e640 1 -- 172.21.15.46:0/1684228036 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77e00a0300 con 0x7f77e00a2ee0 2024-09-16T04:44:35.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77d7fff640 4 set_mon_vals no callback set 2024-09-16T04:44:35.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77e7e5e640 10 monclient: _renew_subs 2024-09-16T04:44:35.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77e7e5e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:35.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.575+0000 7f77e7e5e640 1 -- 172.21.15.46:0/1684228036 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77e0126fd0 con 0x7f77e00a2ee0 2024-09-16T04:44:35.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77e7e5e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:35.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77e7e5e640 1 --2- 172.21.15.46:0/1684228036 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77e01cbd20 0x7f77e0127210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77e7e5e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77e7e5e640 1 -- 172.21.15.46:0/1684228036 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f77e00a3620 con 0x7f77e01cbd20 2024-09-16T04:44:35.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77d67fc640 1 -- 172.21.15.46:0/1684228036 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f77d001c460 con 0x7f77e00a2ee0 2024-09-16T04:44:35.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77e5bfa640 1 --2- 172.21.15.46:0/1684228036 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77e01cbd20 0x7f77e0127210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77e5bfa640 10 monclient: get_auth_request con 0x7f77e01cbd20 auth_method 0 2024-09-16T04:44:35.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77e5bfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77e5bfa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77e5bfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77e5bfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77e5bfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77e5bfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77e5bfa640 10 monclient(hunting): handle_auth_done global_id 64331 payload 824 2024-09-16T04:44:35.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77d67fc640 1 --2- 172.21.15.46:0/1684228036 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f77c40383b0 0x7f77c403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77e5bfa640 1 -- 172.21.15.46:0/1684228036 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f77e01374a0 con 0x7f77e01cbd20 2024-09-16T04:44:35.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77d67fc640 1 -- 172.21.15.46:0/1684228036 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 4738+0+0 (secure 0 0 0) 0x7f77d004b5b0 con 0x7f77e00a2ee0 2024-09-16T04:44:35.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77e5bfa640 1 --2- 172.21.15.46:0/1684228036 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77e01cbd20 0x7f77e0127210 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f77e009fec0 tx=0x7f77dc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:35.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77e63fb640 1 --2- 172.21.15.46:0/1684228036 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f77c40383b0 0x7f77c403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77e63fb640 10 monclient: get_auth_request con 0x7f77c40383b0 auth_method 0 2024-09-16T04:44:35.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77e63fb640 1 -- 172.21.15.46:0/1684228036 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f77c40383b0 msgr2=0x7f77c403a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:44:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77d67fc640 1 -- 172.21.15.46:0/1684228036 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77dc042070 con 0x7f77e01cbd20 2024-09-16T04:44:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77e63fb640 1 -- 172.21.15.46:0/1684228036 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f77c40383b0 msgr2=0x7f77c403a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:44:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77e63fb640 1 --2- 172.21.15.46:0/1684228036 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f77c40383b0 0x7f77c403a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:44:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77d67fc640 10 monclient: dump: 2024-09-16T04:44:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.602 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.602 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.576+0000 7f77e63fb640 1 --2- 172.21.15.46:0/1684228036 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f77c40383b0 0x7f77c403a870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:35.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.577+0000 7f77d67fc640 1 -- 172.21.15.46:0/1684228036 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f77dc003d80 con 0x7f77e01cbd20 2024-09-16T04:44:35.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.577+0000 7f77d67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:35.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.577+0000 7f77d67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:44:35.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.577+0000 7f77d67fc640 1 -- 172.21.15.46:0/1684228036 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77e01cbd20 msgr2=0x7f77e0127210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.577+0000 7f77d67fc640 1 --2- 172.21.15.46:0/1684228036 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77e01cbd20 0x7f77e0127210 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f77e009fec0 tx=0x7f77dc0046c0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.577+0000 7f77d67fc640 1 -- 172.21.15.46:0/1684228036 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77e01cbd20 msgr2=0x7f77e0127210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:44:35.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.577+0000 7f77d67fc640 1 --2- 172.21.15.46:0/1684228036 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77e01cbd20 0x7f77e0127210 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f77e009fec0 tx=0x7f77dc0046c0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.583+0000 7f77e7e5e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:44:35.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.583+0000 7f77e7e5e640 1 --2- 172.21.15.46:0/1684228036 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77e0123b40 0x7f77e0123f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.583+0000 7f77e7e5e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.583+0000 7f77e7e5e640 1 -- 172.21.15.46:0/1684228036 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f77e0124490 con 0x7f77e0123b40 2024-09-16T04:44:35.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.583+0000 7f77e5bfa640 1 --2- 172.21.15.46:0/1684228036 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77e0123b40 0x7f77e0123f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.583+0000 7f77e5bfa640 10 monclient: get_auth_request con 0x7f77e0123b40 auth_method 0 2024-09-16T04:44:35.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.583+0000 7f77e5bfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.583+0000 7f77e5bfa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.583+0000 7f77e5bfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.583+0000 7f77e5bfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.583+0000 7f77e5bfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.583+0000 7f77e5bfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.583+0000 7f77e5bfa640 10 monclient(hunting): handle_auth_done global_id 64334 payload 824 2024-09-16T04:44:35.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.583+0000 7f77e5bfa640 1 -- 172.21.15.46:0/1684228036 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f77e0067e00 con 0x7f77e0123b40 2024-09-16T04:44:35.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.583+0000 7f77e5bfa640 1 --2- 172.21.15.46:0/1684228036 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77e0123b40 0x7f77e0123f50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f77dc02a170 tx=0x7f77dc008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:35.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77d67fc640 1 -- 172.21.15.46:0/1684228036 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77dc037040 con 0x7f77e0123b40 2024-09-16T04:44:35.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:44:35.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77d67fc640 10 monclient: dump: 2024-09-16T04:44:35.605 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.605 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.605 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.605 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.605 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.605 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.606 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77d67fc640 1 -- 172.21.15.46:0/1684228036 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f77dc03c430 con 0x7f77e0123b40 2024-09-16T04:44:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77d67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:44:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77d67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:44:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77d67fc640 1 -- 172.21.15.46:0/1684228036 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77e0123b40 msgr2=0x7f77e0123f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77d67fc640 1 --2- 172.21.15.46:0/1684228036 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77e0123b40 0x7f77e0123f50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f77dc02a170 tx=0x7f77dc008030 comp rx=0 tx=0).stop 2024-09-16T04:44:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77d67fc640 1 -- 172.21.15.46:0/1684228036 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77e0123b40 msgr2=0x7f77e0123f50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:44:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77d67fc640 1 --2- 172.21.15.46:0/1684228036 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f77e0123b40 0x7f77e0123f50 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f77dc02a170 tx=0x7f77dc008030 comp rx=0 tx=0).stop 2024-09-16T04:44:35.607 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:44:35.607 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:44:35.607 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:44:35.607 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:44:35.607 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 68, 2024-09-16T04:44:35.607 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 63521, 2024-09-16T04:44:35.607 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:44:35.607 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:44:35.607 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:44:35.607 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:44:35.607 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:35.607 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 2, 2024-09-16T04:44:35.608 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:44:35.608 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:44:35.608 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:44:35.608 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:35.608 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:35.608 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:35.608 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:35.608 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:35.608 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:35.608 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:35.609 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:35.609 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:35.609 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:35.609 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:35.609 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:35.609 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:44:35.609 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:44:35.609 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:35.609 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:35.609 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:35.609 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:35.609 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:35.609 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:35.610 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:35.610 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:35.610 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:35.610 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:35.610 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:35.610 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:35.610 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.610 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:44:35.610 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:44:35.610 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:44:35.610 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:44:35.610 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:44:35.610 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:44:35.611 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:44:35.611 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:44:35.611 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:44:35.611 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:44:35.611 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:44:35.611 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:44:35.611 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:44:35.611 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:44:35.611 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:44:35.611 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:44:35.611 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:44:35.611 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:35.611 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:35.611 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:35.612 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:35.612 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:35.612 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:35.612 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:35.612 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:35.612 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:35.612 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:35.612 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:35.612 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:35.612 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:44:35.612 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.612 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:44:35.612 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.612 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:44:35.613 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:44:35.613 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:35.613 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:35.613 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.613 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:35.613 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:44:35.613 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:35.613 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.613 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.613 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:35.613 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:44:35.613 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:35.613 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:35.614 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:35.614 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.614 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:44:35.614 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:44:35.614 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:35.614 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:35.614 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:35.614 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.614 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.614 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:44:35.614 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:44:35.614 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:35.614 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:35.615 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.615 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:35.615 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:44:35.615 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:35.615 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.615 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.615 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:35.615 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:44:35.615 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:35.615 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:35.615 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:35.615 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.615 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:44:35.616 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:44:35.616 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:35.616 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:35.616 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:35.616 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.616 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.616 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:44:35.616 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:44:35.616 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:35.616 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:35.616 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.616 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:35.616 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:44:35.616 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:35.617 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.617 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.617 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:35.617 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:44:35.617 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:35.617 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:35.617 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:35.617 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.617 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:44:35.617 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:44:35.617 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:35.617 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:35.617 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:35.618 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:35.618 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:35.618 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.618 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:44:35.618 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:44:35.618 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.618 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:35.618 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:35.618 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:44:35.618 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:35.618 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:35.618 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:44:35.618 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.619 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:35.619 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:35.619 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:44:35.619 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:35.619 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:35.619 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:44:35.619 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.619 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:35.619 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:35.619 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:44:35.619 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:35.619 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:35.619 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.619 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:44:35.620 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:44:35.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77e7e5e640 1 -- 172.21.15.46:0/1684228036 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f77c40383b0 msgr2=0x7f77c403a870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:44:35.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77e7e5e640 1 --2- 172.21.15.46:0/1684228036 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f77c40383b0 0x7f77c403a870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77e7e5e640 10 monclient: shutdown 2024-09-16T04:44:35.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77e7e5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:35.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77e7e5e640 1 -- 172.21.15.46:0/1684228036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77e00a2ee0 msgr2=0x7f77e01360d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77e63fb640 1 -- 172.21.15.46:0/1684228036 reap_dead start 2024-09-16T04:44:35.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77e7e5e640 1 --2- 172.21.15.46:0/1684228036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77e00a2ee0 0x7f77e01360d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f77d000c430 tx=0x7f77d000c8f0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77e7e5e640 1 -- 172.21.15.46:0/1684228036 shutdown_connections 2024-09-16T04:44:35.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77e7e5e640 1 --2- 172.21.15.46:0/1684228036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77e00a2ee0 0x7f77e01360d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77e7e5e640 1 -- 172.21.15.46:0/1684228036 >> 172.21.15.46:0/1684228036 conn(0x7f77e00a5c10 msgr2=0x7f77e009d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:35.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77e7e5e640 1 -- 172.21.15.46:0/1684228036 shutdown_connections 2024-09-16T04:44:35.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.584+0000 7f77e7e5e640 1 -- 172.21.15.46:0/1684228036 wait complete. 2024-09-16T04:44:35.622 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:44:35.622 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:44:35.622 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:44:35.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.732+0000 7f1071b42640 10 monclient: get_monmap_and_config 2024-09-16T04:44:35.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.732+0000 7f1071b42640 10 monclient: build_initial_monmap 2024-09-16T04:44:35.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.732+0000 7f1071b42640 10 monclient: monmap: 2024-09-16T04:44:35.734 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:35.734 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.734 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:44:35.733980+0000 2024-09-16T04:44:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:44:35.733980+0000 2024-09-16T04:44:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:44:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:44:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:44:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:44:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:44:35.735 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.733+0000 7f1071b42640 1 Processor -- start 2024-09-16T04:44:35.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.733+0000 7f1071b42640 1 -- start start 2024-09-16T04:44:35.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.733+0000 7f1071b42640 10 monclient: init 2024-09-16T04:44:35.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.733+0000 7f1071b42640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:35.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.733+0000 7f1071b42640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-16T04:44:35.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.733+0000 7f1071b42640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f106c0a3c10 0x7f106c09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.733+0000 7f1071b42640 10 monclient(hunting): picked mon.noname-b con 0x7f106c0a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:35.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.733+0000 7f1071b42640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f106c0a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:44:35.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.733+0000 7f1071b42640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f106c13be90 0x7f106c13e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.733+0000 7f1071b42640 10 monclient(hunting): picked mon.noname-c con 0x7f106c13be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:44:35.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.733+0000 7f1071b42640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f106c09b940 con 0x7f106c0a3ff0 2024-09-16T04:44:35.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.733+0000 7f1071b42640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.733+0000 7f1071b42640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f106c09ab00 con 0x7f106c13be90 2024-09-16T04:44:35.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.733+0000 7f1071b42640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.733+0000 7f1071b42640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f106c09b030 con 0x7f106c0a3c10 2024-09-16T04:44:35.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f1071b42640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:35.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f1071b42640 10 monclient(hunting): authenticate will time out at 1074.150028s 2024-09-16T04:44:35.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106b7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f106c0a3c10 0x7f106c09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106b7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f106c0a3c10 0x7f106c09bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:60918/0 (socket says 172.21.15.46:60918) 2024-09-16T04:44:35.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106affd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f106c0a3ff0 0x7f106c0a43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.46:60346/0 (socket says 172.21.15.46:60346) 2024-09-16T04:44:35.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106b7fe640 1 -- 172.21.15.46:0/1194003995 learned_addr learned my addr 172.21.15.46:0/1194003995 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:35.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106b7fe640 10 monclient(hunting): get_auth_request con 0x7f106c0a3c10 auth_method 0 2024-09-16T04:44:35.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106bfff640 1 --2- 172.21.15.46:0/1194003995 >> v2:172.21.15.114:3300/0 conn(0x7f106c13be90 0x7f106c13e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106bfff640 10 monclient(hunting): get_auth_request con 0x7f106c13be90 auth_method 0 2024-09-16T04:44:35.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106b7fe640 10 monclient(hunting): handle_auth_done global_id 64337 payload 274 2024-09-16T04:44:35.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106b7fe640 1 -- 172.21.15.46:0/1194003995 >> v2:172.21.15.114:3300/0 conn(0x7f106c13be90 msgr2=0x7f106c13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:44:35.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106b7fe640 1 --2- 172.21.15.46:0/1194003995 >> v2:172.21.15.114:3300/0 conn(0x7f106c13be90 0x7f106c13e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106b7fe640 1 -- 172.21.15.46:0/1194003995 >> v1:172.21.15.114:6789/0 conn(0x7f106c0a3ff0 legacy=0x7f106c0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:44:35.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106b7fe640 1 monclient: found mon.noname-b 2024-09-16T04:44:35.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:35.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:44:35.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.734+0000 7f106b7fe640 1 -- 172.21.15.46:0/1194003995 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f106c09b310 con 0x7f106c0a3c10 2024-09-16T04:44:35.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f106b7fe640 1 --2- 172.21.15.46:0/1194003995 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f106c0a3c10 0x7f106c09bcf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f105c009870 tx=0x7f105c02a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fad8941a5668329a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:35.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f106a7fc640 1 -- 172.21.15.46:0/1194003995 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c037070 con 0x7f106c0a3c10 2024-09-16T04:44:35.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f106a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f106a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:44:35.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f106a7fc640 10 monclient: dump: 2024-09-16T04:44:35.750 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.750 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.750 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.750 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.750 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.750 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.750 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.750 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.751 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.751 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f106a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:44:35.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f106a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:44:35.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f106a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:35.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f1071b42640 5 monclient: authenticate success, global_id 64337 2024-09-16T04:44:35.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f1071b42640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:44:35.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f106a7fc640 1 -- 172.21.15.46:0/1194003995 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f105c032440 con 0x7f106c0a3c10 2024-09-16T04:44:35.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f106a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:35.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f1071b42640 10 monclient: get_monmap_and_config success 2024-09-16T04:44:35.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f1071b42640 4 set_mon_vals no callback set 2024-09-16T04:44:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f106a7fc640 1 -- 172.21.15.46:0/1194003995 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c032810 con 0x7f106c0a3c10 2024-09-16T04:44:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f106a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f106a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:44:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f106a7fc640 10 monclient: dump: 2024-09-16T04:44:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.753 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.753 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.753 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f1071b42640 10 monclient: shutdown 2024-09-16T04:44:35.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f1071b42640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:35.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f1071b42640 1 -- 172.21.15.46:0/1194003995 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f106c0a3c10 msgr2=0x7f106c09bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.735+0000 7f1071b42640 1 --2- 172.21.15.46:0/1194003995 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f106c0a3c10 0x7f106c09bcf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f105c009870 tx=0x7f105c02a130 comp rx=0 tx=0).stop 2024-09-16T04:44:35.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.736+0000 7f1071b42640 1 -- 172.21.15.46:0/1194003995 shutdown_connections 2024-09-16T04:44:35.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.736+0000 7f1071b42640 1 --2- 172.21.15.46:0/1194003995 >> v2:172.21.15.114:3300/0 conn(0x7f106c13be90 0x7f106c13e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.736+0000 7f1071b42640 1 --2- 172.21.15.46:0/1194003995 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f106c0a3c10 0x7f106c09bcf0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.736+0000 7f1071b42640 1 -- 172.21.15.46:0/1194003995 >> 172.21.15.46:0/1194003995 conn(0x7f106c0a6340 msgr2=0x7f106c06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:35.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.736+0000 7f1071b42640 1 -- 172.21.15.46:0/1194003995 shutdown_connections 2024-09-16T04:44:35.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.736+0000 7f1071b42640 1 -- 172.21.15.46:0/1194003995 wait complete. 2024-09-16T04:44:35.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.736+0000 7f1071b42640 10 monclient: build_initial_monmap 2024-09-16T04:44:35.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.736+0000 7f1071b42640 10 monclient: monmap: 2024-09-16T04:44:35.754 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:35.754 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:44:35.754 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:44:35.754 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:44:35.754 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:35.754 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.754 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:44:35.754 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:44:35.754 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:44:35.755 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f1071b42640 1 Processor -- start 2024-09-16T04:44:35.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f1071b42640 1 -- start start 2024-09-16T04:44:35.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f1071b42640 10 monclient: init 2024-09-16T04:44:35.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f1071b42640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:35.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f1071b42640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:44:35.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f1071b42640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f106c0a3c10 0x7f106c133cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f1071b42640 10 monclient(hunting): picked mon.noname-a con 0x7f106c0a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:44:35.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f1071b42640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c0a3ff0 0x7f106c134230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f1071b42640 10 monclient(hunting): picked mon.noname-b con 0x7f106c0a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:44:35.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f1071b42640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f106c13be90 0x7f106c1cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f1071b42640 10 monclient(hunting): picked mon.noname-c con 0x7f106c13be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:35.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f1071b42640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f1071b42640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f106c0a6b40 con 0x7f106c0a3c10 2024-09-16T04:44:35.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f1071b42640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f1071b42640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f106c0a69c0 con 0x7f106c0a3ff0 2024-09-16T04:44:35.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f1071b42640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f1071b42640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f106c13e7c0 con 0x7f106c13be90 2024-09-16T04:44:35.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f1071b42640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:35.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f1071b42640 10 monclient(hunting): authenticate will time out at 1074.153910s 2024-09-16T04:44:35.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f106b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f106c0a3c10 0x7f106c133cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f106bfff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f106c13be90 0x7f106c1cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f106b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f106c0a3c10 0x7f106c133cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43266/0 (socket says 172.21.15.46:43266) 2024-09-16T04:44:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f106bfff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f106c13be90 0x7f106c1cbac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:60928/0 (socket says 172.21.15.46:60928) 2024-09-16T04:44:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f106b7fe640 1 -- 172.21.15.46:0/1818247817 learned_addr learned my addr 172.21.15.46:0/1818247817 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f106b7fe640 10 monclient(hunting): get_auth_request con 0x7f106c0a3c10 auth_method 0 2024-09-16T04:44:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f106b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f106b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.737+0000 7f106b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106bfff640 10 monclient(hunting): get_auth_request con 0x7f106c13be90 auth_method 0 2024-09-16T04:44:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106affd640 1 --2- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c0a3ff0 0x7f106c134230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106affd640 10 monclient(hunting): get_auth_request con 0x7f106c0a3ff0 auth_method 0 2024-09-16T04:44:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106b7fe640 10 monclient(hunting): handle_auth_done global_id 74169 payload 824 2024-09-16T04:44:35.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106b7fe640 1 -- 172.21.15.46:0/1818247817 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f106c13be90 msgr2=0x7f106c1cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106b7fe640 1 --2- 172.21.15.46:0/1818247817 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f106c13be90 0x7f106c1cbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106b7fe640 1 -- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c0a3ff0 msgr2=0x7f106c134230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106b7fe640 1 --2- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c0a3ff0 0x7f106c134230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:44:35.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106b7fe640 1 monclient: found mon.noname-a 2024-09-16T04:44:35.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:35.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:44:35.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106b7fe640 1 -- 172.21.15.46:0/1818247817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f106c09f0b0 con 0x7f106c0a3c10 2024-09-16T04:44:35.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106bfff640 1 --2- 172.21.15.46:0/1818247817 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f106c13be90 0x7f106c1cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:44:35.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f106b7fe640 1 --2- 172.21.15.46:0/1818247817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f106c0a3c10 0x7f106c133cf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f105c000c00 tx=0x7f105c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:35.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f1053fff640 1 -- 172.21.15.46:0/1818247817 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c041020 con 0x7f106c0a3c10 2024-09-16T04:44:35.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f1053fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f1053fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:44:35.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f1053fff640 10 monclient: dump: 2024-09-16T04:44:35.760 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.760 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.760 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.760 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.760 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.760 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.761 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.761 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.761 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.761 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f1053fff640 10 monclient: _finish_auth 0 2024-09-16T04:44:35.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f1053fff640 10 monclient: _check_auth_tickets 2024-09-16T04:44:35.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f1053fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:35.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f1071b42640 5 monclient: authenticate success, global_id 74169 2024-09-16T04:44:35.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f1053fff640 1 -- 172.21.15.46:0/1818247817 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f105c030640 con 0x7f106c0a3c10 2024-09-16T04:44:35.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f1053fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:35.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f1053fff640 1 -- 172.21.15.46:0/1818247817 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c037040 con 0x7f106c0a3c10 2024-09-16T04:44:35.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f1071b42640 10 monclient: _renew_subs 2024-09-16T04:44:35.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f1071b42640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:35.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f1071b42640 1 -- 172.21.15.46:0/1818247817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f106c13c600 con 0x7f106c0a3c10 2024-09-16T04:44:35.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f1053fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f1053fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:35.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f1053fff640 10 monclient: dump: 2024-09-16T04:44:35.762 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.763 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.763 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.763 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.763 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.763 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.763 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.763 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.763 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.763 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f1069ffb640 4 set_mon_vals no callback set 2024-09-16T04:44:35.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f1071b42640 10 monclient: _renew_subs 2024-09-16T04:44:35.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f1071b42640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:35.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.738+0000 7f1071b42640 1 -- 172.21.15.46:0/1818247817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f106c09edd0 con 0x7f106c0a3c10 2024-09-16T04:44:35.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.739+0000 7f1053fff640 1 -- 172.21.15.46:0/1818247817 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f105c044020 con 0x7f106c0a3c10 2024-09-16T04:44:35.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.739+0000 7f1071b42640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:35.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.739+0000 7f1071b42640 1 --2- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c1cc000 0x7f106c1cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.739+0000 7f1071b42640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.739+0000 7f1071b42640 1 -- 172.21.15.46:0/1818247817 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f106c1390e0 con 0x7f106c1cc000 2024-09-16T04:44:35.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.739+0000 7f1053fff640 1 --2- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f10580383b0 0x7f105803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.739+0000 7f1053fff640 1 -- 172.21.15.46:0/1818247817 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 4738+0+0 (secure 0 0 0) 0x7f105c06bf30 con 0x7f106c0a3c10 2024-09-16T04:44:35.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.739+0000 7f106affd640 1 --2- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c1cc000 0x7f106c1cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.740+0000 7f106affd640 10 monclient: get_auth_request con 0x7f106c1cc000 auth_method 0 2024-09-16T04:44:35.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.740+0000 7f106affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.740+0000 7f106affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.740+0000 7f106affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.740+0000 7f106affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.740+0000 7f106affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.740+0000 7f106affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.740+0000 7f106bfff640 1 --2- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f10580383b0 0x7f105803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.740+0000 7f106bfff640 10 monclient: get_auth_request con 0x7f10580383b0 auth_method 0 2024-09-16T04:44:35.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.740+0000 7f106affd640 10 monclient(hunting): handle_auth_done global_id 6274 payload 824 2024-09-16T04:44:35.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.740+0000 7f106affd640 1 -- 172.21.15.46:0/1818247817 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f106c1cc920 con 0x7f106c1cc000 2024-09-16T04:44:35.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.740+0000 7f106affd640 1 --2- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c1cc000 0x7f106c1cc3e0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f1060000f60 tx=0x7f1060001970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:35.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.740+0000 7f106bfff640 1 --2- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f10580383b0 0x7f105803a870 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f106c09df40 tx=0x7f106400b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:35.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.740+0000 7f1053fff640 1 -- 172.21.15.46:0/1818247817 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1060011580 con 0x7f106c1cc000 2024-09-16T04:44:35.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.740+0000 7f1053fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.740+0000 7f1053fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:44:35.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.740+0000 7f1053fff640 10 monclient: dump: 2024-09-16T04:44:35.766 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.766 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.766 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.766 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.766 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.766 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.766 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.766 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.766 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.767 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.741+0000 7f1053fff640 1 -- 172.21.15.46:0/1818247817 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1060018070 con 0x7f106c1cc000 2024-09-16T04:44:35.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.741+0000 7f1053fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:35.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.741+0000 7f1053fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:44:35.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.741+0000 7f1053fff640 1 -- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c1cc000 msgr2=0x7f106c1cc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.741+0000 7f1053fff640 1 --2- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c1cc000 0x7f106c1cc3e0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f1060000f60 tx=0x7f1060001970 comp rx=0 tx=0).stop 2024-09-16T04:44:35.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.741+0000 7f1053fff640 1 -- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c1cc000 msgr2=0x7f106c1cc3e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:44:35.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.741+0000 7f1053fff640 1 --2- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c1cc000 0x7f106c1cc3e0 secure :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f1060000f60 tx=0x7f1060001970 comp rx=0 tx=0).stop 2024-09-16T04:44:35.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.747+0000 7f1071b42640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:44:35.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.747+0000 7f1071b42640 1 --2- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c0a0610 0x7f106c0a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.747+0000 7f1071b42640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.747+0000 7f1071b42640 1 -- 172.21.15.46:0/1818247817 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f106c09e810 con 0x7f106c0a0610 2024-09-16T04:44:35.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.747+0000 7f106affd640 1 --2- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c0a0610 0x7f106c0a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.747+0000 7f106affd640 10 monclient: get_auth_request con 0x7f106c0a0610 auth_method 0 2024-09-16T04:44:35.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.747+0000 7f106affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.747+0000 7f106affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.747+0000 7f106affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.748+0000 7f106affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.748+0000 7f106affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.748+0000 7f106affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.748+0000 7f106affd640 10 monclient(hunting): handle_auth_done global_id 6277 payload 824 2024-09-16T04:44:35.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.748+0000 7f106affd640 1 -- 172.21.15.46:0/1818247817 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f106c069460 con 0x7f106c0a0610 2024-09-16T04:44:35.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.748+0000 7f106affd640 1 --2- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c0a0610 0x7f106c0a0a20 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f1060000f60 tx=0x7f1060007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:35.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.748+0000 7f1053fff640 1 -- 172.21.15.46:0/1818247817 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1060011d50 con 0x7f106c0a0610 2024-09-16T04:44:35.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.748+0000 7f1053fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.748+0000 7f1053fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:44:35.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.748+0000 7f1053fff640 10 monclient: dump: 2024-09-16T04:44:35.771 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.771 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.772 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.772 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.772 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.772 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.772 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.772 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.772 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.772 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.748+0000 7f1053fff640 1 -- 172.21.15.46:0/1818247817 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f1060011d50 con 0x7f106c0a0610 2024-09-16T04:44:35.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.748+0000 7f1053fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:44:35.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.748+0000 7f1053fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:44:35.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.748+0000 7f1053fff640 1 -- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c0a0610 msgr2=0x7f106c0a0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.748+0000 7f1053fff640 1 --2- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c0a0610 0x7f106c0a0a20 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f1060000f60 tx=0x7f1060007f90 comp rx=0 tx=0).stop 2024-09-16T04:44:35.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.748+0000 7f1053fff640 1 -- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c0a0610 msgr2=0x7f106c0a0a20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:44:35.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.748+0000 7f1053fff640 1 --2- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f106c0a0610 0x7f106c0a0a20 secure :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f1060000f60 tx=0x7f1060007f90 comp rx=0 tx=0).stop 2024-09-16T04:44:35.773 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:44:35.773 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:44:35.773 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:44:35.773 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:44:35.773 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 68, 2024-09-16T04:44:35.774 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 361651, 2024-09-16T04:44:35.774 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:44:35.774 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:44:35.774 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:44:35.774 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:44:35.774 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:35.774 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 3, 2024-09-16T04:44:35.774 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:44:35.774 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:44:35.774 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:44:35.774 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:35.775 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:35.775 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:35.775 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:35.775 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:35.775 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:35.775 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:35.775 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:35.775 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:35.775 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:35.775 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:35.775 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:35.776 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:44:35.776 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:44:35.776 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:35.776 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:35.776 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:35.776 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:35.776 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:35.776 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:35.776 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:35.776 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:35.776 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:35.777 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:35.777 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:35.777 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:35.777 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.777 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:44:35.777 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:44:35.777 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:44:35.777 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:44:35.777 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:44:35.777 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:44:35.777 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:44:35.778 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:44:35.778 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:44:35.778 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:44:35.778 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:44:35.778 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:44:35.778 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:44:35.778 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:44:35.778 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:44:35.778 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:44:35.778 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:44:35.778 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:35.779 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:35.779 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:35.779 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:35.779 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:35.779 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:35.779 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:35.779 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:35.779 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:35.779 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:35.779 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:35.780 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:35.780 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:44:35.780 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.780 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:44:35.780 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.780 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:44:35.780 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:44:35.780 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:35.780 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:35.780 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.780 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:35.780 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:44:35.781 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:35.781 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.781 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.781 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:35.781 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:44:35.781 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:35.781 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:35.781 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:35.781 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.781 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:44:35.781 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:44:35.782 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:35.782 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:35.782 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:35.782 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.782 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.782 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:44:35.782 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:44:35.782 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:35.782 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:35.782 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.782 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:35.783 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:44:35.783 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:35.783 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.783 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.783 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:35.783 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:44:35.783 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:35.783 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:35.783 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:35.784 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.784 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:44:35.784 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:44:35.784 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:35.784 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:35.784 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:35.784 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.784 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.784 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:44:35.784 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:44:35.784 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:35.784 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:35.785 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.785 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:35.785 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:44:35.785 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:35.785 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.785 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.785 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:35.785 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:44:35.785 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:35.785 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:35.785 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:35.786 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.786 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:44:35.786 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:44:35.786 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:35.786 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:35.786 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:35.786 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:35.786 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:35.786 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.786 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:44:35.786 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:44:35.787 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.787 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:35.787 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:35.787 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:44:35.787 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:35.787 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:35.787 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:44:35.787 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.787 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:35.787 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:35.787 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5 2024-09-16T04:44:35.787 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:35.788 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:35.788 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:44:35.788 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.788 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:35.788 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:35.788 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6 2024-09-16T04:44:35.788 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:35.788 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:35.788 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:44:35.788 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:35.788 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:35.789 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:35.789 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:44:35.789 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:35.789 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:35.789 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:35.789 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:44:35.789 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:44:35.790 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38891]: starting. 2024-09-16T04:44:35.791 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38891]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-38890 2024-09-16T04:44:35.791 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38891]: created object 0... 2024-09-16T04:44:35.791 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38891]: created object 25... 2024-09-16T04:44:35.791 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38891]: created object 49... 2024-09-16T04:44:35.791 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38891]: finishing. 2024-09-16T04:44:35.791 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38891]: shutting down. 2024-09-16T04:44:35.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.749+0000 7f1071b42640 1 -- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f10580383b0 msgr2=0x7f105803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.749+0000 7f1071b42640 1 --2- 172.21.15.46:0/1818247817 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f10580383b0 0x7f105803a870 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f106c09df40 tx=0x7f106400b040 comp rx=0 tx=0).stop 2024-09-16T04:44:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.749+0000 7f1071b42640 10 monclient: shutdown 2024-09-16T04:44:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.749+0000 7f1071b42640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.749+0000 7f1071b42640 1 -- 172.21.15.46:0/1818247817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f106c0a3c10 msgr2=0x7f106c133cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.749+0000 7f1071b42640 1 --2- 172.21.15.46:0/1818247817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f106c0a3c10 0x7f106c133cf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f105c000c00 tx=0x7f105c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.749+0000 7f106bfff640 1 -- 172.21.15.46:0/1818247817 reap_dead start 2024-09-16T04:44:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.749+0000 7f106bfff640 1 -- 172.21.15.46:0/1818247817 reap_dead start 2024-09-16T04:44:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.749+0000 7f1071b42640 1 -- 172.21.15.46:0/1818247817 shutdown_connections 2024-09-16T04:44:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.749+0000 7f1071b42640 1 -- 172.21.15.46:0/1818247817 >> 172.21.15.46:0/1818247817 conn(0x7f106c0a6340 msgr2=0x7f106c13cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.749+0000 7f1071b42640 1 -- 172.21.15.46:0/1818247817 shutdown_connections 2024-09-16T04:44:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.749+0000 7f1071b42640 1 -- 172.21.15.46:0/1818247817 wait complete. 2024-09-16T04:44:35.793 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:44:35.793 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T04:44:35.793 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:44:35.793 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T04:44:35.793 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:44:35.794 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:44:35.794 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:35.804 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:44:35.805 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:44:35.805 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:44:35.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.943+0000 7f16cb1d6640 10 monclient: get_monmap_and_config 2024-09-16T04:44:35.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.943+0000 7f16cb1d6640 10 monclient: build_initial_monmap 2024-09-16T04:44:35.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.943+0000 7f16cb1d6640 10 monclient: monmap: 2024-09-16T04:44:35.946 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:35.946 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.946 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:44:35.945215+0000 2024-09-16T04:44:35.946 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:44:35.945215+0000 2024-09-16T04:44:35.946 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:35.946 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.946 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:44:35.947 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:44:35.947 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:44:35.947 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:44:35.947 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:44:35.947 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.944+0000 7f16cb1d6640 1 Processor -- start 2024-09-16T04:44:35.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.944+0000 7f16cb1d6640 1 -- start start 2024-09-16T04:44:35.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.944+0000 7f16cb1d6640 10 monclient: init 2024-09-16T04:44:35.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16cb1d6640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:35.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16cb1d6640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-16T04:44:35.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16cb1d6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f16c40a3c10 0x7f16c409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16cb1d6640 10 monclient(hunting): picked mon.noname-a con 0x7f16c40a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:44:35.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16cb1d6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f16c40a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:44:35.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16cb1d6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16c413be90 addr v1:172.21.15.114:6789/0 2024-09-16T04:44:35.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16cb1d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16c409b940 con 0x7f16c40a3ff0 2024-09-16T04:44:35.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16cb1d6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16c409ab00 con 0x7f16c413be90 2024-09-16T04:44:35.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16cb1d6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16cb1d6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f16c409b030 con 0x7f16c40a3c10 2024-09-16T04:44:35.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16cb1d6640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:35.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16cb1d6640 10 monclient(hunting): authenticate will time out at 1074.361221s 2024-09-16T04:44:35.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16c8f72640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f16c40a3c10 0x7f16c409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16c3fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f16c40a3ff0 0x7f16c40a43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49578/0 (socket says 172.21.15.46:49578) 2024-09-16T04:44:35.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16c8f72640 10 monclient(hunting): get_auth_request con 0x7f16c40a3c10 auth_method 0 2024-09-16T04:44:35.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16c3fff640 1 -- 172.21.15.46:0/3756389664 learned_addr learned my addr 172.21.15.46:0/3756389664 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:44:35.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16c8f72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16c8f72640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16c8f72640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16c8f72640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16c8f72640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16c8f72640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16c8f72640 10 monclient(hunting): handle_auth_done global_id 74172 payload 274 2024-09-16T04:44:35.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16c8f72640 1 -- 172.21.15.46:0/3756389664 >> v1:172.21.15.114:6789/0 conn(0x7f16c413be90 legacy=0x7f16c413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:44:35.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16c8f72640 1 -- 172.21.15.46:0/3756389664 >> v1:172.21.15.46:6789/0 conn(0x7f16c40a3ff0 legacy=0x7f16c40a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16c37fe640 1 -- 172.21.15.46:0/3756389664 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2780144610 0 0) 0x7f16c409b940 con 0x7f16c40a3ff0 2024-09-16T04:44:35.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16c8f72640 10 monclient: _finish_hunting 0 2024-09-16T04:44:35.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16c8f72640 1 monclient: found mon.noname-a 2024-09-16T04:44:35.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16c8f72640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:35.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16c8f72640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:44:35.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16c8f72640 1 -- 172.21.15.46:0/3756389664 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c409b310 con 0x7f16c40a3c10 2024-09-16T04:44:35.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.945+0000 7f16c37fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:44:35.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16c8f72640 1 --2- 172.21.15.46:0/3756389664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16c40a3c10 0x7f16c409bcf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f16b80097a0 tx=0x7f16b802a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=705837d59e141fb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:35.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16c37fe640 1 -- 172.21.15.46:0/3756389664 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b8037070 con 0x7f16c40a3c10 2024-09-16T04:44:35.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16c37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:44:35.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16c37fe640 10 monclient: dump: 2024-09-16T04:44:35.958 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.959 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16c37fe640 10 monclient: _finish_auth 0 2024-09-16T04:44:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16c37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:44:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16cb1d6640 5 monclient: authenticate success, global_id 74172 2024-09-16T04:44:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16c37fe640 1 -- 172.21.15.46:0/3756389664 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16b802adc0 con 0x7f16c40a3c10 2024-09-16T04:44:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16cb1d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:44:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16c37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16cb1d6640 10 monclient: get_monmap_and_config success 2024-09-16T04:44:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16cb1d6640 4 set_mon_vals no callback set 2024-09-16T04:44:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16c37fe640 1 -- 172.21.15.46:0/3756389664 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b8032660 con 0x7f16c40a3c10 2024-09-16T04:44:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16c37fe640 10 monclient: dump: 2024-09-16T04:44:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.961 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.961 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.961 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.961 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.961 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.961 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.961 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.961 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.961 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16cb1d6640 10 monclient: shutdown 2024-09-16T04:44:35.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16cb1d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:35.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16cb1d6640 1 -- 172.21.15.46:0/3756389664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16c40a3c10 msgr2=0x7f16c409bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.946+0000 7f16cb1d6640 1 --2- 172.21.15.46:0/3756389664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16c40a3c10 0x7f16c409bcf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f16b80097a0 tx=0x7f16b802a0b0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.947+0000 7f16cb1d6640 1 -- 172.21.15.46:0/3756389664 shutdown_connections 2024-09-16T04:44:35.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.947+0000 7f16cb1d6640 1 --2- 172.21.15.46:0/3756389664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16c40a3c10 0x7f16c409bcf0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.947+0000 7f16cb1d6640 1 -- 172.21.15.46:0/3756389664 >> 172.21.15.46:0/3756389664 conn(0x7f16c40a6340 msgr2=0x7f16c406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:35.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.947+0000 7f16cb1d6640 1 -- 172.21.15.46:0/3756389664 shutdown_connections 2024-09-16T04:44:35.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.947+0000 7f16cb1d6640 1 -- 172.21.15.46:0/3756389664 wait complete. 2024-09-16T04:44:35.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.947+0000 7f16cb1d6640 10 monclient: build_initial_monmap 2024-09-16T04:44:35.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.947+0000 7f16cb1d6640 10 monclient: monmap: 2024-09-16T04:44:35.962 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:35.962 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:44:35.962 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:44:35.962 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:44:35.963 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:35.963 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.963 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:44:35.963 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:44:35.963 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:44:35.963 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.947+0000 7f16cb1d6640 1 Processor -- start 2024-09-16T04:44:35.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16cb1d6640 1 -- start start 2024-09-16T04:44:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16cb1d6640 10 monclient: init 2024-09-16T04:44:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16cb1d6640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16cb1d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:44:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16cb1d6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16c40a3c10 0x7f16c41cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16cb1d6640 10 monclient(hunting): picked mon.noname-b con 0x7f16c40a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:44:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16cb1d6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f16c40a3ff0 0x7f16c4136e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16cb1d6640 10 monclient(hunting): picked mon.noname-c con 0x7f16c40a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16cb1d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16c413be90 0x7f16c4138e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16cb1d6640 10 monclient(hunting): picked mon.noname-a con 0x7f16c413be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:44:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16cb1d6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16cb1d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f16c40a6b40 con 0x7f16c413be90 2024-09-16T04:44:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16cb1d6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16cb1d6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f16c40a69c0 con 0x7f16c40a3c10 2024-09-16T04:44:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16cb1d6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16cb1d6640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f16c413f9a0 con 0x7f16c40a3ff0 2024-09-16T04:44:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16cb1d6640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16cb1d6640 10 monclient(hunting): authenticate will time out at 1074.364644s 2024-09-16T04:44:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16c3fff640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f16c40a3ff0 msgr2=0x7f16c4136e70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:44:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16c3fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f16c40a3ff0 0x7f16c4136e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16c9773640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16c413be90 0x7f16c4138e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16c9773640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16c413be90 0x7f16c4138e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43284/0 (socket says 172.21.15.46:43284) 2024-09-16T04:44:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16c9773640 1 -- 172.21.15.46:0/2865583544 learned_addr learned my addr 172.21.15.46:0/2865583544 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16c8f72640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16c40a3c10 0x7f16c41cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16c9773640 10 monclient(hunting): get_auth_request con 0x7f16c413be90 auth_method 0 2024-09-16T04:44:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16c9773640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16c9773640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16c9773640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16c8f72640 10 monclient(hunting): get_auth_request con 0x7f16c40a3c10 auth_method 0 2024-09-16T04:44:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16c8f72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16c8f72640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16c8f72640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16c9773640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16c9773640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16c9773640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16c8f72640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16c8f72640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.948+0000 7f16c8f72640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c9773640 10 monclient(hunting): handle_auth_done global_id 74178 payload 824 2024-09-16T04:44:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c9773640 1 -- 172.21.15.46:0/2865583544 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f16c40a3ff0 msgr2=0x7f16c4136e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:44:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c9773640 1 --2- 172.21.15.46:0/2865583544 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f16c40a3ff0 0x7f16c4136e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c9773640 1 -- 172.21.15.46:0/2865583544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16c40a3c10 msgr2=0x7f16c41cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c9773640 1 --2- 172.21.15.46:0/2865583544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16c40a3c10 0x7f16c41cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c9773640 10 monclient: _finish_hunting 0 2024-09-16T04:44:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c9773640 1 monclient: found mon.noname-a 2024-09-16T04:44:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c9773640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c9773640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:44:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c9773640 1 -- 172.21.15.46:0/2865583544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c41207f0 con 0x7f16c413be90 2024-09-16T04:44:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c9773640 1 --2- 172.21.15.46:0/2865583544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16c413be90 0x7f16c4138e40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f16bc00c780 tx=0x7f16bc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c17fa640 1 -- 172.21.15.46:0/2865583544 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc004b10 con 0x7f16c413be90 2024-09-16T04:44:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:44:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c17fa640 10 monclient: dump: 2024-09-16T04:44:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.969 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c17fa640 10 monclient: _finish_auth 0 2024-09-16T04:44:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c17fa640 10 monclient: _check_auth_tickets 2024-09-16T04:44:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16cb1d6640 5 monclient: authenticate success, global_id 74178 2024-09-16T04:44:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c17fa640 1 -- 172.21.15.46:0/2865583544 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16bc004cb0 con 0x7f16c413be90 2024-09-16T04:44:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16cb1d6640 10 monclient: _renew_subs 2024-09-16T04:44:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16cb1d6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16cb1d6640 1 -- 172.21.15.46:0/2865583544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16c40a0960 con 0x7f16c413be90 2024-09-16T04:44:35.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:35.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c17fa640 1 -- 172.21.15.46:0/2865583544 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc00aa90 con 0x7f16c413be90 2024-09-16T04:44:35.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:35.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:35.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c17fa640 10 monclient: dump: 2024-09-16T04:44:35.970 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:35.970 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:35.970 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.970 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:35.970 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:35.970 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:35.970 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:35.971 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:35.971 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:35.971 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:35.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16cb1d6640 10 monclient: _renew_subs 2024-09-16T04:44:35.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16c27fc640 4 set_mon_vals no callback set 2024-09-16T04:44:35.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16cb1d6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:35.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.949+0000 7f16cb1d6640 1 -- 172.21.15.46:0/2865583544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16c4120230 con 0x7f16c413be90 2024-09-16T04:44:35.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.950+0000 7f16c17fa640 1 -- 172.21.15.46:0/2865583544 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f16bc00ac30 con 0x7f16c413be90 2024-09-16T04:44:35.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.950+0000 7f16cb1d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:35.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.950+0000 7f16cb1d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:35.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.950+0000 7f16cb1d6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:35.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.950+0000 7f16cb1d6640 1 -- 172.21.15.46:0/2865583544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16c409bd70 con 0x7f16c413be90 2024-09-16T04:44:35.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.950+0000 7f16c17fa640 1 --2- 172.21.15.46:0/2865583544 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f16a8038360 0x7f16a803a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:35.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.950+0000 7f16c17fa640 1 -- 172.21.15.46:0/2865583544 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 4738+0+0 (secure 0 0 0) 0x7f16bc04b850 con 0x7f16c413be90 2024-09-16T04:44:35.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.950+0000 7f16c8f72640 1 --2- 172.21.15.46:0/2865583544 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f16a8038360 0x7f16a803a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:35.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.950+0000 7f16c8f72640 10 monclient: get_auth_request con 0x7f16a8038360 auth_method 0 2024-09-16T04:44:35.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.951+0000 7f16c8f72640 1 --2- 172.21.15.46:0/2865583544 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f16a8038360 0x7f16a803a820 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f16b8034000 tx=0x7f16b802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:35.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.955+0000 7f16c17fa640 1 -- 172.21.15.46:0/2865583544 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f16bc020070 con 0x7f16c413be90 2024-09-16T04:44:35.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.955+0000 7f16c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:35.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:35.955+0000 7f16c17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:44:36.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:36.205+0000 7f16cb1d6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:44:36.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:36.205+0000 7f16cb1d6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:44:36.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:36.205+0000 7f16cb1d6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:36.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:36.205+0000 7f16cb1d6640 1 -- 172.21.15.46:0/2865583544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f16c413b3c0 con 0x7f16c413be90 2024-09-16T04:44:36.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:36.205+0000 7f16c17fa640 1 -- 172.21.15.46:0/2865583544 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f16bc011540 con 0x7f16c413be90 2024-09-16T04:44:36.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:36.205+0000 7f16c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:44:36.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:36.205+0000 7f16c17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:44:36.210 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:44:36.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:36.207+0000 7f16cb1d6640 1 -- 172.21.15.46:0/2865583544 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f16a8038360 msgr2=0x7f16a803a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:36.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:36.207+0000 7f16cb1d6640 1 --2- 172.21.15.46:0/2865583544 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f16a8038360 0x7f16a803a820 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f16b8034000 tx=0x7f16b802e000 comp rx=0 tx=0).stop 2024-09-16T04:44:36.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:36.208+0000 7f16cb1d6640 10 monclient: shutdown 2024-09-16T04:44:36.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:36.208+0000 7f16cb1d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:36.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:36.208+0000 7f16cb1d6640 1 -- 172.21.15.46:0/2865583544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16c413be90 msgr2=0x7f16c4138e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:36.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:36.208+0000 7f16cb1d6640 1 --2- 172.21.15.46:0/2865583544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16c413be90 0x7f16c4138e40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f16bc00c780 tx=0x7f16bc00cc40 comp rx=0 tx=0).stop 2024-09-16T04:44:36.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:36.208+0000 7f16cb1d6640 1 -- 172.21.15.46:0/2865583544 shutdown_connections 2024-09-16T04:44:36.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:36.208+0000 7f16cb1d6640 1 --2- 172.21.15.46:0/2865583544 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f16a8038360 0x7f16a803a820 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:36.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:36.208+0000 7f16cb1d6640 1 --2- 172.21.15.46:0/2865583544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16c413be90 0x7f16c4138e40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:36.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:36.208+0000 7f16cb1d6640 1 --2- 172.21.15.46:0/2865583544 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f16c40a3ff0 0x7f16c4136e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:36.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:36.208+0000 7f16cb1d6640 1 --2- 172.21.15.46:0/2865583544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16c40a3c10 0x7f16c41cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:36.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:36.208+0000 7f16cb1d6640 1 -- 172.21.15.46:0/2865583544 >> 172.21.15.46:0/2865583544 conn(0x7f16c40a6340 msgr2=0x7f16c40a27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:36.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:36.208+0000 7f16cb1d6640 1 -- 172.21.15.46:0/2865583544 shutdown_connections 2024-09-16T04:44:36.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:36.208+0000 7f16cb1d6640 1 -- 172.21.15.46:0/2865583544 wait complete. 2024-09-16T04:44:39.221 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:44:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.333+0000 7fbc0c3cf640 10 monclient: get_monmap_and_config 2024-09-16T04:44:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.333+0000 7fbc0c3cf640 10 monclient: build_initial_monmap 2024-09-16T04:44:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.333+0000 7fbc0c3cf640 10 monclient: monmap: 2024-09-16T04:44:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:44:39.334961+0000 2024-09-16T04:44:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:44:39.334961+0000 2024-09-16T04:44:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:44:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:44:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:44:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:44:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:44:39.336 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:39.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.334+0000 7fbc0c3cf640 1 Processor -- start 2024-09-16T04:44:39.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.334+0000 7fbc0c3cf640 1 -- start start 2024-09-16T04:44:39.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.334+0000 7fbc0c3cf640 10 monclient: init 2024-09-16T04:44:39.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.334+0000 7fbc0c3cf640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:39.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.334+0000 7fbc0c3cf640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-16T04:44:39.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.334+0000 7fbc0c3cf640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbc0406e160 0x7fbc040a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:39.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.334+0000 7fbc0c3cf640 10 monclient(hunting): picked mon.noname-a con 0x7fbc0406e160 addr v2:172.21.15.46:3300/0 2024-09-16T04:44:39.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.334+0000 7fbc0c3cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc040a2ee0 addr v1:172.21.15.46:6789/0 2024-09-16T04:44:39.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.334+0000 7fbc0c3cf640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbc04137050 0x7fbc0413ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:39.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.334+0000 7fbc0c3cf640 10 monclient(hunting): picked mon.noname-c con 0x7fbc04137050 addr v2:172.21.15.114:3300/0 2024-09-16T04:44:39.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.334+0000 7fbc0c3cf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc040a3fd0 con 0x7fbc040a2ee0 2024-09-16T04:44:39.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.334+0000 7fbc0c3cf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:39.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.334+0000 7fbc0c3cf640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fbc040a3300 con 0x7fbc0406e160 2024-09-16T04:44:39.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.334+0000 7fbc0c3cf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:39.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.334+0000 7fbc0c3cf640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fbc0406e540 con 0x7fbc04137050 2024-09-16T04:44:39.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.334+0000 7fbc0c3cf640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:39.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.334+0000 7fbc0c3cf640 10 monclient(hunting): authenticate will time out at 1077.750893s 2024-09-16T04:44:39.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a16b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbc0406e160 0x7fbc040a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:39.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a16b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbc0406e160 0x7fbc040a29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43294/0 (socket says 172.21.15.46:43294) 2024-09-16T04:44:39.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a16b640 1 -- 172.21.15.46:0/2957589900 learned_addr learned my addr 172.21.15.46:0/2957589900 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:39.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a16b640 10 monclient(hunting): get_auth_request con 0x7fbc0406e160 auth_method 0 2024-09-16T04:44:39.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a16b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:39.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a16b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:39.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a16b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:39.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a16b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:39.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a16b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:39.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a96c640 1 --2- 172.21.15.46:0/2957589900 >> v2:172.21.15.114:3300/0 conn(0x7fbc04137050 0x7fbc0413ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:39.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a16b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:39.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a96c640 10 monclient(hunting): get_auth_request con 0x7fbc04137050 auth_method 0 2024-09-16T04:44:39.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a96c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:39.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a96c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:39.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a96c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:39.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a96c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:39.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a96c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:39.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a96c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a16b640 10 monclient(hunting): handle_auth_done global_id 74181 payload 274 2024-09-16T04:44:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a16b640 1 -- 172.21.15.46:0/2957589900 >> v2:172.21.15.114:3300/0 conn(0x7fbc04137050 msgr2=0x7fbc0413ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:44:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a16b640 1 --2- 172.21.15.46:0/2957589900 >> v2:172.21.15.114:3300/0 conn(0x7fbc04137050 0x7fbc0413ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a16b640 1 -- 172.21.15.46:0/2957589900 >> v1:172.21.15.46:6789/0 conn(0x7fbc040a2ee0 legacy=0x7fbc04136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a16b640 10 monclient: _finish_hunting 0 2024-09-16T04:44:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a16b640 1 monclient: found mon.noname-a 2024-09-16T04:44:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a16b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a16b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:44:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc09169640 1 -- 172.21.15.46:0/2957589900 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3537063884 0 0) 0x7fbbf8001110 con 0x7fbc040a2ee0 2024-09-16T04:44:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc0a16b640 1 -- 172.21.15.46:0/2957589900 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc040a0bd0 con 0x7fbc0406e160 2024-09-16T04:44:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.335+0000 7fbc09169640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:44:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc0a16b640 1 --2- 172.21.15.46:0/2957589900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc0406e160 0x7fbc040a29a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbc04067c50 tx=0x7fbc0002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd3ef3c86af5e75c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc09169640 1 -- 172.21.15.46:0/2957589900 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc00037070 con 0x7fbc0406e160 2024-09-16T04:44:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc09169640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc09169640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:44:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc09169640 10 monclient: dump: 2024-09-16T04:44:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:39.350 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc09169640 10 monclient: _finish_auth 0 2024-09-16T04:44:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc09169640 10 monclient: _check_auth_tickets 2024-09-16T04:44:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc09169640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc09169640 1 -- 172.21.15.46:0/2957589900 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc00032440 con 0x7fbc0406e160 2024-09-16T04:44:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc0c3cf640 5 monclient: authenticate success, global_id 74181 2024-09-16T04:44:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc0c3cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:44:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc09169640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc0c3cf640 10 monclient: get_monmap_and_config success 2024-09-16T04:44:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc0c3cf640 4 set_mon_vals no callback set 2024-09-16T04:44:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc09169640 1 -- 172.21.15.46:0/2957589900 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc00032810 con 0x7fbc0406e160 2024-09-16T04:44:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc09169640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc09169640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc09169640 10 monclient: dump: 2024-09-16T04:44:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:39.352 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc0c3cf640 10 monclient: shutdown 2024-09-16T04:44:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc0c3cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc0c3cf640 1 -- 172.21.15.46:0/2957589900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc0406e160 msgr2=0x7fbc040a29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.336+0000 7fbc0c3cf640 1 --2- 172.21.15.46:0/2957589900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc0406e160 0x7fbc040a29a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbc04067c50 tx=0x7fbc0002a180 comp rx=0 tx=0).stop 2024-09-16T04:44:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.337+0000 7fbc0c3cf640 1 -- 172.21.15.46:0/2957589900 shutdown_connections 2024-09-16T04:44:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.337+0000 7fbc0c3cf640 1 --2- 172.21.15.46:0/2957589900 >> v2:172.21.15.114:3300/0 conn(0x7fbc04137050 0x7fbc0413ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.337+0000 7fbc0c3cf640 1 --2- 172.21.15.46:0/2957589900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc0406e160 0x7fbc040a29a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.337+0000 7fbc0c3cf640 1 -- 172.21.15.46:0/2957589900 >> 172.21.15.46:0/2957589900 conn(0x7fbc040a5c10 msgr2=0x7fbc040a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.337+0000 7fbc0c3cf640 1 -- 172.21.15.46:0/2957589900 shutdown_connections 2024-09-16T04:44:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.337+0000 7fbc0c3cf640 1 -- 172.21.15.46:0/2957589900 wait complete. 2024-09-16T04:44:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.337+0000 7fbc0c3cf640 10 monclient: build_initial_monmap 2024-09-16T04:44:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.337+0000 7fbc0c3cf640 10 monclient: monmap: 2024-09-16T04:44:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:44:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:44:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:44:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:44:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:44:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:44:39.353 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:39.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.337+0000 7fbc0c3cf640 1 Processor -- start 2024-09-16T04:44:39.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0c3cf640 1 -- start start 2024-09-16T04:44:39.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0c3cf640 10 monclient: init 2024-09-16T04:44:39.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0c3cf640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:39.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0c3cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:44:39.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0c3cf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc0406e160 0x7fbc04135ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:39.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0c3cf640 10 monclient(hunting): picked mon.noname-b con 0x7fbc0406e160 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:44:39.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0c3cf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc040a2ee0 0x7fbc04135ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:39.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0c3cf640 10 monclient(hunting): picked mon.noname-a con 0x7fbc040a2ee0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:44:39.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0c3cf640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fbc04137050 0x7fbc0411f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:39.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0c3cf640 10 monclient(hunting): picked mon.noname-c con 0x7fbc04137050 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:39.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0c3cf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:39.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0c3cf640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbc040a34a0 con 0x7fbc040a2ee0 2024-09-16T04:44:39.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0c3cf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:39.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0c3cf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbc040a5130 con 0x7fbc0406e160 2024-09-16T04:44:39.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0c3cf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:39.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0c3cf640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fbc04123470 con 0x7fbc04137050 2024-09-16T04:44:39.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0c3cf640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:39.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0c3cf640 10 monclient(hunting): authenticate will time out at 1077.754531s 2024-09-16T04:44:39.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0a96c640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fbc04137050 msgr2=0x7fbc0411f760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:44:39.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0a96c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fbc04137050 0x7fbc0411f760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:39.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0996a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc040a2ee0 0x7fbc04135ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:39.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0996a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc040a2ee0 0x7fbc04135ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43300/0 (socket says 172.21.15.46:43300) 2024-09-16T04:44:39.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0996a640 1 -- 172.21.15.46:0/3863463655 learned_addr learned my addr 172.21.15.46:0/3863463655 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:39.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0996a640 10 monclient(hunting): get_auth_request con 0x7fbc040a2ee0 auth_method 0 2024-09-16T04:44:39.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0996a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:39.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0996a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:39.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0996a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:39.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0a16b640 1 --2- 172.21.15.46:0/3863463655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc0406e160 0x7fbc04135ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:39.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0a16b640 10 monclient(hunting): get_auth_request con 0x7fbc0406e160 auth_method 0 2024-09-16T04:44:39.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0a16b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:39.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0a16b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:39.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0a16b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:39.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0996a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:39.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0996a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:39.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0996a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:39.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0a16b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:39.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0a16b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:39.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0a16b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:39.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0996a640 10 monclient(hunting): handle_auth_done global_id 74187 payload 824 2024-09-16T04:44:39.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0996a640 1 -- 172.21.15.46:0/3863463655 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fbc04137050 msgr2=0x7fbc0411f760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:44:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0996a640 1 --2- 172.21.15.46:0/3863463655 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fbc04137050 0x7fbc0411f760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0996a640 1 -- 172.21.15.46:0/3863463655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc0406e160 msgr2=0x7fbc04135ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0996a640 1 --2- 172.21.15.46:0/3863463655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc0406e160 0x7fbc04135ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0996a640 10 monclient: _finish_hunting 0 2024-09-16T04:44:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0996a640 1 monclient: found mon.noname-a 2024-09-16T04:44:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0996a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0996a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:44:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.338+0000 7fbc0996a640 1 -- 172.21.15.46:0/3863463655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc040a8190 con 0x7fbc040a2ee0 2024-09-16T04:44:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbc0996a640 1 --2- 172.21.15.46:0/3863463655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc040a2ee0 0x7fbc04135ff0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbbf800c780 tx=0x7fbbf800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbbf6ffd640 1 -- 172.21.15.46:0/3863463655 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8017070 con 0x7fbc040a2ee0 2024-09-16T04:44:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbbf6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbbf6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:44:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbbf6ffd640 10 monclient: dump: 2024-09-16T04:44:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:39.358 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbbf6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:44:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbbf6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:44:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbbf6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbbf6ffd640 1 -- 172.21.15.46:0/3863463655 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbf80052b0 con 0x7fbc040a2ee0 2024-09-16T04:44:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbc0c3cf640 5 monclient: authenticate success, global_id 74187 2024-09-16T04:44:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbc0c3cf640 10 monclient: _renew_subs 2024-09-16T04:44:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbc0c3cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbc0c3cf640 1 -- 172.21.15.46:0/3863463655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc04122eb0 con 0x7fbc040a2ee0 2024-09-16T04:44:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbbf6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbbf6ffd640 1 -- 172.21.15.46:0/3863463655 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf800a880 con 0x7fbc040a2ee0 2024-09-16T04:44:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbbf6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbbf6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbbf6ffd640 10 monclient: dump: 2024-09-16T04:44:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:39.360 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbc0c3cf640 10 monclient: _renew_subs 2024-09-16T04:44:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbc0c3cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbc0c3cf640 1 -- 172.21.15.46:0/3863463655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc040a0300 con 0x7fbc040a2ee0 2024-09-16T04:44:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.339+0000 7fbbf7fff640 4 set_mon_vals no callback set 2024-09-16T04:44:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.340+0000 7fbbf6ffd640 1 -- 172.21.15.46:0/3863463655 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fbbf801a070 con 0x7fbc040a2ee0 2024-09-16T04:44:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.340+0000 7fbbf6ffd640 1 --2- 172.21.15.46:0/3863463655 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fbbe8038390 0x7fbbe803a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.340+0000 7fbbf6ffd640 1 -- 172.21.15.46:0/3863463655 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 4738+0+0 (secure 0 0 0) 0x7fbbf804aa10 con 0x7fbc040a2ee0 2024-09-16T04:44:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.340+0000 7fbc0c3cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.340+0000 7fbc0c3cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.340+0000 7fbc0c3cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:39.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.340+0000 7fbc0c3cf640 1 -- 172.21.15.46:0/3863463655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc04121c70 con 0x7fbc040a2ee0 2024-09-16T04:44:39.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.340+0000 7fbc0a16b640 1 --2- 172.21.15.46:0/3863463655 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fbbe8038390 0x7fbbe803a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:39.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.340+0000 7fbc0a16b640 10 monclient: get_auth_request con 0x7fbbe8038390 auth_method 0 2024-09-16T04:44:39.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.341+0000 7fbc0a16b640 1 --2- 172.21.15.46:0/3863463655 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fbbe8038390 0x7fbbe803a850 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbc00034000 tx=0x7fbc0002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:39.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.345+0000 7fbbf6ffd640 1 -- 172.21.15.46:0/3863463655 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fbc040a8190 con 0x7fbc040a2ee0 2024-09-16T04:44:39.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.345+0000 7fbbf6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:39.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.345+0000 7fbbf6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:44:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.595+0000 7fbc0c3cf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:44:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.595+0000 7fbc0c3cf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:44:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.595+0000 7fbc0c3cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.595+0000 7fbc0c3cf640 1 -- 172.21.15.46:0/3863463655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbc041374a0 con 0x7fbc040a2ee0 2024-09-16T04:44:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.595+0000 7fbbf6ffd640 1 -- 172.21.15.46:0/3863463655 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fbc040a3fd0 con 0x7fbc040a2ee0 2024-09-16T04:44:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.595+0000 7fbbf6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:44:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.595+0000 7fbbf6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:44:39.598 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:44:39.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.598+0000 7fbc0c3cf640 1 -- 172.21.15.46:0/3863463655 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fbbe8038390 msgr2=0x7fbbe803a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:39.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.598+0000 7fbc0c3cf640 1 --2- 172.21.15.46:0/3863463655 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fbbe8038390 0x7fbbe803a850 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbc00034000 tx=0x7fbc0002e000 comp rx=0 tx=0).stop 2024-09-16T04:44:39.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.598+0000 7fbc0c3cf640 10 monclient: shutdown 2024-09-16T04:44:39.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.598+0000 7fbc0c3cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:39.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.598+0000 7fbc0c3cf640 1 -- 172.21.15.46:0/3863463655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc040a2ee0 msgr2=0x7fbc04135ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:39.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.598+0000 7fbc0c3cf640 1 --2- 172.21.15.46:0/3863463655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc040a2ee0 0x7fbc04135ff0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbbf800c780 tx=0x7fbbf800cc40 comp rx=0 tx=0).stop 2024-09-16T04:44:39.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.598+0000 7fbc0c3cf640 1 -- 172.21.15.46:0/3863463655 shutdown_connections 2024-09-16T04:44:39.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.598+0000 7fbc0c3cf640 1 --2- 172.21.15.46:0/3863463655 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fbc04137050 0x7fbc0411f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:39.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.598+0000 7fbc0c3cf640 1 --2- 172.21.15.46:0/3863463655 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fbbe8038390 0x7fbbe803a850 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:39.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.598+0000 7fbc0c3cf640 1 --2- 172.21.15.46:0/3863463655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc040a2ee0 0x7fbc04135ff0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:39.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.598+0000 7fbc0c3cf640 1 --2- 172.21.15.46:0/3863463655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc0406e160 0x7fbc04135ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:39.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.598+0000 7fbc0c3cf640 1 -- 172.21.15.46:0/3863463655 >> 172.21.15.46:0/3863463655 conn(0x7fbc040a5c10 msgr2=0x7fbc040a8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:39.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.598+0000 7fbc0c3cf640 1 -- 172.21.15.46:0/3863463655 shutdown_connections 2024-09-16T04:44:39.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:39.598+0000 7fbc0c3cf640 1 -- 172.21.15.46:0/3863463655 wait complete. 2024-09-16T04:44:42.612 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:44:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.723+0000 7f9c0e378640 10 monclient: get_monmap_and_config 2024-09-16T04:44:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.723+0000 7f9c0e378640 10 monclient: build_initial_monmap 2024-09-16T04:44:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.723+0000 7f9c0e378640 10 monclient: monmap: 2024-09-16T04:44:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:44:42.725189+0000 2024-09-16T04:44:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:44:42.725189+0000 2024-09-16T04:44:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:44:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:44:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:44:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:44:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:44:42.726 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.724+0000 7f9c0e378640 1 Processor -- start 2024-09-16T04:44:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.724+0000 7f9c0e378640 1 -- start start 2024-09-16T04:44:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.724+0000 7f9c0e378640 10 monclient: init 2024-09-16T04:44:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c0e378640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c0e378640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-16T04:44:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c0e378640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9c08131a10 0x7f9c0809ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c0e378640 10 monclient(hunting): picked mon.noname-a con 0x7f9c08131a10 addr v2:172.21.15.46:3300/0 2024-09-16T04:44:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c0e378640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9c0809e760 0x7f9c08131df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c0e378640 10 monclient(hunting): picked mon.noname-b con 0x7f9c0809e760 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c0e378640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9c081347d0 0x7f9c08134c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c0e378640 10 monclient(hunting): picked mon.noname-c con 0x7f9c081347d0 addr v2:172.21.15.114:3300/0 2024-09-16T04:44:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c0e378640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c0e378640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f9c080a0f40 con 0x7f9c08131a10 2024-09-16T04:44:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c0e378640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c0e378640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f9c080a1220 con 0x7f9c081347d0 2024-09-16T04:44:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c0e378640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c0e378640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f9c08132330 con 0x7f9c0809e760 2024-09-16T04:44:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c0e378640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c0e378640 10 monclient(hunting): authenticate will time out at 1081.141140s 2024-09-16T04:44:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c077fe640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9c0809e760 msgr2=0x7f9c08131df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:44:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c077fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9c0809e760 0x7f9c08131df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c07fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9c08131a10 0x7f9c0809ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c07fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9c08131a10 0x7f9c0809ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43304/0 (socket says 172.21.15.46:43304) 2024-09-16T04:44:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c07fff640 1 -- 172.21.15.46:0/2929422824 learned_addr learned my addr 172.21.15.46:0/2929422824 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c07fff640 10 monclient(hunting): get_auth_request con 0x7f9c08131a10 auth_method 0 2024-09-16T04:44:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c07fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:42.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:42.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:42.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:42.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:42.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c0c915640 1 --2- 172.21.15.46:0/2929422824 >> v2:172.21.15.114:3300/0 conn(0x7f9c081347d0 0x7f9c08134c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:42.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c0c915640 10 monclient(hunting): get_auth_request con 0x7f9c081347d0 auth_method 0 2024-09-16T04:44:42.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c0c915640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:42.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c0c915640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:42.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c0c915640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:42.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c07fff640 10 monclient(hunting): handle_auth_done global_id 74190 payload 274 2024-09-16T04:44:42.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c07fff640 1 -- 172.21.15.46:0/2929422824 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9c0809e760 msgr2=0x7f9c08131df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:44:42.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c07fff640 1 --2- 172.21.15.46:0/2929422824 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9c0809e760 0x7f9c08131df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:42.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c07fff640 1 -- 172.21.15.46:0/2929422824 >> v2:172.21.15.114:3300/0 conn(0x7f9c081347d0 msgr2=0x7f9c08134c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:44:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c07fff640 1 --2- 172.21.15.46:0/2929422824 >> v2:172.21.15.114:3300/0 conn(0x7f9c081347d0 0x7f9c08134c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c07fff640 10 monclient: _finish_hunting 0 2024-09-16T04:44:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c07fff640 1 monclient: found mon.noname-a 2024-09-16T04:44:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c07fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:44:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.725+0000 7f9c07fff640 1 -- 172.21.15.46:0/2929422824 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c0809f810 con 0x7f9c08131a10 2024-09-16T04:44:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.726+0000 7f9c07fff640 1 --2- 172.21.15.46:0/2929422824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c08131a10 0x7f9c0809ec70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9c080a2cc0 tx=0x7f9bf802a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97a32cb7d50f7c46 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.726+0000 7f9c06ffd640 1 -- 172.21.15.46:0/2929422824 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf8037070 con 0x7f9c08131a10 2024-09-16T04:44:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.726+0000 7f9c06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.726+0000 7f9c06ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:44:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.726+0000 7f9c06ffd640 10 monclient: dump: 2024-09-16T04:44:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:42.739 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:42.739 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:42.739 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:42.739 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:42.739 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:42.739 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:42.739 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:42.739 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:42.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.726+0000 7f9c06ffd640 10 monclient: _finish_auth 0 2024-09-16T04:44:42.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.726+0000 7f9c06ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:44:42.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.726+0000 7f9c06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:42.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.726+0000 7f9c0e378640 5 monclient: authenticate success, global_id 74190 2024-09-16T04:44:42.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.726+0000 7f9c06ffd640 1 -- 172.21.15.46:0/2929422824 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bf802ae20 con 0x7f9c08131a10 2024-09-16T04:44:42.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.726+0000 7f9c0e378640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:44:42.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.726+0000 7f9c06ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:42.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.726+0000 7f9c0e378640 10 monclient: get_monmap_and_config success 2024-09-16T04:44:42.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.726+0000 7f9c0e378640 4 set_mon_vals no callback set 2024-09-16T04:44:42.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.726+0000 7f9c06ffd640 1 -- 172.21.15.46:0/2929422824 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf8032660 con 0x7f9c08131a10 2024-09-16T04:44:42.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.726+0000 7f9c06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:42.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.726+0000 7f9c06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:42.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.726+0000 7f9c06ffd640 10 monclient: dump: 2024-09-16T04:44:42.740 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:42.740 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:42.740 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:42.740 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:42.740 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:42.740 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:42.740 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:42.741 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.727+0000 7f9c0e378640 10 monclient: shutdown 2024-09-16T04:44:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.727+0000 7f9c0e378640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.727+0000 7f9c0e378640 1 -- 172.21.15.46:0/2929422824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c08131a10 msgr2=0x7f9c0809ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.727+0000 7f9c0e378640 1 --2- 172.21.15.46:0/2929422824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c08131a10 0x7f9c0809ec70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9c080a2cc0 tx=0x7f9bf802a0d0 comp rx=0 tx=0).stop 2024-09-16T04:44:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.727+0000 7f9c0e378640 1 -- 172.21.15.46:0/2929422824 shutdown_connections 2024-09-16T04:44:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.727+0000 7f9c0e378640 1 --2- 172.21.15.46:0/2929422824 >> v2:172.21.15.114:3300/0 conn(0x7f9c081347d0 0x7f9c08134c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.727+0000 7f9c0e378640 1 --2- 172.21.15.46:0/2929422824 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9c0809e760 0x7f9c08131df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.727+0000 7f9c0e378640 1 --2- 172.21.15.46:0/2929422824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c08131a10 0x7f9c0809ec70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.727+0000 7f9c0e378640 1 -- 172.21.15.46:0/2929422824 >> 172.21.15.46:0/2929422824 conn(0x7f9c080a5960 msgr2=0x7f9c080a1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:42.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.727+0000 7f9c0e378640 1 -- 172.21.15.46:0/2929422824 shutdown_connections 2024-09-16T04:44:42.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.727+0000 7f9c0e378640 1 -- 172.21.15.46:0/2929422824 wait complete. 2024-09-16T04:44:42.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.727+0000 7f9c0e378640 10 monclient: build_initial_monmap 2024-09-16T04:44:42.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.727+0000 7f9c0e378640 10 monclient: monmap: 2024-09-16T04:44:42.742 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:42.742 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:44:42.742 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:44:42.742 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:44:42.742 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:42.742 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:42.742 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:44:42.742 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:44:42.742 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:44:42.742 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:42.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c0e378640 1 Processor -- start 2024-09-16T04:44:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c0e378640 1 -- start start 2024-09-16T04:44:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c0e378640 10 monclient: init 2024-09-16T04:44:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c0e378640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c0e378640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:44:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c0e378640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c0809e760 0x7f9c081c9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c0e378640 10 monclient(hunting): picked mon.noname-b con 0x7f9c0809e760 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:44:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c0e378640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c08131a10 0x7f9c081308e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c0e378640 10 monclient(hunting): picked mon.noname-a con 0x7f9c08131a10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:44:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c0e378640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9c081347d0 0x7f9c0811e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c0e378640 10 monclient(hunting): picked mon.noname-c con 0x7f9c081347d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c0e378640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c0e378640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9c08135320 con 0x7f9c08131a10 2024-09-16T04:44:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c0e378640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c0e378640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9c081351a0 con 0x7f9c0809e760 2024-09-16T04:44:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c0e378640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c0e378640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f9c081354a0 con 0x7f9c081347d0 2024-09-16T04:44:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c0e378640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c0e378640 10 monclient(hunting): authenticate will time out at 1081.144888s 2024-09-16T04:44:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c0c915640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9c081347d0 msgr2=0x7f9c0811e350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:44:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c0c915640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9c081347d0 0x7f9c0811e350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.728+0000 7f9c077fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c08131a10 0x7f9c081308e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c077fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c08131a10 0x7f9c081308e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43308/0 (socket says 172.21.15.46:43308) 2024-09-16T04:44:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c077fe640 1 -- 172.21.15.46:0/992879555 learned_addr learned my addr 172.21.15.46:0/992879555 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c077fe640 10 monclient(hunting): get_auth_request con 0x7f9c08131a10 auth_method 0 2024-09-16T04:44:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c07fff640 1 --2- 172.21.15.46:0/992879555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c0809e760 0x7f9c081c9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c07fff640 10 monclient(hunting): get_auth_request con 0x7f9c0809e760 auth_method 0 2024-09-16T04:44:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c07fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c077fe640 10 monclient(hunting): handle_auth_done global_id 74193 payload 824 2024-09-16T04:44:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c077fe640 1 -- 172.21.15.46:0/992879555 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9c081347d0 msgr2=0x7f9c0811e350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:44:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c077fe640 1 --2- 172.21.15.46:0/992879555 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9c081347d0 0x7f9c0811e350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c077fe640 1 -- 172.21.15.46:0/992879555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c0809e760 msgr2=0x7f9c081c9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c077fe640 1 --2- 172.21.15.46:0/992879555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c0809e760 0x7f9c081c9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c077fe640 10 monclient: _finish_hunting 0 2024-09-16T04:44:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c077fe640 1 monclient: found mon.noname-a 2024-09-16T04:44:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c077fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:44:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c077fe640 1 -- 172.21.15.46:0/992879555 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c081222e0 con 0x7f9c08131a10 2024-09-16T04:44:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c077fe640 1 --2- 172.21.15.46:0/992879555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c08131a10 0x7f9c081308e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9bf000b550 tx=0x7f9bf000ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c04ff9640 1 -- 172.21.15.46:0/992879555 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0003bc0 con 0x7f9c08131a10 2024-09-16T04:44:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c04ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:44:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c04ff9640 10 monclient: dump: 2024-09-16T04:44:42.747 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:42.747 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:42.747 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:42.747 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:42.747 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:42.747 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:42.747 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:42.747 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:42.747 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:42.747 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:42.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c04ff9640 10 monclient: _finish_auth 0 2024-09-16T04:44:42.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c04ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:44:42.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c04ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c04ff9640 1 -- 172.21.15.46:0/992879555 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bf0015070 con 0x7f9c08131a10 2024-09-16T04:44:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c0e378640 5 monclient: authenticate success, global_id 74193 2024-09-16T04:44:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c04ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c04ff9640 1 -- 172.21.15.46:0/992879555 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0005070 con 0x7f9c08131a10 2024-09-16T04:44:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c04ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c04ff9640 10 monclient: dump: 2024-09-16T04:44:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:42.749 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:42.749 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:42.749 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:42.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c0e378640 10 monclient: _renew_subs 2024-09-16T04:44:42.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c0e378640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:42.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c0e378640 1 -- 172.21.15.46:0/992879555 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c0809c100 con 0x7f9c08131a10 2024-09-16T04:44:42.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c067fc640 4 set_mon_vals no callback set 2024-09-16T04:44:42.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c0e378640 10 monclient: _renew_subs 2024-09-16T04:44:42.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c0e378640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:42.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.729+0000 7f9c0e378640 1 -- 172.21.15.46:0/992879555 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c08122000 con 0x7f9c08131a10 2024-09-16T04:44:42.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.730+0000 7f9c04ff9640 1 -- 172.21.15.46:0/992879555 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f9bf0005210 con 0x7f9c08131a10 2024-09-16T04:44:42.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.730+0000 7f9c0e378640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:42.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.730+0000 7f9c0e378640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.730+0000 7f9c0e378640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.730+0000 7f9c0e378640 1 -- 172.21.15.46:0/992879555 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c0809ec70 con 0x7f9c08131a10 2024-09-16T04:44:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.730+0000 7f9c04ff9640 1 --2- 172.21.15.46:0/992879555 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9be8038390 0x7f9be803a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.730+0000 7f9c04ff9640 1 -- 172.21.15.46:0/992879555 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 4738+0+0 (secure 0 0 0) 0x7f9bf004b060 con 0x7f9c08131a10 2024-09-16T04:44:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.731+0000 7f9c07fff640 1 --2- 172.21.15.46:0/992879555 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9be8038390 0x7f9be803a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.731+0000 7f9c07fff640 10 monclient: get_auth_request con 0x7f9be8038390 auth_method 0 2024-09-16T04:44:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.731+0000 7f9c07fff640 1 --2- 172.21.15.46:0/992879555 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9be8038390 0x7f9be803a850 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8034000 tx=0x7f9bf802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.735+0000 7f9c04ff9640 1 -- 172.21.15.46:0/992879555 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9bf0005790 con 0x7f9c08131a10 2024-09-16T04:44:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.735+0000 7f9c04ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.735+0000 7f9c04ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:44:42.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.979+0000 7f9c0e378640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:44:42.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.979+0000 7f9c0e378640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:44:42.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.979+0000 7f9c0e378640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:42.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.979+0000 7f9c0e378640 1 -- 172.21.15.46:0/992879555 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9c08134330 con 0x7f9c08131a10 2024-09-16T04:44:42.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.979+0000 7f9c04ff9640 1 -- 172.21.15.46:0/992879555 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f9bf001a980 con 0x7f9c08131a10 2024-09-16T04:44:42.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.979+0000 7f9c04ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:44:42.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.979+0000 7f9c04ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:44:42.982 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:44:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.981+0000 7f9c0e378640 1 -- 172.21.15.46:0/992879555 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9be8038390 msgr2=0x7f9be803a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.981+0000 7f9c0e378640 1 --2- 172.21.15.46:0/992879555 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9be8038390 0x7f9be803a850 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8034000 tx=0x7f9bf802e000 comp rx=0 tx=0).stop 2024-09-16T04:44:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.981+0000 7f9c0e378640 10 monclient: shutdown 2024-09-16T04:44:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.981+0000 7f9c0e378640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.981+0000 7f9c0e378640 1 -- 172.21.15.46:0/992879555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c08131a10 msgr2=0x7f9c081308e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.981+0000 7f9c0e378640 1 --2- 172.21.15.46:0/992879555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c08131a10 0x7f9c081308e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9bf000b550 tx=0x7f9bf000ba10 comp rx=0 tx=0).stop 2024-09-16T04:44:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.981+0000 7f9c0e378640 1 -- 172.21.15.46:0/992879555 shutdown_connections 2024-09-16T04:44:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.981+0000 7f9c0e378640 1 --2- 172.21.15.46:0/992879555 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f9c081347d0 0x7f9c0811e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.981+0000 7f9c0e378640 1 --2- 172.21.15.46:0/992879555 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9be8038390 0x7f9be803a850 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.981+0000 7f9c0e378640 1 --2- 172.21.15.46:0/992879555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c08131a10 0x7f9c081308e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.981+0000 7f9c0e378640 1 --2- 172.21.15.46:0/992879555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c0809e760 0x7f9c081c9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.981+0000 7f9c0e378640 1 -- 172.21.15.46:0/992879555 >> 172.21.15.46:0/992879555 conn(0x7f9c080a5960 msgr2=0x7f9c08132820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:42.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.981+0000 7f9c0e378640 1 -- 172.21.15.46:0/992879555 shutdown_connections 2024-09-16T04:44:42.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:42.981+0000 7f9c0e378640 1 -- 172.21.15.46:0/992879555 wait complete. 2024-09-16T04:44:45.995 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:44:46.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.106+0000 7f546cf31640 10 monclient: get_monmap_and_config 2024-09-16T04:44:46.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.106+0000 7f546cf31640 10 monclient: build_initial_monmap 2024-09-16T04:44:46.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.106+0000 7f546cf31640 10 monclient: monmap: 2024-09-16T04:44:46.108 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:46.108 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:46.108 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:44:46.107899+0000 2024-09-16T04:44:46.109 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:44:46.107899+0000 2024-09-16T04:44:46.109 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:46.109 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:46.109 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:44:46.109 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:44:46.109 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:44:46.109 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:44:46.109 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:44:46.109 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:46.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f546cf31640 1 Processor -- start 2024-09-16T04:44:46.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f546cf31640 1 -- start start 2024-09-16T04:44:46.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f546cf31640 10 monclient: init 2024-09-16T04:44:46.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f546cf31640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:46.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f546cf31640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-16T04:44:46.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f546cf31640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 0x7f546809bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:46.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f546cf31640 10 monclient(hunting): picked mon.noname-c con 0x7f54680a3ba0 addr v2:172.21.15.114:3300/0 2024-09-16T04:44:46.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f546cf31640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 0x7f54680a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:46.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f546cf31640 10 monclient(hunting): picked mon.noname-a con 0x7f546809c1c0 addr v2:172.21.15.46:3300/0 2024-09-16T04:44:46.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f546cf31640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54680a44c0 0x7f5468135950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:46.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f546cf31640 10 monclient(hunting): picked mon.noname-b con 0x7f54680a44c0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:46.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f546cf31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:46.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f546cf31640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f546809b860 con 0x7f546809c1c0 2024-09-16T04:44:46.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f546cf31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f546cf31640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f546809b2a0 con 0x7f54680a3ba0 2024-09-16T04:44:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f546cf31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f546cf31640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f546809afc0 con 0x7f54680a44c0 2024-09-16T04:44:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f546cf31640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f546cf31640 10 monclient(hunting): authenticate will time out at 1084.523718s 2024-09-16T04:44:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f5466d9d640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54680a44c0 msgr2=0x7f5468135950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:44:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f5466d9d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54680a44c0 0x7f5468135950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f545dd9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 0x7f54680a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f545dd9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 0x7f54680a3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60392/0 (socket says 172.21.15.46:60392) 2024-09-16T04:44:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f545dd9b640 1 -- 172.21.15.46:0/3299635994 learned_addr learned my addr 172.21.15.46:0/3299635994 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f545dd9b640 10 monclient(hunting): get_auth_request con 0x7f546809c1c0 auth_method 0 2024-09-16T04:44:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f545dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f545dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f545dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f545dd9b640 1 -- 172.21.15.46:0/3299635994 >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 msgr2=0x7f54680a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:44:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f545dd9b640 1 -- 172.21.15.46:0/3299635994 >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 msgr2=0x7f54680a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f546659c640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 0x7f546809bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.107+0000 7f545dd9b640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 0x7f54680a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.108+0000 7f546659c640 10 monclient(hunting): get_auth_request con 0x7f54680a3ba0 auth_method 0 2024-09-16T04:44:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.108+0000 7f546659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.108+0000 7f546659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.108+0000 7f546659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.108+0000 7f545dd9b640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 0x7f54680a3f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.108+0000 7f546659c640 1 -- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 msgr2=0x7f546809bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:44:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.108+0000 7f546659c640 1 -- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 msgr2=0x7f546809bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:46.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.108+0000 7f546659c640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 0x7f546809bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:46.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.108+0000 7f546659c640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 0x7f546809bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:46.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.308+0000 7f5466d9d640 1 -- 172.21.15.46:0/3299635994 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54680a44c0 msgr2=0x7f5468135950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:44:46.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.308+0000 7f5466d9d640 1 --2- 172.21.15.46:0/3299635994 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54680a44c0 0x7f5468135950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:44:46.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.309+0000 7f545dd9b640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 0x7f54680a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:46.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.309+0000 7f545dd9b640 10 monclient(hunting): get_auth_request con 0x7f546809c1c0 auth_method 0 2024-09-16T04:44:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.309+0000 7f545dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.309+0000 7f545dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.309+0000 7f545dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:44:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.309+0000 7f545dd9b640 1 -- 172.21.15.46:0/3299635994 >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 msgr2=0x7f54680a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:44:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.309+0000 7f545dd9b640 1 -- 172.21.15.46:0/3299635994 >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 msgr2=0x7f54680a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.309+0000 7f545dd9b640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 0x7f54680a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.309+0000 7f545dd9b640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 0x7f54680a3f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:44:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.309+0000 7f546659c640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 0x7f546809bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.309+0000 7f546659c640 10 monclient(hunting): get_auth_request con 0x7f54680a3ba0 auth_method 0 2024-09-16T04:44:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.309+0000 7f546659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.309+0000 7f546659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.309+0000 7f546659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:44:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.309+0000 7f546659c640 1 -- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 msgr2=0x7f546809bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:44:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.309+0000 7f546659c640 1 -- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 msgr2=0x7f546809bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:46.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.309+0000 7f546659c640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 0x7f546809bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:46.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.309+0000 7f546659c640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 0x7f546809bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:44:46.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.709+0000 7f5466d9d640 1 -- 172.21.15.46:0/3299635994 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54680a44c0 msgr2=0x7f5468135950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:44:46.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.709+0000 7f5466d9d640 1 --2- 172.21.15.46:0/3299635994 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54680a44c0 0x7f5468135950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:44:46.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.710+0000 7f545dd9b640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 0x7f54680a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:46.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.710+0000 7f545dd9b640 10 monclient(hunting): get_auth_request con 0x7f546809c1c0 auth_method 0 2024-09-16T04:44:46.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.710+0000 7f545dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:46.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.710+0000 7f545dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:46.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.710+0000 7f545dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:44:46.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.710+0000 7f546659c640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 0x7f546809bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:46.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.710+0000 7f545dd9b640 1 -- 172.21.15.46:0/3299635994 >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 msgr2=0x7f54680a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:44:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.710+0000 7f545dd9b640 1 -- 172.21.15.46:0/3299635994 >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 msgr2=0x7f54680a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.710+0000 7f546659c640 10 monclient(hunting): get_auth_request con 0x7f54680a3ba0 auth_method 0 2024-09-16T04:44:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.710+0000 7f545dd9b640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 0x7f54680a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.710+0000 7f546659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.710+0000 7f546659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.710+0000 7f546659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:44:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.710+0000 7f545dd9b640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 0x7f54680a3f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:44:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.710+0000 7f546659c640 1 -- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 msgr2=0x7f546809bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:44:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.710+0000 7f546659c640 1 -- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 msgr2=0x7f546809bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.710+0000 7f546659c640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 0x7f546809bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:46.710+0000 7f546659c640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 0x7f546809bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:44:47.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:47.510+0000 7f5466d9d640 1 -- 172.21.15.46:0/3299635994 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54680a44c0 msgr2=0x7f5468135950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:44:47.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:47.510+0000 7f5466d9d640 1 --2- 172.21.15.46:0/3299635994 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54680a44c0 0x7f5468135950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:44:47.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:47.511+0000 7f546659c640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 0x7f546809bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:47.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:47.511+0000 7f546659c640 10 monclient(hunting): get_auth_request con 0x7f54680a3ba0 auth_method 0 2024-09-16T04:44:47.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:47.511+0000 7f546659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:47.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:47.511+0000 7f546659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:47.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:47.511+0000 7f546659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:44:47.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:47.511+0000 7f545dd9b640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 0x7f54680a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:47.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:47.511+0000 7f545dd9b640 10 monclient(hunting): get_auth_request con 0x7f546809c1c0 auth_method 0 2024-09-16T04:44:47.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:47.511+0000 7f545dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:47.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:47.511+0000 7f545dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:47.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:47.511+0000 7f545dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:44:47.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:47.511+0000 7f546659c640 1 -- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 msgr2=0x7f546809bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:44:47.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:47.511+0000 7f546659c640 1 -- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 msgr2=0x7f546809bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:47.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:47.511+0000 7f546659c640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 0x7f546809bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:47.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:47.511+0000 7f546659c640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 0x7f546809bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:44:47.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:47.511+0000 7f545dd9b640 1 -- 172.21.15.46:0/3299635994 >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 msgr2=0x7f54680a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:44:47.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:47.511+0000 7f545dd9b640 1 -- 172.21.15.46:0/3299635994 >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 msgr2=0x7f54680a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:44:47.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:47.511+0000 7f545dd9b640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 0x7f54680a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:44:47.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:47.511+0000 7f545dd9b640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 0x7f54680a3f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:44:49.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:49.108+0000 7f5464d99640 10 monclient(hunting): tick 2024-09-16T04:44:49.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:49.108+0000 7f5464d99640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:44:49.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:49.108+0000 7f5464d99640 1 monclient(hunting): continuing hunt 2024-09-16T04:44:49.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:49.108+0000 7f5464d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:44:49.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:49.108+0000 7f5464d99640 1 -- 172.21.15.46:0/3299635994 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54680a44c0 msgr2=0x7f5468135950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:44:49.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:49.108+0000 7f5464d99640 1 --2- 172.21.15.46:0/3299635994 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54680a44c0 0x7f5468135950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:49.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:49.108+0000 7f5464d99640 1 -- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 msgr2=0x7f546809bc80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:44:49.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:49.108+0000 7f5464d99640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.114:3300/0 conn(0x7f54680a3ba0 0x7f546809bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:49.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:49.108+0000 7f5464d99640 1 -- 172.21.15.46:0/3299635994 >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 msgr2=0x7f54680a3f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:44:49.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:49.108+0000 7f5464d99640 1 --2- 172.21.15.46:0/3299635994 >> v2:172.21.15.46:3300/0 conn(0x7f546809c1c0 0x7f54680a3f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:49.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:49.108+0000 7f5464d99640 10 monclient: _add_conns ranks=[0,1] 2024-09-16T04:44:49.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:49.108+0000 7f5464d99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5448003690 addr v1:172.21.15.46:6789/0 2024-09-16T04:44:49.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:49.108+0000 7f5464d99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5448007320 addr v1:172.21.15.114:6789/0 2024-09-16T04:44:49.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:49.108+0000 7f5464d99640 1 -- 172.21.15.46:0/3299635994 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f546809b580 con 0x7f5448003690 2024-09-16T04:44:49.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:49.108+0000 7f5464d99640 1 -- 172.21.15.46:0/3299635994 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f544800c0a0 con 0x7f5448007320 2024-09-16T04:44:49.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:49.108+0000 7f5464d99640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:50.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.707+0000 7f5465d9b640 1 -- 172.21.15.46:0/3299635994 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2920646764 0 0) 0x7f546809b580 con 0x7f5448003690 2024-09-16T04:44:50.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.707+0000 7f5465d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:44:50.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.707+0000 7f5465d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:50.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.707+0000 7f5465d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:50.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.707+0000 7f5465d9b640 10 monclient(hunting): my global_id is 74196 2024-09-16T04:44:50.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.707+0000 7f5465d9b640 1 -- 172.21.15.46:0/3299635994 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5454003520 con 0x7f5448003690 2024-09-16T04:44:50.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 1 -- 172.21.15.46:0/3299635994 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2500219146 0 0) 0x7f5454003520 con 0x7f5448003690 2024-09-16T04:44:50.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 1 -- 172.21.15.46:0/3299635994 >> v1:172.21.15.114:6789/0 conn(0x7f5448007320 legacy=0x7f54480097e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:50.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:44:50.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:44:50.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:50.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:44:50.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 1 -- 172.21.15.46:0/3299635994 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f544800d340 con 0x7f5448003690 2024-09-16T04:44:50.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 10 monclient: _finish_auth 0 2024-09-16T04:44:50.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:44:50.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:50.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 1 -- 172.21.15.46:0/3299635994 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f545800d580 con 0x7f5448003690 2024-09-16T04:44:50.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f546cf31640 5 monclient: authenticate success, global_id 74196 2024-09-16T04:44:50.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f546cf31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:44:50.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:50.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:44:50.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 10 monclient: dump: 2024-09-16T04:44:50.714 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:50.714 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:50.714 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:50.714 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:50.714 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:50.715 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:50.715 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:50.715 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:50.715 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:50.715 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:50.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:44:50.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:50.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 1 -- 172.21.15.46:0/3299635994 >> v1:172.21.15.46:6789/0 conn(0x7f5448003690 legacy=0x7f5448005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:50.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5466d9d640 1 -- 172.21.15.46:0/3299635994 reap_dead start 2024-09-16T04:44:50.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:44:50.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 1 --2- 172.21.15.46:0/3299635994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54540039e0 0x7f5454003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:50.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 10 monclient(hunting): picked mon.a con 0x7f54540039e0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:44:50.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 1 --2- 172.21.15.46:0/3299635994 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5454004800 0x7f5454004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:50.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 10 monclient(hunting): picked mon.c con 0x7f5454004800 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:50.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 1 --2- 172.21.15.46:0/3299635994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54540053c0 0x7f545400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:50.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 10 monclient(hunting): picked mon.b con 0x7f54540053c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:44:50.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:50.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 1 -- 172.21.15.46:0/3299635994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f545400c250 con 0x7f54540039e0 2024-09-16T04:44:50.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:50.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 1 -- 172.21.15.46:0/3299635994 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f545400c400 con 0x7f54540053c0 2024-09-16T04:44:50.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:50.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 1 -- 172.21.15.46:0/3299635994 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f545400c600 con 0x7f5454004800 2024-09-16T04:44:50.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:50.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5465d9b640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:44:50.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f546cf31640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:44:50.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f545dd9b640 1 -- 172.21.15.46:0/3299635994 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5454004800 msgr2=0x7f5454004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:44:50.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f545dd9b640 1 --2- 172.21.15.46:0/3299635994 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5454004800 0x7f5454004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:50.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f546659c640 1 --2- 172.21.15.46:0/3299635994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54540039e0 0x7f5454003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:50.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f546659c640 10 monclient(hunting): get_auth_request con 0x7f54540039e0 auth_method 0 2024-09-16T04:44:50.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f546659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:50.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f546659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:50.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f546659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:44:50.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5466d9d640 1 --2- 172.21.15.46:0/3299635994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54540053c0 0x7f545400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:50.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5466d9d640 10 monclient(hunting): get_auth_request con 0x7f54540053c0 auth_method 0 2024-09-16T04:44:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5466d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5466d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5466d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:44:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f546659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f546659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f546659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:44:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5466d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5466d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f5466d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:44:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f546659c640 10 monclient(hunting): handle_auth_done global_id 74196 payload 293 2024-09-16T04:44:50.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f546659c640 1 -- 172.21.15.46:0/3299635994 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5454004800 msgr2=0x7f5454004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:44:50.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f546659c640 1 --2- 172.21.15.46:0/3299635994 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5454004800 0x7f5454004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:50.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f546659c640 1 -- 172.21.15.46:0/3299635994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54540053c0 msgr2=0x7f545400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:44:50.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f546659c640 1 --2- 172.21.15.46:0/3299635994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54540053c0 0x7f545400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:50.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f546659c640 10 monclient: _finish_hunting 0 2024-09-16T04:44:50.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.708+0000 7f546659c640 1 monclient: found mon.a 2024-09-16T04:44:50.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.709+0000 7f546659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:50.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.709+0000 7f546659c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:50.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.709+0000 7f546659c640 1 -- 172.21.15.46:0/3299635994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5454003520 con 0x7f54540039e0 2024-09-16T04:44:50.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.709+0000 7f546659c640 10 monclient: _finish_auth 0 2024-09-16T04:44:50.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.709+0000 7f546659c640 10 monclient: _check_auth_tickets 2024-09-16T04:44:50.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.709+0000 7f546659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:50.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.709+0000 7f546659c640 1 --2- 172.21.15.46:0/3299635994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54540039e0 0x7f5454003de0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5458013360 tx=0x7f5454005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4425886ae7a93ac8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:50.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.710+0000 7f5465d9b640 1 -- 172.21.15.46:0/3299635994 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5458031070 con 0x7f54540039e0 2024-09-16T04:44:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.710+0000 7f5465d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.710+0000 7f5465d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.710+0000 7f5465d9b640 10 monclient: dump: 2024-09-16T04:44:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:50.721 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:50.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.710+0000 7f546cf31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:44:50.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.711+0000 7f5465d9b640 1 -- 172.21.15.46:0/3299635994 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f545802f070 con 0x7f54540039e0 2024-09-16T04:44:50.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.711+0000 7f5465d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:50.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.711+0000 7f5465d9b640 1 -- 172.21.15.46:0/3299635994 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545800dcf0 con 0x7f54540039e0 2024-09-16T04:44:50.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.711+0000 7f546cf31640 10 monclient: get_monmap_and_config success 2024-09-16T04:44:50.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.711+0000 7f5465d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:50.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.711+0000 7f546cf31640 4 set_mon_vals no callback set 2024-09-16T04:44:50.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.711+0000 7f5465d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:50.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.711+0000 7f5465d9b640 10 monclient: dump: 2024-09-16T04:44:50.722 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:50.722 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:50.722 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:50.722 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:50.722 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:50.722 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:50.722 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:50.722 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:50.722 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:50.722 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.711+0000 7f546cf31640 10 monclient: shutdown 2024-09-16T04:44:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.711+0000 7f546cf31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.711+0000 7f546cf31640 1 -- 172.21.15.46:0/3299635994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54540039e0 msgr2=0x7f5454003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.711+0000 7f546cf31640 1 --2- 172.21.15.46:0/3299635994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54540039e0 0x7f5454003de0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5458013360 tx=0x7f5454005830 comp rx=0 tx=0).stop 2024-09-16T04:44:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.712+0000 7f546cf31640 1 -- 172.21.15.46:0/3299635994 shutdown_connections 2024-09-16T04:44:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.712+0000 7f546cf31640 1 --2- 172.21.15.46:0/3299635994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54540053c0 0x7f545400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.712+0000 7f546cf31640 1 --2- 172.21.15.46:0/3299635994 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5454004800 0x7f5454004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:50.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.712+0000 7f546cf31640 1 --2- 172.21.15.46:0/3299635994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54540039e0 0x7f5454003de0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:50.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.712+0000 7f546cf31640 1 -- 172.21.15.46:0/3299635994 >> 172.21.15.46:0/3299635994 conn(0x7f54680a60c0 msgr2=0x7f54680a64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:50.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.712+0000 7f546cf31640 1 -- 172.21.15.46:0/3299635994 shutdown_connections 2024-09-16T04:44:50.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.712+0000 7f546cf31640 1 -- 172.21.15.46:0/3299635994 wait complete. 2024-09-16T04:44:50.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.713+0000 7f546cf31640 10 monclient: build_initial_monmap 2024-09-16T04:44:50.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.713+0000 7f546cf31640 10 monclient: monmap: 2024-09-16T04:44:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:44:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:44:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:44:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:44:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:44:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:44:50.730 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.714+0000 7f546cf31640 1 Processor -- start 2024-09-16T04:44:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.714+0000 7f546cf31640 1 -- start start 2024-09-16T04:44:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.714+0000 7f546cf31640 10 monclient: init 2024-09-16T04:44:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546cf31640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:50.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546cf31640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:44:50.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546cf31640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54540039e0 0x7f54681cb550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:50.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546cf31640 10 monclient(hunting): picked mon.noname-a con 0x7f54540039e0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:44:50.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546cf31640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5454004800 0x7f5468136e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:50.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546cf31640 10 monclient(hunting): picked mon.noname-b con 0x7f5454004800 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:44:50.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546cf31640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54540053c0 0x7f54681373a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:50.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546cf31640 10 monclient(hunting): picked mon.noname-c con 0x7f54540053c0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:50.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546cf31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:50.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546cf31640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f54680a5540 con 0x7f54540039e0 2024-09-16T04:44:50.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546cf31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:50.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546cf31640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f54680a53c0 con 0x7f5454004800 2024-09-16T04:44:50.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546cf31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:50.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546cf31640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f54680a3f80 con 0x7f54540053c0 2024-09-16T04:44:50.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546cf31640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:50.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546cf31640 10 monclient(hunting): authenticate will time out at 1089.131108s 2024-09-16T04:44:50.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f5466d9d640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54540053c0 msgr2=0x7f54681373a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:44:50.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f5466d9d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54540053c0 0x7f54681373a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:50.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54540039e0 0x7f54681cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:50.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54540039e0 0x7f54681cb550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60430/0 (socket says 172.21.15.46:60430) 2024-09-16T04:44:50.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f545dd9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5454004800 0x7f5468136e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:50.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546659c640 1 -- 172.21.15.46:0/2423859132 learned_addr learned my addr 172.21.15.46:0/2423859132 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:50.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546659c640 10 monclient(hunting): get_auth_request con 0x7f54540039e0 auth_method 0 2024-09-16T04:44:50.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:50.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:50.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:50.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f545dd9b640 10 monclient(hunting): get_auth_request con 0x7f5454004800 auth_method 0 2024-09-16T04:44:50.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f545dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:50.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f545dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:50.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f545dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:50.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:50.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:50.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:50.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f545dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:50.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f545dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:50.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f545dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:50.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546659c640 10 monclient(hunting): handle_auth_done global_id 74199 payload 824 2024-09-16T04:44:50.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546659c640 1 -- 172.21.15.46:0/2423859132 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54540053c0 msgr2=0x7f54681373a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:44:50.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546659c640 1 --2- 172.21.15.46:0/2423859132 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54540053c0 0x7f54681373a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:50.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546659c640 1 -- 172.21.15.46:0/2423859132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5454004800 msgr2=0x7f5468136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:50.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546659c640 1 --2- 172.21.15.46:0/2423859132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5454004800 0x7f5468136e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:50.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546659c640 10 monclient: _finish_hunting 0 2024-09-16T04:44:50.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546659c640 1 monclient: found mon.noname-a 2024-09-16T04:44:50.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:50.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:44:50.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546659c640 1 -- 172.21.15.46:0/2423859132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54680a0350 con 0x7f54540039e0 2024-09-16T04:44:50.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.715+0000 7f546659c640 1 --2- 172.21.15.46:0/2423859132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54540039e0 0x7f54681cb550 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5458013480 tx=0x7f54580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:50.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.716+0000 7f545f7fe640 1 -- 172.21.15.46:0/2423859132 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5458044070 con 0x7f54540039e0 2024-09-16T04:44:50.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.716+0000 7f545f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:50.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.716+0000 7f545f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:44:50.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.716+0000 7f545f7fe640 10 monclient: dump: 2024-09-16T04:44:50.735 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:50.735 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:50.735 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:50.735 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:50.735 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:50.735 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:50.735 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:50.736 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:50.736 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:50.736 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:50.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.716+0000 7f545f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:44:50.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.716+0000 7f545f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:44:50.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.716+0000 7f545f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:50.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.716+0000 7f545f7fe640 1 -- 172.21.15.46:0/2423859132 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5458004250 con 0x7f54540039e0 2024-09-16T04:44:50.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.716+0000 7f546cf31640 5 monclient: authenticate success, global_id 74199 2024-09-16T04:44:50.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.716+0000 7f545f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:50.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.716+0000 7f545f7fe640 1 -- 172.21.15.46:0/2423859132 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5458003610 con 0x7f54540039e0 2024-09-16T04:44:50.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.716+0000 7f546cf31640 10 monclient: _renew_subs 2024-09-16T04:44:50.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.716+0000 7f546cf31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:50.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.716+0000 7f546cf31640 1 -- 172.21.15.46:0/2423859132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f546809d110 con 0x7f54540039e0 2024-09-16T04:44:50.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.716+0000 7f545f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:50.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.716+0000 7f545f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:50.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.716+0000 7f545f7fe640 10 monclient: dump: 2024-09-16T04:44:50.738 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:50.738 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:50.738 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:50.738 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:50.738 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:50.738 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:50.738 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:50.738 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:50.738 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:50.739 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:50.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.716+0000 7f546559a640 4 set_mon_vals no callback set 2024-09-16T04:44:50.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.716+0000 7f546cf31640 10 monclient: _renew_subs 2024-09-16T04:44:50.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.716+0000 7f546cf31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:50.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.716+0000 7f546cf31640 1 -- 172.21.15.46:0/2423859132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54681cba90 con 0x7f54540039e0 2024-09-16T04:44:50.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.717+0000 7f545f7fe640 1 -- 172.21.15.46:0/2423859132 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f5458047070 con 0x7f54540039e0 2024-09-16T04:44:50.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.718+0000 7f546cf31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:50.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.718+0000 7f545f7fe640 1 --2- 172.21.15.46:0/2423859132 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5448038d20 0x7f544803b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:50.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.718+0000 7f546cf31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:50.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.718+0000 7f546cf31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:50.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.718+0000 7f545f7fe640 1 -- 172.21.15.46:0/2423859132 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 4738+0+0 (secure 0 0 0) 0x7f545806b760 con 0x7f54540039e0 2024-09-16T04:44:50.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.718+0000 7f546cf31640 1 -- 172.21.15.46:0/2423859132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f546809bc80 con 0x7f54540039e0 2024-09-16T04:44:50.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.718+0000 7f545dd9b640 1 --2- 172.21.15.46:0/2423859132 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5448038d20 0x7f544803b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:50.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.718+0000 7f545dd9b640 10 monclient: get_auth_request con 0x7f5448038d20 auth_method 0 2024-09-16T04:44:50.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.718+0000 7f545dd9b640 1 --2- 172.21.15.46:0/2423859132 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5448038d20 0x7f544803b1e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f54681bfe00 tx=0x7f5450008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:50.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.723+0000 7f545f7fe640 1 -- 172.21.15.46:0/2423859132 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5458036460 con 0x7f54540039e0 2024-09-16T04:44:50.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.723+0000 7f545f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:50.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.723+0000 7f545f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:44:50.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.971+0000 7f546cf31640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:44:50.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.971+0000 7f546cf31640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:44:50.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.971+0000 7f546cf31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:50.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.971+0000 7f546cf31640 1 -- 172.21.15.46:0/2423859132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f54681241a0 con 0x7f54540039e0 2024-09-16T04:44:50.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.971+0000 7f545f7fe640 1 -- 172.21.15.46:0/2423859132 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f545804e070 con 0x7f54540039e0 2024-09-16T04:44:50.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.971+0000 7f545f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:44:50.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.971+0000 7f545f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:44:50.974 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":72,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:44:50.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.973+0000 7f546cf31640 1 -- 172.21.15.46:0/2423859132 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5448038d20 msgr2=0x7f544803b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:50.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.973+0000 7f546cf31640 1 --2- 172.21.15.46:0/2423859132 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5448038d20 0x7f544803b1e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f54681bfe00 tx=0x7f5450008040 comp rx=0 tx=0).stop 2024-09-16T04:44:50.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.973+0000 7f546cf31640 10 monclient: shutdown 2024-09-16T04:44:50.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.973+0000 7f546cf31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:50.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.973+0000 7f546cf31640 1 -- 172.21.15.46:0/2423859132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54540039e0 msgr2=0x7f54681cb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:50.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.973+0000 7f546cf31640 1 --2- 172.21.15.46:0/2423859132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54540039e0 0x7f54681cb550 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5458013480 tx=0x7f54580046c0 comp rx=0 tx=0).stop 2024-09-16T04:44:50.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.973+0000 7f546cf31640 1 -- 172.21.15.46:0/2423859132 shutdown_connections 2024-09-16T04:44:50.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.973+0000 7f546cf31640 1 --2- 172.21.15.46:0/2423859132 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f54540053c0 0x7f54681373a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:50.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.973+0000 7f546cf31640 1 --2- 172.21.15.46:0/2423859132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5454004800 0x7f5468136e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.973+0000 7f546cf31640 1 --2- 172.21.15.46:0/2423859132 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5448038d20 0x7f544803b1e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.973+0000 7f546cf31640 1 --2- 172.21.15.46:0/2423859132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54540039e0 0x7f54681cb550 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.973+0000 7f546cf31640 1 -- 172.21.15.46:0/2423859132 >> 172.21.15.46:0/2423859132 conn(0x7f54680a60c0 msgr2=0x7f546809f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.973+0000 7f546cf31640 1 -- 172.21.15.46:0/2423859132 shutdown_connections 2024-09-16T04:44:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:50.973+0000 7f546cf31640 1 -- 172.21.15.46:0/2423859132 wait complete. 2024-09-16T04:44:50.987 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:44:50.987 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:44:50.987 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:44:51.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.087+0000 7f5730bcf640 10 monclient: get_monmap_and_config 2024-09-16T04:44:51.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.087+0000 7f5730bcf640 10 monclient: build_initial_monmap 2024-09-16T04:44:51.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.087+0000 7f5730bcf640 10 monclient: monmap: 2024-09-16T04:44:51.090 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:51.090 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:51.090 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:44:51.089155+0000 2024-09-16T04:44:51.090 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:44:51.089155+0000 2024-09-16T04:44:51.090 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:51.090 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:51.090 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:44:51.090 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:44:51.090 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:44:51.090 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:44:51.090 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:44:51.090 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:51.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.088+0000 7f5730bcf640 1 Processor -- start 2024-09-16T04:44:51.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.088+0000 7f5730bcf640 1 -- start start 2024-09-16T04:44:51.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.088+0000 7f5730bcf640 10 monclient: init 2024-09-16T04:44:51.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.088+0000 7f5730bcf640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:51.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.088+0000 7f5730bcf640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-16T04:44:51.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.088+0000 7f5730bcf640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f572c06e160 0x7f572c0a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:51.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.088+0000 7f5730bcf640 10 monclient(hunting): picked mon.noname-c con 0x7f572c06e160 addr v2:172.21.15.114:3300/0 2024-09-16T04:44:51.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.088+0000 7f5730bcf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f572c0a2ee0 addr v1:172.21.15.114:6789/0 2024-09-16T04:44:51.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.088+0000 7f5730bcf640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f572c137050 0x7f572c13ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:51.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.088+0000 7f5730bcf640 10 monclient(hunting): picked mon.noname-b con 0x7f572c137050 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:51.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.088+0000 7f5730bcf640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f572c0a3fd0 con 0x7f572c0a2ee0 2024-09-16T04:44:51.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.088+0000 7f5730bcf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:51.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.088+0000 7f5730bcf640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f572c0a3300 con 0x7f572c06e160 2024-09-16T04:44:51.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.088+0000 7f5730bcf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:51.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.088+0000 7f5730bcf640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f572c06e540 con 0x7f572c137050 2024-09-16T04:44:51.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.088+0000 7f5730bcf640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:51.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.088+0000 7f5730bcf640 10 monclient(hunting): authenticate will time out at 1089.504950s 2024-09-16T04:44:51.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f572ad9d640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f572c137050 msgr2=0x7f572c13ec80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:44:51.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f572ad9d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f572c137050 0x7f572c13ec80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:51.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f572a59c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f572c06e160 0x7f572c0a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:51.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f5729d9b640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f572c0a2ee0 0x7f572c136940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.46:33260/0 (socket says 172.21.15.46:33260) 2024-09-16T04:44:51.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f572a59c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f572c06e160 0x7f572c0a29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.46:46408/0 (socket says 172.21.15.46:46408) 2024-09-16T04:44:51.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f5729d9b640 1 -- 172.21.15.46:0/1157836872 learned_addr learned my addr 172.21.15.46:0/1157836872 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:44:51.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f572a59c640 10 monclient(hunting): get_auth_request con 0x7f572c06e160 auth_method 0 2024-09-16T04:44:51.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f572a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:51.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f572a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:51.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f572a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:51.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f572a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:51.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f572a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:51.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f572a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:51.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f572a59c640 10 monclient(hunting): handle_auth_done global_id 6298 payload 274 2024-09-16T04:44:51.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f572a59c640 1 -- 172.21.15.46:0/1157836872 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f572c137050 msgr2=0x7f572c13ec80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:44:51.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f572a59c640 1 --2- 172.21.15.46:0/1157836872 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f572c137050 0x7f572c13ec80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:51.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f572a59c640 1 -- 172.21.15.46:0/1157836872 >> v1:172.21.15.114:6789/0 conn(0x7f572c0a2ee0 legacy=0x7f572c136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:51.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f572a59c640 10 monclient: _finish_hunting 0 2024-09-16T04:44:51.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f572a59c640 1 monclient: found mon.noname-c 2024-09-16T04:44:51.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f572a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:51.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f572a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:44:51.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.089+0000 7f572a59c640 1 -- 172.21.15.46:0/1157836872 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f572c0a0bd0 con 0x7f572c06e160 2024-09-16T04:44:51.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.090+0000 7f572a59c640 1 --2- 172.21.15.46:0/1157836872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f572c06e160 0x7f572c0a29a0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f571c0097d0 tx=0x7f571c02a0b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9cc68814e9345b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:51.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.090+0000 7f572959a640 1 -- 172.21.15.46:0/1157836872 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c037070 con 0x7f572c06e160 2024-09-16T04:44:51.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.090+0000 7f572959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:51.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.090+0000 7f572959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:44:51.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.090+0000 7f572959a640 10 monclient: dump: 2024-09-16T04:44:51.105 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:51.105 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:51.105 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.105 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.105 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:51.105 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:51.105 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:51.105 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:51.105 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:51.106 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:51.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.090+0000 7f572959a640 10 monclient: _finish_auth 0 2024-09-16T04:44:51.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.090+0000 7f572959a640 10 monclient: _check_auth_tickets 2024-09-16T04:44:51.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.090+0000 7f572959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:51.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.090+0000 7f572959a640 1 -- 172.21.15.46:0/1157836872 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f571c02adc0 con 0x7f572c06e160 2024-09-16T04:44:51.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.090+0000 7f572959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:51.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.090+0000 7f572959a640 1 -- 172.21.15.46:0/1157836872 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c032660 con 0x7f572c06e160 2024-09-16T04:44:51.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.090+0000 7f5730bcf640 5 monclient: authenticate success, global_id 6298 2024-09-16T04:44:51.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.090+0000 7f5730bcf640 10 monclient: get_monmap_and_config success 2024-09-16T04:44:51.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.090+0000 7f5730bcf640 4 set_mon_vals no callback set 2024-09-16T04:44:51.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.090+0000 7f572959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:51.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.090+0000 7f572959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:44:51.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.090+0000 7f572959a640 10 monclient: dump: 2024-09-16T04:44:51.106 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:51.107 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:51.107 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.107 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.107 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:51.107 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:51.107 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:51.107 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:51.107 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:51.107 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:51.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.091+0000 7f5730bcf640 10 monclient: shutdown 2024-09-16T04:44:51.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.091+0000 7f5730bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:51.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.091+0000 7f5730bcf640 1 -- 172.21.15.46:0/1157836872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f572c06e160 msgr2=0x7f572c0a29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:51.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.091+0000 7f5730bcf640 1 --2- 172.21.15.46:0/1157836872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f572c06e160 0x7f572c0a29a0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f571c0097d0 tx=0x7f571c02a0b0 comp rx=0 tx=0).stop 2024-09-16T04:44:51.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.091+0000 7f5730bcf640 1 -- 172.21.15.46:0/1157836872 shutdown_connections 2024-09-16T04:44:51.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.091+0000 7f5730bcf640 1 --2- 172.21.15.46:0/1157836872 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f572c137050 0x7f572c13ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:51.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.091+0000 7f5730bcf640 1 --2- 172.21.15.46:0/1157836872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f572c06e160 0x7f572c0a29a0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:51.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.091+0000 7f5730bcf640 1 -- 172.21.15.46:0/1157836872 >> 172.21.15.46:0/1157836872 conn(0x7f572c0a5c10 msgr2=0x7f572c0a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:51.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.091+0000 7f5730bcf640 1 -- 172.21.15.46:0/1157836872 shutdown_connections 2024-09-16T04:44:51.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.091+0000 7f5730bcf640 1 -- 172.21.15.46:0/1157836872 wait complete. 2024-09-16T04:44:51.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.092+0000 7f5730bcf640 10 monclient: build_initial_monmap 2024-09-16T04:44:51.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.092+0000 7f5730bcf640 10 monclient: monmap: 2024-09-16T04:44:51.108 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:51.108 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:44:51.108 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:44:51.108 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:44:51.108 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:51.108 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:51.109 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:44:51.109 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:44:51.109 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:44:51.109 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:51.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.092+0000 7f5730bcf640 1 Processor -- start 2024-09-16T04:44:51.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.092+0000 7f5730bcf640 1 -- start start 2024-09-16T04:44:51.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.092+0000 7f5730bcf640 10 monclient: init 2024-09-16T04:44:51.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5730bcf640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:51.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5730bcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:44:51.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5730bcf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f572c06e160 0x7f572c135bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:51.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5730bcf640 10 monclient(hunting): picked mon.noname-a con 0x7f572c06e160 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:44:51.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5730bcf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f572c0a2ee0 0x7f572c136110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:51.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5730bcf640 10 monclient(hunting): picked mon.noname-b con 0x7f572c0a2ee0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:44:51.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5730bcf640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f572c137050 0x7f572c11f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:51.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5730bcf640 10 monclient(hunting): picked mon.noname-c con 0x7f572c137050 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:51.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5730bcf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:51.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5730bcf640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f572c0a34a0 con 0x7f572c06e160 2024-09-16T04:44:51.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5730bcf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:51.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5730bcf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f572c0a5130 con 0x7f572c0a2ee0 2024-09-16T04:44:51.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5730bcf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:51.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5730bcf640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f572c1369d0 con 0x7f572c137050 2024-09-16T04:44:51.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5730bcf640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:51.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5730bcf640 10 monclient(hunting): authenticate will time out at 1089.509142s 2024-09-16T04:44:51.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572ad9d640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f572c137050 msgr2=0x7f572c11f760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:44:51.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572ad9d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f572c137050 0x7f572c11f760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:51.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f572c06e160 0x7f572c135bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:51.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f572c06e160 0x7f572c135bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60434/0 (socket says 172.21.15.46:60434) 2024-09-16T04:44:51.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572a59c640 1 -- 172.21.15.46:0/1296764251 learned_addr learned my addr 172.21.15.46:0/1296764251 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:51.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572a59c640 10 monclient(hunting): get_auth_request con 0x7f572c06e160 auth_method 0 2024-09-16T04:44:51.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:51.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:51.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:51.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:51.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:51.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:51.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5729d9b640 1 --2- 172.21.15.46:0/1296764251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f572c0a2ee0 0x7f572c136110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:51.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5729d9b640 10 monclient(hunting): get_auth_request con 0x7f572c0a2ee0 auth_method 0 2024-09-16T04:44:51.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5729d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:51.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5729d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:51.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5729d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:51.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572a59c640 10 monclient(hunting): handle_auth_done global_id 74202 payload 824 2024-09-16T04:44:51.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572a59c640 1 -- 172.21.15.46:0/1296764251 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f572c137050 msgr2=0x7f572c11f760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:44:51.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572a59c640 1 --2- 172.21.15.46:0/1296764251 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f572c137050 0x7f572c11f760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:51.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572a59c640 1 -- 172.21.15.46:0/1296764251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f572c0a2ee0 msgr2=0x7f572c136110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:51.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572a59c640 1 --2- 172.21.15.46:0/1296764251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f572c0a2ee0 0x7f572c136110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:51.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572a59c640 10 monclient: _finish_hunting 0 2024-09-16T04:44:51.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572a59c640 1 monclient: found mon.noname-a 2024-09-16T04:44:51.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:51.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:44:51.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572a59c640 1 -- 172.21.15.46:0/1296764251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f572c1232d0 con 0x7f572c06e160 2024-09-16T04:44:51.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f572a59c640 1 --2- 172.21.15.46:0/1296764251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f572c06e160 0x7f572c135bd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f571c009720 tx=0x7f571c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:51.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f571affd640 1 -- 172.21.15.46:0/1296764251 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c041070 con 0x7f572c06e160 2024-09-16T04:44:51.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f571affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:51.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f571affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:44:51.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f571affd640 10 monclient: dump: 2024-09-16T04:44:51.114 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:51.114 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:51.114 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.114 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.114 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:51.114 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:51.114 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:51.114 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:51.114 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:51.114 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:51.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f571affd640 10 monclient: _finish_auth 0 2024-09-16T04:44:51.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f571affd640 10 monclient: _check_auth_tickets 2024-09-16T04:44:51.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f571affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:51.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f571affd640 1 -- 172.21.15.46:0/1296764251 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f571c03c670 con 0x7f572c06e160 2024-09-16T04:44:51.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5730bcf640 5 monclient: authenticate success, global_id 74202 2024-09-16T04:44:51.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5730bcf640 10 monclient: _renew_subs 2024-09-16T04:44:51.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5730bcf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:51.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5730bcf640 1 -- 172.21.15.46:0/1296764251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f572c1235b0 con 0x7f572c06e160 2024-09-16T04:44:51.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f571affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:51.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f571affd640 1 -- 172.21.15.46:0/1296764251 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c037040 con 0x7f572c06e160 2024-09-16T04:44:51.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f571affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:51.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f571affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:51.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f571affd640 10 monclient: dump: 2024-09-16T04:44:51.115 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:51.115 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:51.115 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.115 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.115 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:51.116 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5730bcf640 10 monclient: _renew_subs 2024-09-16T04:44:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.093+0000 7f5728d99640 4 set_mon_vals no callback set 2024-09-16T04:44:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.094+0000 7f5730bcf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.094+0000 7f5730bcf640 1 -- 172.21.15.46:0/1296764251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f572c121ba0 con 0x7f572c06e160 2024-09-16T04:44:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.094+0000 7f5730bcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.094+0000 7f5730bcf640 1 --2- 172.21.15.46:0/1296764251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f572c1cbe20 0x7f572c11fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.094+0000 7f5730bcf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.094+0000 7f5730bcf640 1 -- 172.21.15.46:0/1296764251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f572c0a3620 con 0x7f572c1cbe20 2024-09-16T04:44:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.094+0000 7f571affd640 1 -- 172.21.15.46:0/1296764251 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f571c044070 con 0x7f572c06e160 2024-09-16T04:44:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.094+0000 7f5729d9b640 1 --2- 172.21.15.46:0/1296764251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f572c1cbe20 0x7f572c11fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f5729d9b640 10 monclient: get_auth_request con 0x7f572c1cbe20 auth_method 0 2024-09-16T04:44:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f5729d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f5729d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f5729d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:51.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f5729d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:51.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f5729d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:51.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f5729d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:51.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f571affd640 1 --2- 172.21.15.46:0/1296764251 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f570c038390 0x7f570c03a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:51.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f571affd640 1 -- 172.21.15.46:0/1296764251 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 4738+0+0 (secure 0 0 0) 0x7f571c06be80 con 0x7f572c06e160 2024-09-16T04:44:51.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f5729d9b640 10 monclient(hunting): handle_auth_done global_id 74205 payload 824 2024-09-16T04:44:51.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f5729d9b640 1 -- 172.21.15.46:0/1296764251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f572c0a8610 con 0x7f572c1cbe20 2024-09-16T04:44:51.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f572ad9d640 1 --2- 172.21.15.46:0/1296764251 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f570c038390 0x7f570c03a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:51.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f5729d9b640 1 --2- 172.21.15.46:0/1296764251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f572c1cbe20 0x7f572c11fca0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f572c120ec0 tx=0x7f572000e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:51.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f572ad9d640 10 monclient: get_auth_request con 0x7f570c038390 auth_method 0 2024-09-16T04:44:51.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f571affd640 1 -- 172.21.15.46:0/1296764251 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5720019070 con 0x7f572c1cbe20 2024-09-16T04:44:51.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f571affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:51.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f571affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:51.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f571affd640 10 monclient: dump: 2024-09-16T04:44:51.119 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:51.119 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:51.119 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.119 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.119 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:51.119 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:51.119 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:51.119 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:51.120 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:51.120 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:51.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f571affd640 1 -- 172.21.15.46:0/1296764251 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5720005ac0 con 0x7f572c1cbe20 2024-09-16T04:44:51.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f571affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:51.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f571affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:44:51.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f571affd640 1 -- 172.21.15.46:0/1296764251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f572c1cbe20 msgr2=0x7f572c11fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:51.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f571affd640 1 --2- 172.21.15.46:0/1296764251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f572c1cbe20 0x7f572c11fca0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f572c120ec0 tx=0x7f572000e960 comp rx=0 tx=0).stop 2024-09-16T04:44:51.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f571affd640 1 -- 172.21.15.46:0/1296764251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f572c1cbe20 msgr2=0x7f572c11fca0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:44:51.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f571affd640 1 --2- 172.21.15.46:0/1296764251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f572c1cbe20 0x7f572c11fca0 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f572c120ec0 tx=0x7f572000e960 comp rx=0 tx=0).stop 2024-09-16T04:44:51.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.095+0000 7f572ad9d640 1 --2- 172.21.15.46:0/1296764251 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f570c038390 0x7f570c03a850 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5724003d00 tx=0x7f5724009270 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:51.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f5730bcf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:44:51.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f5730bcf640 1 --2- 172.21.15.46:0/1296764251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f572c1201e0 0x7f572c1bc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:51.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f5730bcf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:51.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f5730bcf640 1 -- 172.21.15.46:0/1296764251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f572c0a0300 con 0x7f572c1201e0 2024-09-16T04:44:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f5729d9b640 1 --2- 172.21.15.46:0/1296764251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f572c1201e0 0x7f572c1bc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f5729d9b640 10 monclient: get_auth_request con 0x7f572c1201e0 auth_method 0 2024-09-16T04:44:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f5729d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f5729d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f5729d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f5729d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f5729d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f5729d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f5729d9b640 10 monclient(hunting): handle_auth_done global_id 74208 payload 824 2024-09-16T04:44:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f5729d9b640 1 -- 172.21.15.46:0/1296764251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f572c1205c0 con 0x7f572c1201e0 2024-09-16T04:44:51.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f5729d9b640 1 --2- 172.21.15.46:0/1296764251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f572c1201e0 0x7f572c1bc740 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5720016000 tx=0x7f5720000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:51.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f571affd640 1 -- 172.21.15.46:0/1296764251 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5720019040 con 0x7f572c1201e0 2024-09-16T04:44:51.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f571affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:51.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f571affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:51.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f571affd640 10 monclient: dump: 2024-09-16T04:44:51.125 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:51.125 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:51.125 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.125 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.125 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:51.125 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:51.126 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f571affd640 1 -- 172.21.15.46:0/1296764251 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4702 (secure 0 0 0) 0x7f5720007410 con 0x7f572c1201e0 2024-09-16T04:44:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f571affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:44:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f571affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:44:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f571affd640 1 -- 172.21.15.46:0/1296764251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f572c1201e0 msgr2=0x7f572c1bc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f571affd640 1 --2- 172.21.15.46:0/1296764251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f572c1201e0 0x7f572c1bc740 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5720016000 tx=0x7f5720000ed0 comp rx=0 tx=0).stop 2024-09-16T04:44:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f571affd640 1 -- 172.21.15.46:0/1296764251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f572c1201e0 msgr2=0x7f572c1bc740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:44:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.102+0000 7f571affd640 1 --2- 172.21.15.46:0/1296764251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f572c1201e0 0x7f572c1bc740 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5720016000 tx=0x7f5720000ed0 comp rx=0 tx=0).stop 2024-09-16T04:44:51.127 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:44:51.127 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:44:51.127 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:44:51.127 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:44:51.127 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 72, 2024-09-16T04:44:51.127 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 31098, 2024-09-16T04:44:51.127 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:44:51.127 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:44:51.127 INFO:teuthology.orchestra.run.smithi046.stdout: 1 2024-09-16T04:44:51.127 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:51.127 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:44:51.128 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:44:51.128 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:44:51.128 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:44:51.128 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:51.128 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:51.128 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:51.128 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:51.128 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:51.128 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:51.128 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:51.129 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:51.129 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:51.129 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:51.129 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:51.129 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:51.129 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:44:51.129 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:44:51.129 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:51.129 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:51.129 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:51.129 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:51.129 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:51.129 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:51.130 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:51.130 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:51.130 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:51.130 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:51.130 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:51.130 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:51.130 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.131 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:44:51.131 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:44:51.131 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:44:51.131 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:44:51.131 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:44:51.131 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:44:51.131 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:44:51.131 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:44:51.132 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:44:51.132 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:44:51.132 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:44:51.132 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:44:51.132 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:44:51.132 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:44:51.132 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:44:51.132 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:44:51.132 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:44:51.132 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:51.132 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:51.133 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:51.133 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:51.133 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:51.133 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:51.133 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:51.133 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:51.133 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:51.133 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:51.134 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:51.134 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:51.134 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:44:51.134 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.134 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:44:51.134 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.134 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:44:51.134 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:44:51.134 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:51.134 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:51.134 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.134 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:51.134 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:44:51.135 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:51.135 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.135 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.135 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:51.135 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:44:51.135 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:51.135 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:51.135 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:51.135 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.135 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:44:51.135 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:44:51.135 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:51.136 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:51.136 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:51.136 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.136 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.136 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:44:51.136 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:44:51.136 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:51.136 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:51.136 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.136 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:51.136 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:44:51.136 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:51.137 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.137 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.137 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:51.137 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:44:51.137 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:51.137 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:51.138 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:51.138 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.138 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:44:51.138 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:44:51.138 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:51.138 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:51.138 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:51.138 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.138 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.138 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:44:51.138 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:44:51.138 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:51.138 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:51.139 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.139 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:51.139 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:44:51.139 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:51.139 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.139 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.139 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:51.139 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:44:51.139 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:51.139 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:51.139 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:51.139 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.139 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:44:51.140 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:44:51.140 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:51.140 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:51.140 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:51.140 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:51.140 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:51.140 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.140 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:44:51.140 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:44:51.140 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.140 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:51.140 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:51.140 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:44:51.141 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:51.141 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:51.141 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:44:51.141 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.141 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:51.141 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:51.141 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:44:51.141 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:51.141 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:51.141 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:44:51.141 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.141 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:51.141 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:51.141 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:44:51.141 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:51.142 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:51.142 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.142 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:44:51.142 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:44:51.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.103+0000 7f5730bcf640 1 -- 172.21.15.46:0/1296764251 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f570c038390 msgr2=0x7f570c03a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:51.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.103+0000 7f5730bcf640 1 --2- 172.21.15.46:0/1296764251 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f570c038390 0x7f570c03a850 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5724003d00 tx=0x7f5724009270 comp rx=0 tx=0).stop 2024-09-16T04:44:51.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.103+0000 7f5730bcf640 10 monclient: shutdown 2024-09-16T04:44:51.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.103+0000 7f5730bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:51.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.103+0000 7f5730bcf640 1 -- 172.21.15.46:0/1296764251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f572c06e160 msgr2=0x7f572c135bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:51.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.103+0000 7f5730bcf640 1 --2- 172.21.15.46:0/1296764251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f572c06e160 0x7f572c135bd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f571c009720 tx=0x7f571c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:44:51.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.103+0000 7f572ad9d640 1 -- 172.21.15.46:0/1296764251 reap_dead start 2024-09-16T04:44:51.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.103+0000 7f572ad9d640 1 -- 172.21.15.46:0/1296764251 reap_dead start 2024-09-16T04:44:51.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.103+0000 7f5730bcf640 1 -- 172.21.15.46:0/1296764251 shutdown_connections 2024-09-16T04:44:51.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.103+0000 7f5730bcf640 1 -- 172.21.15.46:0/1296764251 >> 172.21.15.46:0/1296764251 conn(0x7f572c0a5c10 msgr2=0x7f572c09f520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:51.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.103+0000 7f5730bcf640 1 -- 172.21.15.46:0/1296764251 shutdown_connections 2024-09-16T04:44:51.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.103+0000 7f5730bcf640 1 -- 172.21.15.46:0/1296764251 wait complete. 2024-09-16T04:44:51.145 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:44:51.145 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:44:51.145 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:44:51.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.245+0000 7fda63552640 10 monclient: get_monmap_and_config 2024-09-16T04:44:51.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.245+0000 7fda63552640 10 monclient: build_initial_monmap 2024-09-16T04:44:51.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.245+0000 7fda63552640 10 monclient: monmap: 2024-09-16T04:44:51.248 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:51.248 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:51.248 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:44:51.247052+0000 2024-09-16T04:44:51.248 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:44:51.247052+0000 2024-09-16T04:44:51.248 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:51.248 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:51.249 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:44:51.249 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:44:51.249 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:44:51.249 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:44:51.249 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:44:51.249 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:51.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.246+0000 7fda63552640 1 Processor -- start 2024-09-16T04:44:51.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.246+0000 7fda63552640 1 -- start start 2024-09-16T04:44:51.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.246+0000 7fda63552640 10 monclient: init 2024-09-16T04:44:51.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.246+0000 7fda63552640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:51.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.246+0000 7fda63552640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-16T04:44:51.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.246+0000 7fda63552640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fda5c0a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:44:51.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.246+0000 7fda63552640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda5c0a3ff0 0x7fda5c0a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:51.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.246+0000 7fda63552640 10 monclient(hunting): picked mon.noname-a con 0x7fda5c0a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:44:51.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.246+0000 7fda63552640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c13be90 0x7fda5c13e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:51.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.246+0000 7fda63552640 10 monclient(hunting): picked mon.noname-b con 0x7fda5c13be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:51.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.246+0000 7fda63552640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda5c09b940 con 0x7fda5c0a3c10 2024-09-16T04:44:51.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.246+0000 7fda63552640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:51.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.246+0000 7fda63552640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fda5c09ab00 con 0x7fda5c0a3ff0 2024-09-16T04:44:51.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.246+0000 7fda63552640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:51.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.246+0000 7fda63552640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fda5c09b030 con 0x7fda5c13be90 2024-09-16T04:44:51.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.246+0000 7fda63552640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:51.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.246+0000 7fda63552640 10 monclient(hunting): authenticate will time out at 1089.662864s 2024-09-16T04:44:51.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.246+0000 7fda61aef640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c13be90 msgr2=0x7fda5c13e280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:44:51.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.246+0000 7fda61aef640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c13be90 0x7fda5c13e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:51.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.246+0000 7fda60aed640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda5c0a3ff0 0x7fda5c0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:51.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda60aed640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda5c0a3ff0 0x7fda5c0a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60464/0 (socket says 172.21.15.46:60464) 2024-09-16T04:44:51.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda60aed640 1 -- 172.21.15.46:0/157947359 learned_addr learned my addr 172.21.15.46:0/157947359 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:51.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda60aed640 10 monclient(hunting): get_auth_request con 0x7fda5c0a3ff0 auth_method 0 2024-09-16T04:44:51.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda612ee640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fda5c0a3c10 0x7fda5c09bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56246/0 (socket says 172.21.15.46:56246) 2024-09-16T04:44:51.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda60aed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:51.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda60aed640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:51.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda60aed640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:51.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda60aed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:51.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda60aed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:51.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda60aed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:51.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda60aed640 10 monclient(hunting): handle_auth_done global_id 74211 payload 274 2024-09-16T04:44:51.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda60aed640 1 -- 172.21.15.46:0/157947359 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c13be90 msgr2=0x7fda5c13e280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:44:51.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda60aed640 1 --2- 172.21.15.46:0/157947359 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c13be90 0x7fda5c13e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:51.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda60aed640 1 -- 172.21.15.46:0/157947359 >> v1:172.21.15.46:6789/0 conn(0x7fda5c0a3c10 legacy=0x7fda5c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:51.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda60aed640 10 monclient: _finish_hunting 0 2024-09-16T04:44:51.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda60aed640 1 monclient: found mon.noname-a 2024-09-16T04:44:51.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda60aed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:51.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda60aed640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:44:51.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda60aed640 1 -- 172.21.15.46:0/157947359 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda5c09b310 con 0x7fda5c0a3ff0 2024-09-16T04:44:51.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda53fff640 1 -- 172.21.15.46:0/157947359 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2368409536 0 0) 0x7fda5c09b940 con 0x7fda5c0a3c10 2024-09-16T04:44:51.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda53fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:44:51.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.247+0000 7fda60aed640 1 --2- 172.21.15.46:0/157947359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda5c0a3ff0 0x7fda5c0a43d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fda4c009870 tx=0x7fda4c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83c799183b0bdba7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:51.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda53fff640 1 -- 172.21.15.46:0/157947359 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4c037070 con 0x7fda5c0a3ff0 2024-09-16T04:44:51.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:51.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda53fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:44:51.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda53fff640 10 monclient: dump: 2024-09-16T04:44:51.264 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:51.264 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:51.264 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.264 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.264 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:51.264 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:51.264 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:51.264 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:51.265 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:51.265 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:51.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda53fff640 10 monclient: _finish_auth 0 2024-09-16T04:44:51.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda53fff640 10 monclient: _check_auth_tickets 2024-09-16T04:44:51.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:51.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda63552640 5 monclient: authenticate success, global_id 74211 2024-09-16T04:44:51.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda53fff640 1 -- 172.21.15.46:0/157947359 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda4c032440 con 0x7fda5c0a3ff0 2024-09-16T04:44:51.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda53fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:51.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda63552640 10 monclient: get_monmap_and_config success 2024-09-16T04:44:51.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda53fff640 1 -- 172.21.15.46:0/157947359 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4c032810 con 0x7fda5c0a3ff0 2024-09-16T04:44:51.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:51.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda63552640 4 set_mon_vals no callback set 2024-09-16T04:44:51.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda53fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:51.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda53fff640 10 monclient: dump: 2024-09-16T04:44:51.266 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:51.266 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:51.266 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.266 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.266 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:51.266 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:51.266 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:51.266 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:51.266 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:51.266 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:51.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda63552640 10 monclient: shutdown 2024-09-16T04:44:51.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda63552640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:51.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda63552640 1 -- 172.21.15.46:0/157947359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda5c0a3ff0 msgr2=0x7fda5c0a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:51.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda63552640 1 --2- 172.21.15.46:0/157947359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda5c0a3ff0 0x7fda5c0a43d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fda4c009870 tx=0x7fda4c02a1e0 comp rx=0 tx=0).stop 2024-09-16T04:44:51.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda63552640 1 -- 172.21.15.46:0/157947359 shutdown_connections 2024-09-16T04:44:51.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda63552640 1 --2- 172.21.15.46:0/157947359 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c13be90 0x7fda5c13e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:51.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda63552640 1 --2- 172.21.15.46:0/157947359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda5c0a3ff0 0x7fda5c0a43d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:51.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda63552640 1 -- 172.21.15.46:0/157947359 >> 172.21.15.46:0/157947359 conn(0x7fda5c0a6340 msgr2=0x7fda5c06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:51.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda63552640 1 -- 172.21.15.46:0/157947359 shutdown_connections 2024-09-16T04:44:51.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.248+0000 7fda63552640 1 -- 172.21.15.46:0/157947359 wait complete. 2024-09-16T04:44:51.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.249+0000 7fda63552640 10 monclient: build_initial_monmap 2024-09-16T04:44:51.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.249+0000 7fda63552640 10 monclient: monmap: 2024-09-16T04:44:51.267 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:44:51.267 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:44:51.268 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:44:51.268 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:44:51.268 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:44:51.268 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:51.268 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:44:51.268 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:44:51.268 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:44:51.268 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:51.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.249+0000 7fda63552640 1 Processor -- start 2024-09-16T04:44:51.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.249+0000 7fda63552640 1 -- start start 2024-09-16T04:44:51.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.249+0000 7fda63552640 10 monclient: init 2024-09-16T04:44:51.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda63552640 10 monclient: _reopen_session rank -1 2024-09-16T04:44:51.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda63552640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:44:51.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda63552640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c0a3c10 0x7fda5c133cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:51.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda63552640 10 monclient(hunting): picked mon.noname-b con 0x7fda5c0a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:44:51.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda63552640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda5c0a3ff0 0x7fda5c134230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:51.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda63552640 10 monclient(hunting): picked mon.noname-a con 0x7fda5c0a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:44:51.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda63552640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c13be90 0x7fda5c1cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:51.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda63552640 10 monclient(hunting): picked mon.noname-c con 0x7fda5c13be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:44:51.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda63552640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:51.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda63552640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fda5c0a6b40 con 0x7fda5c0a3ff0 2024-09-16T04:44:51.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda63552640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:51.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda63552640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fda5c0a69c0 con 0x7fda5c0a3c10 2024-09-16T04:44:51.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda61aef640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c13be90 msgr2=0x7fda5c1cbbf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:44:51.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda61aef640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c13be90 0x7fda5c1cbbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:44:51.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda63552640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:51.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda63552640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fda5c13e7c0 con 0x7fda5c13be90 2024-09-16T04:44:51.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda60aed640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda5c0a3ff0 0x7fda5c134230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:51.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda63552640 10 monclient(hunting): _renew_subs 2024-09-16T04:44:51.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda63552640 10 monclient(hunting): authenticate will time out at 1089.666375s 2024-09-16T04:44:51.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda60aed640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda5c0a3ff0 0x7fda5c134230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60480/0 (socket says 172.21.15.46:60480) 2024-09-16T04:44:51.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda60aed640 1 -- 172.21.15.46:0/663621785 learned_addr learned my addr 172.21.15.46:0/663621785 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:44:51.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda60aed640 10 monclient(hunting): get_auth_request con 0x7fda5c0a3ff0 auth_method 0 2024-09-16T04:44:51.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda60aed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:51.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda60aed640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:51.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda60aed640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:51.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda612ee640 1 --2- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c0a3c10 0x7fda5c133cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:51.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda60aed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:51.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda60aed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:51.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda60aed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:51.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda612ee640 10 monclient(hunting): get_auth_request con 0x7fda5c0a3c10 auth_method 0 2024-09-16T04:44:51.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda612ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:51.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda612ee640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:51.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda612ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:51.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda612ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:51.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda612ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:51.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda612ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:51.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda60aed640 10 monclient(hunting): handle_auth_done global_id 74217 payload 824 2024-09-16T04:44:51.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda60aed640 1 -- 172.21.15.46:0/663621785 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c13be90 msgr2=0x7fda5c1cbbf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:44:51.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda60aed640 1 --2- 172.21.15.46:0/663621785 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5c13be90 0x7fda5c1cbbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:51.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda60aed640 1 -- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c0a3c10 msgr2=0x7fda5c133cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:51.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda60aed640 1 --2- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c0a3c10 0x7fda5c133cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:51.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda60aed640 10 monclient: _finish_hunting 0 2024-09-16T04:44:51.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda60aed640 1 monclient: found mon.noname-a 2024-09-16T04:44:51.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda60aed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:44:51.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda60aed640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:44:51.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda60aed640 1 -- 172.21.15.46:0/663621785 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda5c13a140 con 0x7fda5c0a3ff0 2024-09-16T04:44:51.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.250+0000 7fda60aed640 1 --2- 172.21.15.46:0/663621785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda5c0a3ff0 0x7fda5c134230 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fda4c000c00 tx=0x7fda4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:51.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.251+0000 7fda51ffb640 1 -- 172.21.15.46:0/663621785 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4c041070 con 0x7fda5c0a3ff0 2024-09-16T04:44:51.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.251+0000 7fda51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:51.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.251+0000 7fda51ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:44:51.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.251+0000 7fda51ffb640 10 monclient: dump: 2024-09-16T04:44:51.272 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:51.272 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:51.272 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.273 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.273 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:51.273 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:51.273 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:51.273 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:51.273 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:51.273 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:51.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.251+0000 7fda51ffb640 10 monclient: _finish_auth 0 2024-09-16T04:44:51.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.251+0000 7fda51ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:44:51.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.251+0000 7fda51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:44:51.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.251+0000 7fda51ffb640 1 -- 172.21.15.46:0/663621785 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda4c03c870 con 0x7fda5c0a3ff0 2024-09-16T04:44:51.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.251+0000 7fda51ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:44:51.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.251+0000 7fda51ffb640 1 -- 172.21.15.46:0/663621785 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4c037040 con 0x7fda5c0a3ff0 2024-09-16T04:44:51.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.251+0000 7fda51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:51.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.251+0000 7fda51ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:44:51.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.251+0000 7fda51ffb640 10 monclient: dump: 2024-09-16T04:44:51.274 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:51.274 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:51.274 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.274 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.274 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:51.274 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:51.274 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:51.274 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:51.274 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:51.274 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:51.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.251+0000 7fda63552640 5 monclient: authenticate success, global_id 74217 2024-09-16T04:44:51.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.251+0000 7fda63552640 10 monclient: _renew_subs 2024-09-16T04:44:51.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.251+0000 7fda537fe640 4 set_mon_vals no callback set 2024-09-16T04:44:51.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.251+0000 7fda63552640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:51.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.251+0000 7fda63552640 1 -- 172.21.15.46:0/663621785 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda5c13c270 con 0x7fda5c0a3ff0 2024-09-16T04:44:51.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.251+0000 7fda63552640 10 monclient: _renew_subs 2024-09-16T04:44:51.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.251+0000 7fda63552640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:44:51.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.251+0000 7fda63552640 1 -- 172.21.15.46:0/663621785 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda5c139950 con 0x7fda5c0a3ff0 2024-09-16T04:44:51.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.252+0000 7fda51ffb640 1 -- 172.21.15.46:0/663621785 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fda4c044070 con 0x7fda5c0a3ff0 2024-09-16T04:44:51.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.252+0000 7fda63552640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:51.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.252+0000 7fda63552640 1 --2- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c13a380 0x7fda5c1cc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:51.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.252+0000 7fda63552640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:51.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.252+0000 7fda63552640 1 -- 172.21.15.46:0/663621785 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fda5c09faa0 con 0x7fda5c13a380 2024-09-16T04:44:51.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.252+0000 7fda51ffb640 1 --2- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda40038360 0x7fda4003a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:51.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.252+0000 7fda51ffb640 1 -- 172.21.15.46:0/663621785 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 4738+0+0 (secure 0 0 0) 0x7fda4c06c2a0 con 0x7fda5c0a3ff0 2024-09-16T04:44:51.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.252+0000 7fda612ee640 1 --2- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c13a380 0x7fda5c1cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:51.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.252+0000 7fda612ee640 10 monclient: get_auth_request con 0x7fda5c13a380 auth_method 0 2024-09-16T04:44:51.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.252+0000 7fda612ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:51.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.252+0000 7fda612ee640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:51.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.252+0000 7fda612ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:51.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.252+0000 7fda61aef640 1 --2- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda40038360 0x7fda4003a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:51.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.252+0000 7fda612ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:51.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.252+0000 7fda612ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:51.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.252+0000 7fda612ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:51.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.252+0000 7fda61aef640 10 monclient: get_auth_request con 0x7fda40038360 auth_method 0 2024-09-16T04:44:51.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.252+0000 7fda612ee640 10 monclient(hunting): handle_auth_done global_id 6310 payload 824 2024-09-16T04:44:51.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.252+0000 7fda612ee640 1 -- 172.21.15.46:0/663621785 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda5c0a4450 con 0x7fda5c13a380 2024-09-16T04:44:51.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.253+0000 7fda612ee640 1 --2- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c13a380 0x7fda5c1cc130 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fda580011e0 tx=0x7fda58001990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:51.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.253+0000 7fda61aef640 1 --2- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda40038360 0x7fda4003a820 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fda5c06eeb0 tx=0x7fda540091f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:51.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.253+0000 7fda51ffb640 1 -- 172.21.15.46:0/663621785 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda5800f600 con 0x7fda5c13a380 2024-09-16T04:44:51.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.253+0000 7fda51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:51.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.253+0000 7fda51ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:44:51.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.253+0000 7fda51ffb640 10 monclient: dump: 2024-09-16T04:44:51.277 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:51.277 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:51.278 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.278 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.278 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:51.278 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:51.278 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:51.278 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:51.278 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:51.278 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:51.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.253+0000 7fda51ffb640 1 -- 172.21.15.46:0/663621785 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fda58018070 con 0x7fda5c13a380 2024-09-16T04:44:51.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.253+0000 7fda51ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:44:51.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.253+0000 7fda51ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:44:51.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.253+0000 7fda51ffb640 1 -- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c13a380 msgr2=0x7fda5c1cc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:51.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.253+0000 7fda51ffb640 1 --2- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c13a380 0x7fda5c1cc130 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fda580011e0 tx=0x7fda58001990 comp rx=0 tx=0).stop 2024-09-16T04:44:51.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.253+0000 7fda51ffb640 1 -- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c13a380 msgr2=0x7fda5c1cc130 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:44:51.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.253+0000 7fda51ffb640 1 --2- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c13a380 0x7fda5c1cc130 secure :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fda580011e0 tx=0x7fda58001990 comp rx=0 tx=0).stop 2024-09-16T04:44:51.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.259+0000 7fda63552640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:44:51.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.259+0000 7fda63552640 1 --2- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c1cc670 0x7fda5c0a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:44:51.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.259+0000 7fda63552640 10 monclient(hunting): start opening mon connection 2024-09-16T04:44:51.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.259+0000 7fda63552640 1 -- 172.21.15.46:0/663621785 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fda5c1cca80 con 0x7fda5c1cc670 2024-09-16T04:44:51.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda612ee640 1 --2- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c1cc670 0x7fda5c0a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:44:51.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda612ee640 10 monclient: get_auth_request con 0x7fda5c1cc670 auth_method 0 2024-09-16T04:44:51.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda612ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:44:51.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda612ee640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:44:51.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda612ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:44:51.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda612ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:44:51.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda612ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:44:51.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda612ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:44:51.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda612ee640 10 monclient(hunting): handle_auth_done global_id 6313 payload 824 2024-09-16T04:44:51.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda612ee640 1 -- 172.21.15.46:0/663621785 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fda5c13a760 con 0x7fda5c1cc670 2024-09-16T04:44:51.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda612ee640 1 --2- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c1cc670 0x7fda5c0a05e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fda580011e0 tx=0x7fda58001080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:44:51.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda51ffb640 1 -- 172.21.15.46:0/663621785 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda5800fdb0 con 0x7fda5c1cc670 2024-09-16T04:44:51.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:44:51.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda51ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:44:51.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda51ffb640 10 monclient: dump: 2024-09-16T04:44:51.282 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:44:51.282 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:44:51.282 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.282 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:44:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:44:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:44:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:44:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:44:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:44:51.283 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:44:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda51ffb640 1 -- 172.21.15.46:0/663621785 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4868 (secure 0 0 0) 0x7fda5800fdb0 con 0x7fda5c1cc670 2024-09-16T04:44:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda51ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:44:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda51ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:44:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda51ffb640 1 -- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c1cc670 msgr2=0x7fda5c0a05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda51ffb640 1 --2- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c1cc670 0x7fda5c0a05e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fda580011e0 tx=0x7fda58001080 comp rx=0 tx=0).stop 2024-09-16T04:44:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda51ffb640 1 -- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c1cc670 msgr2=0x7fda5c0a05e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:44:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.260+0000 7fda51ffb640 1 --2- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c1cc670 0x7fda5c0a05e0 secure :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fda580011e0 tx=0x7fda58001080 comp rx=0 tx=0).stop 2024-09-16T04:44:51.284 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:44:51.284 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:44:51.284 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:44:51.284 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:44:51.284 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 72, 2024-09-16T04:44:51.284 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 377163, 2024-09-16T04:44:51.284 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:44:51.284 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:44:51.284 INFO:teuthology.orchestra.run.smithi046.stdout: 1 2024-09-16T04:44:51.284 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:51.285 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:44:51.285 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:44:51.285 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:44:51.285 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:44:51.285 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:51.285 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:51.285 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:51.285 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:51.285 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:51.285 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:51.286 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:51.286 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:51.286 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:51.286 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:51.286 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:51.286 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:51.286 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:44:51.286 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:44:51.286 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:51.286 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:51.286 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:51.287 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:51.287 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:51.287 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:51.287 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:51.287 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:51.287 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:51.287 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:51.287 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:51.287 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:51.288 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.288 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:44:51.288 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:44:51.288 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:44:51.288 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:44:51.288 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:44:51.288 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:44:51.289 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:44:51.289 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:44:51.289 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:44:51.289 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:44:51.289 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:44:51.289 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:44:51.289 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:44:51.289 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:44:51.289 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:44:51.289 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:44:51.289 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:44:51.289 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:44:51.290 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:44:51.290 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:44:51.290 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:44:51.290 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:44:51.290 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:44:51.290 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:44:51.290 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:44:51.290 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:44:51.290 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:44:51.290 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:44:51.290 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:51.290 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:44:51.290 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.291 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:44:51.291 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.291 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:44:51.291 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:44:51.291 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:51.291 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:51.291 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.291 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:51.291 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:44:51.291 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:51.291 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.291 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.291 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:51.291 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:44:51.292 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:51.292 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:51.292 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:51.292 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.292 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:44:51.292 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:44:51.292 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:51.292 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:51.292 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:51.292 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.292 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.292 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:44:51.292 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:44:51.292 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:51.293 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:51.293 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.293 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:51.293 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:44:51.293 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:51.293 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.293 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.293 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:51.293 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:44:51.293 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:51.293 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:51.293 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:51.293 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.294 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:44:51.294 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:44:51.294 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:51.294 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:51.294 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:51.294 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.294 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.294 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:44:51.294 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:44:51.294 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:44:51.294 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:44:51.294 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.294 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:44:51.294 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:44:51.295 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:51.295 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.295 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.295 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:44:51.295 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:44:51.295 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:44:51.295 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:51.295 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:51.295 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.295 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:44:51.295 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:44:51.295 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:44:51.295 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:44:51.295 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:44:51.296 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:51.296 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:51.296 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.296 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:44:51.296 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:44:51.296 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.296 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:51.296 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:51.296 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:44:51.296 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:51.296 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:51.296 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:44:51.296 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.296 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:51.297 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:51.297 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5 2024-09-16T04:44:51.297 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:51.297 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:51.297 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:44:51.297 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.297 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:51.297 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:51.297 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5 2024-09-16T04:44:51.297 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:51.297 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:44:51.297 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:44:51.297 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:44:51.298 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:44:51.298 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:44:51.298 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:44:51.298 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:44:51.298 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:44:51.298 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:44:51.298 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:44:51.298 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:44:51.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.261+0000 7fda63552640 1 -- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda40038360 msgr2=0x7fda4003a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:51.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.261+0000 7fda63552640 1 --2- 172.21.15.46:0/663621785 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda40038360 0x7fda4003a820 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fda5c06eeb0 tx=0x7fda540091f0 comp rx=0 tx=0).stop 2024-09-16T04:44:51.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.261+0000 7fda63552640 10 monclient: shutdown 2024-09-16T04:44:51.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.261+0000 7fda63552640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:44:51.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.261+0000 7fda63552640 1 -- 172.21.15.46:0/663621785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda5c0a3ff0 msgr2=0x7fda5c134230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:44:51.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.261+0000 7fda63552640 1 --2- 172.21.15.46:0/663621785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda5c0a3ff0 0x7fda5c134230 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fda4c000c00 tx=0x7fda4c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:44:51.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.261+0000 7fda61aef640 1 -- 172.21.15.46:0/663621785 reap_dead start 2024-09-16T04:44:51.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.261+0000 7fda63552640 1 -- 172.21.15.46:0/663621785 shutdown_connections 2024-09-16T04:44:51.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.261+0000 7fda63552640 1 --2- 172.21.15.46:0/663621785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda5c0a3ff0 0x7fda5c134230 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:44:51.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.261+0000 7fda63552640 1 -- 172.21.15.46:0/663621785 >> 172.21.15.46:0/663621785 conn(0x7fda5c0a6340 msgr2=0x7fda5c0a7110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:44:51.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.261+0000 7fda63552640 1 -- 172.21.15.46:0/663621785 shutdown_connections 2024-09-16T04:44:51.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:44:51.261+0000 7fda63552640 1 -- 172.21.15.46:0/663621785 wait complete. 2024-09-16T04:44:51.300 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:44:51.301 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:44:51.717 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38892]: starting. 2024-09-16T04:44:51.717 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38892]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-38890 2024-09-16T04:44:51.717 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38892]: shutting down. 2024-09-16T04:44:53.807 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:53.807 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38925]: starting. 2024-09-16T04:44:53.807 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38925]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-38890 2024-09-16T04:44:53.807 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38925]: created object 0... 2024-09-16T04:44:53.807 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38925]: created object 25... 2024-09-16T04:44:53.808 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38925]: created object 49... 2024-09-16T04:44:53.808 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38925]: finishing. 2024-09-16T04:44:53.808 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[38925]: shutting down. 2024-09-16T04:44:54.429 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:54.429 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38927]: starting. 2024-09-16T04:44:54.429 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38927]: listing objects. 2024-09-16T04:44:54.429 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38927]: listed object 0... 2024-09-16T04:44:54.430 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38927]: listed object 25... 2024-09-16T04:44:54.430 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38927]: saw 50 objects 2024-09-16T04:44:54.430 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[38927]: shutting down. 2024-09-16T04:44:54.729 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:54.729 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38926]: starting. 2024-09-16T04:44:54.729 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38926]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-38890 2024-09-16T04:44:54.729 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[38926]: shutting down. 2024-09-16T04:44:54.730 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:54.730 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:54.730 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:44:54.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:44:56.118 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38994]: starting. 2024-09-16T04:44:56.118 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38994]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-38993 2024-09-16T04:44:56.118 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38994]: created object 0... 2024-09-16T04:44:56.118 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38994]: created object 25... 2024-09-16T04:44:56.118 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38994]: created object 49... 2024-09-16T04:44:56.119 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38994]: finishing. 2024-09-16T04:44:56.119 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[38994]: shutting down. 2024-09-16T04:44:57.025 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38995]: starting. 2024-09-16T04:44:57.026 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38995]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-38993 2024-09-16T04:44:57.026 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[38995]: shutting down. 2024-09-16T04:44:59.121 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:59.121 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39028]: starting. 2024-09-16T04:44:59.121 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39028]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-38993 2024-09-16T04:44:59.121 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39028]: created object 0... 2024-09-16T04:44:59.121 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39028]: created object 25... 2024-09-16T04:44:59.122 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39028]: created object 49... 2024-09-16T04:44:59.122 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39028]: finishing. 2024-09-16T04:44:59.122 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39028]: shutting down. 2024-09-16T04:44:59.133 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:44:59.133 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39030]: starting. 2024-09-16T04:44:59.133 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39030]: listing objects. 2024-09-16T04:44:59.133 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39030]: listed object 0... 2024-09-16T04:44:59.133 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39030]: listed object 25... 2024-09-16T04:44:59.133 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39030]: saw 50 objects 2024-09-16T04:44:59.134 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39030]: shutting down. 2024-09-16T04:45:00.042 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:00.042 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39029]: starting. 2024-09-16T04:45:00.042 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39029]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-38993 2024-09-16T04:45:00.042 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39029]: shutting down. 2024-09-16T04:45:00.044 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:00.044 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:00.044 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:45:00.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:45:02.134 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39097]: starting. 2024-09-16T04:45:02.134 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39097]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-39096 2024-09-16T04:45:02.135 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39097]: created object 0... 2024-09-16T04:45:02.135 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39097]: created object 25... 2024-09-16T04:45:02.135 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39097]: created object 49... 2024-09-16T04:45:02.135 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39097]: finishing. 2024-09-16T04:45:02.135 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39097]: shutting down. 2024-09-16T04:45:03.054 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39098]: starting. 2024-09-16T04:45:03.055 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39098]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-39096 2024-09-16T04:45:03.055 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39098]: shutting down. 2024-09-16T04:45:05.130 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:05.130 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39131]: starting. 2024-09-16T04:45:05.130 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39131]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-39096 2024-09-16T04:45:05.130 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39131]: created object 0... 2024-09-16T04:45:05.130 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39131]: created object 25... 2024-09-16T04:45:05.131 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39131]: created object 49... 2024-09-16T04:45:05.131 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39131]: finishing. 2024-09-16T04:45:05.131 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39131]: shutting down. 2024-09-16T04:45:05.142 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:05.142 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39133]: starting. 2024-09-16T04:45:05.142 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39133]: listing objects. 2024-09-16T04:45:05.142 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39133]: listed object 0... 2024-09-16T04:45:05.143 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39133]: listed object 25... 2024-09-16T04:45:05.143 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39133]: saw 50 objects 2024-09-16T04:45:05.143 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39133]: shutting down. 2024-09-16T04:45:06.032 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:06.032 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39132]: starting. 2024-09-16T04:45:06.032 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39132]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-39096 2024-09-16T04:45:06.032 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39132]: shutting down. 2024-09-16T04:45:06.033 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:06.034 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:06.034 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:45:06.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:45:08.163 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39200]: starting. 2024-09-16T04:45:08.164 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39200]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-39199 2024-09-16T04:45:08.164 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39200]: created object 0... 2024-09-16T04:45:08.164 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39200]: created object 25... 2024-09-16T04:45:08.164 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39200]: created object 49... 2024-09-16T04:45:08.164 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39200]: finishing. 2024-09-16T04:45:08.164 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39200]: shutting down. 2024-09-16T04:45:09.077 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39201]: starting. 2024-09-16T04:45:09.077 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39201]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-39199 2024-09-16T04:45:09.077 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39201]: shutting down. 2024-09-16T04:45:11.173 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:11.174 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39234]: starting. 2024-09-16T04:45:11.174 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39234]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-39199 2024-09-16T04:45:11.174 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39234]: created object 0... 2024-09-16T04:45:11.174 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39234]: created object 25... 2024-09-16T04:45:11.174 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39234]: created object 49... 2024-09-16T04:45:11.174 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39234]: finishing. 2024-09-16T04:45:11.174 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39234]: shutting down. 2024-09-16T04:45:11.186 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:11.186 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39236]: starting. 2024-09-16T04:45:11.186 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39236]: listing objects. 2024-09-16T04:45:11.186 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39236]: listed object 0... 2024-09-16T04:45:11.186 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39236]: listed object 25... 2024-09-16T04:45:11.186 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39236]: saw 50 objects 2024-09-16T04:45:11.186 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39236]: shutting down. 2024-09-16T04:45:11.301 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:45:11.301 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T04:45:11.301 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:45:11.302 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:45:11.304 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:45:11.305 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:45:11.305 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:45:11.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.409+0000 7fcbc70d0640 10 monclient: get_monmap_and_config 2024-09-16T04:45:11.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.409+0000 7fcbc70d0640 10 monclient: build_initial_monmap 2024-09-16T04:45:11.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.409+0000 7fcbc70d0640 10 monclient: monmap: 2024-09-16T04:45:11.410 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:11.410 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:11.410 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:45:11.409887+0000 2024-09-16T04:45:11.410 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:45:11.409887+0000 2024-09-16T04:45:11.411 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:11.411 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:11.411 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:45:11.411 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:45:11.411 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:45:11.411 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:45:11.411 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:45:11.411 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:11.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc70d0640 1 Processor -- start 2024-09-16T04:45:11.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc70d0640 1 -- start start 2024-09-16T04:45:11.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc70d0640 10 monclient: init 2024-09-16T04:45:11.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc70d0640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:11.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc70d0640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-16T04:45:11.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc70d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcbc00a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:45:11.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc70d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcbc00a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:45:11.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc70d0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcbc013cea0 0x7fcbc013f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:11.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc70d0640 10 monclient(hunting): picked mon.noname-a con 0x7fcbc013cea0 addr v2:172.21.15.46:3300/0 2024-09-16T04:45:11.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc70d0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbc009b940 con 0x7fcbc00a3c10 2024-09-16T04:45:11.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc70d0640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbc009ab00 con 0x7fcbc00a3ff0 2024-09-16T04:45:11.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc70d0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:11.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc70d0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fcbc009b030 con 0x7fcbc013cea0 2024-09-16T04:45:11.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc70d0640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:11.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc70d0640 10 monclient(hunting): authenticate will time out at 1109.825766s 2024-09-16T04:45:11.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc4e6c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcbc00a3c10 0x7fcbc009bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50654/0 (socket says 172.21.15.46:50654) 2024-09-16T04:45:11.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc4e6c640 1 -- 172.21.15.46:0/4043536099 learned_addr learned my addr 172.21.15.46:0/4043536099 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:45:11.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc566d640 1 --2- 172.21.15.46:0/4043536099 >> v2:172.21.15.46:3300/0 conn(0x7fcbc013cea0 0x7fcbc013f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:11.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc566d640 10 monclient(hunting): get_auth_request con 0x7fcbc013cea0 auth_method 0 2024-09-16T04:45:11.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc566d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:11.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc566d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:11.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.410+0000 7fcbc566d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:11.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.411+0000 7fcbc566d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:11.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.411+0000 7fcbc566d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.411+0000 7fcbc566d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.411+0000 7fcbbf7fe640 1 -- 172.21.15.46:0/4043536099 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1631173986 0 0) 0x7fcbc009b940 con 0x7fcbc00a3c10 2024-09-16T04:45:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.411+0000 7fcbbf7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:45:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.411+0000 7fcbbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.411+0000 7fcbbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.411+0000 7fcbbf7fe640 10 monclient(hunting): my global_id is 74337 2024-09-16T04:45:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.411+0000 7fcbbf7fe640 1 -- 172.21.15.46:0/4043536099 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbb0003520 con 0x7fcbc00a3c10 2024-09-16T04:45:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.411+0000 7fcbc566d640 10 monclient(hunting): handle_auth_done global_id 74334 payload 274 2024-09-16T04:45:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.411+0000 7fcbc566d640 1 -- 172.21.15.46:0/4043536099 >> v1:172.21.15.114:6789/0 conn(0x7fcbc00a3ff0 legacy=0x7fcbc00a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:45:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.411+0000 7fcbc566d640 1 -- 172.21.15.46:0/4043536099 >> v1:172.21.15.46:6789/0 conn(0x7fcbc00a3c10 legacy=0x7fcbc009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.411+0000 7fcbc566d640 10 monclient: _finish_hunting 0 2024-09-16T04:45:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.411+0000 7fcbc566d640 1 monclient: found mon.noname-a 2024-09-16T04:45:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.411+0000 7fcbc566d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.411+0000 7fcbc566d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.411+0000 7fcbc566d640 1 -- 172.21.15.46:0/4043536099 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc009b310 con 0x7fcbc013cea0 2024-09-16T04:45:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.411+0000 7fcbc566d640 1 --2- 172.21.15.46:0/4043536099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc013cea0 0x7fcbc013f290 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8009d90 tx=0x7fcbb802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2eea38996ae0b19 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbbf7fe640 1 -- 172.21.15.46:0/4043536099 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8031600 con 0x7fcbc013cea0 2024-09-16T04:45:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbbf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbbf7fe640 10 monclient: dump: 2024-09-16T04:45:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:11.424 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:11.424 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:11.424 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:11.424 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:11.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbbf7fe640 10 monclient: _finish_auth 0 2024-09-16T04:45:11.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbbf7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:45:11.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:11.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbbf7fe640 1 -- 172.21.15.46:0/4043536099 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbb80317a0 con 0x7fcbc013cea0 2024-09-16T04:45:11.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbc70d0640 5 monclient: authenticate success, global_id 74334 2024-09-16T04:45:11.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbc70d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:45:11.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbbf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:11.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbc70d0640 10 monclient: get_monmap_and_config success 2024-09-16T04:45:11.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbbf7fe640 1 -- 172.21.15.46:0/4043536099 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8031b70 con 0x7fcbc013cea0 2024-09-16T04:45:11.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbc70d0640 4 set_mon_vals no callback set 2024-09-16T04:45:11.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:11.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbbf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:11.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbbf7fe640 10 monclient: dump: 2024-09-16T04:45:11.425 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:11.425 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:11.425 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:11.425 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:11.425 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:11.425 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:11.425 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:11.425 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:11.425 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:11.425 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:11.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbc70d0640 10 monclient: shutdown 2024-09-16T04:45:11.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbc70d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:11.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbc70d0640 1 -- 172.21.15.46:0/4043536099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc013cea0 msgr2=0x7fcbc013f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:11.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbc70d0640 1 --2- 172.21.15.46:0/4043536099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc013cea0 0x7fcbc013f290 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8009d90 tx=0x7fcbb802a610 comp rx=0 tx=0).stop 2024-09-16T04:45:11.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbc70d0640 1 -- 172.21.15.46:0/4043536099 shutdown_connections 2024-09-16T04:45:11.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbc70d0640 1 --2- 172.21.15.46:0/4043536099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc013cea0 0x7fcbc013f290 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:11.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbc70d0640 1 -- 172.21.15.46:0/4043536099 >> 172.21.15.46:0/4043536099 conn(0x7fcbc00a6340 msgr2=0x7fcbc006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:11.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbc70d0640 1 -- 172.21.15.46:0/4043536099 shutdown_connections 2024-09-16T04:45:11.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.412+0000 7fcbc70d0640 1 -- 172.21.15.46:0/4043536099 wait complete. 2024-09-16T04:45:11.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.413+0000 7fcbc70d0640 10 monclient: build_initial_monmap 2024-09-16T04:45:11.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.413+0000 7fcbc70d0640 10 monclient: monmap: 2024-09-16T04:45:11.426 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:11.426 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:45:11.426 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:45:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:45:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:45:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:45:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:45:11.427 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.413+0000 7fcbc70d0640 1 Processor -- start 2024-09-16T04:45:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.413+0000 7fcbc70d0640 1 -- start start 2024-09-16T04:45:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.413+0000 7fcbc70d0640 10 monclient: init 2024-09-16T04:45:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc70d0640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc70d0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:45:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc70d0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc00a3c10 0x7fcbc01cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc70d0640 10 monclient(hunting): picked mon.noname-a con 0x7fcbc00a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:45:11.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc70d0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbc00a3ff0 0x7fcbc01cc280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:11.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc70d0640 10 monclient(hunting): picked mon.noname-b con 0x7fcbc00a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:45:11.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc70d0640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcbc013cea0 0x7fcbc0138eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:11.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc70d0640 10 monclient(hunting): picked mon.noname-c con 0x7fcbc013cea0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:11.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc70d0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:11.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc70d0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc013f950 con 0x7fcbc00a3c10 2024-09-16T04:45:11.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc70d0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:11.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc70d0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc013f7d0 con 0x7fcbc00a3ff0 2024-09-16T04:45:11.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc70d0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:11.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc70d0640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fcbc013fad0 con 0x7fcbc013cea0 2024-09-16T04:45:11.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc70d0640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:11.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc70d0640 10 monclient(hunting): authenticate will time out at 1109.829253s 2024-09-16T04:45:11.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc4e6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc00a3c10 0x7fcbc01cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:11.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc4e6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc00a3c10 0x7fcbc01cbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41110/0 (socket says 172.21.15.46:41110) 2024-09-16T04:45:11.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc4e6c640 1 -- 172.21.15.46:0/4252382641 learned_addr learned my addr 172.21.15.46:0/4252382641 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:11.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc4e6c640 10 monclient(hunting): get_auth_request con 0x7fcbc00a3c10 auth_method 0 2024-09-16T04:45:11.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc4e6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:11.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc4e6c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:11.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc4e6c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:11.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc566d640 1 --2- 172.21.15.46:0/4252382641 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcbc013cea0 0x7fcbc0138eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:11.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc566d640 10 monclient(hunting): get_auth_request con 0x7fcbc013cea0 auth_method 0 2024-09-16T04:45:11.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc566d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:11.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc566d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:11.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc566d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:11.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbbffff640 1 --2- 172.21.15.46:0/4252382641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbc00a3ff0 0x7fcbc01cc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:11.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbbffff640 10 monclient(hunting): get_auth_request con 0x7fcbc00a3ff0 auth_method 0 2024-09-16T04:45:11.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:11.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbbffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:11.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:11.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc4e6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:11.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc4e6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:11.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc4e6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:11.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc566d640 1 -- 172.21.15.46:0/4252382641 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcbc013cea0 msgr2=0x7fcbc0138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:45:11.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc566d640 1 -- 172.21.15.46:0/4252382641 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcbc013cea0 msgr2=0x7fcbc0138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:45:11.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc566d640 1 --2- 172.21.15.46:0/4252382641 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcbc013cea0 0x7fcbc0138eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:11.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc566d640 1 --2- 172.21.15.46:0/4252382641 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcbc013cea0 0x7fcbc0138eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:11.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:11.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:11.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:11.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc4e6c640 10 monclient(hunting): handle_auth_done global_id 74340 payload 824 2024-09-16T04:45:11.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc4e6c640 1 -- 172.21.15.46:0/4252382641 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcbc013cea0 msgr2=0x7fcbc0138eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:45:11.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc4e6c640 1 --2- 172.21.15.46:0/4252382641 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcbc013cea0 0x7fcbc0138eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:11.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc4e6c640 1 -- 172.21.15.46:0/4252382641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbc00a3ff0 msgr2=0x7fcbc01cc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:11.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc4e6c640 1 --2- 172.21.15.46:0/4252382641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbc00a3ff0 0x7fcbc01cc280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:11.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc4e6c640 10 monclient: _finish_hunting 0 2024-09-16T04:45:11.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc4e6c640 1 monclient: found mon.noname-a 2024-09-16T04:45:11.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc4e6c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:11.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc4e6c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:11.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc4e6c640 1 -- 172.21.15.46:0/4252382641 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc00a0400 con 0x7fcbc00a3c10 2024-09-16T04:45:11.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbc4e6c640 1 --2- 172.21.15.46:0/4252382641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc00a3c10 0x7fcbc01cbab0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcbb400c430 tx=0x7fcbb400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:11.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbbd7fa640 1 -- 172.21.15.46:0/4252382641 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4017070 con 0x7fcbc00a3c10 2024-09-16T04:45:11.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbbd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:11.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbbd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:11.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbbd7fa640 10 monclient: dump: 2024-09-16T04:45:11.432 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:11.432 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:11.432 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:11.432 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:11.432 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:11.432 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:11.432 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:11.432 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:11.432 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:11.432 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:11.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbbd7fa640 10 monclient: _finish_auth 0 2024-09-16T04:45:11.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbbd7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:45:11.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbbd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:11.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.414+0000 7fcbbd7fa640 1 -- 172.21.15.46:0/4252382641 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbb4005ce0 con 0x7fcbc00a3c10 2024-09-16T04:45:11.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.415+0000 7fcbc70d0640 5 monclient: authenticate success, global_id 74340 2024-09-16T04:45:11.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.415+0000 7fcbbd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:11.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.415+0000 7fcbbd7fa640 1 -- 172.21.15.46:0/4252382641 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb40072f0 con 0x7fcbc00a3c10 2024-09-16T04:45:11.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.415+0000 7fcbc70d0640 10 monclient: _renew_subs 2024-09-16T04:45:11.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.415+0000 7fcbc70d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:11.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.415+0000 7fcbc70d0640 1 -- 172.21.15.46:0/4252382641 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbc011f860 con 0x7fcbc00a3c10 2024-09-16T04:45:11.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.415+0000 7fcbbd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:11.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.415+0000 7fcbbd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:11.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.415+0000 7fcbbd7fa640 10 monclient: dump: 2024-09-16T04:45:11.433 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:11.433 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:11.433 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:11.433 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:11.434 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:11.434 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:11.434 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:11.434 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:11.434 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:11.434 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:11.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.415+0000 7fcbbeffd640 4 set_mon_vals no callback set 2024-09-16T04:45:11.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.415+0000 7fcbc70d0640 10 monclient: _renew_subs 2024-09-16T04:45:11.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.415+0000 7fcbc70d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:11.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.415+0000 7fcbc70d0640 1 -- 172.21.15.46:0/4252382641 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbc0120180 con 0x7fcbc00a3c10 2024-09-16T04:45:11.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.415+0000 7fcbbd7fa640 1 -- 172.21.15.46:0/4252382641 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fcbb401a070 con 0x7fcbc00a3c10 2024-09-16T04:45:11.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.416+0000 7fcbc70d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:11.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.416+0000 7fcbc70d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:11.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.416+0000 7fcbc70d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:11.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.416+0000 7fcbc70d0640 1 -- 172.21.15.46:0/4252382641 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbc009bcf0 con 0x7fcbc00a3c10 2024-09-16T04:45:11.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.416+0000 7fcbbd7fa640 1 --2- 172.21.15.46:0/4252382641 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcba40383b0 0x7fcba403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:11.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.416+0000 7fcbbd7fa640 1 -- 172.21.15.46:0/4252382641 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 4738+0+0 (secure 0 0 0) 0x7fcbb404ac20 con 0x7fcbc00a3c10 2024-09-16T04:45:11.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.416+0000 7fcbbffff640 1 --2- 172.21.15.46:0/4252382641 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcba40383b0 0x7fcba403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:11.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.416+0000 7fcbbffff640 10 monclient: get_auth_request con 0x7fcba40383b0 auth_method 0 2024-09-16T04:45:11.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.416+0000 7fcbbffff640 1 --2- 172.21.15.46:0/4252382641 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcba40383b0 0x7fcba403a870 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00a8620 tx=0x7fcbac007e30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:11.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.421+0000 7fcbbd7fa640 1 -- 172.21.15.46:0/4252382641 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fcbc00a0400 con 0x7fcbc00a3c10 2024-09-16T04:45:11.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.421+0000 7fcbbd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:11.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.421+0000 7fcbbd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:45:11.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.665+0000 7fcbc70d0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:45:11.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.665+0000 7fcbc70d0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:45:11.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.665+0000 7fcbc70d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:11.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.665+0000 7fcbc70d0640 1 -- 172.21.15.46:0/4252382641 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcbc013b3e0 con 0x7fcbc00a3c10 2024-09-16T04:45:11.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.665+0000 7fcbbd7fa640 1 -- 172.21.15.46:0/4252382641 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fcbb4019e10 con 0x7fcbc00a3c10 2024-09-16T04:45:11.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.665+0000 7fcbbd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:45:11.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.665+0000 7fcbbd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:45:11.667 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":72,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:45:11.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.667+0000 7fcbc70d0640 1 -- 172.21.15.46:0/4252382641 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcba40383b0 msgr2=0x7fcba403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:11.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.667+0000 7fcbc70d0640 1 --2- 172.21.15.46:0/4252382641 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcba40383b0 0x7fcba403a870 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00a8620 tx=0x7fcbac007e30 comp rx=0 tx=0).stop 2024-09-16T04:45:11.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.667+0000 7fcbc70d0640 10 monclient: shutdown 2024-09-16T04:45:11.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.667+0000 7fcbc70d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:11.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.667+0000 7fcbc70d0640 1 -- 172.21.15.46:0/4252382641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc00a3c10 msgr2=0x7fcbc01cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:11.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.667+0000 7fcbc70d0640 1 --2- 172.21.15.46:0/4252382641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc00a3c10 0x7fcbc01cbab0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcbb400c430 tx=0x7fcbb400c8f0 comp rx=0 tx=0).stop 2024-09-16T04:45:11.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.667+0000 7fcbc70d0640 1 -- 172.21.15.46:0/4252382641 shutdown_connections 2024-09-16T04:45:11.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.667+0000 7fcbc70d0640 1 --2- 172.21.15.46:0/4252382641 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcbc013cea0 0x7fcbc0138eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:11.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.667+0000 7fcbc70d0640 1 --2- 172.21.15.46:0/4252382641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbc00a3ff0 0x7fcbc01cc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:11.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.667+0000 7fcbc70d0640 1 --2- 172.21.15.46:0/4252382641 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcba40383b0 0x7fcba403a870 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:11.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.667+0000 7fcbc70d0640 1 --2- 172.21.15.46:0/4252382641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc00a3c10 0x7fcbc01cbab0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:11.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.667+0000 7fcbc70d0640 1 -- 172.21.15.46:0/4252382641 >> 172.21.15.46:0/4252382641 conn(0x7fcbc00a6340 msgr2=0x7fcbc013e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:11.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.667+0000 7fcbc70d0640 1 -- 172.21.15.46:0/4252382641 shutdown_connections 2024-09-16T04:45:11.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:11.667+0000 7fcbc70d0640 1 -- 172.21.15.46:0/4252382641 wait complete. 2024-09-16T04:45:14.679 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:45:14.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.792+0000 7f5d591f1640 10 monclient: get_monmap_and_config 2024-09-16T04:45:14.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.792+0000 7f5d591f1640 10 monclient: build_initial_monmap 2024-09-16T04:45:14.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.792+0000 7f5d591f1640 10 monclient: monmap: 2024-09-16T04:45:14.793 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:14.793 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:14.794 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:45:14.792925+0000 2024-09-16T04:45:14.794 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:45:14.792925+0000 2024-09-16T04:45:14.794 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:14.794 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:14.794 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:45:14.794 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:45:14.794 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:45:14.794 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:45:14.794 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:45:14.794 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:14.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d591f1640 1 Processor -- start 2024-09-16T04:45:14.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d591f1640 1 -- start start 2024-09-16T04:45:14.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d591f1640 10 monclient: init 2024-09-16T04:45:14.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d591f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:14.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d591f1640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-16T04:45:14.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d591f1640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 0x7f5d540a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:14.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d591f1640 10 monclient(hunting): picked mon.noname-c con 0x7f5d5406e160 addr v2:172.21.15.114:3300/0 2024-09-16T04:45:14.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d591f1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d540a2ee0 addr v1:172.21.15.46:6789/0 2024-09-16T04:45:14.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d591f1640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 0x7f5d5413ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:14.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d591f1640 10 monclient(hunting): picked mon.noname-b con 0x7f5d54137050 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:14.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d591f1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d540a3fd0 con 0x7f5d540a2ee0 2024-09-16T04:45:14.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d591f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:14.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d591f1640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f5d540a3300 con 0x7f5d5406e160 2024-09-16T04:45:14.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d591f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:14.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d591f1640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f5d5406e540 con 0x7f5d54137050 2024-09-16T04:45:14.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d591f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:14.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d591f1640 10 monclient(hunting): authenticate will time out at 1113.208656s 2024-09-16T04:45:14.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d5259c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5d540a2ee0 0x7f5d54136940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45444/0 (socket says 172.21.15.46:45444) 2024-09-16T04:45:14.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d5259c640 1 -- 172.21.15.46:0/921577918 learned_addr learned my addr 172.21.15.46:0/921577918 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:45:14.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d5359e640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 0x7f5d5413ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:14.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d5359e640 10 monclient(hunting): get_auth_request con 0x7f5d54137050 auth_method 0 2024-09-16T04:45:14.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d5359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:14.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d5359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:14.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d5359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:14.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d52d9d640 1 --2- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 0x7f5d540a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:14.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d52d9d640 10 monclient(hunting): get_auth_request con 0x7f5d5406e160 auth_method 0 2024-09-16T04:45:14.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:14.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:14.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d52d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:14.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.793+0000 7f5d5359e640 1 -- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 msgr2=0x7f5d5413ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:45:14.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.794+0000 7f5d5359e640 1 -- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 msgr2=0x7f5d5413ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:14.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.794+0000 7f5d5359e640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 0x7f5d5413ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:14.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.794+0000 7f5d5359e640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 0x7f5d5413ec80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:14.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.794+0000 7f5d52d9d640 1 -- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 msgr2=0x7f5d540a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:45:14.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.794+0000 7f5d52d9d640 1 -- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 msgr2=0x7f5d540a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:14.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.794+0000 7f5d52d9d640 1 --2- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 0x7f5d540a29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:14.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.794+0000 7f5d52d9d640 1 --2- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 0x7f5d540a29a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:14.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.995+0000 7f5d5359e640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 0x7f5d5413ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:14.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.995+0000 7f5d5359e640 10 monclient(hunting): get_auth_request con 0x7f5d54137050 auth_method 0 2024-09-16T04:45:14.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.995+0000 7f5d5359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:14.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.995+0000 7f5d5359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:14.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.995+0000 7f5d5359e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:14.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.995+0000 7f5d5359e640 1 -- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 msgr2=0x7f5d5413ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:45:14.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.995+0000 7f5d5359e640 1 -- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 msgr2=0x7f5d5413ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:14.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.995+0000 7f5d5359e640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 0x7f5d5413ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:14.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.995+0000 7f5d5359e640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 0x7f5d5413ec80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:45:14.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.995+0000 7f5d52d9d640 1 --2- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 0x7f5d540a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:14.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.995+0000 7f5d52d9d640 10 monclient(hunting): get_auth_request con 0x7f5d5406e160 auth_method 0 2024-09-16T04:45:14.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.995+0000 7f5d52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:14.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.995+0000 7f5d52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:14.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.995+0000 7f5d52d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:14.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.995+0000 7f5d52d9d640 1 -- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 msgr2=0x7f5d540a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:45:14.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.995+0000 7f5d52d9d640 1 -- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 msgr2=0x7f5d540a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:14.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.995+0000 7f5d52d9d640 1 --2- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 0x7f5d540a29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:14.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:14.995+0000 7f5d52d9d640 1 --2- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 0x7f5d540a29a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:45:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:15.396+0000 7f5d5359e640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 0x7f5d5413ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:15.396+0000 7f5d5359e640 10 monclient(hunting): get_auth_request con 0x7f5d54137050 auth_method 0 2024-09-16T04:45:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:15.396+0000 7f5d5359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:15.396+0000 7f5d5359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:15.396+0000 7f5d5359e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:15.396+0000 7f5d5359e640 1 -- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 msgr2=0x7f5d5413ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:45:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:15.396+0000 7f5d5359e640 1 -- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 msgr2=0x7f5d5413ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:15.396+0000 7f5d5359e640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 0x7f5d5413ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:15.396+0000 7f5d5359e640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 0x7f5d5413ec80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:45:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:15.396+0000 7f5d52d9d640 1 --2- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 0x7f5d540a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:15.396+0000 7f5d52d9d640 10 monclient(hunting): get_auth_request con 0x7f5d5406e160 auth_method 0 2024-09-16T04:45:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:15.396+0000 7f5d52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:15.396+0000 7f5d52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:15.396+0000 7f5d52d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:15.396+0000 7f5d52d9d640 1 -- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 msgr2=0x7f5d540a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:45:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:15.396+0000 7f5d52d9d640 1 -- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 msgr2=0x7f5d540a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:15.396+0000 7f5d52d9d640 1 --2- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 0x7f5d540a29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:15.396+0000 7f5d52d9d640 1 --2- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 0x7f5d540a29a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:45:16.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:16.197+0000 7f5d5359e640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 0x7f5d5413ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:16.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:16.197+0000 7f5d5359e640 10 monclient(hunting): get_auth_request con 0x7f5d54137050 auth_method 0 2024-09-16T04:45:16.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:16.197+0000 7f5d5359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:16.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:16.197+0000 7f5d5359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:16.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:16.197+0000 7f5d5359e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:16.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:16.197+0000 7f5d5359e640 1 -- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 msgr2=0x7f5d5413ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:45:16.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:16.197+0000 7f5d5359e640 1 -- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 msgr2=0x7f5d5413ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:16.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:16.197+0000 7f5d5359e640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 0x7f5d5413ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:16.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:16.197+0000 7f5d5359e640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 0x7f5d5413ec80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:45:16.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:16.197+0000 7f5d52d9d640 1 --2- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 0x7f5d540a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:16.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:16.197+0000 7f5d52d9d640 10 monclient(hunting): get_auth_request con 0x7f5d5406e160 auth_method 0 2024-09-16T04:45:16.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:16.197+0000 7f5d52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:16.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:16.197+0000 7f5d52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:16.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:16.197+0000 7f5d52d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:16.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:16.197+0000 7f5d52d9d640 1 -- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 msgr2=0x7f5d540a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:45:16.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:16.197+0000 7f5d52d9d640 1 -- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 msgr2=0x7f5d540a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:16.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:16.197+0000 7f5d52d9d640 1 --2- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 0x7f5d540a29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:16.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:16.197+0000 7f5d52d9d640 1 --2- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 0x7f5d540a29a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:45:17.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.017+0000 7f5d51d9b640 1 -- 172.21.15.46:0/921577918 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3825196249 0 0) 0x7f5d44001110 con 0x7f5d540a2ee0 2024-09-16T04:45:17.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.017+0000 7f5d51d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:45:17.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.017+0000 7f5d51d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:17.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.017+0000 7f5d51d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:17.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.017+0000 7f5d51d9b640 10 monclient(hunting): my global_id is 74343 2024-09-16T04:45:17.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.017+0000 7f5d51d9b640 1 -- 172.21.15.46:0/921577918 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d40003520 con 0x7f5d540a2ee0 2024-09-16T04:45:17.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 1 -- 172.21.15.46:0/921577918 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 282980178 0 0) 0x7f5d40003520 con 0x7f5d540a2ee0 2024-09-16T04:45:17.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 1 -- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 msgr2=0x7f5d5413ec80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:45:17.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 0x7f5d5413ec80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:17.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 1 -- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 msgr2=0x7f5d540a29a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:45:17.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 1 --2- 172.21.15.46:0/921577918 >> v2:172.21.15.114:3300/0 conn(0x7f5d5406e160 0x7f5d540a29a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:17.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:45:17.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:45:17.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:17.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:45:17.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 1 -- 172.21.15.46:0/921577918 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d540a0bd0 con 0x7f5d540a2ee0 2024-09-16T04:45:17.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 10 monclient: _finish_auth 0 2024-09-16T04:45:17.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:45:17.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:17.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 1 -- 172.21.15.46:0/921577918 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f5d44001840 con 0x7f5d540a2ee0 2024-09-16T04:45:17.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d591f1640 5 monclient: authenticate success, global_id 74343 2024-09-16T04:45:17.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d591f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:45:17.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:17.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:45:17.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 10 monclient: dump: 2024-09-16T04:45:17.026 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:17.026 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:17.026 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:17.026 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:17.027 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:17.027 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:17.027 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:17.027 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:17.027 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:17.027 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:17.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:45:17.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:17.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 1 -- 172.21.15.46:0/921577918 >> v1:172.21.15.46:6789/0 conn(0x7f5d540a2ee0 legacy=0x7f5d54136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:17.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:45:17.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d400039e0 0x7f5d40003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:17.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 10 monclient(hunting): picked mon.b con 0x7f5d400039e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:45:17.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d40004800 0x7f5d40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 10 monclient(hunting): picked mon.a con 0x7f5d40004800 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:45:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 0x7f5d4000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 10 monclient(hunting): picked mon.c con 0x7f5d54137050 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 1 -- 172.21.15.46:0/921577918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5d540a3300 con 0x7f5d40004800 2024-09-16T04:45:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 1 -- 172.21.15.46:0/921577918 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5406e540 con 0x7f5d400039e0 2024-09-16T04:45:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 1 -- 172.21.15.46:0/921577918 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f5d400055a0 con 0x7f5d54137050 2024-09-16T04:45:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d51d9b640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:45:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.019+0000 7f5d591f1640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:45:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5259c640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d40004800 0x7f5d40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5259c640 10 monclient(hunting): get_auth_request con 0x7f5d40004800 auth_method 0 2024-09-16T04:45:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5359e640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 0x7f5d4000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5359e640 10 monclient(hunting): get_auth_request con 0x7f5d54137050 auth_method 0 2024-09-16T04:45:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5359e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d52d9d640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d400039e0 0x7f5d40003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d52d9d640 10 monclient(hunting): get_auth_request con 0x7f5d400039e0 auth_method 0 2024-09-16T04:45:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d52d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5359e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:45:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:45:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d52d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d52d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d52d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:45:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5359e640 10 monclient(hunting): handle_auth_done global_id 74343 payload 293 2024-09-16T04:45:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5359e640 1 -- 172.21.15.46:0/921577918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d400039e0 msgr2=0x7f5d40003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:45:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5359e640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d400039e0 0x7f5d40003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:17.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5359e640 1 -- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d40004800 msgr2=0x7f5d40004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:45:17.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5359e640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d40004800 0x7f5d40004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:17.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5359e640 10 monclient: _finish_hunting 0 2024-09-16T04:45:17.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5359e640 1 monclient: found mon.c 2024-09-16T04:45:17.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:17.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:45:17.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5359e640 1 -- 172.21.15.46:0/921577918 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d40003520 con 0x7f5d54137050 2024-09-16T04:45:17.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5359e640 10 monclient: _finish_auth 0 2024-09-16T04:45:17.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5359e640 10 monclient: _check_auth_tickets 2024-09-16T04:45:17.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5359e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:17.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.020+0000 7f5d5259c640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d40004800 0x7f5d40004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:45:17.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.021+0000 7f5d5359e640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 0x7f5d4000b9c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c004360 tx=0x7f5d40005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e330056a3eaa89d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:17.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.021+0000 7f5d51d9b640 1 -- 172.21.15.46:0/921577918 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c031070 con 0x7f5d54137050 2024-09-16T04:45:17.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.021+0000 7f5d51d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:17.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.021+0000 7f5d51d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:45:17.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.021+0000 7f5d51d9b640 10 monclient: dump: 2024-09-16T04:45:17.032 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:17.032 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:17.032 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:17.032 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:17.032 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:17.032 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:17.032 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:17.032 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:17.032 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:17.033 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.021+0000 7f5d51d9b640 1 -- 172.21.15.46:0/921577918 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d4c02ba80 con 0x7f5d54137050 2024-09-16T04:45:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.021+0000 7f5d591f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:45:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.021+0000 7f5d51d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.021+0000 7f5d51d9b640 1 -- 172.21.15.46:0/921577918 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c004040 con 0x7f5d54137050 2024-09-16T04:45:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.021+0000 7f5d591f1640 10 monclient: get_monmap_and_config success 2024-09-16T04:45:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.021+0000 7f5d591f1640 4 set_mon_vals no callback set 2024-09-16T04:45:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.021+0000 7f5d51d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.021+0000 7f5d51d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:45:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.021+0000 7f5d51d9b640 10 monclient: dump: 2024-09-16T04:45:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:17.034 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.022+0000 7f5d591f1640 10 monclient: shutdown 2024-09-16T04:45:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.022+0000 7f5d591f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.022+0000 7f5d591f1640 1 -- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 msgr2=0x7f5d4000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.022+0000 7f5d591f1640 1 --2- 172.21.15.46:0/921577918 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d54137050 0x7f5d4000b9c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c004360 tx=0x7f5d40005830 comp rx=0 tx=0).stop 2024-09-16T04:45:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.022+0000 7f5d5359e640 1 -- 172.21.15.46:0/921577918 reap_dead start 2024-09-16T04:45:17.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.022+0000 7f5d591f1640 1 -- 172.21.15.46:0/921577918 shutdown_connections 2024-09-16T04:45:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.022+0000 7f5d591f1640 1 -- 172.21.15.46:0/921577918 >> 172.21.15.46:0/921577918 conn(0x7f5d540a5c10 msgr2=0x7f5d540a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.022+0000 7f5d591f1640 1 -- 172.21.15.46:0/921577918 shutdown_connections 2024-09-16T04:45:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.022+0000 7f5d591f1640 1 -- 172.21.15.46:0/921577918 wait complete. 2024-09-16T04:45:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.023+0000 7f5d591f1640 10 monclient: build_initial_monmap 2024-09-16T04:45:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.023+0000 7f5d591f1640 10 monclient: monmap: 2024-09-16T04:45:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:45:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:45:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:45:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:45:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:45:17.035 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:45:17.036 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:17.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.024+0000 7f5d591f1640 1 Processor -- start 2024-09-16T04:45:17.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.024+0000 7f5d591f1640 1 -- start start 2024-09-16T04:45:17.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.024+0000 7f5d591f1640 10 monclient: init 2024-09-16T04:45:17.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d591f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:17.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d591f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:45:17.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d591f1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d54137050 0x7f5d541c8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:17.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d591f1640 10 monclient(hunting): picked mon.noname-b con 0x7f5d54137050 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:45:17.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d591f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d54135f00 0x7f5d54136340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:17.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d591f1640 10 monclient(hunting): picked mon.noname-a con 0x7f5d54135f00 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:45:17.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d591f1640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d541cbc20 0x7f5d541cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:17.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d591f1640 10 monclient(hunting): picked mon.noname-c con 0x7f5d541cbc20 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:17.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d591f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:17.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d591f1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5d540a34a0 con 0x7f5d54135f00 2024-09-16T04:45:17.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d591f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:17.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d591f1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5d540a5130 con 0x7f5d54137050 2024-09-16T04:45:17.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d591f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:17.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d591f1640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f5d54136940 con 0x7f5d541cbc20 2024-09-16T04:45:17.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d591f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:17.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d591f1640 10 monclient(hunting): authenticate will time out at 1115.440232s 2024-09-16T04:45:17.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d54135f00 0x7f5d54136340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:17.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5359e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d541cbc20 0x7f5d541cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:17.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5359e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d541cbc20 0x7f5d541cc0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:34672/0 (socket says 172.21.15.46:34672) 2024-09-16T04:45:17.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5359e640 1 -- 172.21.15.46:0/3989530734 learned_addr learned my addr 172.21.15.46:0/3989530734 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:17.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5359e640 10 monclient(hunting): get_auth_request con 0x7f5d541cbc20 auth_method 0 2024-09-16T04:45:17.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d54135f00 0x7f5d54136340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60958/0 (socket says 172.21.15.46:60958) 2024-09-16T04:45:17.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:17.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:17.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:17.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5259c640 10 monclient(hunting): get_auth_request con 0x7f5d54135f00 auth_method 0 2024-09-16T04:45:17.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:17.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:17.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:17.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d52d9d640 1 --2- 172.21.15.46:0/3989530734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d54137050 0x7f5d541c8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:17.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d52d9d640 10 monclient(hunting): get_auth_request con 0x7f5d54137050 auth_method 0 2024-09-16T04:45:17.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:17.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:17.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d52d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:17.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5359e640 1 -- 172.21.15.46:0/3989530734 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d541cbc20 msgr2=0x7f5d541cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:45:17.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5359e640 1 -- 172.21.15.46:0/3989530734 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d541cbc20 msgr2=0x7f5d541cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:45:17.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5359e640 1 --2- 172.21.15.46:0/3989530734 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d541cbc20 0x7f5d541cc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:17.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:17.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:17.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:17.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5359e640 1 --2- 172.21.15.46:0/3989530734 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d541cbc20 0x7f5d541cc0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:17.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d52d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:17.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d52d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:17.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d52d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:17.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5259c640 10 monclient(hunting): handle_auth_done global_id 74346 payload 824 2024-09-16T04:45:17.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5259c640 1 -- 172.21.15.46:0/3989530734 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d541cbc20 msgr2=0x7f5d541cc0a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:45:17.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5259c640 1 --2- 172.21.15.46:0/3989530734 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d541cbc20 0x7f5d541cc0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5259c640 1 -- 172.21.15.46:0/3989530734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d54137050 msgr2=0x7f5d541c8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5259c640 1 --2- 172.21.15.46:0/3989530734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d54137050 0x7f5d541c8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5259c640 10 monclient: _finish_hunting 0 2024-09-16T04:45:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5259c640 1 monclient: found mon.noname-a 2024-09-16T04:45:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.025+0000 7f5d5259c640 1 -- 172.21.15.46:0/3989530734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d540a8830 con 0x7f5d54135f00 2024-09-16T04:45:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d5259c640 1 --2- 172.21.15.46:0/3989530734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d54135f00 0x7f5d54136340 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5d44001eb0 tx=0x7f5d44007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d4b7fe640 1 -- 172.21.15.46:0/3989530734 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4400f990 con 0x7f5d54135f00 2024-09-16T04:45:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d4b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d4b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d4b7fe640 10 monclient: dump: 2024-09-16T04:45:17.045 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:17.046 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d4b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:45:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d4b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:45:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d4b7fe640 1 -- 172.21.15.46:0/3989530734 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d44018070 con 0x7f5d54135f00 2024-09-16T04:45:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d4b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d4b7fe640 1 -- 172.21.15.46:0/3989530734 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4400fd30 con 0x7f5d54135f00 2024-09-16T04:45:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d591f1640 5 monclient: authenticate success, global_id 74346 2024-09-16T04:45:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d591f1640 10 monclient: _renew_subs 2024-09-16T04:45:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d591f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d591f1640 1 -- 172.21.15.46:0/3989530734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d540a0300 con 0x7f5d54135f00 2024-09-16T04:45:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d4b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d4b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d4b7fe640 10 monclient: dump: 2024-09-16T04:45:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:17.048 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d5159a640 4 set_mon_vals no callback set 2024-09-16T04:45:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d591f1640 10 monclient: _renew_subs 2024-09-16T04:45:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d591f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.026+0000 7f5d591f1640 1 -- 172.21.15.46:0/3989530734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d541235e0 con 0x7f5d54135f00 2024-09-16T04:45:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.027+0000 7f5d4b7fe640 1 -- 172.21.15.46:0/3989530734 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f5d4401d460 con 0x7f5d54135f00 2024-09-16T04:45:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.027+0000 7f5d591f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.027+0000 7f5d591f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.027+0000 7f5d591f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.027+0000 7f5d591f1640 1 -- 172.21.15.46:0/3989530734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d540a4dd0 con 0x7f5d54135f00 2024-09-16T04:45:17.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.027+0000 7f5d4b7fe640 1 --2- 172.21.15.46:0/3989530734 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5d340383b0 0x7f5d3403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:17.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.028+0000 7f5d4b7fe640 1 -- 172.21.15.46:0/3989530734 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 4738+0+0 (secure 0 0 0) 0x7f5d4404b8b0 con 0x7f5d54135f00 2024-09-16T04:45:17.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.028+0000 7f5d52d9d640 1 --2- 172.21.15.46:0/3989530734 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5d340383b0 0x7f5d3403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:17.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.028+0000 7f5d52d9d640 10 monclient: get_auth_request con 0x7f5d340383b0 auth_method 0 2024-09-16T04:45:17.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.028+0000 7f5d52d9d640 1 --2- 172.21.15.46:0/3989530734 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5d340383b0 0x7f5d3403a870 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c006fd0 tx=0x7f5d3c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:17.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.032+0000 7f5d4b7fe640 1 -- 172.21.15.46:0/3989530734 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5d540a3fd0 con 0x7f5d54135f00 2024-09-16T04:45:17.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.032+0000 7f5d4b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:17.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.032+0000 7f5d4b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:45:17.086 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:17.086 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39235]: starting. 2024-09-16T04:45:17.086 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39235]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-39199 2024-09-16T04:45:17.086 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39235]: shutting down. 2024-09-16T04:45:17.087 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:17.087 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:17.087 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:45:17.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:45:17.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.287+0000 7f5d591f1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:45:17.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.287+0000 7f5d591f1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:45:17.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.287+0000 7f5d591f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:17.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.287+0000 7f5d591f1640 1 -- 172.21.15.46:0/3989530734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5d540008d0 con 0x7f5d54135f00 2024-09-16T04:45:17.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.288+0000 7f5d4b7fe640 1 -- 172.21.15.46:0/3989530734 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f5d440133f0 con 0x7f5d54135f00 2024-09-16T04:45:17.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.288+0000 7f5d4b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:45:17.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.288+0000 7f5d4b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:45:17.290 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":76,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:45:17.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.290+0000 7f5d591f1640 1 -- 172.21.15.46:0/3989530734 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5d340383b0 msgr2=0x7f5d3403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:17.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.290+0000 7f5d591f1640 1 --2- 172.21.15.46:0/3989530734 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5d340383b0 0x7f5d3403a870 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c006fd0 tx=0x7f5d3c008040 comp rx=0 tx=0).stop 2024-09-16T04:45:17.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.290+0000 7f5d591f1640 10 monclient: shutdown 2024-09-16T04:45:17.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.290+0000 7f5d591f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:17.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.290+0000 7f5d591f1640 1 -- 172.21.15.46:0/3989530734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d54135f00 msgr2=0x7f5d54136340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:17.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.290+0000 7f5d591f1640 1 --2- 172.21.15.46:0/3989530734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d54135f00 0x7f5d54136340 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5d44001eb0 tx=0x7f5d44007720 comp rx=0 tx=0).stop 2024-09-16T04:45:17.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.290+0000 7f5d591f1640 1 -- 172.21.15.46:0/3989530734 shutdown_connections 2024-09-16T04:45:17.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.290+0000 7f5d591f1640 1 --2- 172.21.15.46:0/3989530734 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5d541cbc20 0x7f5d541cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:17.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.290+0000 7f5d591f1640 1 --2- 172.21.15.46:0/3989530734 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5d340383b0 0x7f5d3403a870 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:17.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.290+0000 7f5d591f1640 1 --2- 172.21.15.46:0/3989530734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d54135f00 0x7f5d54136340 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:17.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.290+0000 7f5d591f1640 1 --2- 172.21.15.46:0/3989530734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d54137050 0x7f5d541c8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:17.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.290+0000 7f5d591f1640 1 -- 172.21.15.46:0/3989530734 >> 172.21.15.46:0/3989530734 conn(0x7f5d540a5c10 msgr2=0x7f5d541bc170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:17.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.290+0000 7f5d591f1640 1 -- 172.21.15.46:0/3989530734 shutdown_connections 2024-09-16T04:45:17.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:17.290+0000 7f5d591f1640 1 -- 172.21.15.46:0/3989530734 wait complete. 2024-09-16T04:45:19.177 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39303]: starting. 2024-09-16T04:45:19.177 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39303]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-39302 2024-09-16T04:45:19.177 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39303]: created object 0... 2024-09-16T04:45:19.177 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39303]: created object 25... 2024-09-16T04:45:19.178 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39303]: created object 49... 2024-09-16T04:45:19.178 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39303]: finishing. 2024-09-16T04:45:19.178 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39303]: shutting down. 2024-09-16T04:45:20.025 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39304]: starting. 2024-09-16T04:45:20.026 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39304]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-39302 2024-09-16T04:45:20.026 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39304]: shutting down. 2024-09-16T04:45:20.302 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:45:20.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.418+0000 7f526199f640 10 monclient: get_monmap_and_config 2024-09-16T04:45:20.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.418+0000 7f526199f640 10 monclient: build_initial_monmap 2024-09-16T04:45:20.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.418+0000 7f526199f640 10 monclient: monmap: 2024-09-16T04:45:20.420 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:20.420 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:20.420 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:45:20.419413+0000 2024-09-16T04:45:20.420 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:45:20.419413+0000 2024-09-16T04:45:20.420 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:20.420 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:20.420 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:45:20.421 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:45:20.421 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:45:20.421 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:45:20.421 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:45:20.421 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:20.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.419+0000 7f526199f640 1 Processor -- start 2024-09-16T04:45:20.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.419+0000 7f526199f640 1 -- start start 2024-09-16T04:45:20.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.419+0000 7f526199f640 10 monclient: init 2024-09-16T04:45:20.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526199f640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:20.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526199f640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-16T04:45:20.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526199f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f525c0a3a30 0x7f525c0a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526199f640 10 monclient(hunting): picked mon.noname-a con 0x7f525c0a3a30 addr v2:172.21.15.46:3300/0 2024-09-16T04:45:20.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526199f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f525c0a1d40 addr v1:172.21.15.114:6789/0 2024-09-16T04:45:20.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526199f640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f525c0a4350 0x7f525c136f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526199f640 10 monclient(hunting): picked mon.noname-b con 0x7f525c0a4350 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:20.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526199f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f525c0a0ac0 con 0x7f525c0a1d40 2024-09-16T04:45:20.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526199f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526199f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f525c0a50c0 con 0x7f525c0a3a30 2024-09-16T04:45:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526199f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526199f640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f525c0a0da0 con 0x7f525c0a4350 2024-09-16T04:45:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526199f640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526199f640 10 monclient(hunting): authenticate will time out at 1118.835433s 2024-09-16T04:45:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526099d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f525c0a3a30 0x7f525c0a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526119e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f525c0a4350 0x7f525c136f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526099d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f525c0a3a30 0x7f525c0a3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60966/0 (socket says 172.21.15.46:60966) 2024-09-16T04:45:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526099d640 1 -- 172.21.15.46:0/3141632634 learned_addr learned my addr 172.21.15.46:0/3141632634 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526099d640 10 monclient(hunting): get_auth_request con 0x7f525c0a3a30 auth_method 0 2024-09-16T04:45:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526099d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526099d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526099d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526119e640 10 monclient(hunting): get_auth_request con 0x7f525c0a4350 auth_method 0 2024-09-16T04:45:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526119e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526119e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526119e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526099d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526099d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526099d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526119e640 1 -- 172.21.15.46:0/3141632634 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f525c0a4350 msgr2=0x7f525c136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:45:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526119e640 1 -- 172.21.15.46:0/3141632634 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f525c0a4350 msgr2=0x7f525c136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526119e640 1 --2- 172.21.15.46:0/3141632634 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f525c0a4350 0x7f525c136f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526119e640 1 --2- 172.21.15.46:0/3141632634 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f525c0a4350 0x7f525c136f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526099d640 10 monclient(hunting): handle_auth_done global_id 74382 payload 274 2024-09-16T04:45:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526099d640 1 -- 172.21.15.46:0/3141632634 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f525c0a4350 msgr2=0x7f525c136f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:45:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526099d640 1 --2- 172.21.15.46:0/3141632634 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f525c0a4350 0x7f525c136f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526099d640 1 -- 172.21.15.46:0/3141632634 >> v1:172.21.15.114:6789/0 conn(0x7f525c0a1d40 legacy=0x7f525c136880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:45:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526099d640 10 monclient: _finish_hunting 0 2024-09-16T04:45:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526099d640 1 monclient: found mon.noname-a 2024-09-16T04:45:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526099d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526099d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.420+0000 7f526099d640 1 -- 172.21.15.46:0/3141632634 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f525c0a1080 con 0x7f525c0a3a30 2024-09-16T04:45:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.421+0000 7f526099d640 1 --2- 172.21.15.46:0/3141632634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0a3a30 0x7f525c0a3e10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f524c009870 tx=0x7f524c02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d0281334ae05e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.421+0000 7f525b7fe640 1 -- 172.21.15.46:0/3141632634 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524c037070 con 0x7f525c0a3a30 2024-09-16T04:45:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.421+0000 7f525b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.421+0000 7f525b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:20.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.421+0000 7f525b7fe640 10 monclient: dump: 2024-09-16T04:45:20.433 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:20.433 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:20.433 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.433 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.433 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:20.433 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:20.433 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:20.433 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:20.433 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:20.433 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:20.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.421+0000 7f525b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:45:20.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.421+0000 7f525b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:45:20.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.421+0000 7f525b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.421+0000 7f525b7fe640 1 -- 172.21.15.46:0/3141632634 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f524c02ae10 con 0x7f525c0a3a30 2024-09-16T04:45:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.421+0000 7f526199f640 5 monclient: authenticate success, global_id 74382 2024-09-16T04:45:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.421+0000 7f525b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.421+0000 7f525b7fe640 1 -- 172.21.15.46:0/3141632634 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524c032660 con 0x7f525c0a3a30 2024-09-16T04:45:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.421+0000 7f526199f640 10 monclient: get_monmap_and_config success 2024-09-16T04:45:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.421+0000 7f526199f640 4 set_mon_vals no callback set 2024-09-16T04:45:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.421+0000 7f525b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.421+0000 7f525b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.421+0000 7f525b7fe640 10 monclient: dump: 2024-09-16T04:45:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:20.435 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.422+0000 7f526199f640 10 monclient: shutdown 2024-09-16T04:45:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.422+0000 7f526199f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.422+0000 7f526199f640 1 -- 172.21.15.46:0/3141632634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0a3a30 msgr2=0x7f525c0a3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.422+0000 7f526199f640 1 --2- 172.21.15.46:0/3141632634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0a3a30 0x7f525c0a3e10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f524c009870 tx=0x7f524c02a100 comp rx=0 tx=0).stop 2024-09-16T04:45:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.422+0000 7f526199f640 1 -- 172.21.15.46:0/3141632634 shutdown_connections 2024-09-16T04:45:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.422+0000 7f526199f640 1 --2- 172.21.15.46:0/3141632634 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f525c0a4350 0x7f525c136f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.422+0000 7f526199f640 1 --2- 172.21.15.46:0/3141632634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0a3a30 0x7f525c0a3e10 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.422+0000 7f526199f640 1 -- 172.21.15.46:0/3141632634 >> 172.21.15.46:0/3141632634 conn(0x7f525c0a5b30 msgr2=0x7f525c0a5f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.422+0000 7f526199f640 1 -- 172.21.15.46:0/3141632634 shutdown_connections 2024-09-16T04:45:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.422+0000 7f526199f640 1 -- 172.21.15.46:0/3141632634 wait complete. 2024-09-16T04:45:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.423+0000 7f526199f640 10 monclient: build_initial_monmap 2024-09-16T04:45:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.423+0000 7f526199f640 10 monclient: monmap: 2024-09-16T04:45:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:45:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:45:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:45:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:45:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:45:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:45:20.436 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:20.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.423+0000 7f526199f640 1 Processor -- start 2024-09-16T04:45:20.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.423+0000 7f526199f640 1 -- start start 2024-09-16T04:45:20.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.423+0000 7f526199f640 10 monclient: init 2024-09-16T04:45:20.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526199f640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:20.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526199f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:45:20.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526199f640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f525c0a1d40 0x7f525c1cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526199f640 10 monclient(hunting): picked mon.noname-c con 0x7f525c0a1d40 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:20.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526199f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f525c0a3a30 0x7f525c135950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526199f640 10 monclient(hunting): picked mon.noname-b con 0x7f525c0a3a30 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:45:20.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526199f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0a4350 0x7f525c135e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526199f640 10 monclient(hunting): picked mon.noname-a con 0x7f525c0a4350 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:45:20.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526199f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526199f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f525c06e1d0 con 0x7f525c0a4350 2024-09-16T04:45:20.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526199f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526199f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f525c0a0530 con 0x7f525c0a3a30 2024-09-16T04:45:20.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526199f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526199f640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f525c06e350 con 0x7f525c0a1d40 2024-09-16T04:45:20.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526199f640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:20.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526199f640 10 monclient(hunting): authenticate will time out at 1118.839129s 2024-09-16T04:45:20.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526099d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f525c0a1d40 0x7f525c1cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526119e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0a4350 0x7f525c135e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526099d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f525c0a1d40 0x7f525c1cbc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:34704/0 (socket says 172.21.15.46:34704) 2024-09-16T04:45:20.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526099d640 1 -- 172.21.15.46:0/259395111 learned_addr learned my addr 172.21.15.46:0/259395111 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:20.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526099d640 10 monclient(hunting): get_auth_request con 0x7f525c0a1d40 auth_method 0 2024-09-16T04:45:20.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526099d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526099d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526099d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:20.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526119e640 10 monclient(hunting): get_auth_request con 0x7f525c0a4350 auth_method 0 2024-09-16T04:45:20.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526119e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526119e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526119e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:20.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526099d640 1 -- 172.21.15.46:0/259395111 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f525c0a1d40 msgr2=0x7f525c1cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:45:20.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526099d640 1 -- 172.21.15.46:0/259395111 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f525c0a1d40 msgr2=0x7f525c1cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:45:20.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526119e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:20.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526119e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:20.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526099d640 1 --2- 172.21.15.46:0/259395111 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f525c0a1d40 0x7f525c1cbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:20.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526119e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:20.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526099d640 1 --2- 172.21.15.46:0/259395111 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f525c0a1d40 0x7f525c1cbc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:20.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f525bfff640 1 --2- 172.21.15.46:0/259395111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f525c0a3a30 0x7f525c135950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f525bfff640 10 monclient(hunting): get_auth_request con 0x7f525c0a3a30 auth_method 0 2024-09-16T04:45:20.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f525bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f525bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f525bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:20.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526119e640 10 monclient(hunting): handle_auth_done global_id 74385 payload 824 2024-09-16T04:45:20.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526119e640 1 -- 172.21.15.46:0/259395111 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f525c0a1d40 msgr2=0x7f525c1cbc50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:45:20.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526119e640 1 --2- 172.21.15.46:0/259395111 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f525c0a1d40 0x7f525c1cbc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526119e640 1 -- 172.21.15.46:0/259395111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f525c0a3a30 msgr2=0x7f525c135950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:20.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526119e640 1 --2- 172.21.15.46:0/259395111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f525c0a3a30 0x7f525c135950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526119e640 10 monclient: _finish_hunting 0 2024-09-16T04:45:20.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526119e640 1 monclient: found mon.noname-a 2024-09-16T04:45:20.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526119e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:20.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526119e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:20.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526119e640 1 -- 172.21.15.46:0/259395111 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f525c1cc190 con 0x7f525c0a4350 2024-09-16T04:45:20.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526119e640 1 --2- 172.21.15.46:0/259395111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0a4350 0x7f525c135e90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5254007cf0 tx=0x7f525400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:20.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f52597fa640 1 -- 172.21.15.46:0/259395111 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525400edd0 con 0x7f525c0a4350 2024-09-16T04:45:20.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f52597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:20.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f52597fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:20.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f52597fa640 10 monclient: dump: 2024-09-16T04:45:20.441 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:20.441 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:20.441 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.441 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.441 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:20.441 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:20.441 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:20.442 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:20.442 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:20.442 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:20.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f52597fa640 10 monclient: _finish_auth 0 2024-09-16T04:45:20.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f52597fa640 10 monclient: _check_auth_tickets 2024-09-16T04:45:20.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f52597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:20.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f52597fa640 1 -- 172.21.15.46:0/259395111 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5254003990 con 0x7f525c0a4350 2024-09-16T04:45:20.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.424+0000 7f526199f640 5 monclient: authenticate success, global_id 74385 2024-09-16T04:45:20.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.425+0000 7f526199f640 10 monclient: _renew_subs 2024-09-16T04:45:20.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.425+0000 7f526199f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:20.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.425+0000 7f526199f640 1 -- 172.21.15.46:0/259395111 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f525c125520 con 0x7f525c0a4350 2024-09-16T04:45:20.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.425+0000 7f52597fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:20.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.425+0000 7f52597fa640 1 -- 172.21.15.46:0/259395111 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5254003d80 con 0x7f525c0a4350 2024-09-16T04:45:20.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.425+0000 7f52597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:20.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.425+0000 7f52597fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:20.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.425+0000 7f52597fa640 10 monclient: dump: 2024-09-16T04:45:20.443 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:20.443 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:20.443 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.443 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.443 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:20.443 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:20.443 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:20.443 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:20.443 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:20.443 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:20.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.425+0000 7f526199f640 10 monclient: _renew_subs 2024-09-16T04:45:20.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.425+0000 7f525a7fc640 4 set_mon_vals no callback set 2024-09-16T04:45:20.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.425+0000 7f526199f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:20.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.425+0000 7f526199f640 1 -- 172.21.15.46:0/259395111 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f525c0a86b0 con 0x7f525c0a4350 2024-09-16T04:45:20.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.426+0000 7f52597fa640 1 -- 172.21.15.46:0/259395111 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f525401b020 con 0x7f525c0a4350 2024-09-16T04:45:20.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.426+0000 7f526199f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:20.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.426+0000 7f526199f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:20.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.426+0000 7f526199f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:20.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.426+0000 7f526199f640 1 -- 172.21.15.46:0/259395111 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f525c0a5630 con 0x7f525c0a4350 2024-09-16T04:45:20.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.426+0000 7f52597fa640 1 --2- 172.21.15.46:0/259395111 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f52480383b0 0x7f524803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.426+0000 7f52597fa640 1 -- 172.21.15.46:0/259395111 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(251..251 src has 1..251) ==== 4346+0+0 (secure 0 0 0) 0x7f525404b7c0 con 0x7f525c0a4350 2024-09-16T04:45:20.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.426+0000 7f526099d640 1 --2- 172.21.15.46:0/259395111 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f52480383b0 0x7f524803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.426+0000 7f526099d640 10 monclient: get_auth_request con 0x7f52480383b0 auth_method 0 2024-09-16T04:45:20.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.427+0000 7f526099d640 1 --2- 172.21.15.46:0/259395111 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f52480383b0 0x7f524803a870 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f524c035040 tx=0x7f524c02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:20.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.431+0000 7f52597fa640 1 -- 172.21.15.46:0/259395111 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f525c0a5630 con 0x7f525c0a4350 2024-09-16T04:45:20.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.431+0000 7f52597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:20.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.431+0000 7f52597fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:45:20.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.679+0000 7f526199f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:45:20.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.679+0000 7f526199f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:45:20.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.679+0000 7f526199f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:20.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.679+0000 7f526199f640 1 -- 172.21.15.46:0/259395111 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f525c136880 con 0x7f525c0a4350 2024-09-16T04:45:20.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.680+0000 7f52597fa640 1 -- 172.21.15.46:0/259395111 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f5254014420 con 0x7f525c0a4350 2024-09-16T04:45:20.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.680+0000 7f52597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:45:20.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.680+0000 7f52597fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:45:20.682 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":78,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:45:20.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.682+0000 7f526199f640 1 -- 172.21.15.46:0/259395111 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f52480383b0 msgr2=0x7f524803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:20.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.682+0000 7f526199f640 1 --2- 172.21.15.46:0/259395111 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f52480383b0 0x7f524803a870 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f524c035040 tx=0x7f524c02e000 comp rx=0 tx=0).stop 2024-09-16T04:45:20.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.682+0000 7f526199f640 10 monclient: shutdown 2024-09-16T04:45:20.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.682+0000 7f526199f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:20.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.682+0000 7f526199f640 1 -- 172.21.15.46:0/259395111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0a4350 msgr2=0x7f525c135e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:20.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.682+0000 7f526199f640 1 --2- 172.21.15.46:0/259395111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0a4350 0x7f525c135e90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5254007cf0 tx=0x7f525400e4b0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.682+0000 7f526199f640 1 -- 172.21.15.46:0/259395111 shutdown_connections 2024-09-16T04:45:20.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.682+0000 7f526199f640 1 --2- 172.21.15.46:0/259395111 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f52480383b0 0x7f524803a870 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.682+0000 7f526199f640 1 --2- 172.21.15.46:0/259395111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f525c0a4350 0x7f525c135e90 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.682+0000 7f526199f640 1 --2- 172.21.15.46:0/259395111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f525c0a3a30 0x7f525c135950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.682+0000 7f526199f640 1 --2- 172.21.15.46:0/259395111 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f525c0a1d40 0x7f525c1cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.682+0000 7f526199f640 1 -- 172.21.15.46:0/259395111 >> 172.21.15.46:0/259395111 conn(0x7f525c0a5b30 msgr2=0x7f525c0a2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:20.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.682+0000 7f526199f640 1 -- 172.21.15.46:0/259395111 shutdown_connections 2024-09-16T04:45:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.682+0000 7f526199f640 1 -- 172.21.15.46:0/259395111 wait complete. 2024-09-16T04:45:20.694 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:45:20.694 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:45:20.694 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:45:20.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.795+0000 7fa9615fd640 10 monclient: get_monmap_and_config 2024-09-16T04:45:20.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.795+0000 7fa9615fd640 10 monclient: build_initial_monmap 2024-09-16T04:45:20.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.795+0000 7fa9615fd640 10 monclient: monmap: 2024-09-16T04:45:20.797 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:20.797 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:20.797 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:45:20.796205+0000 2024-09-16T04:45:20.797 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:45:20.796205+0000 2024-09-16T04:45:20.797 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:20.797 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:20.797 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:45:20.797 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:45:20.797 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:45:20.797 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:45:20.798 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:45:20.798 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:20.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.796+0000 7fa9615fd640 1 Processor -- start 2024-09-16T04:45:20.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.796+0000 7fa9615fd640 1 -- start start 2024-09-16T04:45:20.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.796+0000 7fa9615fd640 10 monclient: init 2024-09-16T04:45:20.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.796+0000 7fa9615fd640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:20.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.796+0000 7fa9615fd640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-16T04:45:20.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.796+0000 7fa9615fd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa95c131a10 0x7fa95c09ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.796+0000 7fa9615fd640 10 monclient(hunting): picked mon.noname-b con 0x7fa95c131a10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:20.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.796+0000 7fa9615fd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa95c09e760 addr v1:172.21.15.46:6789/0 2024-09-16T04:45:20.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.796+0000 7fa9615fd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa95c139af0 0x7fa95c13bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.796+0000 7fa9615fd640 10 monclient(hunting): picked mon.noname-a con 0x7fa95c139af0 addr v2:172.21.15.46:3300/0 2024-09-16T04:45:20.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.796+0000 7fa9615fd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa95c0a0f40 con 0x7fa95c09e760 2024-09-16T04:45:20.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.796+0000 7fa9615fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.796+0000 7fa9615fd640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa95c0a1220 con 0x7fa95c139af0 2024-09-16T04:45:20.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.796+0000 7fa9615fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.796+0000 7fa9615fd640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fa95c09f810 con 0x7fa95c131a10 2024-09-16T04:45:20.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.796+0000 7fa9615fd640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:20.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa9615fd640 10 monclient(hunting): authenticate will time out at 1119.212006s 2024-09-16T04:45:20.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95affd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa95c131a10 0x7fa95c09ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95a7fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa95c09e760 0x7fa95c131df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45446/0 (socket says 172.21.15.46:45446) 2024-09-16T04:45:20.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95a7fc640 1 -- 172.21.15.46:0/3953509565 learned_addr learned my addr 172.21.15.46:0/3953509565 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:45:20.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95affd640 10 monclient(hunting): get_auth_request con 0x7fa95c131a10 auth_method 0 2024-09-16T04:45:20.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:20.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95b7fe640 1 --2- 172.21.15.46:0/3953509565 >> v2:172.21.15.46:3300/0 conn(0x7fa95c139af0 0x7fa95c13bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95b7fe640 10 monclient(hunting): get_auth_request con 0x7fa95c139af0 auth_method 0 2024-09-16T04:45:20.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:20.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95affd640 1 -- 172.21.15.46:0/3953509565 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa95c131a10 msgr2=0x7fa95c09ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:45:20.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95affd640 1 -- 172.21.15.46:0/3953509565 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa95c131a10 msgr2=0x7fa95c09ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:20.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95affd640 1 --2- 172.21.15.46:0/3953509565 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa95c131a10 0x7fa95c09ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:20.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:20.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:20.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95affd640 1 --2- 172.21.15.46:0/3953509565 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa95c131a10 0x7fa95c09ec70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95b7fe640 10 monclient(hunting): handle_auth_done global_id 74388 payload 274 2024-09-16T04:45:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95b7fe640 1 -- 172.21.15.46:0/3953509565 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa95c131a10 msgr2=0x7fa95c09ec70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:45:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95b7fe640 1 --2- 172.21.15.46:0/3953509565 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa95c131a10 0x7fa95c09ec70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95b7fe640 1 -- 172.21.15.46:0/3953509565 >> v1:172.21.15.46:6789/0 conn(0x7fa95c09e760 legacy=0x7fa95c131df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:45:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95b7fe640 1 monclient: found mon.noname-a 2024-09-16T04:45:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa95b7fe640 1 -- 172.21.15.46:0/3953509565 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa95c0a2290 con 0x7fa95c139af0 2024-09-16T04:45:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa959ffb640 1 -- 172.21.15.46:0/3953509565 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1715848609 0 0) 0x7fa95c0a0f40 con 0x7fa95c09e760 2024-09-16T04:45:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.797+0000 7fa959ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:45:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa95b7fe640 1 --2- 172.21.15.46:0/3953509565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c139af0 0x7fa95c13bee0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa954009d90 tx=0x7fa95402a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14782bc86924981b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:20.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa959ffb640 1 -- 172.21.15.46:0/3953509565 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa954007740 con 0x7fa95c139af0 2024-09-16T04:45:20.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa959ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:20.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa959ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:20.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa959ffb640 10 monclient: dump: 2024-09-16T04:45:20.811 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:20.811 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:20.811 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.811 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.811 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:20.811 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:20.811 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:20.811 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:20.811 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:20.811 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:20.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa959ffb640 10 monclient: _finish_auth 0 2024-09-16T04:45:20.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa959ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:45:20.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa959ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:20.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa9615fd640 5 monclient: authenticate success, global_id 74388 2024-09-16T04:45:20.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa959ffb640 1 -- 172.21.15.46:0/3953509565 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9540078e0 con 0x7fa95c139af0 2024-09-16T04:45:20.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa9615fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:45:20.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa959ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:20.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa9615fd640 10 monclient: get_monmap_and_config success 2024-09-16T04:45:20.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa9615fd640 4 set_mon_vals no callback set 2024-09-16T04:45:20.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa959ffb640 1 -- 172.21.15.46:0/3953509565 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa954007cb0 con 0x7fa95c139af0 2024-09-16T04:45:20.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa959ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:20.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa959ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:20.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa959ffb640 10 monclient: dump: 2024-09-16T04:45:20.812 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:20.813 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:20.813 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.813 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.813 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:20.813 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:20.813 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:20.813 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:20.813 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:20.813 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:20.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa9615fd640 10 monclient: shutdown 2024-09-16T04:45:20.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa9615fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:20.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa9615fd640 1 -- 172.21.15.46:0/3953509565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c139af0 msgr2=0x7fa95c13bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:20.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa9615fd640 1 --2- 172.21.15.46:0/3953509565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c139af0 0x7fa95c13bee0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa954009d90 tx=0x7fa95402a530 comp rx=0 tx=0).stop 2024-09-16T04:45:20.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa9615fd640 1 -- 172.21.15.46:0/3953509565 shutdown_connections 2024-09-16T04:45:20.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa9615fd640 1 --2- 172.21.15.46:0/3953509565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c139af0 0x7fa95c13bee0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa9615fd640 1 --2- 172.21.15.46:0/3953509565 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa95c131a10 0x7fa95c09ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa9615fd640 1 -- 172.21.15.46:0/3953509565 >> 172.21.15.46:0/3953509565 conn(0x7fa95c0a5960 msgr2=0x7fa95c0a1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:20.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa9615fd640 1 -- 172.21.15.46:0/3953509565 shutdown_connections 2024-09-16T04:45:20.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.798+0000 7fa9615fd640 1 -- 172.21.15.46:0/3953509565 wait complete. 2024-09-16T04:45:20.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.799+0000 7fa9615fd640 10 monclient: build_initial_monmap 2024-09-16T04:45:20.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.799+0000 7fa9615fd640 10 monclient: monmap: 2024-09-16T04:45:20.814 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:20.814 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:45:20.814 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:45:20.814 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:45:20.815 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:20.815 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:20.815 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:45:20.815 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:45:20.815 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:45:20.815 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:20.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.799+0000 7fa9615fd640 1 Processor -- start 2024-09-16T04:45:20.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.799+0000 7fa9615fd640 1 -- start start 2024-09-16T04:45:20.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.799+0000 7fa9615fd640 10 monclient: init 2024-09-16T04:45:20.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa9615fd640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:20.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa9615fd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:45:20.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa9615fd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c09e760 0x7fa95c0720d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa9615fd640 10 monclient(hunting): picked mon.noname-a con 0x7fa95c09e760 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:45:20.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa9615fd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa95c131a10 0x7fa95c09c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa9615fd640 10 monclient(hunting): picked mon.noname-c con 0x7fa95c131a10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:20.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa9615fd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa95c139af0 0x7fa95c09c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa9615fd640 10 monclient(hunting): picked mon.noname-b con 0x7fa95c139af0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:45:20.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa9615fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa9615fd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa95c13c7b0 con 0x7fa95c09e760 2024-09-16T04:45:20.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa9615fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa9615fd640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa95c13c630 con 0x7fa95c139af0 2024-09-16T04:45:20.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa9615fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa9615fd640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fa95c13c930 con 0x7fa95c131a10 2024-09-16T04:45:20.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa9615fd640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:20.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa9615fd640 10 monclient(hunting): authenticate will time out at 1119.215433s 2024-09-16T04:45:20.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c09e760 0x7fa95c0720d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95a7fc640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa95c131a10 0x7fa95c09c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c09e760 0x7fa95c0720d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60988/0 (socket says 172.21.15.46:60988) 2024-09-16T04:45:20.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95a7fc640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa95c131a10 0x7fa95c09c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:34714/0 (socket says 172.21.15.46:34714) 2024-09-16T04:45:20.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95affd640 1 -- 172.21.15.46:0/2998219378 learned_addr learned my addr 172.21.15.46:0/2998219378 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:20.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95affd640 10 monclient(hunting): get_auth_request con 0x7fa95c09e760 auth_method 0 2024-09-16T04:45:20.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:20.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95a7fc640 10 monclient(hunting): get_auth_request con 0x7fa95c131a10 auth_method 0 2024-09-16T04:45:20.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:20.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:20.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:20.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95a7fc640 1 -- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa95c131a10 msgr2=0x7fa95c09c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:45:20.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95a7fc640 1 -- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa95c131a10 msgr2=0x7fa95c09c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:45:20.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:20.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95a7fc640 1 --2- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa95c131a10 0x7fa95c09c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:20.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95a7fc640 1 --2- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa95c131a10 0x7fa95c09c100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:20.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95b7fe640 1 --2- 172.21.15.46:0/2998219378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa95c139af0 0x7fa95c09c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95b7fe640 10 monclient(hunting): get_auth_request con 0x7fa95c139af0 auth_method 0 2024-09-16T04:45:20.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:20.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95affd640 10 monclient(hunting): handle_auth_done global_id 74394 payload 824 2024-09-16T04:45:20.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95affd640 1 -- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa95c131a10 msgr2=0x7fa95c09c100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:45:20.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95affd640 1 --2- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa95c131a10 0x7fa95c09c100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95affd640 1 -- 172.21.15.46:0/2998219378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa95c139af0 msgr2=0x7fa95c09c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:20.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95affd640 1 --2- 172.21.15.46:0/2998219378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa95c139af0 0x7fa95c09c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95affd640 10 monclient: _finish_hunting 0 2024-09-16T04:45:20.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95affd640 1 monclient: found mon.noname-a 2024-09-16T04:45:20.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:20.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:20.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.800+0000 7fa95affd640 1 -- 172.21.15.46:0/2998219378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa95c09dcd0 con 0x7fa95c09e760 2024-09-16T04:45:20.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa95affd640 1 --2- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c09e760 0x7fa95c0720d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa94c00d7d0 tx=0x7fa94c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:20.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa9437fe640 1 -- 172.21.15.46:0/2998219378 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94c00b6c0 con 0x7fa95c09e760 2024-09-16T04:45:20.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa9437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:20.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa9437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:20.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa9437fe640 10 monclient: dump: 2024-09-16T04:45:20.819 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:20.819 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:20.819 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.819 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.819 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:20.820 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:20.820 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:20.820 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:20.820 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:20.820 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:20.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa9437fe640 10 monclient: _finish_auth 0 2024-09-16T04:45:20.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa9437fe640 10 monclient: _check_auth_tickets 2024-09-16T04:45:20.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa9437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:20.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa9615fd640 5 monclient: authenticate success, global_id 74394 2024-09-16T04:45:20.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa9437fe640 1 -- 172.21.15.46:0/2998219378 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa94c019070 con 0x7fa95c09e760 2024-09-16T04:45:20.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa9437fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:20.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa9437fe640 1 -- 172.21.15.46:0/2998219378 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94c00bac0 con 0x7fa95c09e760 2024-09-16T04:45:20.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa9615fd640 10 monclient: _renew_subs 2024-09-16T04:45:20.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa9615fd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:20.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa9615fd640 1 -- 172.21.15.46:0/2998219378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa95c0a3470 con 0x7fa95c09e760 2024-09-16T04:45:20.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa9437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:20.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa9437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:20.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa9437fe640 10 monclient: dump: 2024-09-16T04:45:20.821 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:20.821 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:20.821 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.821 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.821 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:20.821 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:20.821 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:20.821 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:20.821 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:20.822 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:20.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa9597fa640 4 set_mon_vals no callback set 2024-09-16T04:45:20.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa9615fd640 10 monclient: _renew_subs 2024-09-16T04:45:20.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa9615fd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:20.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.801+0000 7fa9615fd640 1 -- 172.21.15.46:0/2998219378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa95c06ee60 con 0x7fa95c09e760 2024-09-16T04:45:20.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa9615fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:20.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa9615fd640 1 --2- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c09cb80 0x7fa95c1358c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa9615fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa9615fd640 1 -- 172.21.15.46:0/2998219378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa95c072fe0 con 0x7fa95c09cb80 2024-09-16T04:45:20.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa9437fe640 1 -- 172.21.15.46:0/2998219378 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fa94c00bc60 con 0x7fa95c09e760 2024-09-16T04:45:20.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa95a7fc640 1 --2- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c09cb80 0x7fa95c1358c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa95a7fc640 10 monclient: get_auth_request con 0x7fa95c09cb80 auth_method 0 2024-09-16T04:45:20.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa95a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa95a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa95a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:20.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa95a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:20.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa95a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:20.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa95a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:20.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa95a7fc640 10 monclient(hunting): handle_auth_done global_id 74397 payload 824 2024-09-16T04:45:20.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa95a7fc640 1 -- 172.21.15.46:0/2998219378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa95c09ec70 con 0x7fa95c09cb80 2024-09-16T04:45:20.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa9437fe640 1 --2- 172.21.15.46:0/2998219378 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa9480383e0 0x7fa94803a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa9437fe640 1 -- 172.21.15.46:0/2998219378 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(251..251 src has 1..251) ==== 4346+0+0 (secure 0 0 0) 0x7fa94c04baa0 con 0x7fa95c09e760 2024-09-16T04:45:20.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa95a7fc640 1 --2- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c09cb80 0x7fa95c1358c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa95c13a870 tx=0x7fa950001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:20.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa9437fe640 1 -- 172.21.15.46:0/2998219378 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa950012590 con 0x7fa95c09cb80 2024-09-16T04:45:20.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa9437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:20.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa9437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:20.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa9437fe640 10 monclient: dump: 2024-09-16T04:45:20.824 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:20.824 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:20.824 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.824 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.824 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:20.824 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:20.824 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:20.824 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:20.824 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:20.824 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:20.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa95b7fe640 1 --2- 172.21.15.46:0/2998219378 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa9480383e0 0x7fa94803a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa95b7fe640 10 monclient: get_auth_request con 0x7fa9480383e0 auth_method 0 2024-09-16T04:45:20.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa9437fe640 1 -- 172.21.15.46:0/2998219378 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa950018070 con 0x7fa95c09cb80 2024-09-16T04:45:20.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa9437fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:20.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa9437fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:45:20.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa9437fe640 1 -- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c09cb80 msgr2=0x7fa95c1358c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:20.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa9437fe640 1 --2- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c09cb80 0x7fa95c1358c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa95c13a870 tx=0x7fa950001970 comp rx=0 tx=0).stop 2024-09-16T04:45:20.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa9437fe640 1 -- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c09cb80 msgr2=0x7fa95c1358c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:45:20.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.802+0000 7fa9437fe640 1 --2- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c09cb80 0x7fa95c1358c0 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa95c13a870 tx=0x7fa950001970 comp rx=0 tx=0).stop 2024-09-16T04:45:20.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.803+0000 7fa95b7fe640 1 --2- 172.21.15.46:0/2998219378 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa9480383e0 0x7fa94803a8a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa954034000 tx=0x7fa95402f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.809+0000 7fa9615fd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:45:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.809+0000 7fa9615fd640 1 --2- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c09cf60 0x7fa95c135e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.809+0000 7fa9615fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.809+0000 7fa9615fd640 1 -- 172.21.15.46:0/2998219378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa95c0700f0 con 0x7fa95c09cf60 2024-09-16T04:45:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.809+0000 7fa95a7fc640 1 --2- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c09cf60 0x7fa95c135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.809+0000 7fa95a7fc640 10 monclient: get_auth_request con 0x7fa95c09cf60 auth_method 0 2024-09-16T04:45:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.809+0000 7fa95a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.809+0000 7fa95a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.809+0000 7fa95a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.809+0000 7fa95a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:20.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.809+0000 7fa95a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:20.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.809+0000 7fa95a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:20.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.809+0000 7fa95a7fc640 10 monclient(hunting): handle_auth_done global_id 74400 payload 824 2024-09-16T04:45:20.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.809+0000 7fa95a7fc640 1 -- 172.21.15.46:0/2998219378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa95c06f0e0 con 0x7fa95c09cf60 2024-09-16T04:45:20.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.809+0000 7fa95a7fc640 1 --2- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c09cf60 0x7fa95c135e00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa950014000 tx=0x7fa950001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:20.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.809+0000 7fa9437fe640 1 -- 172.21.15.46:0/2998219378 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa950012b60 con 0x7fa95c09cf60 2024-09-16T04:45:20.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.809+0000 7fa9437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:20.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.809+0000 7fa9437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:20.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.809+0000 7fa9437fe640 10 monclient: dump: 2024-09-16T04:45:20.828 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:20.828 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:20.828 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.828 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.829 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:20.829 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:20.829 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:20.829 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:20.829 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:20.829 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:20.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.809+0000 7fa9437fe640 1 -- 172.21.15.46:0/2998219378 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7fa950012b60 con 0x7fa95c09cf60 2024-09-16T04:45:20.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.809+0000 7fa9437fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:45:20.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.810+0000 7fa9437fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:45:20.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.810+0000 7fa9437fe640 1 -- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c09cf60 msgr2=0x7fa95c135e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:20.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.810+0000 7fa9437fe640 1 --2- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c09cf60 0x7fa95c135e00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa950014000 tx=0x7fa950001080 comp rx=0 tx=0).stop 2024-09-16T04:45:20.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.810+0000 7fa9437fe640 1 -- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c09cf60 msgr2=0x7fa95c135e00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:45:20.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.810+0000 7fa9437fe640 1 --2- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c09cf60 0x7fa95c135e00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.830 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:45:20.830 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:45:20.830 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:45:20.830 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:45:20.830 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 78, 2024-09-16T04:45:20.830 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 60805, 2024-09-16T04:45:20.830 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:45:20.830 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:45:20.830 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:45:20.830 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:45:20.831 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:20.831 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 2, 2024-09-16T04:45:20.831 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:45:20.831 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:45:20.831 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:45:20.831 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:20.831 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:20.831 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:20.831 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:20.831 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:20.831 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:20.831 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:20.831 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:20.831 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:20.832 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:20.832 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:20.832 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:20.832 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:45:20.832 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:45:20.832 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:20.832 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:20.832 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:20.832 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:20.832 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:20.832 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:20.832 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:20.832 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:20.833 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:20.833 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:20.833 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:20.833 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:20.833 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:20.833 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:45:20.833 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:45:20.833 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:45:20.833 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:45:20.833 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:45:20.833 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:45:20.833 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:45:20.833 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:45:20.834 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:45:20.834 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:45:20.834 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:45:20.834 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:45:20.834 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:45:20.834 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:45:20.834 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:45:20.834 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:45:20.834 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:45:20.834 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:20.834 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:20.834 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:20.834 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:20.835 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:20.835 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:20.835 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:20.835 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:20.835 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:20.835 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:20.835 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:20.835 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:20.835 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:45:20.835 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:20.835 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:45:20.835 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:20.835 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:45:20.835 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:45:20.836 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:20.836 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:20.836 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:20.836 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:20.836 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:45:20.836 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:20.836 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:20.836 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:20.836 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:20.836 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:45:20.836 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:20.836 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:20.837 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:20.837 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:20.837 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:45:20.837 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:45:20.837 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:20.837 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:20.837 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:20.837 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:20.837 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:20.837 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:45:20.837 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:45:20.837 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:20.837 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:20.838 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:20.838 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:20.838 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:45:20.838 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:20.838 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:20.838 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:20.838 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:20.838 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:45:20.838 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:20.838 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:20.838 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:20.838 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:20.838 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:45:20.838 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:45:20.839 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:20.839 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:20.839 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:20.839 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:20.839 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:20.839 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:45:20.839 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:45:20.839 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:20.839 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:20.839 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:20.839 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:20.839 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:45:20.839 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:20.839 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:20.839 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:20.840 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:20.840 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:45:20.840 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:20.840 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:20.840 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:20.840 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:20.840 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:45:20.840 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:45:20.840 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:20.840 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:20.840 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:20.840 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:20.841 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:20.841 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:20.841 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:45:20.841 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:45:20.841 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:20.841 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:20.841 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:20.841 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:45:20.841 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:20.841 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:20.841 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:45:20.841 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:20.841 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:20.841 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:20.842 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:45:20.842 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:20.842 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:20.842 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:45:20.842 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:20.842 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:20.842 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:20.842 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:45:20.842 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:20.842 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:20.842 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:20.842 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:45:20.842 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:45:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.810+0000 7fa9615fd640 1 -- 172.21.15.46:0/2998219378 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa9480383e0 msgr2=0x7fa94803a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.810+0000 7fa9615fd640 1 --2- 172.21.15.46:0/2998219378 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa9480383e0 0x7fa94803a8a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa954034000 tx=0x7fa95402f000 comp rx=0 tx=0).stop 2024-09-16T04:45:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.810+0000 7fa9615fd640 10 monclient: shutdown 2024-09-16T04:45:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.810+0000 7fa9615fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.810+0000 7fa9615fd640 1 -- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c09e760 msgr2=0x7fa95c0720d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.810+0000 7fa9615fd640 1 --2- 172.21.15.46:0/2998219378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa95c09e760 0x7fa95c0720d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa94c00d7d0 tx=0x7fa94c00dc90 comp rx=0 tx=0).stop 2024-09-16T04:45:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.810+0000 7fa95b7fe640 1 -- 172.21.15.46:0/2998219378 reap_dead start 2024-09-16T04:45:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.810+0000 7fa95b7fe640 1 -- 172.21.15.46:0/2998219378 reap_dead start 2024-09-16T04:45:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.810+0000 7fa9615fd640 1 -- 172.21.15.46:0/2998219378 shutdown_connections 2024-09-16T04:45:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.810+0000 7fa9615fd640 1 -- 172.21.15.46:0/2998219378 >> 172.21.15.46:0/2998219378 conn(0x7fa95c0a5960 msgr2=0x7fa95c13b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.810+0000 7fa9615fd640 1 -- 172.21.15.46:0/2998219378 shutdown_connections 2024-09-16T04:45:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.810+0000 7fa9615fd640 1 -- 172.21.15.46:0/2998219378 wait complete. 2024-09-16T04:45:20.845 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:45:20.845 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:45:20.845 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:45:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.946+0000 7f70e705c640 10 monclient: get_monmap_and_config 2024-09-16T04:45:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.946+0000 7f70e705c640 10 monclient: build_initial_monmap 2024-09-16T04:45:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.946+0000 7f70e705c640 10 monclient: monmap: 2024-09-16T04:45:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:45:20.947445+0000 2024-09-16T04:45:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:45:20.947445+0000 2024-09-16T04:45:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:20.949 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:45:20.949 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:45:20.949 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:45:20.949 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:45:20.949 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:45:20.949 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:20.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.947+0000 7f70e705c640 1 Processor -- start 2024-09-16T04:45:20.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.947+0000 7f70e705c640 1 -- start start 2024-09-16T04:45:20.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.947+0000 7f70e705c640 10 monclient: init 2024-09-16T04:45:20.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70e705c640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:20.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70e705c640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-16T04:45:20.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70e705c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f70e009e2c0 addr v1:172.21.15.46:6789/0 2024-09-16T04:45:20.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70e705c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0136700 0x7f70e0138af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70e705c640 10 monclient(hunting): picked mon.noname-b con 0x7f70e0136700 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:20.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70e705c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f70e0139030 addr v1:172.21.15.114:6789/0 2024-09-16T04:45:20.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70e705c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70e009c310 con 0x7f70e009e2c0 2024-09-16T04:45:20.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70e705c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70e009c840 con 0x7f70e0139030 2024-09-16T04:45:20.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70e705c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70e705c640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f70e00a2730 con 0x7f70e0136700 2024-09-16T04:45:20.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70e705c640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:20.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70e705c640 10 monclient(hunting): authenticate will time out at 1119.363457s 2024-09-16T04:45:20.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70e4df8640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f70e009e2c0 0x7f70e00a2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45454/0 (socket says 172.21.15.46:45454) 2024-09-16T04:45:20.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70e4df8640 1 -- 172.21.15.46:0/796754040 learned_addr learned my addr 172.21.15.46:0/796754040 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:45:20.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70dffff640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0136700 0x7f70e0138af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70dffff640 10 monclient(hunting): get_auth_request con 0x7f70e0136700 auth_method 0 2024-09-16T04:45:20.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:20.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70dffff640 1 -- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0136700 msgr2=0x7f70e0138af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:45:20.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70dffff640 1 -- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0136700 msgr2=0x7f70e0138af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:20.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70dffff640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0136700 0x7f70e0138af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:20.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70dffff640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0136700 0x7f70e0138af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:20.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70df7fe640 1 -- 172.21.15.46:0/796754040 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3818566268 0 0) 0x7f70e009c310 con 0x7f70e009e2c0 2024-09-16T04:45:20.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70df7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:45:20.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:20.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70df7fe640 10 monclient(hunting): my global_id is 74403 2024-09-16T04:45:20.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70df7fe640 1 -- 172.21.15.46:0/796754040 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70c8005730 con 0x7f70e009e2c0 2024-09-16T04:45:20.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70df7fe640 1 -- 172.21.15.46:0/796754040 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1732958029 0 0) 0x7f70e009c840 con 0x7f70e0139030 2024-09-16T04:45:20.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70df7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:45:20.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:20.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70df7fe640 10 monclient(hunting): my global_id is 6475 2024-09-16T04:45:20.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70df7fe640 1 -- 172.21.15.46:0/796754040 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70e009c310 con 0x7f70e0139030 2024-09-16T04:45:20.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.948+0000 7f70df7fe640 1 -- 172.21.15.46:0/796754040 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1852426163 0 0) 0x7f70c8005730 con 0x7f70e009e2c0 2024-09-16T04:45:20.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 1 -- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0136700 msgr2=0x7f70e0138af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:45:20.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0136700 0x7f70e0138af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 1 -- 172.21.15.46:0/796754040 >> v1:172.21.15.114:6789/0 conn(0x7f70e0139030 legacy=0x7f70e013b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:20.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:45:20.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:45:20.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:20.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:45:20.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 1 -- 172.21.15.46:0/796754040 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70e009d850 con 0x7f70e009e2c0 2024-09-16T04:45:20.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 10 monclient: _finish_auth 0 2024-09-16T04:45:20.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:45:20.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:20.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 1 -- 172.21.15.46:0/796754040 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f70d0001670 con 0x7f70e009e2c0 2024-09-16T04:45:20.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e705c640 5 monclient: authenticate success, global_id 74403 2024-09-16T04:45:20.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e705c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:45:20.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:20.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:45:20.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 10 monclient: dump: 2024-09-16T04:45:20.966 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:20.966 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:20.966 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.966 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.966 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:20.966 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:20.966 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:20.966 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:20.966 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:20.966 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:20.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:45:20.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:20.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 1 -- 172.21.15.46:0/796754040 >> v1:172.21.15.46:6789/0 conn(0x7f70e009e2c0 legacy=0x7f70e00a2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:20.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:45:20.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0136700 0x7f70c8005d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 10 monclient(hunting): picked mon.c con 0x7f70e0136700 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:20.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70c8006330 0x7f70c8006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 10 monclient(hunting): picked mon.b con 0x7f70c8006330 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:45:20.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70c8006f50 0x7f70c80073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 10 monclient(hunting): picked mon.a con 0x7f70c8006f50 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:45:20.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 1 -- 172.21.15.46:0/796754040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f70e00a2730 con 0x7f70c8006f50 2024-09-16T04:45:20.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 1 -- 172.21.15.46:0/796754040 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f70c800de30 con 0x7f70c8006330 2024-09-16T04:45:20.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 1 -- 172.21.15.46:0/796754040 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f70c800e030 con 0x7f70e0136700 2024-09-16T04:45:20.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:20.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70df7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:45:20.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e705c640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:45:20.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e4df8640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0136700 0x7f70c8005d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e4df8640 10 monclient(hunting): get_auth_request con 0x7f70e0136700 auth_method 0 2024-09-16T04:45:20.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e55f9640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70c8006f50 0x7f70c80073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e4df8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e4df8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e4df8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:20.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e55f9640 10 monclient(hunting): get_auth_request con 0x7f70c8006f50 auth_method 0 2024-09-16T04:45:20.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e55f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e55f9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e55f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:20.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e4df8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:20.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e4df8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:20.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e4df8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:45:20.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70dffff640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70c8006330 0x7f70c8006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e55f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:20.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e55f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:20.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e55f9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:45:20.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70dffff640 10 monclient(hunting): get_auth_request con 0x7f70c8006330 auth_method 0 2024-09-16T04:45:20.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70dffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:20.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e4df8640 10 monclient(hunting): handle_auth_done global_id 74403 payload 293 2024-09-16T04:45:20.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e4df8640 1 -- 172.21.15.46:0/796754040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70c8006330 msgr2=0x7f70c8006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:45:20.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e4df8640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70c8006330 0x7f70c8006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e4df8640 1 -- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70c8006f50 msgr2=0x7f70c80073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:45:20.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e4df8640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70c8006f50 0x7f70c80073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e4df8640 10 monclient: _finish_hunting 0 2024-09-16T04:45:20.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e4df8640 1 monclient: found mon.c 2024-09-16T04:45:20.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e4df8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:20.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e4df8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:45:20.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e4df8640 1 -- 172.21.15.46:0/796754040 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c8005730 con 0x7f70e0136700 2024-09-16T04:45:20.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e4df8640 10 monclient: _finish_auth 0 2024-09-16T04:45:20.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e4df8640 10 monclient: _check_auth_tickets 2024-09-16T04:45:20.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e4df8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:20.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70dffff640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70c8006330 0x7f70c8006a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:45:20.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.949+0000 7f70e55f9640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70c8006f50 0x7f70c80073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:45:20.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70e4df8640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0136700 0x7f70c8005d80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f70d0007fc0 tx=0x7f70c8007a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4216509085d8526 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:20.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70e4df8640 1 -- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0136700 msgr2=0x7f70c8005d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T04:45:20.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70e4df8640 1 -- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0136700 msgr2=0x7f70c8005d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:45:20.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70e4df8640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0136700 0x7f70c8005d80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f70d0007fc0 tx=0x7f70c8007a40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:20.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70e4df8640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0136700 0x7f70c8005d80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f70d0007fc0 tx=0x7f70c8007a40 comp rx=0 tx=0).stop 2024-09-16T04:45:20.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70e55f9640 1 -- 172.21.15.46:0/796754040 reap_dead start 2024-09-16T04:45:20.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70df7fe640 10 monclient: ms_handle_reset current mon [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:20.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70df7fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:20.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70df7fe640 1 -- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0136700 msgr2=0x7f70c8005d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:45:20.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70df7fe640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0136700 0x7f70c8005d80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70df7fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:45:20.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70df7fe640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70c8006a10 0x7f70c80073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70df7fe640 10 monclient(hunting): picked mon.b con 0x7f70c8006a10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:45:20.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70df7fe640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70c800bc70 0x7f70c800c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70df7fe640 10 monclient(hunting): picked mon.a con 0x7f70c800bc70 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:45:20.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70df7fe640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70c800c8a0 0x7f70c800cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70df7fe640 10 monclient(hunting): picked mon.c con 0x7f70c800c8a0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:20.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70df7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70df7fe640 1 -- 172.21.15.46:0/796754040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f70c800d5d0 con 0x7f70c800bc70 2024-09-16T04:45:20.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70df7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70df7fe640 1 -- 172.21.15.46:0/796754040 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f70c800d780 con 0x7f70c8006a10 2024-09-16T04:45:20.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70df7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70df7fe640 1 -- 172.21.15.46:0/796754040 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f70c800d980 con 0x7f70c800c8a0 2024-09-16T04:45:20.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70df7fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:20.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70df7fe640 10 monclient(hunting): ms_handle_reset hunted mon [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:20.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70dffff640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70c800bc70 0x7f70c800c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70e55f9640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70c800c8a0 0x7f70c800cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70dffff640 10 monclient(hunting): get_auth_request con 0x7f70c800bc70 auth_method 0 2024-09-16T04:45:20.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70dffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:20.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70e55f9640 10 monclient(hunting): get_auth_request con 0x7f70c800c8a0 auth_method 0 2024-09-16T04:45:20.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70e55f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70e55f9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70e55f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:20.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:20.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:20.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70dffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:45:20.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70e55f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:20.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70e55f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:20.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70e55f9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:45:20.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70e4df8640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70c8006a10 0x7f70c80073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70e4df8640 10 monclient(hunting): get_auth_request con 0x7f70c8006a10 auth_method 0 2024-09-16T04:45:20.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70e4df8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70e4df8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70e4df8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:20.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70dffff640 10 monclient(hunting): handle_auth_done global_id 74403 payload 293 2024-09-16T04:45:20.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70dffff640 1 -- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70c800c8a0 msgr2=0x7f70c800cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:45:20.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70dffff640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70c800c8a0 0x7f70c800cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70dffff640 1 -- 172.21.15.46:0/796754040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70c8006a10 msgr2=0x7f70c80073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:45:20.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70dffff640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70c8006a10 0x7f70c80073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70dffff640 10 monclient: _finish_hunting 0 2024-09-16T04:45:20.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70dffff640 1 monclient: found mon.a 2024-09-16T04:45:20.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:20.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70dffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:20.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70dffff640 1 -- 172.21.15.46:0/796754040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=2+}) -- 0x7f70e009c840 con 0x7f70c800bc70 2024-09-16T04:45:20.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70dffff640 10 monclient: _finish_auth 0 2024-09-16T04:45:20.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70dffff640 10 monclient: _check_auth_tickets 2024-09-16T04:45:20.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:20.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.950+0000 7f70e55f9640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70c800c8a0 0x7f70c800cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:45:20.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.951+0000 7f70dffff640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70c800bc70 0x7f70c800c050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f70d400da00 tx=0x7f70d0001d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a38512a65d050ae7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:20.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.951+0000 7f70df7fe640 1 -- 172.21.15.46:0/796754040 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d4019030 con 0x7f70c800bc70 2024-09-16T04:45:20.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.951+0000 7f70df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:20.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.951+0000 7f70df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:20.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.951+0000 7f70df7fe640 10 monclient: dump: 2024-09-16T04:45:20.976 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:20.976 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:20.976 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.976 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.976 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:20.976 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:20.976 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:20.976 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:20.976 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:20.976 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:20.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.951+0000 7f70e705c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:45:20.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.951+0000 7f70df7fe640 1 -- 172.21.15.46:0/796754040 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70d4004020 con 0x7f70c800bc70 2024-09-16T04:45:20.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.951+0000 7f70df7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:20.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.951+0000 7f70e705c640 10 monclient: get_monmap_and_config success 2024-09-16T04:45:20.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.951+0000 7f70e705c640 4 set_mon_vals no callback set 2024-09-16T04:45:20.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.951+0000 7f70e705c640 10 monclient: shutdown 2024-09-16T04:45:20.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.951+0000 7f70e705c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:20.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.951+0000 7f70e705c640 1 -- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70c800bc70 msgr2=0x7f70c800c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:20.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.951+0000 7f70e705c640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70c800bc70 0x7f70c800c050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f70d400da00 tx=0x7f70d0001d90 comp rx=0 tx=0).stop 2024-09-16T04:45:20.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.951+0000 7f70e705c640 1 -- 172.21.15.46:0/796754040 shutdown_connections 2024-09-16T04:45:20.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.951+0000 7f70e705c640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70c800c8a0 0x7f70c800cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.951+0000 7f70e705c640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70c800bc70 0x7f70c800c050 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.951+0000 7f70e705c640 1 --2- 172.21.15.46:0/796754040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70c8006a10 0x7f70c80073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.951+0000 7f70e705c640 1 -- 172.21.15.46:0/796754040 >> 172.21.15.46:0/796754040 conn(0x7f70e00a5960 msgr2=0x7f70e00a1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:20.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.951+0000 7f70e705c640 1 -- 172.21.15.46:0/796754040 shutdown_connections 2024-09-16T04:45:20.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.951+0000 7f70e705c640 1 -- 172.21.15.46:0/796754040 wait complete. 2024-09-16T04:45:20.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.952+0000 7f70e705c640 10 monclient: build_initial_monmap 2024-09-16T04:45:20.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.952+0000 7f70e705c640 10 monclient: monmap: 2024-09-16T04:45:20.978 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:20.978 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:45:20.978 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:45:20.978 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:45:20.978 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:20.978 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:20.978 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:45:20.978 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:45:20.978 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:45:20.978 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:20.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.952+0000 7f70e705c640 1 Processor -- start 2024-09-16T04:45:20.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.952+0000 7f70e705c640 1 -- start start 2024-09-16T04:45:20.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.952+0000 7f70e705c640 10 monclient: init 2024-09-16T04:45:20.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e705c640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:20.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e705c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:45:20.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e705c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70c8006a10 0x7f70e01c7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e705c640 10 monclient(hunting): picked mon.noname-a con 0x7f70c8006a10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:45:20.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e705c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70c800bc70 0x7f70e0138410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e705c640 10 monclient(hunting): picked mon.noname-c con 0x7f70c800bc70 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:20.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e705c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70c800c8a0 0x7f70e011fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e705c640 10 monclient(hunting): picked mon.noname-b con 0x7f70c800c8a0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:45:20.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e705c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e705c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f70e009e6a0 con 0x7f70c8006a10 2024-09-16T04:45:20.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e705c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e705c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f70e009f090 con 0x7f70c800c8a0 2024-09-16T04:45:20.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e705c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e705c640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f70e00a2020 con 0x7f70c800bc70 2024-09-16T04:45:20.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e705c640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:20.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e705c640 10 monclient(hunting): authenticate will time out at 1119.368475s 2024-09-16T04:45:20.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e4df8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70c8006a10 0x7f70e01c7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70dffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70c800bc70 0x7f70e0138410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e4df8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70c8006a10 0x7f70e01c7530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32786/0 (socket says 172.21.15.46:32786) 2024-09-16T04:45:20.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e4df8640 1 -- 172.21.15.46:0/1129404353 learned_addr learned my addr 172.21.15.46:0/1129404353 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:20.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e4df8640 10 monclient(hunting): get_auth_request con 0x7f70c8006a10 auth_method 0 2024-09-16T04:45:20.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e4df8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e4df8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e4df8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:20.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70dffff640 10 monclient(hunting): get_auth_request con 0x7f70c800bc70 auth_method 0 2024-09-16T04:45:20.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:20.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e4df8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:20.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e4df8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:20.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70dffff640 1 -- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70c800bc70 msgr2=0x7f70e0138410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:45:20.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e4df8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:20.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70dffff640 1 -- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70c800bc70 msgr2=0x7f70e0138410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:45:20.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70dffff640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70c800bc70 0x7f70e0138410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:20.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70dffff640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70c800bc70 0x7f70e0138410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:20.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e4df8640 10 monclient(hunting): handle_auth_done global_id 74406 payload 824 2024-09-16T04:45:20.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e55f9640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70c800c8a0 0x7f70e011fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e4df8640 1 -- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70c800bc70 msgr2=0x7f70e0138410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:45:20.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e4df8640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70c800bc70 0x7f70e0138410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e4df8640 1 -- 172.21.15.46:0/1129404353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70c800c8a0 msgr2=0x7f70e011fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:20.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e4df8640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70c800c8a0 0x7f70e011fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:20.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e4df8640 10 monclient: _finish_hunting 0 2024-09-16T04:45:20.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e4df8640 1 monclient: found mon.noname-a 2024-09-16T04:45:20.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e4df8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:20.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e4df8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:20.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.953+0000 7f70e4df8640 1 -- 172.21.15.46:0/1129404353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70e01c8430 con 0x7f70c8006a10 2024-09-16T04:45:20.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70e4df8640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70c8006a10 0x7f70e01c7530 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f70d002acf0 tx=0x7f70e006be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:20.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70dd7fa640 1 -- 172.21.15.46:0/1129404353 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d0041020 con 0x7f70c8006a10 2024-09-16T04:45:20.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:20.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70dd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:20.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70dd7fa640 10 monclient: dump: 2024-09-16T04:45:20.982 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:20.983 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:20.983 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.983 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.983 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:20.983 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:20.983 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:20.983 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:20.983 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:20.983 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:20.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70dd7fa640 10 monclient: _finish_auth 0 2024-09-16T04:45:20.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70dd7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:45:20.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:20.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70dd7fa640 1 -- 172.21.15.46:0/1129404353 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70d00049f0 con 0x7f70c8006a10 2024-09-16T04:45:20.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70e705c640 5 monclient: authenticate success, global_id 74406 2024-09-16T04:45:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70e705c640 10 monclient: _renew_subs 2024-09-16T04:45:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70e705c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70e705c640 1 -- 172.21.15.46:0/1129404353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70e01237f0 con 0x7f70c8006a10 2024-09-16T04:45:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70dd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70dd7fa640 1 -- 172.21.15.46:0/1129404353 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d0007be0 con 0x7f70c8006a10 2024-09-16T04:45:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70dd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70dd7fa640 10 monclient: dump: 2024-09-16T04:45:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:20.985 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70e705c640 10 monclient: _renew_subs 2024-09-16T04:45:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70e705c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70e705c640 1 -- 172.21.15.46:0/1129404353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70e01223a0 con 0x7f70c8006a10 2024-09-16T04:45:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.954+0000 7f70deffd640 4 set_mon_vals no callback set 2024-09-16T04:45:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.955+0000 7f70dd7fa640 1 -- 172.21.15.46:0/1129404353 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f70d0044020 con 0x7f70c8006a10 2024-09-16T04:45:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.955+0000 7f70e705c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.955+0000 7f70e705c640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0138950 0x7f70e0120360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.955+0000 7f70e705c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.955+0000 7f70e705c640 1 -- 172.21.15.46:0/1129404353 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f70e009d290 con 0x7f70e0138950 2024-09-16T04:45:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.955+0000 7f70dffff640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0138950 0x7f70e0120360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.955+0000 7f70dffff640 10 monclient: get_auth_request con 0x7f70e0138950 auth_method 0 2024-09-16T04:45:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.955+0000 7f70dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.955+0000 7f70dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.955+0000 7f70dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.955+0000 7f70dd7fa640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f70cc0383b0 0x7f70cc03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.955+0000 7f70dffff640 1 -- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0138950 msgr2=0x7f70e0120360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:45:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.955+0000 7f70dffff640 1 -- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0138950 msgr2=0x7f70e0120360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:45:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.955+0000 7f70dffff640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0138950 0x7f70e0120360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.955+0000 7f70dd7fa640 1 -- 172.21.15.46:0/1129404353 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(251..251 src has 1..251) ==== 4346+0+0 (secure 0 0 0) 0x7f70d006b940 con 0x7f70c8006a10 2024-09-16T04:45:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.955+0000 7f70dffff640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0138950 0x7f70e0120360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.955+0000 7f70e55f9640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f70cc0383b0 0x7f70cc03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.955+0000 7f70e55f9640 10 monclient: get_auth_request con 0x7f70cc0383b0 auth_method 0 2024-09-16T04:45:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:20.956+0000 7f70e55f9640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f70cc0383b0 0x7f70cc03a870 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f70e009bfe0 tx=0x7f70d8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:21.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.156+0000 7f70dffff640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0138950 0x7f70e0120360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:21.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.156+0000 7f70dffff640 10 monclient: get_auth_request con 0x7f70e0138950 auth_method 0 2024-09-16T04:45:21.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.156+0000 7f70dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:21.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.156+0000 7f70dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:21.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.156+0000 7f70dffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:21.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.156+0000 7f70dffff640 1 -- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0138950 msgr2=0x7f70e0120360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:45:21.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.156+0000 7f70dffff640 1 -- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0138950 msgr2=0x7f70e0120360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:45:21.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.156+0000 7f70dffff640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0138950 0x7f70e0120360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:21.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.156+0000 7f70dffff640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0138950 0x7f70e0120360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:45:21.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.557+0000 7f70dffff640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0138950 0x7f70e0120360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:21.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.557+0000 7f70dffff640 10 monclient: get_auth_request con 0x7f70e0138950 auth_method 0 2024-09-16T04:45:21.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.557+0000 7f70dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:21.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.557+0000 7f70dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:21.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.557+0000 7f70dffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:21.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.557+0000 7f70dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:21.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.557+0000 7f70dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:21.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.557+0000 7f70dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:21.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.557+0000 7f70dffff640 10 monclient(hunting): handle_auth_done global_id 84101 payload 824 2024-09-16T04:45:21.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.557+0000 7f70dffff640 1 -- 172.21.15.46:0/1129404353 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f70e006e620 con 0x7f70e0138950 2024-09-16T04:45:21.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.557+0000 7f70dffff640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0138950 0x7f70e0120360 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f70d400da00 tx=0x7f70d40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:21.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.557+0000 7f70dd7fa640 1 -- 172.21.15.46:0/1129404353 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d402c070 con 0x7f70e0138950 2024-09-16T04:45:21.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.557+0000 7f70dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:21.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.557+0000 7f70dd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:45:21.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.557+0000 7f70dd7fa640 10 monclient: dump: 2024-09-16T04:45:21.560 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:21.560 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:21.560 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.560 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.561 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:21.561 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:21.561 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:21.561 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:21.561 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:21.561 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:21.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.558+0000 7f70dd7fa640 1 -- 172.21.15.46:0/1129404353 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f70d400fa70 con 0x7f70e0138950 2024-09-16T04:45:21.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.558+0000 7f70dd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:21.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.558+0000 7f70dd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:45:21.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.558+0000 7f70dd7fa640 1 -- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0138950 msgr2=0x7f70e0120360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:21.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.558+0000 7f70dd7fa640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0138950 0x7f70e0120360 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f70d400da00 tx=0x7f70d40046c0 comp rx=0 tx=0).stop 2024-09-16T04:45:21.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.558+0000 7f70dd7fa640 1 -- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0138950 msgr2=0x7f70e0120360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:45:21.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.558+0000 7f70dd7fa640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0138950 0x7f70e0120360 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f70d400da00 tx=0x7f70d40046c0 comp rx=0 tx=0).stop 2024-09-16T04:45:21.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.564+0000 7f70e705c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:45:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.564+0000 7f70e705c640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0123b50 0x7f70e01208a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.564+0000 7f70e705c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.564+0000 7f70e705c640 1 -- 172.21.15.46:0/1129404353 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f70e00a2400 con 0x7f70e0123b50 2024-09-16T04:45:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.565+0000 7f70dffff640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0123b50 0x7f70e01208a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.565+0000 7f70dffff640 10 monclient: get_auth_request con 0x7f70e0123b50 auth_method 0 2024-09-16T04:45:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.565+0000 7f70dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.565+0000 7f70dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.565+0000 7f70dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.565+0000 7f70dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.565+0000 7f70dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.565+0000 7f70dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:21.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.565+0000 7f70dffff640 10 monclient(hunting): handle_auth_done global_id 84104 payload 824 2024-09-16T04:45:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.565+0000 7f70dffff640 1 -- 172.21.15.46:0/1129404353 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f70e006c020 con 0x7f70e0123b50 2024-09-16T04:45:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.565+0000 7f70dffff640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0123b50 0x7f70e01208a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f70e0075000 tx=0x7f70d4016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.565+0000 7f70dd7fa640 1 -- 172.21.15.46:0/1129404353 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d402c040 con 0x7f70e0123b50 2024-09-16T04:45:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.565+0000 7f70dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.565+0000 7f70dd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:45:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.565+0000 7f70dd7fa640 10 monclient: dump: 2024-09-16T04:45:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:21.569 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.565+0000 7f70dd7fa640 1 -- 172.21.15.46:0/1129404353 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f70d4003e40 con 0x7f70e0123b50 2024-09-16T04:45:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.565+0000 7f70dd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:45:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.565+0000 7f70dd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:45:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.566+0000 7f70dd7fa640 1 -- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0123b50 msgr2=0x7f70e01208a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.566+0000 7f70dd7fa640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0123b50 0x7f70e01208a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f70e0075000 tx=0x7f70d4016000 comp rx=0 tx=0).stop 2024-09-16T04:45:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.566+0000 7f70dd7fa640 1 -- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0123b50 msgr2=0x7f70e01208a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:45:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.566+0000 7f70dd7fa640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f70e0123b50 0x7f70e01208a0 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f70e0075000 tx=0x7f70d4016000 comp rx=0 tx=0).stop 2024-09-16T04:45:21.570 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:45:21.570 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:45:21.570 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:45:21.570 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:45:21.570 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 78, 2024-09-16T04:45:21.570 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 10176, 2024-09-16T04:45:21.570 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:45:21.570 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:45:21.570 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:45:21.570 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:45:21.571 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:21.571 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 3, 2024-09-16T04:45:21.571 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:45:21.571 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:45:21.571 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:45:21.571 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:21.571 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:21.571 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:21.571 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:21.571 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:21.571 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:21.571 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:21.572 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:21.572 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:21.572 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:21.572 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:21.572 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:21.572 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:45:21.572 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:45:21.572 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:21.572 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:21.572 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:21.572 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:21.572 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:21.572 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:21.572 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:21.572 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:21.573 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:21.573 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:21.573 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:21.573 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:21.573 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.573 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:45:21.573 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:45:21.573 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:45:21.573 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:45:21.573 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:45:21.573 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:45:21.573 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:45:21.573 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:45:21.574 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:45:21.574 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:45:21.574 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:45:21.574 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:45:21.574 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:45:21.574 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:45:21.574 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:45:21.574 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:45:21.574 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:45:21.574 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:21.574 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:21.574 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:21.574 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:21.574 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:21.575 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:21.575 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:21.575 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:21.575 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:21.575 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:21.575 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:21.575 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:21.575 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:45:21.575 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.575 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:45:21.575 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.575 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:45:21.576 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:45:21.576 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:21.576 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:21.576 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.576 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:21.576 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:45:21.576 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:21.576 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.576 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.576 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:21.576 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:45:21.576 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:21.576 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:21.576 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:21.577 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.577 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:45:21.577 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:45:21.577 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:21.577 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:21.577 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:21.577 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.577 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.577 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:45:21.577 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:45:21.577 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:21.577 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:21.577 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.578 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:21.578 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:45:21.578 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:21.578 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.578 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.578 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:21.578 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:45:21.578 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:21.578 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:21.578 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:21.578 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.578 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:45:21.578 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:45:21.579 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:21.579 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:21.579 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:21.579 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.579 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.579 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:45:21.579 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:45:21.579 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:21.579 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:21.579 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.579 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:21.579 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:45:21.579 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:21.579 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.580 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.580 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:21.580 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:45:21.580 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:21.580 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:21.580 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:21.580 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.580 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:45:21.580 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:45:21.580 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:21.580 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:21.581 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:21.581 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:21.581 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:21.581 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.581 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:45:21.581 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:45:21.581 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.581 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:21.581 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:21.581 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:45:21.581 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:21.581 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:21.581 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:45:21.582 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.582 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:21.582 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:21.582 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:45:21.582 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:21.582 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:21.582 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.582 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:45:21.582 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:45:21.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.566+0000 7f70e705c640 1 -- 172.21.15.46:0/1129404353 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f70cc0383b0 msgr2=0x7f70cc03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.566+0000 7f70e705c640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f70cc0383b0 0x7f70cc03a870 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f70e009bfe0 tx=0x7f70d8009040 comp rx=0 tx=0).stop 2024-09-16T04:45:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.566+0000 7f70e705c640 10 monclient: shutdown 2024-09-16T04:45:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.566+0000 7f70e705c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.566+0000 7f70e705c640 1 -- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70c8006a10 msgr2=0x7f70e01c7530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.566+0000 7f70e705c640 1 --2- 172.21.15.46:0/1129404353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70c8006a10 0x7f70e01c7530 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f70d002acf0 tx=0x7f70e006be50 comp rx=0 tx=0).stop 2024-09-16T04:45:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.566+0000 7f70e55f9640 1 -- 172.21.15.46:0/1129404353 reap_dead start 2024-09-16T04:45:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.566+0000 7f70e55f9640 1 -- 172.21.15.46:0/1129404353 reap_dead start 2024-09-16T04:45:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.566+0000 7f70e705c640 1 -- 172.21.15.46:0/1129404353 shutdown_connections 2024-09-16T04:45:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.566+0000 7f70e705c640 1 -- 172.21.15.46:0/1129404353 >> 172.21.15.46:0/1129404353 conn(0x7f70e00a5960 msgr2=0x7f70e0070ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.566+0000 7f70e705c640 1 -- 172.21.15.46:0/1129404353 shutdown_connections 2024-09-16T04:45:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.566+0000 7f70e705c640 1 -- 172.21.15.46:0/1129404353 wait complete. 2024-09-16T04:45:21.585 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:45:21.585 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:45:21.585 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:45:21.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.691+0000 7fb021eba640 10 monclient: get_monmap_and_config 2024-09-16T04:45:21.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.691+0000 7fb021eba640 10 monclient: build_initial_monmap 2024-09-16T04:45:21.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.691+0000 7fb021eba640 10 monclient: monmap: 2024-09-16T04:45:21.692 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:21.692 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:21.693 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:45:21.691942+0000 2024-09-16T04:45:21.693 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:45:21.691942+0000 2024-09-16T04:45:21.693 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:21.693 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:21.693 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:45:21.693 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:45:21.693 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:45:21.693 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:45:21.693 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:45:21.693 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:21.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb021eba640 1 Processor -- start 2024-09-16T04:45:21.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb021eba640 1 -- start start 2024-09-16T04:45:21.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb021eba640 10 monclient: init 2024-09-16T04:45:21.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb021eba640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:21.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb021eba640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-16T04:45:21.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb021eba640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb01c0a3c10 0x7fb01c09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:21.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb021eba640 10 monclient(hunting): picked mon.noname-c con 0x7fb01c0a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:45:21.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb021eba640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb01c0a3ff0 0x7fb01c0a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:21.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb021eba640 10 monclient(hunting): picked mon.noname-a con 0x7fb01c0a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:45:21.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb021eba640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb01c1369d0 0x7fb01c136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:21.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb021eba640 10 monclient(hunting): picked mon.noname-b con 0x7fb01c1369d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:21.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb021eba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:21.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb021eba640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb01c09b940 con 0x7fb01c0a3ff0 2024-09-16T04:45:21.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb021eba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:21.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb021eba640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fb01c09ab00 con 0x7fb01c0a3c10 2024-09-16T04:45:21.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb021eba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:21.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb021eba640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fb01c09b030 con 0x7fb01c1369d0 2024-09-16T04:45:21.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb021eba640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:21.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb021eba640 10 monclient(hunting): authenticate will time out at 1120.107824s 2024-09-16T04:45:21.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb01affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb01c0a3ff0 0x7fb01c0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:21.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb01bfff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb01c1369d0 0x7fb01c136dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:21.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.692+0000 7fb01affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb01c0a3ff0 0x7fb01c0a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32794/0 (socket says 172.21.15.46:32794) 2024-09-16T04:45:21.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01bfff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb01c1369d0 0x7fb01c136dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:34806/0 (socket says 172.21.15.46:34806) 2024-09-16T04:45:21.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01affd640 1 -- 172.21.15.46:0/2772566170 learned_addr learned my addr 172.21.15.46:0/2772566170 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:21.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01affd640 10 monclient(hunting): get_auth_request con 0x7fb01c0a3ff0 auth_method 0 2024-09-16T04:45:21.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:21.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:21.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:21.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01bfff640 10 monclient(hunting): get_auth_request con 0x7fb01c1369d0 auth_method 0 2024-09-16T04:45:21.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:21.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:21.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:21.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:21.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01b7fe640 1 --2- 172.21.15.46:0/2772566170 >> v2:172.21.15.114:3300/0 conn(0x7fb01c0a3c10 0x7fb01c09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01b7fe640 10 monclient(hunting): get_auth_request con 0x7fb01c0a3c10 auth_method 0 2024-09-16T04:45:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01affd640 10 monclient(hunting): handle_auth_done global_id 74409 payload 274 2024-09-16T04:45:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01affd640 1 -- 172.21.15.46:0/2772566170 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb01c1369d0 msgr2=0x7fb01c136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:45:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01affd640 1 --2- 172.21.15.46:0/2772566170 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb01c1369d0 0x7fb01c136dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01affd640 1 -- 172.21.15.46:0/2772566170 >> v2:172.21.15.114:3300/0 conn(0x7fb01c0a3c10 msgr2=0x7fb01c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:45:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01affd640 1 --2- 172.21.15.46:0/2772566170 >> v2:172.21.15.114:3300/0 conn(0x7fb01c0a3c10 0x7fb01c09bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01affd640 10 monclient: _finish_hunting 0 2024-09-16T04:45:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01affd640 1 monclient: found mon.noname-a 2024-09-16T04:45:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01affd640 1 -- 172.21.15.46:0/2772566170 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb01c09b310 con 0x7fb01c0a3ff0 2024-09-16T04:45:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01bfff640 1 --2- 172.21.15.46:0/2772566170 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb01c1369d0 0x7fb01c136dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:45:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.693+0000 7fb01b7fe640 1 --2- 172.21.15.46:0/2772566170 >> v2:172.21.15.114:3300/0 conn(0x7fb01c0a3c10 0x7fb01c09bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:45:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb01affd640 1 --2- 172.21.15.46:0/2772566170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb01c0a3ff0 0x7fb01c0a43d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb01c068240 tx=0x7fb00402a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17953cc4af3cb9ca server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb01a7fc640 1 -- 172.21.15.46:0/2772566170 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb004037070 con 0x7fb01c0a3ff0 2024-09-16T04:45:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb01a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb01a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb01a7fc640 10 monclient: dump: 2024-09-16T04:45:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:21.709 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb01a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:45:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb01a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:45:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb01a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:21.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb01a7fc640 1 -- 172.21.15.46:0/2772566170 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb00402ae00 con 0x7fb01c0a3ff0 2024-09-16T04:45:21.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb01a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:21.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb01a7fc640 1 -- 172.21.15.46:0/2772566170 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb004032660 con 0x7fb01c0a3ff0 2024-09-16T04:45:21.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb021eba640 5 monclient: authenticate success, global_id 74409 2024-09-16T04:45:21.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb021eba640 10 monclient: get_monmap_and_config success 2024-09-16T04:45:21.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb021eba640 4 set_mon_vals no callback set 2024-09-16T04:45:21.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb01a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:21.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb01a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:21.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb01a7fc640 10 monclient: dump: 2024-09-16T04:45:21.710 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:21.710 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:21.710 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.711 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.711 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:21.711 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:21.711 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:21.711 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:21.711 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:21.711 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:21.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb021eba640 10 monclient: shutdown 2024-09-16T04:45:21.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb021eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:21.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb021eba640 1 -- 172.21.15.46:0/2772566170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb01c0a3ff0 msgr2=0x7fb01c0a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:21.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb021eba640 1 --2- 172.21.15.46:0/2772566170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb01c0a3ff0 0x7fb01c0a43d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb01c068240 tx=0x7fb00402a0b0 comp rx=0 tx=0).stop 2024-09-16T04:45:21.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb021eba640 1 -- 172.21.15.46:0/2772566170 shutdown_connections 2024-09-16T04:45:21.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb021eba640 1 --2- 172.21.15.46:0/2772566170 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb01c1369d0 0x7fb01c136dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:21.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb021eba640 1 --2- 172.21.15.46:0/2772566170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb01c0a3ff0 0x7fb01c0a43d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:21.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb021eba640 1 --2- 172.21.15.46:0/2772566170 >> v2:172.21.15.114:3300/0 conn(0x7fb01c0a3c10 0x7fb01c09bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:21.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb021eba640 1 -- 172.21.15.46:0/2772566170 >> 172.21.15.46:0/2772566170 conn(0x7fb01c0a6340 msgr2=0x7fb01c06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:21.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb021eba640 1 -- 172.21.15.46:0/2772566170 shutdown_connections 2024-09-16T04:45:21.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.694+0000 7fb021eba640 1 -- 172.21.15.46:0/2772566170 wait complete. 2024-09-16T04:45:21.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.695+0000 7fb021eba640 10 monclient: build_initial_monmap 2024-09-16T04:45:21.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.695+0000 7fb021eba640 10 monclient: monmap: 2024-09-16T04:45:21.712 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:21.712 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:45:21.712 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:45:21.712 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:45:21.712 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:21.712 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:45:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:45:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:45:21.713 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.695+0000 7fb021eba640 1 Processor -- start 2024-09-16T04:45:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb021eba640 1 -- start start 2024-09-16T04:45:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb021eba640 10 monclient: init 2024-09-16T04:45:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb021eba640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb021eba640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:45:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb021eba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb01c0a3c10 0x7fb01c1cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb021eba640 10 monclient(hunting): picked mon.noname-a con 0x7fb01c0a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:45:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb021eba640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb01c0a3ff0 0x7fb01c1cc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb021eba640 10 monclient(hunting): picked mon.noname-b con 0x7fb01c0a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:45:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb021eba640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb01c1369d0 0x7fb01c123190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:21.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb021eba640 10 monclient(hunting): picked mon.noname-c con 0x7fb01c1369d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:21.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb021eba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:21.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb021eba640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c0a4910 con 0x7fb01c0a3c10 2024-09-16T04:45:21.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb021eba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:21.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb021eba640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c0a69c0 con 0x7fb01c0a3ff0 2024-09-16T04:45:21.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb021eba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:21.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb021eba640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fb01c127b00 con 0x7fb01c1369d0 2024-09-16T04:45:21.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb021eba640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:21.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb021eba640 10 monclient(hunting): authenticate will time out at 1120.111473s 2024-09-16T04:45:21.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb01c0a3c10 0x7fb01c1cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:21.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb01c0a3c10 0x7fb01c1cbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32808/0 (socket says 172.21.15.46:32808) 2024-09-16T04:45:21.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01b7fe640 1 -- 172.21.15.46:0/3729610666 learned_addr learned my addr 172.21.15.46:0/3729610666 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:21.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01b7fe640 10 monclient(hunting): get_auth_request con 0x7fb01c0a3c10 auth_method 0 2024-09-16T04:45:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01bfff640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb01c1369d0 0x7fb01c123190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01bfff640 10 monclient(hunting): get_auth_request con 0x7fb01c1369d0 auth_method 0 2024-09-16T04:45:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01affd640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb01c0a3ff0 0x7fb01c1cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01affd640 10 monclient(hunting): get_auth_request con 0x7fb01c0a3ff0 auth_method 0 2024-09-16T04:45:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01b7fe640 10 monclient(hunting): handle_auth_done global_id 74412 payload 824 2024-09-16T04:45:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01b7fe640 1 -- 172.21.15.46:0/3729610666 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb01c1369d0 msgr2=0x7fb01c123190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01b7fe640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb01c1369d0 0x7fb01c123190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01b7fe640 1 -- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb01c0a3ff0 msgr2=0x7fb01c1cc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01b7fe640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb01c0a3ff0 0x7fb01c1cc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:45:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01b7fe640 1 monclient: found mon.noname-a 2024-09-16T04:45:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01b7fe640 1 -- 172.21.15.46:0/3729610666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb01c125990 con 0x7fb01c0a3c10 2024-09-16T04:45:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.696+0000 7fb01bfff640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb01c1369d0 0x7fb01c123190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:45:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb01b7fe640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb01c0a3c10 0x7fb01c1cbf50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb00c00e730 tx=0x7fb00c00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb013fff640 1 -- 172.21.15.46:0/3729610666 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c00f040 con 0x7fb01c0a3c10 2024-09-16T04:45:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb013fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb013fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb013fff640 10 monclient: dump: 2024-09-16T04:45:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:21.718 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb013fff640 10 monclient: _finish_auth 0 2024-09-16T04:45:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb013fff640 10 monclient: _check_auth_tickets 2024-09-16T04:45:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb013fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb013fff640 1 -- 172.21.15.46:0/3729610666 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb00c004030 con 0x7fb01c0a3c10 2024-09-16T04:45:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb021eba640 5 monclient: authenticate success, global_id 74412 2024-09-16T04:45:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb013fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb013fff640 1 -- 172.21.15.46:0/3729610666 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c0043e0 con 0x7fb01c0a3c10 2024-09-16T04:45:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb021eba640 10 monclient: _renew_subs 2024-09-16T04:45:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb021eba640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb021eba640 1 -- 172.21.15.46:0/3729610666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb01c0a2ad0 con 0x7fb01c0a3c10 2024-09-16T04:45:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb013fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb019ffb640 4 set_mon_vals no callback set 2024-09-16T04:45:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb013fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb013fff640 10 monclient: dump: 2024-09-16T04:45:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:21.720 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb021eba640 10 monclient: _renew_subs 2024-09-16T04:45:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb021eba640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.697+0000 7fb021eba640 1 -- 172.21.15.46:0/3729610666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb01c126b60 con 0x7fb01c0a3c10 2024-09-16T04:45:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.698+0000 7fb021eba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.698+0000 7fb021eba640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb01c126da0 0x7fb01c1236d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.698+0000 7fb021eba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.698+0000 7fb021eba640 1 -- 172.21.15.46:0/3729610666 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c0a4a90 con 0x7fb01c126da0 2024-09-16T04:45:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.698+0000 7fb013fff640 1 -- 172.21.15.46:0/3729610666 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fb00c00c670 con 0x7fb01c0a3c10 2024-09-16T04:45:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.698+0000 7fb013fff640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7faffc0383b0 0x7faffc03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.698+0000 7fb01affd640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb01c126da0 0x7fb01c1236d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.698+0000 7fb01affd640 10 monclient: get_auth_request con 0x7fb01c126da0 auth_method 0 2024-09-16T04:45:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.698+0000 7fb01affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.698+0000 7fb01affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.698+0000 7fb01affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.698+0000 7fb013fff640 1 -- 172.21.15.46:0/3729610666 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(252..252 src has 1..252) ==== 4738+0+0 (secure 0 0 0) 0x7fb00c04b470 con 0x7fb01c0a3c10 2024-09-16T04:45:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.698+0000 7fb01affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.698+0000 7fb01affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.698+0000 7fb01affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.698+0000 7fb01bfff640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7faffc0383b0 0x7faffc03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.698+0000 7fb01bfff640 10 monclient: get_auth_request con 0x7faffc0383b0 auth_method 0 2024-09-16T04:45:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.698+0000 7fb01affd640 10 monclient(hunting): handle_auth_done global_id 6484 payload 824 2024-09-16T04:45:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.698+0000 7fb01affd640 1 -- 172.21.15.46:0/3729610666 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb01c0a43d0 con 0x7fb01c126da0 2024-09-16T04:45:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.699+0000 7fb01affd640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb01c126da0 0x7fb01c1236d0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fb01c0a38d0 tx=0x7fb0040046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.699+0000 7fb01bfff640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7faffc0383b0 0x7faffc03a870 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb014003b80 tx=0x7fb01400b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.699+0000 7fb013fff640 1 -- 172.21.15.46:0/3729610666 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb004037040 con 0x7fb01c126da0 2024-09-16T04:45:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.699+0000 7fb013fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.699+0000 7fb013fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:45:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.699+0000 7fb013fff640 10 monclient: dump: 2024-09-16T04:45:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:21.722 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.699+0000 7fb013fff640 1 -- 172.21.15.46:0/3729610666 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb004009490 con 0x7fb01c126da0 2024-09-16T04:45:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.699+0000 7fb013fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.699+0000 7fb013fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:45:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.699+0000 7fb013fff640 1 -- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb01c126da0 msgr2=0x7fb01c1236d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.699+0000 7fb013fff640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb01c126da0 0x7fb01c1236d0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fb01c0a38d0 tx=0x7fb0040046c0 comp rx=0 tx=0).stop 2024-09-16T04:45:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.699+0000 7fb013fff640 1 -- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb01c126da0 msgr2=0x7fb01c1236d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:45:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.699+0000 7fb013fff640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb01c126da0 0x7fb01c1236d0 secure :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fb01c0a38d0 tx=0x7fb0040046c0 comp rx=0 tx=0).stop 2024-09-16T04:45:21.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.705+0000 7fb021eba640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:45:21.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.705+0000 7fb021eba640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb01c123c10 0x7fb01c123ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:21.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.705+0000 7fb021eba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:21.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.705+0000 7fb021eba640 1 -- 172.21.15.46:0/3729610666 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c124530 con 0x7fb01c123c10 2024-09-16T04:45:21.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.706+0000 7fb01affd640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb01c123c10 0x7fb01c123ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:21.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.706+0000 7fb01affd640 10 monclient: get_auth_request con 0x7fb01c123c10 auth_method 0 2024-09-16T04:45:21.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.706+0000 7fb01affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:21.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.706+0000 7fb01affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:21.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.706+0000 7fb01affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:21.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.706+0000 7fb01affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.706+0000 7fb01affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.706+0000 7fb01affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.706+0000 7fb01affd640 10 monclient(hunting): handle_auth_done global_id 6487 payload 824 2024-09-16T04:45:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.706+0000 7fb01affd640 1 -- 172.21.15.46:0/3729610666 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb01c067e20 con 0x7fb01c123c10 2024-09-16T04:45:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.706+0000 7fb01affd640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb01c123c10 0x7fb01c123ff0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fb01c09c8b0 tx=0x7fb004008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb013fff640 1 -- 172.21.15.46:0/3729610666 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb004037070 con 0x7fb01c123c10 2024-09-16T04:45:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb013fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb013fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:45:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb013fff640 10 monclient: dump: 2024-09-16T04:45:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:21.727 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb013fff640 1 -- 172.21.15.46:0/3729610666 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7fb00403e660 con 0x7fb01c123c10 2024-09-16T04:45:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb013fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:45:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb013fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:45:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb013fff640 1 -- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb01c123c10 msgr2=0x7fb01c123ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb013fff640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb01c123c10 0x7fb01c123ff0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fb01c09c8b0 tx=0x7fb004008030 comp rx=0 tx=0).stop 2024-09-16T04:45:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb013fff640 1 -- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb01c123c10 msgr2=0x7fb01c123ff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:45:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb013fff640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb01c123c10 0x7fb01c123ff0 secure :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fb01c09c8b0 tx=0x7fb004008030 comp rx=0 tx=0).stop 2024-09-16T04:45:21.728 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:45:21.728 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:45:21.728 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:45:21.728 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:45:21.728 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 78, 2024-09-16T04:45:21.728 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 407608, 2024-09-16T04:45:21.728 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:45:21.728 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:45:21.728 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:45:21.728 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:45:21.728 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:21.728 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 3, 2024-09-16T04:45:21.729 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:45:21.729 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:45:21.729 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:45:21.729 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:21.729 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:21.729 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:21.729 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:21.729 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:21.729 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:21.729 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:21.729 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:21.729 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:21.729 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:21.730 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:21.730 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:21.730 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:45:21.730 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:45:21.730 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:21.730 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:21.730 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:21.730 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:21.730 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:21.730 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:21.730 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:21.730 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:21.730 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:45:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:45:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:45:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:45:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:45:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:45:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:45:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:45:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:45:21.732 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:45:21.732 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:45:21.732 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:45:21.732 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:45:21.732 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:45:21.732 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:45:21.732 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:45:21.732 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:45:21.732 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:21.732 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:21.732 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:21.732 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:21.732 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:21.733 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:21.733 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:21.733 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:21.733 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:21.733 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:21.733 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:21.733 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:21.733 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:45:21.733 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.733 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:45:21.733 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.733 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:45:21.733 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:45:21.734 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:21.734 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:21.734 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.734 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:21.734 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:45:21.734 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:21.734 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.734 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.734 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:21.734 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:45:21.734 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:21.734 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:21.734 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:21.735 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.735 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:45:21.735 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:45:21.735 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:21.735 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:21.735 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:21.735 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.735 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.735 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:45:21.735 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:45:21.735 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:21.736 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:21.736 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.736 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:21.736 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:45:21.736 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:21.736 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.736 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.736 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:21.736 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:45:21.736 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:21.736 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:21.736 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:21.736 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.737 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:45:21.737 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:45:21.737 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:21.737 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:21.737 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:21.737 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.737 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.737 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:45:21.737 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:45:21.737 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:21.737 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:21.737 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.737 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:21.737 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:45:21.738 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:21.738 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.738 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.738 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:21.738 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:45:21.738 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:21.738 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:21.738 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:21.738 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.738 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:45:21.738 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:45:21.738 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:21.738 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:21.739 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:21.739 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:21.739 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:21.739 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.739 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:45:21.739 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:45:21.739 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.739 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:21.739 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:21.739 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:45:21.739 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:21.739 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:21.739 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:45:21.739 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.740 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:21.740 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:21.740 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5 2024-09-16T04:45:21.740 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:21.740 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:21.740 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:45:21.740 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.740 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:21.740 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:21.740 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 7 2024-09-16T04:45:21.740 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:21.740 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:21.740 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:45:21.741 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:21.741 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:21.741 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:21.741 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:45:21.741 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:21.741 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:21.741 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:21.741 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:45:21.741 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:45:21.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb021eba640 1 -- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7faffc0383b0 msgr2=0x7faffc03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:21.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb021eba640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7faffc0383b0 0x7faffc03a870 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb014003b80 tx=0x7fb01400b040 comp rx=0 tx=0).stop 2024-09-16T04:45:21.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb021eba640 10 monclient: shutdown 2024-09-16T04:45:21.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb021eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:21.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb021eba640 1 -- 172.21.15.46:0/3729610666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb01c0a3c10 msgr2=0x7fb01c1cbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:21.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb021eba640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb01c0a3c10 0x7fb01c1cbf50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb00c00e730 tx=0x7fb00c00ebf0 comp rx=0 tx=0).stop 2024-09-16T04:45:21.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb01bfff640 1 -- 172.21.15.46:0/3729610666 reap_dead start 2024-09-16T04:45:21.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb021eba640 1 -- 172.21.15.46:0/3729610666 shutdown_connections 2024-09-16T04:45:21.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb021eba640 1 --2- 172.21.15.46:0/3729610666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb01c0a3c10 0x7fb01c1cbf50 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:21.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb021eba640 1 -- 172.21.15.46:0/3729610666 >> 172.21.15.46:0/3729610666 conn(0x7fb01c0a6340 msgr2=0x7fb01c0a11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:21.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb021eba640 1 -- 172.21.15.46:0/3729610666 shutdown_connections 2024-09-16T04:45:21.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.707+0000 7fb021eba640 1 -- 172.21.15.46:0/3729610666 wait complete. 2024-09-16T04:45:21.744 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:45:21.744 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:45:21.744 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:45:21.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.844+0000 7f8607647640 10 monclient: get_monmap_and_config 2024-09-16T04:45:21.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.844+0000 7f8607647640 10 monclient: build_initial_monmap 2024-09-16T04:45:21.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.844+0000 7f8607647640 10 monclient: monmap: 2024-09-16T04:45:21.845 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:21.845 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:45:21.845055+0000 2024-09-16T04:45:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:45:21.845055+0000 2024-09-16T04:45:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:45:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:45:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:45:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:45:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:45:21.846 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.845+0000 7f8607647640 1 Processor -- start 2024-09-16T04:45:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.845+0000 7f8607647640 1 -- start start 2024-09-16T04:45:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.845+0000 7f8607647640 10 monclient: init 2024-09-16T04:45:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.845+0000 7f8607647640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.845+0000 7f8607647640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-16T04:45:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.845+0000 7f8607647640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8600136e50 addr v1:172.21.15.46:6789/0 2024-09-16T04:45:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.845+0000 7f8607647640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f86000a3500 0x7f860013bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.845+0000 7f8607647640 10 monclient(hunting): picked mon.noname-c con 0x7f86000a3500 addr v2:172.21.15.114:3300/0 2024-09-16T04:45:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.845+0000 7f8607647640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f860013c1f0 0x7f860013e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.845+0000 7f8607647640 10 monclient(hunting): picked mon.noname-a con 0x7f860013c1f0 addr v2:172.21.15.46:3300/0 2024-09-16T04:45:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.845+0000 7f8607647640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86000a4e90 con 0x7f8600136e50 2024-09-16T04:45:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.845+0000 7f8607647640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.845+0000 7f8607647640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f86000a5410 con 0x7f860013c1f0 2024-09-16T04:45:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.845+0000 7f8607647640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.845+0000 7f8607647640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f86000a4c50 con 0x7f86000a3500 2024-09-16T04:45:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.845+0000 7f8607647640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.845+0000 7f8607647640 10 monclient(hunting): authenticate will time out at 1120.260913s 2024-09-16T04:45:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8605be4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f860013c1f0 0x7f860013e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f86053e3640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8600136e50 0x7f8600137230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45468/0 (socket says 172.21.15.46:45468) 2024-09-16T04:45:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f86053e3640 1 -- 172.21.15.46:0/2535465728 learned_addr learned my addr 172.21.15.46:0/2535465728 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:45:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8605be4640 10 monclient(hunting): get_auth_request con 0x7f860013c1f0 auth_method 0 2024-09-16T04:45:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8605be4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8605be4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8605be4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8605be4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8605be4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8604be2640 1 --2- 172.21.15.46:0/2535465728 >> v2:172.21.15.114:3300/0 conn(0x7f86000a3500 0x7f860013bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8605be4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8604be2640 10 monclient(hunting): get_auth_request con 0x7f86000a3500 auth_method 0 2024-09-16T04:45:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8604be2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8604be2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8604be2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8604be2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8604be2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8604be2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8605be4640 10 monclient(hunting): handle_auth_done global_id 74415 payload 274 2024-09-16T04:45:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8605be4640 1 -- 172.21.15.46:0/2535465728 >> v2:172.21.15.114:3300/0 conn(0x7f86000a3500 msgr2=0x7f860013bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:45:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8605be4640 1 --2- 172.21.15.46:0/2535465728 >> v2:172.21.15.114:3300/0 conn(0x7f86000a3500 0x7f860013bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:21.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8605be4640 1 -- 172.21.15.46:0/2535465728 >> v1:172.21.15.46:6789/0 conn(0x7f8600136e50 legacy=0x7f8600137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:21.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8605be4640 10 monclient: _finish_hunting 0 2024-09-16T04:45:21.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8605be4640 1 monclient: found mon.noname-a 2024-09-16T04:45:21.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8605be4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:21.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8605be4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:21.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.846+0000 7f8605be4640 1 -- 172.21.15.46:0/2535465728 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86000a0dc0 con 0x7f860013c1f0 2024-09-16T04:45:21.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f8605be4640 1 --2- 172.21.15.46:0/2535465728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f860013c1f0 0x7f860013e5e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f86000a70f0 tx=0x7f85f802a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eadb1de583a6d7e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:21.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f85f7fff640 1 -- 172.21.15.46:0/2535465728 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f8007740 con 0x7f860013c1f0 2024-09-16T04:45:21.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f85f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:21.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f85f7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:21.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f85f7fff640 10 monclient: dump: 2024-09-16T04:45:21.857 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:21.857 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:21.858 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.858 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.858 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:21.858 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:21.858 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:21.858 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:21.858 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:21.858 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:21.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f85f7fff640 10 monclient: _finish_auth 0 2024-09-16T04:45:21.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f85f7fff640 10 monclient: _check_auth_tickets 2024-09-16T04:45:21.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f85f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:21.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f8607647640 5 monclient: authenticate success, global_id 74415 2024-09-16T04:45:21.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f85f7fff640 1 -- 172.21.15.46:0/2535465728 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85f80078e0 con 0x7f860013c1f0 2024-09-16T04:45:21.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f8607647640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:45:21.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f85f7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:21.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f8607647640 10 monclient: get_monmap_and_config success 2024-09-16T04:45:21.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f8607647640 4 set_mon_vals no callback set 2024-09-16T04:45:21.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f85f7fff640 1 -- 172.21.15.46:0/2535465728 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f8007cb0 con 0x7f860013c1f0 2024-09-16T04:45:21.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f85f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:21.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f85f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:21.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f85f7fff640 10 monclient: dump: 2024-09-16T04:45:21.859 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:21.859 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:21.859 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.859 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.859 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:21.859 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:21.859 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:21.860 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f8607647640 10 monclient: shutdown 2024-09-16T04:45:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f8607647640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f8607647640 1 -- 172.21.15.46:0/2535465728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f860013c1f0 msgr2=0x7f860013e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f8607647640 1 --2- 172.21.15.46:0/2535465728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f860013c1f0 0x7f860013e5e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f86000a70f0 tx=0x7f85f802a560 comp rx=0 tx=0).stop 2024-09-16T04:45:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f8607647640 1 -- 172.21.15.46:0/2535465728 shutdown_connections 2024-09-16T04:45:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f8607647640 1 --2- 172.21.15.46:0/2535465728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f860013c1f0 0x7f860013e5e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f8607647640 1 --2- 172.21.15.46:0/2535465728 >> v2:172.21.15.114:3300/0 conn(0x7f86000a3500 0x7f860013bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.847+0000 7f8607647640 1 -- 172.21.15.46:0/2535465728 >> 172.21.15.46:0/2535465728 conn(0x7f86000a5f20 msgr2=0x7f86000a6330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.848+0000 7f8607647640 1 -- 172.21.15.46:0/2535465728 shutdown_connections 2024-09-16T04:45:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.848+0000 7f8607647640 1 -- 172.21.15.46:0/2535465728 wait complete. 2024-09-16T04:45:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.848+0000 7f8607647640 10 monclient: build_initial_monmap 2024-09-16T04:45:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.848+0000 7f8607647640 10 monclient: monmap: 2024-09-16T04:45:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:45:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:45:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:45:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:45:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:45:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:45:21.861 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.848+0000 7f8607647640 1 Processor -- start 2024-09-16T04:45:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8607647640 1 -- start start 2024-09-16T04:45:21.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8607647640 10 monclient: init 2024-09-16T04:45:21.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8607647640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:21.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8607647640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:45:21.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8607647640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86000a3500 0x7f86001cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:21.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8607647640 10 monclient(hunting): picked mon.noname-a con 0x7f86000a3500 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:45:21.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8607647640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8600136e50 0x7f86001cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:21.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8607647640 10 monclient(hunting): picked mon.noname-b con 0x7f8600136e50 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:45:21.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8607647640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f860013c1f0 0x7f860009c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:21.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8607647640 10 monclient(hunting): picked mon.noname-c con 0x7f860013c1f0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:21.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8607647640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:21.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8607647640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f86000a0b30 con 0x7f86000a3500 2024-09-16T04:45:21.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8607647640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:21.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8607647640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f86000a09b0 con 0x7f8600136e50 2024-09-16T04:45:21.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8607647640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:21.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8607647640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f86000a1b60 con 0x7f860013c1f0 2024-09-16T04:45:21.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8607647640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:21.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8607647640 10 monclient(hunting): authenticate will time out at 1120.264541s 2024-09-16T04:45:21.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f86053e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86000a3500 0x7f86001cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:21.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f86053e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86000a3500 0x7f86001cb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32838/0 (socket says 172.21.15.46:32838) 2024-09-16T04:45:21.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f86053e3640 1 -- 172.21.15.46:0/65857375 learned_addr learned my addr 172.21.15.46:0/65857375 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:21.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f86053e3640 10 monclient(hunting): get_auth_request con 0x7f86000a3500 auth_method 0 2024-09-16T04:45:21.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f86053e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:21.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f86053e3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:21.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f86053e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:21.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f86053e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:21.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f86053e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:21.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8605be4640 1 --2- 172.21.15.46:0/65857375 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f860013c1f0 0x7f860009c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:21.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f86053e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:21.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8605be4640 10 monclient(hunting): get_auth_request con 0x7f860013c1f0 auth_method 0 2024-09-16T04:45:21.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8605be4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:21.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8605be4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:21.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8605be4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:21.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8605be4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:21.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8605be4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:21.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8605be4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:21.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8604be2640 1 --2- 172.21.15.46:0/65857375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8600136e50 0x7f86001cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:21.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8604be2640 10 monclient(hunting): get_auth_request con 0x7f8600136e50 auth_method 0 2024-09-16T04:45:21.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8604be2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:21.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8604be2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:21.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f8604be2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:21.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f86053e3640 10 monclient(hunting): handle_auth_done global_id 74421 payload 824 2024-09-16T04:45:21.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f86053e3640 1 -- 172.21.15.46:0/65857375 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f860013c1f0 msgr2=0x7f860009c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:21.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f86053e3640 1 --2- 172.21.15.46:0/65857375 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f860013c1f0 0x7f860009c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:21.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f86053e3640 1 -- 172.21.15.46:0/65857375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8600136e50 msgr2=0x7f86001cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:21.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f86053e3640 1 --2- 172.21.15.46:0/65857375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8600136e50 0x7f86001cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:21.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f86053e3640 10 monclient: _finish_hunting 0 2024-09-16T04:45:21.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f86053e3640 1 monclient: found mon.noname-a 2024-09-16T04:45:21.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f86053e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:21.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f86053e3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:21.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.849+0000 7f86053e3640 1 -- 172.21.15.46:0/65857375 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8600120230 con 0x7f86000a3500 2024-09-16T04:45:21.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f86053e3640 1 --2- 172.21.15.46:0/65857375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86000a3500 0x7f86001cb9b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f85fc00c400 tx=0x7f85fc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:21.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f85f5ffb640 1 -- 172.21.15.46:0/65857375 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85fc017070 con 0x7f86000a3500 2024-09-16T04:45:21.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f85f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:21.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f85f5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:21.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f85f5ffb640 10 monclient: dump: 2024-09-16T04:45:21.866 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:21.866 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:21.866 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.866 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.866 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:21.866 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:21.866 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:21.866 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:21.866 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:21.866 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:21.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f85f5ffb640 10 monclient: _finish_auth 0 2024-09-16T04:45:21.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f85f5ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:45:21.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f85f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:21.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f8607647640 5 monclient: authenticate success, global_id 74421 2024-09-16T04:45:21.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f85f5ffb640 1 -- 172.21.15.46:0/65857375 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85fc005ce0 con 0x7f86000a3500 2024-09-16T04:45:21.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f8607647640 10 monclient: _renew_subs 2024-09-16T04:45:21.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f8607647640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:21.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f8607647640 1 -- 172.21.15.46:0/65857375 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8600120510 con 0x7f86000a3500 2024-09-16T04:45:21.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f85f5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:21.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f85f5ffb640 1 -- 172.21.15.46:0/65857375 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85fc0072f0 con 0x7f86000a3500 2024-09-16T04:45:21.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f85f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:21.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f85f5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:21.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f85f5ffb640 10 monclient: dump: 2024-09-16T04:45:21.867 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:21.867 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:21.868 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f85f77fe640 4 set_mon_vals no callback set 2024-09-16T04:45:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f8607647640 10 monclient: _renew_subs 2024-09-16T04:45:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f8607647640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.850+0000 7f8607647640 1 -- 172.21.15.46:0/65857375 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86000a0430 con 0x7f86000a3500 2024-09-16T04:45:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.851+0000 7f85f5ffb640 1 -- 172.21.15.46:0/65857375 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f85fc01a070 con 0x7f86000a3500 2024-09-16T04:45:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.851+0000 7f8607647640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:21.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.851+0000 7f8607647640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:21.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.851+0000 7f8607647640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:21.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.851+0000 7f8607647640 1 -- 172.21.15.46:0/65857375 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8600137230 con 0x7f86000a3500 2024-09-16T04:45:21.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.851+0000 7f85f5ffb640 1 --2- 172.21.15.46:0/65857375 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f85e40383b0 0x7f85e403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:21.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.851+0000 7f85f5ffb640 1 -- 172.21.15.46:0/65857375 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(252..252 src has 1..252) ==== 4738+0+0 (secure 0 0 0) 0x7f85fc04ab80 con 0x7f86000a3500 2024-09-16T04:45:21.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.851+0000 7f8604be2640 1 --2- 172.21.15.46:0/65857375 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f85e40383b0 0x7f85e403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:21.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.851+0000 7f8604be2640 10 monclient: get_auth_request con 0x7f85e40383b0 auth_method 0 2024-09-16T04:45:21.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.852+0000 7f8604be2640 1 --2- 172.21.15.46:0/65857375 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f85e40383b0 0x7f85e403a870 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8600120720 tx=0x7f85f0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:21.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.856+0000 7f85f5ffb640 1 -- 172.21.15.46:0/65857375 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f85fc019e10 con 0x7f86000a3500 2024-09-16T04:45:21.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.856+0000 7f85f5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:21.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:21.856+0000 7f85f5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:45:22.107 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:22.107 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39337]: starting. 2024-09-16T04:45:22.107 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39337]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-39302 2024-09-16T04:45:22.107 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39337]: created object 0... 2024-09-16T04:45:22.107 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39337]: created object 25... 2024-09-16T04:45:22.107 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39337]: created object 49... 2024-09-16T04:45:22.108 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39337]: finishing. 2024-09-16T04:45:22.108 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39337]: shutting down. 2024-09-16T04:45:22.119 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:22.119 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39339]: starting. 2024-09-16T04:45:22.119 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39339]: listing objects. 2024-09-16T04:45:22.119 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39339]: listed object 0... 2024-09-16T04:45:22.119 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39339]: listed object 25... 2024-09-16T04:45:22.120 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39339]: saw 50 objects 2024-09-16T04:45:22.120 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39339]: shutting down. 2024-09-16T04:45:22.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:22.123+0000 7f8607647640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:45:22.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:22.123+0000 7f8607647640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:45:22.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:22.123+0000 7f8607647640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:22.123+0000 7f8607647640 1 -- 172.21.15.46:0/65857375 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f860009ed20 con 0x7f86000a3500 2024-09-16T04:45:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:22.124+0000 7f85f5ffb640 1 -- 172.21.15.46:0/65857375 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f85fc010920 con 0x7f86000a3500 2024-09-16T04:45:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:22.124+0000 7f85f5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:45:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:22.124+0000 7f85f5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:45:22.126 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.125+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.126 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.125+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=10,kv=11,logm=64} crc {auth=4071928632,config=3541143443,health=3943079174,kv=89513504,logm=666298534}) 2024-09-16T04:45:22.126 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.125+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=12,config=2,health=12,kv=11,logm=63} crc {auth=1734103084,config=3541143443,health=208846210,kv=2534035223,logm=230346403}) 2024-09-16T04:45:22.126 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.125+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.126 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.125+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=10,kv=11,logm=64} crc {auth=4071928632,config=3541143443,health=3943079174,kv=89513504,logm=666298534}) 2024-09-16T04:45:22.126 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.125+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=15,config=2,health=11,kv=12,logm=60} crc {auth=256532066,config=3541143443,health=862269741,kv=4017288519,logm=3207627749}) 2024-09-16T04:45:22.127 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.126+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.127 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.126+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=836155612}) 2024-09-16T04:45:22.127 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.126+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2606823481}) 2024-09-16T04:45:22.127 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.126+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.127 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.126+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=836155612}) 2024-09-16T04:45:22.127 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.126+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=808275947}) 2024-09-16T04:45:22.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:22.126+0000 7f8607647640 1 -- 172.21.15.46:0/65857375 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f85e40383b0 msgr2=0x7f85e403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:22.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:22.126+0000 7f8607647640 1 --2- 172.21.15.46:0/65857375 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f85e40383b0 0x7f85e403a870 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8600120720 tx=0x7f85f0008040 comp rx=0 tx=0).stop 2024-09-16T04:45:22.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:22.126+0000 7f8607647640 10 monclient: shutdown 2024-09-16T04:45:22.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:22.126+0000 7f8607647640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:22.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:22.126+0000 7f8607647640 1 -- 172.21.15.46:0/65857375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86000a3500 msgr2=0x7f86001cb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:22.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:22.126+0000 7f8607647640 1 --2- 172.21.15.46:0/65857375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86000a3500 0x7f86001cb9b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f85fc00c400 tx=0x7f85fc00c8c0 comp rx=0 tx=0).stop 2024-09-16T04:45:22.129 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.127+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.129 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.127+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=402399826}) 2024-09-16T04:45:22.129 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.127+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2536804391}) 2024-09-16T04:45:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:22.127+0000 7f8607647640 1 -- 172.21.15.46:0/65857375 shutdown_connections 2024-09-16T04:45:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:22.127+0000 7f8607647640 1 --2- 172.21.15.46:0/65857375 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f860013c1f0 0x7f860009c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:22.127+0000 7f8607647640 1 --2- 172.21.15.46:0/65857375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8600136e50 0x7f86001cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:22.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:22.127+0000 7f8607647640 1 --2- 172.21.15.46:0/65857375 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f85e40383b0 0x7f85e403a870 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:22.130 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.127+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.130 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.127+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=402399826}) 2024-09-16T04:45:22.130 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.127+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2487500304}) 2024-09-16T04:45:22.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:22.127+0000 7f8607647640 1 --2- 172.21.15.46:0/65857375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86000a3500 0x7f86001cb9b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:22.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:22.127+0000 7f8607647640 1 -- 172.21.15.46:0/65857375 >> 172.21.15.46:0/65857375 conn(0x7f86000a5f20 msgr2=0x7f860006d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:22.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:22.127+0000 7f8607647640 1 -- 172.21.15.46:0/65857375 shutdown_connections 2024-09-16T04:45:22.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:22.127+0000 7f8607647640 1 -- 172.21.15.46:0/65857375 wait complete. 2024-09-16T04:45:22.132 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.127+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.133 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.127+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184521951}) 2024-09-16T04:45:22.133 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.127+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3841332559}) 2024-09-16T04:45:22.133 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.127+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.133 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.127+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184521951}) 2024-09-16T04:45:22.133 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.127+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=696464419}) 2024-09-16T04:45:22.133 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.128+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.133 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.128+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2051983139}) 2024-09-16T04:45:22.133 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.128+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2306930637}) 2024-09-16T04:45:22.133 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.128+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.133 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.128+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2051983139}) 2024-09-16T04:45:22.133 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.128+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4170484031}) 2024-09-16T04:45:22.133 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.129+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.133 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.129+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779521505}) 2024-09-16T04:45:22.134 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.129+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2230297749}) 2024-09-16T04:45:22.134 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.129+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.134 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.129+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779521505}) 2024-09-16T04:45:22.134 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.129+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4144864753}) 2024-09-16T04:45:22.134 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.129+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.134 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.129+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3100020627}) 2024-09-16T04:45:22.134 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.129+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1072884827}) 2024-09-16T04:45:22.134 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.129+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.134 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.129+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3100020627}) 2024-09-16T04:45:22.134 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.129+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1649619622}) 2024-09-16T04:45:22.134 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.130+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.134 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.130+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3181158271}) 2024-09-16T04:45:22.134 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.130+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2018720875}) 2024-09-16T04:45:22.134 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.130+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.135 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.130+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3181158271}) 2024-09-16T04:45:22.135 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.130+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2927908283}) 2024-09-16T04:45:22.135 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.131+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.135 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.131+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2179635525}) 2024-09-16T04:45:22.135 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.131+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=97924230}) 2024-09-16T04:45:22.135 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.131+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.135 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.131+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2179635525}) 2024-09-16T04:45:22.135 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.131+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1432545619}) 2024-09-16T04:45:22.137 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.132+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.137 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.132+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3041231839}) 2024-09-16T04:45:22.137 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.132+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=384337731}) 2024-09-16T04:45:22.137 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.132+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.137 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.132+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3041231839}) 2024-09-16T04:45:22.137 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.132+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2341221942}) 2024-09-16T04:45:22.137 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.133+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.137 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.133+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2990746560}) 2024-09-16T04:45:22.137 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.133+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2703826837}) 2024-09-16T04:45:22.137 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.133+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.137 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.133+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2990746560}) 2024-09-16T04:45:22.137 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.133+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2158479964}) 2024-09-16T04:45:22.137 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.134+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.134+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3513212125}) 2024-09-16T04:45:22.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.134+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2088573360}) 2024-09-16T04:45:22.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.134+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.134+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3513212125}) 2024-09-16T04:45:22.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.134+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1723059909}) 2024-09-16T04:45:22.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.135+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.135+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2884663667}) 2024-09-16T04:45:22.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.135+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2203489210}) 2024-09-16T04:45:22.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.135+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.135+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2884663667}) 2024-09-16T04:45:22.138 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.135+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2391025710}) 2024-09-16T04:45:22.139 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.136+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.139 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.136+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=732027496}) 2024-09-16T04:45:22.139 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.136+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=234824559}) 2024-09-16T04:45:22.139 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.136+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.140 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.136+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=732027496}) 2024-09-16T04:45:22.140 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.137+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2895571028}) 2024-09-16T04:45:22.140 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.137+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.140 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.137+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3779246588}) 2024-09-16T04:45:22.140 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.137+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=518417895}) 2024-09-16T04:45:22.140 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.137+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.140 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.138+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3779246588}) 2024-09-16T04:45:22.140 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.138+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2793009523}) 2024-09-16T04:45:22.141 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:45:22.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.138+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.138+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=443183387}) 2024-09-16T04:45:22.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.139+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1756786930}) 2024-09-16T04:45:22.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.139+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.139+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=443183387}) 2024-09-16T04:45:22.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.139+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1813677615}) 2024-09-16T04:45:22.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.139+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.139+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3156290816}) 2024-09-16T04:45:22.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.139+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3075729053}) 2024-09-16T04:45:22.143 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.139+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.143 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.140+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3156290816}) 2024-09-16T04:45:22.143 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.140+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4171148701}) 2024-09-16T04:45:22.143 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.140+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.143 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.141+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2376469930}) 2024-09-16T04:45:22.143 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.141+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2424863374}) 2024-09-16T04:45:22.143 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.141+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.143 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.141+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2376469930}) 2024-09-16T04:45:22.143 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.141+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2378347381}) 2024-09-16T04:45:22.144 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.141+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.144 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.141+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=446941198}) 2024-09-16T04:45:22.144 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.141+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2587003185}) 2024-09-16T04:45:22.144 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.141+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.144 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.141+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=446941198}) 2024-09-16T04:45:22.145 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.141+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1376009483}) 2024-09-16T04:45:22.145 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.143+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.145 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.143+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=26,mdsmap=2,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=27} crc {logm=4013990678,mdsmap=3140514009,mgr=753868082,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=191757757,mon_config_key=1708457055,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=1711428095,osd_snap=1927752934,osdmap=198226479}) 2024-09-16T04:45:22.145 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.143+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=30,mdsmap=3,mgr=6,mgr_command_descs=1,mgrstat=10,mon_config_key=12,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=26} crc {logm=1464882257,mdsmap=3956149183,mgr=1551568835,mgr_command_descs=3961477881,mgrstat=931170864,mon_config_key=1659590655,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=1711428095,osd_snap=723727393,osdmap=3204137563}) 2024-09-16T04:45:22.145 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.143+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.145 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.143+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=26,mdsmap=2,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=27} crc {logm=4013990678,mdsmap=3140514009,mgr=753868082,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=191757757,mon_config_key=1708457055,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=1711428095,osd_snap=1927752934,osdmap=198226479}) 2024-09-16T04:45:22.145 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.143+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=30,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=20} crc {logm=2513158695,mdsmap=3956149183,mgr=1130647210,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=191757757,mon_config_key=1385660123,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=1711428095,osd_snap=723727393,osdmap=3427573232}) 2024-09-16T04:45:22.146 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.144+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.146 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.144+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4190558370}) 2024-09-16T04:45:22.146 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.144+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=400904181}) 2024-09-16T04:45:22.146 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.144+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.146 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.144+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4190558370}) 2024-09-16T04:45:22.146 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.144+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=886585914}) 2024-09-16T04:45:22.146 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.145+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.146 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.145+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2084416497}) 2024-09-16T04:45:22.147 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.145+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1559720550}) 2024-09-16T04:45:22.147 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.145+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.147 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.145+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2084416497}) 2024-09-16T04:45:22.147 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.145+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1794974419}) 2024-09-16T04:45:22.148 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.147+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.148 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.147+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=374772328}) 2024-09-16T04:45:22.148 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.147+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1957446471}) 2024-09-16T04:45:22.148 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.147+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.148 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.147+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=374772328}) 2024-09-16T04:45:22.148 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.147+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2784950021}) 2024-09-16T04:45:22.155 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.154+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.155 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.154+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=47} crc {osdmap=4070569129}) 2024-09-16T04:45:22.155 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.154+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=50} crc {osdmap=2573281574}) 2024-09-16T04:45:22.155 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.154+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:45:22.155 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.154+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=47} crc {osdmap=4070569129}) 2024-09-16T04:45:22.155 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:45:22.154+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=51} crc {osdmap=882804999}) 2024-09-16T04:45:23.029 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:23.030 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39338]: starting. 2024-09-16T04:45:23.030 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39338]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-39302 2024-09-16T04:45:23.030 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39338]: shutting down. 2024-09-16T04:45:23.031 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:23.031 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:23.031 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:45:23.033 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:45:23.141 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:45:23.142 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:45:23.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.249+0000 7f15458f1640 10 monclient: get_monmap_and_config 2024-09-16T04:45:23.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.249+0000 7f15458f1640 10 monclient: build_initial_monmap 2024-09-16T04:45:23.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.249+0000 7f15458f1640 10 monclient: monmap: 2024-09-16T04:45:23.251 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:23.251 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.251 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:45:23.250204+0000 2024-09-16T04:45:23.251 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:45:23.250204+0000 2024-09-16T04:45:23.251 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:23.251 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.251 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:45:23.251 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:45:23.252 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:45:23.252 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:45:23.252 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:45:23.252 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.250+0000 7f15458f1640 1 Processor -- start 2024-09-16T04:45:23.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.250+0000 7f15458f1640 1 -- start start 2024-09-16T04:45:23.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.250+0000 7f15458f1640 10 monclient: init 2024-09-16T04:45:23.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.250+0000 7f15458f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:23.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.250+0000 7f15458f1640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-16T04:45:23.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.250+0000 7f15458f1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f15400a3ba0 0x7f154009bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.250+0000 7f15458f1640 10 monclient(hunting): picked mon.noname-a con 0x7f15400a3ba0 addr v2:172.21.15.46:3300/0 2024-09-16T04:45:23.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.250+0000 7f15458f1640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f154009c1c0 0x7f15400a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.250+0000 7f15458f1640 10 monclient(hunting): picked mon.noname-c con 0x7f154009c1c0 addr v2:172.21.15.114:3300/0 2024-09-16T04:45:23.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f15458f1640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15400a44c0 0x7f1540135950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f15458f1640 10 monclient(hunting): picked mon.noname-b con 0x7f15400a44c0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:23.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f15458f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f15458f1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f154009b860 con 0x7f15400a3ba0 2024-09-16T04:45:23.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f15458f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f15458f1640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f154009b2a0 con 0x7f154009c1c0 2024-09-16T04:45:23.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f15458f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f15458f1640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f154009afc0 con 0x7f15400a44c0 2024-09-16T04:45:23.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f15458f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:23.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f15458f1640 10 monclient(hunting): authenticate will time out at 1121.666048s 2024-09-16T04:45:23.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153f7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15400a44c0 0x7f1540135950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153f7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15400a44c0 0x7f1540135950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:34824/0 (socket says 172.21.15.46:34824) 2024-09-16T04:45:23.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f15400a3ba0 0x7f154009bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153f7fe640 1 -- 172.21.15.46:0/2767724308 learned_addr learned my addr 172.21.15.46:0/2767724308 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:23.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153f7fe640 10 monclient(hunting): get_auth_request con 0x7f15400a44c0 auth_method 0 2024-09-16T04:45:23.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153effd640 10 monclient(hunting): get_auth_request con 0x7f15400a3ba0 auth_method 0 2024-09-16T04:45:23.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:23.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:23.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:23.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:23.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:23.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153e7fc640 1 --2- 172.21.15.46:0/2767724308 >> v2:172.21.15.114:3300/0 conn(0x7f154009c1c0 0x7f15400a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:23.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153e7fc640 10 monclient(hunting): get_auth_request con 0x7f154009c1c0 auth_method 0 2024-09-16T04:45:23.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153f7fe640 10 monclient(hunting): handle_auth_done global_id 84125 payload 274 2024-09-16T04:45:23.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153f7fe640 1 -- 172.21.15.46:0/2767724308 >> v2:172.21.15.114:3300/0 conn(0x7f154009c1c0 msgr2=0x7f15400a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:45:23.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153f7fe640 1 --2- 172.21.15.46:0/2767724308 >> v2:172.21.15.114:3300/0 conn(0x7f154009c1c0 0x7f15400a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153f7fe640 1 -- 172.21.15.46:0/2767724308 >> v2:172.21.15.46:3300/0 conn(0x7f15400a3ba0 msgr2=0x7f154009bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:45:23.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153f7fe640 1 --2- 172.21.15.46:0/2767724308 >> v2:172.21.15.46:3300/0 conn(0x7f15400a3ba0 0x7f154009bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:45:23.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153f7fe640 1 monclient: found mon.noname-b 2024-09-16T04:45:23.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:23.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:45:23.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153f7fe640 1 -- 172.21.15.46:0/2767724308 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f154009b580 con 0x7f15400a44c0 2024-09-16T04:45:23.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153effd640 1 --2- 172.21.15.46:0/2767724308 >> v2:172.21.15.46:3300/0 conn(0x7f15400a3ba0 0x7f154009bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:45:23.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.251+0000 7f153e7fc640 1 --2- 172.21.15.46:0/2767724308 >> v2:172.21.15.114:3300/0 conn(0x7f154009c1c0 0x7f15400a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:45:23.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f153f7fe640 1 --2- 172.21.15.46:0/2767724308 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15400a44c0 0x7f1540135950 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f15400a0360 tx=0x7f153802a610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc67b6caa887d831 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:23.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f153dffb640 1 -- 172.21.15.46:0/2767724308 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1538036070 con 0x7f15400a44c0 2024-09-16T04:45:23.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f153dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f153dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:45:23.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f153dffb640 10 monclient: dump: 2024-09-16T04:45:23.265 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.265 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.265 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.265 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.265 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.265 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.265 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.265 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.265 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.265 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f153dffb640 10 monclient: _finish_auth 0 2024-09-16T04:45:23.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f153dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:45:23.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f153dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:23.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f153dffb640 1 -- 172.21.15.46:0/2767724308 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1538031600 con 0x7f15400a44c0 2024-09-16T04:45:23.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f15458f1640 5 monclient: authenticate success, global_id 84125 2024-09-16T04:45:23.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f15458f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:45:23.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f153dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:23.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f15458f1640 10 monclient: get_monmap_and_config success 2024-09-16T04:45:23.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f153dffb640 1 -- 172.21.15.46:0/2767724308 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15380319d0 con 0x7f15400a44c0 2024-09-16T04:45:23.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f15458f1640 4 set_mon_vals no callback set 2024-09-16T04:45:23.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f153dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f153dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:45:23.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f153dffb640 10 monclient: dump: 2024-09-16T04:45:23.266 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.266 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.266 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.267 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.267 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.267 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.267 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.267 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.267 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.267 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f15458f1640 10 monclient: shutdown 2024-09-16T04:45:23.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f15458f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:23.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f15458f1640 1 -- 172.21.15.46:0/2767724308 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15400a44c0 msgr2=0x7f1540135950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.252+0000 7f15458f1640 1 --2- 172.21.15.46:0/2767724308 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15400a44c0 0x7f1540135950 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f15400a0360 tx=0x7f153802a610 comp rx=0 tx=0).stop 2024-09-16T04:45:23.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.253+0000 7f15458f1640 1 -- 172.21.15.46:0/2767724308 shutdown_connections 2024-09-16T04:45:23.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.253+0000 7f15458f1640 1 --2- 172.21.15.46:0/2767724308 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15400a44c0 0x7f1540135950 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.253+0000 7f15458f1640 1 --2- 172.21.15.46:0/2767724308 >> v2:172.21.15.114:3300/0 conn(0x7f154009c1c0 0x7f15400a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.253+0000 7f15458f1640 1 --2- 172.21.15.46:0/2767724308 >> v2:172.21.15.46:3300/0 conn(0x7f15400a3ba0 0x7f154009bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.253+0000 7f15458f1640 1 -- 172.21.15.46:0/2767724308 >> 172.21.15.46:0/2767724308 conn(0x7f15400a60c0 msgr2=0x7f15400a64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:23.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.253+0000 7f15458f1640 1 -- 172.21.15.46:0/2767724308 shutdown_connections 2024-09-16T04:45:23.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.253+0000 7f15458f1640 1 -- 172.21.15.46:0/2767724308 wait complete. 2024-09-16T04:45:23.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.253+0000 7f15458f1640 10 monclient: build_initial_monmap 2024-09-16T04:45:23.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.253+0000 7f15458f1640 10 monclient: monmap: 2024-09-16T04:45:23.268 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:23.268 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:45:23.268 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:45:23.268 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:45:23.268 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:23.268 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.268 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:45:23.268 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:45:23.269 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:45:23.269 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f15458f1640 1 Processor -- start 2024-09-16T04:45:23.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f15458f1640 1 -- start start 2024-09-16T04:45:23.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f15458f1640 10 monclient: init 2024-09-16T04:45:23.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f15458f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:23.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f15458f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:45:23.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f15458f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f154009c1c0 0x7f15401cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f15458f1640 10 monclient(hunting): picked mon.noname-a con 0x7f154009c1c0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:45:23.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f15458f1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15400a3ba0 0x7f15401cc500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f15458f1640 10 monclient(hunting): picked mon.noname-b con 0x7f15400a3ba0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:45:23.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f15458f1640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15400a44c0 0x7f1540121fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f15458f1640 10 monclient(hunting): picked mon.noname-c con 0x7f15400a44c0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:23.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f15458f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f15458f1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f15400a5540 con 0x7f154009c1c0 2024-09-16T04:45:23.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f15458f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f15458f1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f15400a53c0 con 0x7f15400a3ba0 2024-09-16T04:45:23.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f15458f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f15458f1640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f1540135e90 con 0x7f15400a44c0 2024-09-16T04:45:23.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f15458f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:23.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f15458f1640 10 monclient(hunting): authenticate will time out at 1121.669735s 2024-09-16T04:45:23.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f153effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f154009c1c0 0x7f15401cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f153f7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15400a44c0 0x7f1540121fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f153effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f154009c1c0 0x7f15401cbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32850/0 (socket says 172.21.15.46:32850) 2024-09-16T04:45:23.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f153effd640 1 -- 172.21.15.46:0/2633994822 learned_addr learned my addr 172.21.15.46:0/2633994822 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:23.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f153effd640 10 monclient(hunting): get_auth_request con 0x7f154009c1c0 auth_method 0 2024-09-16T04:45:23.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f153f7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15400a44c0 0x7f1540121fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:34826/0 (socket says 172.21.15.46:34826) 2024-09-16T04:45:23.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f153effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f153effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f153effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f153f7fe640 10 monclient(hunting): get_auth_request con 0x7f15400a44c0 auth_method 0 2024-09-16T04:45:23.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f153f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f153f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f153f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f153f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:23.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f153f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:23.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f153f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:23.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f153effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:23.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f153effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:23.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.254+0000 7f153effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:23.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f153e7fc640 1 --2- 172.21.15.46:0/2633994822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15400a3ba0 0x7f15401cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f153e7fc640 10 monclient(hunting): get_auth_request con 0x7f15400a3ba0 auth_method 0 2024-09-16T04:45:23.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f153e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f153e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f153e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f153f7fe640 10 monclient(hunting): handle_auth_done global_id 84128 payload 824 2024-09-16T04:45:23.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f153f7fe640 1 -- 172.21.15.46:0/2633994822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15400a3ba0 msgr2=0x7f15401cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f153f7fe640 1 --2- 172.21.15.46:0/2633994822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15400a3ba0 0x7f15401cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f153f7fe640 1 -- 172.21.15.46:0/2633994822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f154009c1c0 msgr2=0x7f15401cbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f153f7fe640 1 --2- 172.21.15.46:0/2633994822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f154009c1c0 0x7f15401cbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f153f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:45:23.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f153f7fe640 1 monclient: found mon.noname-c 2024-09-16T04:45:23.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f153f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:23.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f153f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:45:23.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f153f7fe640 1 -- 172.21.15.46:0/2633994822 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15401247b0 con 0x7f15400a44c0 2024-09-16T04:45:23.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f153effd640 1 --2- 172.21.15.46:0/2633994822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f154009c1c0 0x7f15401cbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:45:23.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f153f7fe640 1 --2- 172.21.15.46:0/2633994822 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15400a44c0 0x7f1540121fb0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1538030000 tx=0x7f1538004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:23.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f1527fff640 1 -- 172.21.15.46:0/2633994822 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1538036050 con 0x7f15400a44c0 2024-09-16T04:45:23.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f1527fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f1527fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:45:23.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f1527fff640 10 monclient: dump: 2024-09-16T04:45:23.274 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.274 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.275 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.275 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.275 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.275 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.275 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.275 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.275 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.275 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f1527fff640 10 monclient: _finish_auth 0 2024-09-16T04:45:23.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f1527fff640 10 monclient: _check_auth_tickets 2024-09-16T04:45:23.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f1527fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:23.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f1527fff640 1 -- 172.21.15.46:0/2633994822 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1538042070 con 0x7f15400a44c0 2024-09-16T04:45:23.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f1527fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:23.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f1527fff640 1 -- 172.21.15.46:0/2633994822 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1538009510 con 0x7f15400a44c0 2024-09-16T04:45:23.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f15458f1640 5 monclient: authenticate success, global_id 84128 2024-09-16T04:45:23.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f1527fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f1527fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:45:23.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f1527fff640 10 monclient: dump: 2024-09-16T04:45:23.276 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.276 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.276 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.276 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.277 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.277 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.277 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.277 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.277 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.277 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f15458f1640 10 monclient: _renew_subs 2024-09-16T04:45:23.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f153d7fa640 4 set_mon_vals no callback set 2024-09-16T04:45:23.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f15458f1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:45:23.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f15458f1640 1 -- 172.21.15.46:0/2633994822 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1540125c60 con 0x7f15400a44c0 2024-09-16T04:45:23.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f15458f1640 10 monclient: _renew_subs 2024-09-16T04:45:23.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f15458f1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:45:23.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.255+0000 7f15458f1640 1 -- 172.21.15.46:0/2633994822 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f15400a00f0 con 0x7f15400a44c0 2024-09-16T04:45:23.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.256+0000 7f1527fff640 1 -- 172.21.15.46:0/2633994822 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f1538007bf0 con 0x7f15400a44c0 2024-09-16T04:45:23.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.256+0000 7f15458f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:23.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.256+0000 7f15458f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:23.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.256+0000 7f15458f1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:45:23.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.256+0000 7f1527fff640 1 --2- 172.21.15.46:0/2633994822 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f152c0383b0 0x7f152c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.256+0000 7f15458f1640 1 -- 172.21.15.46:0/2633994822 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f154009bc80 con 0x7f15400a44c0 2024-09-16T04:45:23.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.257+0000 7f1527fff640 1 -- 172.21.15.46:0/2633994822 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(254..254 src has 1..254) ==== 4346+0+0 (secure 0 0 0) 0x7f153806c440 con 0x7f15400a44c0 2024-09-16T04:45:23.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.257+0000 7f153effd640 1 --2- 172.21.15.46:0/2633994822 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f152c0383b0 0x7f152c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.257+0000 7f153effd640 10 monclient: get_auth_request con 0x7f152c0383b0 auth_method 0 2024-09-16T04:45:23.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.257+0000 7f153effd640 1 --2- 172.21.15.46:0/2633994822 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f152c0383b0 0x7f152c03a870 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1530009560 tx=0x7f1530009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:23.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.262+0000 7f1527fff640 1 -- 172.21.15.46:0/2633994822 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f15380463c0 con 0x7f15400a44c0 2024-09-16T04:45:23.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.262+0000 7f1527fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:23.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.262+0000 7f1527fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:45:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.507+0000 7f15458f1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:45:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.507+0000 7f15458f1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:45:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.507+0000 7f15458f1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:45:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.507+0000 7f15458f1640 1 -- 172.21.15.46:0/2633994822 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1540126000 con 0x7f15400a44c0 2024-09-16T04:45:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.507+0000 7f1527fff640 1 -- 172.21.15.46:0/2633994822 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f15380465a0 con 0x7f15400a44c0 2024-09-16T04:45:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.507+0000 7f1527fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:45:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.507+0000 7f1527fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:45:23.509 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":78,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:45:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.509+0000 7f15458f1640 1 -- 172.21.15.46:0/2633994822 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f152c0383b0 msgr2=0x7f152c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.509+0000 7f15458f1640 1 --2- 172.21.15.46:0/2633994822 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f152c0383b0 0x7f152c03a870 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1530009560 tx=0x7f1530009190 comp rx=0 tx=0).stop 2024-09-16T04:45:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.509+0000 7f15458f1640 10 monclient: shutdown 2024-09-16T04:45:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.509+0000 7f15458f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.509+0000 7f15458f1640 1 -- 172.21.15.46:0/2633994822 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15400a44c0 msgr2=0x7f1540121fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.509+0000 7f15458f1640 1 --2- 172.21.15.46:0/2633994822 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15400a44c0 0x7f1540121fb0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1538030000 tx=0x7f1538004a40 comp rx=0 tx=0).stop 2024-09-16T04:45:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.509+0000 7f15458f1640 1 -- 172.21.15.46:0/2633994822 shutdown_connections 2024-09-16T04:45:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.509+0000 7f15458f1640 1 --2- 172.21.15.46:0/2633994822 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15400a44c0 0x7f1540121fb0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.509+0000 7f15458f1640 1 --2- 172.21.15.46:0/2633994822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15400a3ba0 0x7f15401cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.509+0000 7f15458f1640 1 --2- 172.21.15.46:0/2633994822 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f152c0383b0 0x7f152c03a870 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.509+0000 7f15458f1640 1 --2- 172.21.15.46:0/2633994822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f154009c1c0 0x7f15401cbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.509+0000 7f15458f1640 1 -- 172.21.15.46:0/2633994822 >> 172.21.15.46:0/2633994822 conn(0x7f15400a60c0 msgr2=0x7f154009f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.510+0000 7f15458f1640 1 -- 172.21.15.46:0/2633994822 shutdown_connections 2024-09-16T04:45:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.510+0000 7f15458f1640 1 -- 172.21.15.46:0/2633994822 wait complete. 2024-09-16T04:45:23.521 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:45:23.522 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:45:23.522 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:45:23.522 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:45:23.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.627+0000 7f5473c4c640 10 monclient: get_monmap_and_config 2024-09-16T04:45:23.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.627+0000 7f5473c4c640 10 monclient: build_initial_monmap 2024-09-16T04:45:23.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.627+0000 7f5473c4c640 10 monclient: monmap: 2024-09-16T04:45:23.628 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:23.628 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.628 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:45:23.627867+0000 2024-09-16T04:45:23.628 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:45:23.627867+0000 2024-09-16T04:45:23.629 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:23.629 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.629 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:45:23.629 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:45:23.629 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:45:23.629 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:45:23.629 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:45:23.629 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.627+0000 7f5473c4c640 1 Processor -- start 2024-09-16T04:45:23.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f5473c4c640 1 -- start start 2024-09-16T04:45:23.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f5473c4c640 10 monclient: init 2024-09-16T04:45:23.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f5473c4c640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:23.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f5473c4c640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-16T04:45:23.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f5473c4c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f546c073310 0x7f546c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f5473c4c640 10 monclient(hunting): picked mon.noname-c con 0x7f546c073310 addr v2:172.21.15.114:3300/0 2024-09-16T04:45:23.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f5473c4c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f546c09bc10 0x7f546c073c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f5473c4c640 10 monclient(hunting): picked mon.noname-b con 0x7f546c09bc10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:23.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f5473c4c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f546c136a70 addr v1:172.21.15.46:6789/0 2024-09-16T04:45:23.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f5473c4c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f546c0a0e90 con 0x7f546c136a70 2024-09-16T04:45:23.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f5473c4c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f5473c4c640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f546c0a1170 con 0x7f546c073310 2024-09-16T04:45:23.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f5473c4c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f5473c4c640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f546c074170 con 0x7f546c09bc10 2024-09-16T04:45:23.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f5473c4c640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:23.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f5473c4c640 10 monclient(hunting): authenticate will time out at 1122.043768s 2024-09-16T04:45:23.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f54721e9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f546c136a70 0x7f546c136ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45472/0 (socket says 172.21.15.46:45472) 2024-09-16T04:45:23.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f54711e7640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f546c09bc10 0x7f546c073c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f54721e9640 1 -- 172.21.15.46:0/3161190268 learned_addr learned my addr 172.21.15.46:0/3161190268 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:45:23.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f54711e7640 10 monclient(hunting): get_auth_request con 0x7f546c09bc10 auth_method 0 2024-09-16T04:45:23.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f54711e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f54711e7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.628+0000 7f54711e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54711e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:23.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54711e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:23.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54711e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:23.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54719e8640 1 --2- 172.21.15.46:0/3161190268 >> v2:172.21.15.114:3300/0 conn(0x7f546c073310 0x7f546c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54719e8640 10 monclient(hunting): get_auth_request con 0x7f546c073310 auth_method 0 2024-09-16T04:45:23.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54719e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54719e8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54719e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54709e6640 1 -- 172.21.15.46:0/3161190268 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2089930542 0 0) 0x7f546c0a0e90 con 0x7f546c136a70 2024-09-16T04:45:23.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54709e6640 10 monclient(hunting): _negotiate 2024-09-16T04:45:23.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54709e6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54709e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54709e6640 10 monclient(hunting): my global_id is 74442 2024-09-16T04:45:23.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54709e6640 1 -- 172.21.15.46:0/3161190268 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5458003520 con 0x7f546c136a70 2024-09-16T04:45:23.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54711e7640 10 monclient(hunting): handle_auth_done global_id 84131 payload 274 2024-09-16T04:45:23.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54711e7640 1 -- 172.21.15.46:0/3161190268 >> v2:172.21.15.114:3300/0 conn(0x7f546c073310 msgr2=0x7f546c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:45:23.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54711e7640 1 --2- 172.21.15.46:0/3161190268 >> v2:172.21.15.114:3300/0 conn(0x7f546c073310 0x7f546c0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54711e7640 1 -- 172.21.15.46:0/3161190268 >> v1:172.21.15.46:6789/0 conn(0x7f546c136a70 legacy=0x7f546c136ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54711e7640 10 monclient: _finish_hunting 0 2024-09-16T04:45:23.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54711e7640 1 monclient: found mon.noname-b 2024-09-16T04:45:23.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54711e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:23.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54711e7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:45:23.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54711e7640 1 -- 172.21.15.46:0/3161190268 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f546c0a0110 con 0x7f546c09bc10 2024-09-16T04:45:23.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.629+0000 7f54719e8640 1 --2- 172.21.15.46:0/3161190268 >> v2:172.21.15.114:3300/0 conn(0x7f546c073310 0x7f546c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:45:23.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f54711e7640 1 --2- 172.21.15.46:0/3161190268 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f546c09bc10 0x7f546c073c30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f545c0097a0 tx=0x7f545c02a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3fdfba47f32b0de server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:23.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f54709e6640 1 -- 172.21.15.46:0/3161190268 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545c037070 con 0x7f546c09bc10 2024-09-16T04:45:23.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f54709e6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f54709e6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:45:23.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f54709e6640 10 monclient: dump: 2024-09-16T04:45:23.641 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.642 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.642 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.642 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.642 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.642 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.642 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.642 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.642 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.642 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f54709e6640 10 monclient: _finish_auth 0 2024-09-16T04:45:23.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f54709e6640 10 monclient: _check_auth_tickets 2024-09-16T04:45:23.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f54709e6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:23.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f5473c4c640 5 monclient: authenticate success, global_id 84131 2024-09-16T04:45:23.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f5473c4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:45:23.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f54709e6640 1 -- 172.21.15.46:0/3161190268 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f545c02adb0 con 0x7f546c09bc10 2024-09-16T04:45:23.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f54709e6640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:23.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f5473c4c640 10 monclient: get_monmap_and_config success 2024-09-16T04:45:23.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f5473c4c640 4 set_mon_vals no callback set 2024-09-16T04:45:23.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f54709e6640 1 -- 172.21.15.46:0/3161190268 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545c032660 con 0x7f546c09bc10 2024-09-16T04:45:23.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f54709e6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f54709e6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:45:23.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f54709e6640 10 monclient: dump: 2024-09-16T04:45:23.643 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.643 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.643 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.643 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.643 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.643 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.644 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.644 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.644 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.644 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f5473c4c640 10 monclient: shutdown 2024-09-16T04:45:23.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f5473c4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:23.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f5473c4c640 1 -- 172.21.15.46:0/3161190268 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f546c09bc10 msgr2=0x7f546c073c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f5473c4c640 1 --2- 172.21.15.46:0/3161190268 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f546c09bc10 0x7f546c073c30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f545c0097a0 tx=0x7f545c02a060 comp rx=0 tx=0).stop 2024-09-16T04:45:23.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f5473c4c640 1 -- 172.21.15.46:0/3161190268 shutdown_connections 2024-09-16T04:45:23.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f5473c4c640 1 --2- 172.21.15.46:0/3161190268 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f546c09bc10 0x7f546c073c30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f5473c4c640 1 --2- 172.21.15.46:0/3161190268 >> v2:172.21.15.114:3300/0 conn(0x7f546c073310 0x7f546c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f5473c4c640 1 -- 172.21.15.46:0/3161190268 >> 172.21.15.46:0/3161190268 conn(0x7f546c0a1d70 msgr2=0x7f546c0a2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:23.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f5473c4c640 1 -- 172.21.15.46:0/3161190268 shutdown_connections 2024-09-16T04:45:23.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.630+0000 7f5473c4c640 1 -- 172.21.15.46:0/3161190268 wait complete. 2024-09-16T04:45:23.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.631+0000 7f5473c4c640 10 monclient: build_initial_monmap 2024-09-16T04:45:23.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.631+0000 7f5473c4c640 10 monclient: monmap: 2024-09-16T04:45:23.645 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:23.645 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:45:23.645 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:45:23.645 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:45:23.645 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:23.645 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.645 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:45:23.645 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:45:23.645 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:45:23.645 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.631+0000 7f5473c4c640 1 Processor -- start 2024-09-16T04:45:23.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.631+0000 7f5473c4c640 1 -- start start 2024-09-16T04:45:23.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.631+0000 7f5473c4c640 10 monclient: init 2024-09-16T04:45:23.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f5473c4c640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:23.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f5473c4c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:45:23.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f5473c4c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f546c073310 0x7f546c1cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f5473c4c640 10 monclient(hunting): picked mon.noname-b con 0x7f546c073310 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:45:23.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f5473c4c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546c09bc10 0x7f546c1cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f5473c4c640 10 monclient(hunting): picked mon.noname-a con 0x7f546c09bc10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:45:23.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f5473c4c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f546c136a70 0x7f546c122430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f5473c4c640 10 monclient(hunting): picked mon.noname-c con 0x7f546c136a70 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:23.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f5473c4c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f5473c4c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f546c0a1710 con 0x7f546c09bc10 2024-09-16T04:45:23.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f5473c4c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f5473c4c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f546c0a1590 con 0x7f546c073310 2024-09-16T04:45:23.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f5473c4c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f5473c4c640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f546c0a2b00 con 0x7f546c136a70 2024-09-16T04:45:23.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f5473c4c640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:23.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f5473c4c640 10 monclient(hunting): authenticate will time out at 1122.047294s 2024-09-16T04:45:23.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54711e7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546c09bc10 0x7f546c1cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54721e9640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f546c136a70 0x7f546c122430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54711e7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546c09bc10 0x7f546c1cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32858/0 (socket says 172.21.15.46:32858) 2024-09-16T04:45:23.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54711e7640 1 -- 172.21.15.46:0/2482824472 learned_addr learned my addr 172.21.15.46:0/2482824472 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:23.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54711e7640 10 monclient(hunting): get_auth_request con 0x7f546c09bc10 auth_method 0 2024-09-16T04:45:23.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54711e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54711e7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54711e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54721e9640 10 monclient(hunting): get_auth_request con 0x7f546c136a70 auth_method 0 2024-09-16T04:45:23.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54721e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54721e9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54721e9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54711e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:23.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54711e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:23.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54711e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:23.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54721e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:23.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54721e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:23.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54719e8640 1 --2- 172.21.15.46:0/2482824472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f546c073310 0x7f546c1cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54721e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:23.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54719e8640 10 monclient(hunting): get_auth_request con 0x7f546c073310 auth_method 0 2024-09-16T04:45:23.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54719e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54719e8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54719e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54721e9640 10 monclient(hunting): handle_auth_done global_id 84134 payload 824 2024-09-16T04:45:23.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54721e9640 1 -- 172.21.15.46:0/2482824472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f546c073310 msgr2=0x7f546c1cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54721e9640 1 --2- 172.21.15.46:0/2482824472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f546c073310 0x7f546c1cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54721e9640 1 -- 172.21.15.46:0/2482824472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546c09bc10 msgr2=0x7f546c1cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54721e9640 1 --2- 172.21.15.46:0/2482824472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546c09bc10 0x7f546c1cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54721e9640 10 monclient: _finish_hunting 0 2024-09-16T04:45:23.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54721e9640 1 monclient: found mon.noname-c 2024-09-16T04:45:23.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54721e9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:23.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54721e9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:45:23.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54721e9640 1 -- 172.21.15.46:0/2482824472 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f546c125d70 con 0x7f546c136a70 2024-09-16T04:45:23.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54711e7640 1 --2- 172.21.15.46:0/2482824472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546c09bc10 0x7f546c1cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:45:23.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54721e9640 1 --2- 172.21.15.46:0/2482824472 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f546c136a70 0x7f546c122430 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f546400ca80 tx=0x7f546400cf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:23.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.632+0000 7f54627fc640 1 -- 172.21.15.46:0/2482824472 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5464017070 con 0x7f546c136a70 2024-09-16T04:45:23.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f54627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f54627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:45:23.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f54627fc640 10 monclient: dump: 2024-09-16T04:45:23.650 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.650 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.650 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.650 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.650 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.650 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.650 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.650 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.650 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.650 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f54627fc640 10 monclient: _finish_auth 0 2024-09-16T04:45:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f54627fc640 10 monclient: _check_auth_tickets 2024-09-16T04:45:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f54627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f54627fc640 1 -- 172.21.15.46:0/2482824472 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f546400ac30 con 0x7f546c136a70 2024-09-16T04:45:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f5473c4c640 5 monclient: authenticate success, global_id 84134 2024-09-16T04:45:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f5473c4c640 10 monclient: _renew_subs 2024-09-16T04:45:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f5473c4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:45:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f5473c4c640 1 -- 172.21.15.46:0/2482824472 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f546c124ba0 con 0x7f546c136a70 2024-09-16T04:45:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f54627fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f54627fc640 1 -- 172.21.15.46:0/2482824472 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5464012600 con 0x7f546c136a70 2024-09-16T04:45:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f54627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f54627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:45:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f54627fc640 10 monclient: dump: 2024-09-16T04:45:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.652 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.652 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.652 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.652 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.652 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.652 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.652 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.652 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f5473c4c640 10 monclient: _renew_subs 2024-09-16T04:45:23.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f5463fff640 4 set_mon_vals no callback set 2024-09-16T04:45:23.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f5473c4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:45:23.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f5473c4c640 1 -- 172.21.15.46:0/2482824472 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f546c09db40 con 0x7f546c136a70 2024-09-16T04:45:23.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f5473c4c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:23.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f5473c4c640 1 --2- 172.21.15.46:0/2482824472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546c123130 0x7f546c02c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f5473c4c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f5473c4c640 1 -- 172.21.15.46:0/2482824472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f546c0a4540 con 0x7f546c123130 2024-09-16T04:45:23.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f54627fc640 1 -- 172.21.15.46:0/2482824472 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f546401a070 con 0x7f546c136a70 2024-09-16T04:45:23.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f54719e8640 1 --2- 172.21.15.46:0/2482824472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546c123130 0x7f546c02c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f54719e8640 10 monclient: get_auth_request con 0x7f546c123130 auth_method 0 2024-09-16T04:45:23.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f54719e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f54719e8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.633+0000 7f54719e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54719e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:23.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54719e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:23.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54719e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:23.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54627fc640 1 --2- 172.21.15.46:0/2482824472 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f54500383b0 0x7f545003a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54627fc640 1 -- 172.21.15.46:0/2482824472 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(254..254 src has 1..254) ==== 4346+0+0 (secure 0 0 0) 0x7f546404b2d0 con 0x7f546c136a70 2024-09-16T04:45:23.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54719e8640 10 monclient(hunting): handle_auth_done global_id 74448 payload 824 2024-09-16T04:45:23.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54719e8640 1 -- 172.21.15.46:0/2482824472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f546c0736f0 con 0x7f546c123130 2024-09-16T04:45:23.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54719e8640 1 --2- 172.21.15.46:0/2482824472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546c123130 0x7f546c02c640 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5468010790 tx=0x7f5468010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:23.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54711e7640 1 --2- 172.21.15.46:0/2482824472 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f54500383b0 0x7f545003a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54711e7640 10 monclient: get_auth_request con 0x7f54500383b0 auth_method 0 2024-09-16T04:45:23.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54627fc640 1 -- 172.21.15.46:0/2482824472 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546801a020 con 0x7f546c123130 2024-09-16T04:45:23.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:23.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54627fc640 10 monclient: dump: 2024-09-16T04:45:23.654 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.654 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.654 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.654 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.654 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.655 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.655 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.655 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.655 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.655 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54711e7640 1 --2- 172.21.15.46:0/2482824472 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f54500383b0 0x7f545003a870 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f545c02a0a0 tx=0x7f545c02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:23.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54627fc640 1 -- 172.21.15.46:0/2482824472 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f546800e9d0 con 0x7f546c123130 2024-09-16T04:45:23.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54627fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:23.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54627fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:45:23.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54627fc640 1 -- 172.21.15.46:0/2482824472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546c123130 msgr2=0x7f546c02c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54627fc640 1 --2- 172.21.15.46:0/2482824472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546c123130 0x7f546c02c640 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5468010790 tx=0x7f5468010c50 comp rx=0 tx=0).stop 2024-09-16T04:45:23.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54627fc640 1 -- 172.21.15.46:0/2482824472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546c123130 msgr2=0x7f546c02c640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:45:23.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.634+0000 7f54627fc640 1 --2- 172.21.15.46:0/2482824472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546c123130 0x7f546c02c640 secure :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5468010790 tx=0x7f5468010c50 comp rx=0 tx=0).stop 2024-09-16T04:45:23.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f5473c4c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:45:23.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f5473c4c640 1 --2- 172.21.15.46:0/2482824472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546c123510 0x7f546c02cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f5473c4c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f5473c4c640 1 -- 172.21.15.46:0/2482824472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f546c11efe0 con 0x7f546c123510 2024-09-16T04:45:23.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54719e8640 1 --2- 172.21.15.46:0/2482824472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546c123510 0x7f546c02cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54719e8640 10 monclient: get_auth_request con 0x7f546c123510 auth_method 0 2024-09-16T04:45:23.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54719e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54719e8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54719e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54719e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:23.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54719e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:23.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54719e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:23.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54719e8640 10 monclient(hunting): handle_auth_done global_id 74451 payload 824 2024-09-16T04:45:23.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54719e8640 1 -- 172.21.15.46:0/2482824472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f546c124460 con 0x7f546c123510 2024-09-16T04:45:23.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54719e8640 1 --2- 172.21.15.46:0/2482824472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546c123510 0x7f546c02cb80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5468010790 tx=0x7f5468004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:23.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54627fc640 1 -- 172.21.15.46:0/2482824472 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546801a030 con 0x7f546c123510 2024-09-16T04:45:23.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:23.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54627fc640 10 monclient: dump: 2024-09-16T04:45:23.659 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.659 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.659 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.659 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.659 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.659 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.659 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.659 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.659 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.659 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54627fc640 1 -- 172.21.15.46:0/2482824472 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f5468025070 con 0x7f546c123510 2024-09-16T04:45:23.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54627fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:45:23.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54627fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:45:23.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54627fc640 1 -- 172.21.15.46:0/2482824472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546c123510 msgr2=0x7f546c02cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54627fc640 1 --2- 172.21.15.46:0/2482824472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546c123510 0x7f546c02cb80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5468010790 tx=0x7f5468004730 comp rx=0 tx=0).stop 2024-09-16T04:45:23.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54627fc640 1 -- 172.21.15.46:0/2482824472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546c123510 msgr2=0x7f546c02cb80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:45:23.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.641+0000 7f54627fc640 1 --2- 172.21.15.46:0/2482824472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546c123510 0x7f546c02cb80 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5468010790 tx=0x7f5468004730 comp rx=0 tx=0).stop 2024-09-16T04:45:23.660 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:45:23.660 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:45:23.660 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:45:23.661 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:45:23.661 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 78, 2024-09-16T04:45:23.661 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 63637, 2024-09-16T04:45:23.661 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:45:23.661 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:45:23.661 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:45:23.661 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:45:23.661 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:23.661 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 5, 2024-09-16T04:45:23.661 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:45:23.661 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:45:23.661 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:45:23.662 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:23.662 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:23.662 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:23.662 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:23.662 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:23.662 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:23.662 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:23.662 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:23.662 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:23.662 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:23.662 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:23.662 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:23.662 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:45:23.662 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:45:23.663 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:23.663 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:23.663 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:23.663 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:23.663 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:23.663 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:23.663 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:23.663 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:23.663 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:23.663 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:23.663 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:23.663 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:23.663 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.663 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:45:23.664 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:45:23.664 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:45:23.664 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:45:23.664 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:45:23.664 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:45:23.664 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:45:23.664 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:45:23.664 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:45:23.664 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:45:23.664 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:45:23.664 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:45:23.664 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:45:23.664 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:45:23.665 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:45:23.665 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:45:23.665 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:45:23.665 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:23.665 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:23.665 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:23.665 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:23.665 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:23.665 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:23.665 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:23.665 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:23.665 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:23.665 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:23.665 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:23.666 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:23.666 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:45:23.666 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.666 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:45:23.666 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.666 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:45:23.666 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:45:23.666 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:23.666 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:23.666 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.666 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:23.666 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:45:23.666 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:23.666 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.667 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.667 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:23.667 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:45:23.667 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:23.667 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:23.667 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:23.667 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.667 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:45:23.667 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:45:23.667 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:23.667 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:23.667 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:23.667 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.667 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.668 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:45:23.668 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:45:23.668 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:23.668 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:23.668 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.668 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:23.668 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:45:23.668 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:23.668 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.668 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.668 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:23.669 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:45:23.669 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:23.669 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:23.669 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:23.669 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.669 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:45:23.669 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:45:23.669 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:23.669 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:23.669 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:23.669 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.669 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.669 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:45:23.670 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:45:23.670 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:23.670 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:23.670 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.670 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:23.670 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:45:23.670 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:23.670 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.670 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.670 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:23.670 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:45:23.670 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:23.670 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:23.671 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:23.671 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.671 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:45:23.671 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:45:23.671 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:23.671 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:23.671 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:23.671 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:23.671 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:23.672 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.672 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:45:23.672 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:45:23.672 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.672 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:23.672 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:23.672 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:45:23.672 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:23.672 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:23.672 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:45:23.672 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.672 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:23.672 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:23.673 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:45:23.673 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:23.673 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:23.673 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:45:23.673 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.673 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:23.673 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:23.673 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:45:23.673 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:23.673 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:23.673 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.673 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:45:23.673 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:45:23.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.642+0000 7f5473c4c640 1 -- 172.21.15.46:0/2482824472 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f54500383b0 msgr2=0x7f545003a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.642+0000 7f5473c4c640 1 --2- 172.21.15.46:0/2482824472 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f54500383b0 0x7f545003a870 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f545c02a0a0 tx=0x7f545c02e000 comp rx=0 tx=0).stop 2024-09-16T04:45:23.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.642+0000 7f5473c4c640 10 monclient: shutdown 2024-09-16T04:45:23.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.642+0000 7f5473c4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:23.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.642+0000 7f5473c4c640 1 -- 172.21.15.46:0/2482824472 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f546c136a70 msgr2=0x7f546c122430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.642+0000 7f54721e9640 1 -- 172.21.15.46:0/2482824472 reap_dead start 2024-09-16T04:45:23.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.642+0000 7f5473c4c640 1 --2- 172.21.15.46:0/2482824472 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f546c136a70 0x7f546c122430 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f546400ca80 tx=0x7f546400cf40 comp rx=0 tx=0).stop 2024-09-16T04:45:23.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.642+0000 7f5473c4c640 1 -- 172.21.15.46:0/2482824472 shutdown_connections 2024-09-16T04:45:23.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.642+0000 7f5473c4c640 1 --2- 172.21.15.46:0/2482824472 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f546c136a70 0x7f546c122430 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.642+0000 7f5473c4c640 1 -- 172.21.15.46:0/2482824472 >> 172.21.15.46:0/2482824472 conn(0x7f546c0a1d70 msgr2=0x7f546c09e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:23.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.642+0000 7f5473c4c640 1 -- 172.21.15.46:0/2482824472 shutdown_connections 2024-09-16T04:45:23.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.642+0000 7f5473c4c640 1 -- 172.21.15.46:0/2482824472 wait complete. 2024-09-16T04:45:23.676 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:45:23.676 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:45:23.676 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:45:23.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.777+0000 7f6df9a53640 10 monclient: get_monmap_and_config 2024-09-16T04:45:23.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.777+0000 7f6df9a53640 10 monclient: build_initial_monmap 2024-09-16T04:45:23.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.777+0000 7f6df9a53640 10 monclient: monmap: 2024-09-16T04:45:23.778 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:23.778 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:45:23.777920+0000 2024-09-16T04:45:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:45:23.777920+0000 2024-09-16T04:45:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:45:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:45:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:45:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:45:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:45:23.779 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df9a53640 1 Processor -- start 2024-09-16T04:45:23.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df9a53640 1 -- start start 2024-09-16T04:45:23.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df9a53640 10 monclient: init 2024-09-16T04:45:23.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df9a53640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:23.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df9a53640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-16T04:45:23.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df9a53640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6df4136e50 addr v1:172.21.15.114:6789/0 2024-09-16T04:45:23.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df9a53640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df40a3500 0x7f6df413bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df9a53640 10 monclient(hunting): picked mon.noname-b con 0x7f6df40a3500 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:23.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df9a53640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6df413c1f0 addr v1:172.21.15.46:6789/0 2024-09-16T04:45:23.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df9a53640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6df40a4e90 con 0x7f6df413c1f0 2024-09-16T04:45:23.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df9a53640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6df40a4c50 con 0x7f6df4136e50 2024-09-16T04:45:23.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df9a53640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df9a53640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f6df40a5410 con 0x7f6df40a3500 2024-09-16T04:45:23.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df9a53640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:23.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df9a53640 10 monclient(hunting): authenticate will time out at 1122.193688s 2024-09-16T04:45:23.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df27fc640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df40a3500 0x7f6df413bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df37fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6df413c1f0 0x7f6df413e5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45478/0 (socket says 172.21.15.46:45478) 2024-09-16T04:45:23.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df37fe640 1 -- 172.21.15.46:0/529549345 learned_addr learned my addr 172.21.15.46:0/529549345 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:45:23.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df27fc640 10 monclient(hunting): get_auth_request con 0x7f6df40a3500 auth_method 0 2024-09-16T04:45:23.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:23.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:23.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.778+0000 7f6df27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:23.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.779+0000 7f6df1ffb640 1 -- 172.21.15.46:0/529549345 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3639680290 0 0) 0x7f6df40a4e90 con 0x7f6df413c1f0 2024-09-16T04:45:23.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.779+0000 7f6df27fc640 10 monclient(hunting): handle_auth_done global_id 84137 payload 274 2024-09-16T04:45:23.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.779+0000 7f6df27fc640 1 -- 172.21.15.46:0/529549345 >> v1:172.21.15.114:6789/0 conn(0x7f6df4136e50 legacy=0x7f6df4137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.779+0000 7f6df27fc640 1 -- 172.21.15.46:0/529549345 >> v1:172.21.15.46:6789/0 conn(0x7f6df413c1f0 legacy=0x7f6df413e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.779+0000 7f6df27fc640 10 monclient: _finish_hunting 0 2024-09-16T04:45:23.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.779+0000 7f6df27fc640 1 monclient: found mon.noname-b 2024-09-16T04:45:23.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.779+0000 7f6df27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:23.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.779+0000 7f6df27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:45:23.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.779+0000 7f6df27fc640 1 -- 172.21.15.46:0/529549345 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df40a0dc0 con 0x7f6df40a3500 2024-09-16T04:45:23.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.779+0000 7f6df1ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:45:23.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.779+0000 7f6df27fc640 1 --2- 172.21.15.46:0/529549345 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df40a3500 0x7f6df413bcb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6de80097d0 tx=0x7f6de802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e081b5634574617 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:23.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df1ffb640 1 -- 172.21.15.46:0/529549345 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de8037070 con 0x7f6df40a3500 2024-09-16T04:45:23.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:45:23.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df1ffb640 10 monclient: dump: 2024-09-16T04:45:23.790 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.790 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.790 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.790 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.790 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.790 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.790 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.791 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.791 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.791 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df1ffb640 10 monclient: _finish_auth 0 2024-09-16T04:45:23.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df1ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:45:23.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:23.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df1ffb640 1 -- 172.21.15.46:0/529549345 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6de802ae50 con 0x7f6df40a3500 2024-09-16T04:45:23.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df9a53640 5 monclient: authenticate success, global_id 84137 2024-09-16T04:45:23.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:23.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df1ffb640 1 -- 172.21.15.46:0/529549345 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de8032660 con 0x7f6df40a3500 2024-09-16T04:45:23.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:45:23.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df1ffb640 10 monclient: dump: 2024-09-16T04:45:23.791 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.792 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.792 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.792 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.792 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.792 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.792 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.792 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.792 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.792 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df9a53640 10 monclient: get_monmap_and_config success 2024-09-16T04:45:23.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df9a53640 4 set_mon_vals no callback set 2024-09-16T04:45:23.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df9a53640 10 monclient: shutdown 2024-09-16T04:45:23.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df9a53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:23.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df9a53640 1 -- 172.21.15.46:0/529549345 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df40a3500 msgr2=0x7f6df413bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df9a53640 1 --2- 172.21.15.46:0/529549345 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df40a3500 0x7f6df413bcb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6de80097d0 tx=0x7f6de802a100 comp rx=0 tx=0).stop 2024-09-16T04:45:23.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df9a53640 1 -- 172.21.15.46:0/529549345 shutdown_connections 2024-09-16T04:45:23.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df9a53640 1 --2- 172.21.15.46:0/529549345 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df40a3500 0x7f6df413bcb0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df9a53640 1 -- 172.21.15.46:0/529549345 >> 172.21.15.46:0/529549345 conn(0x7f6df40a5f20 msgr2=0x7f6df40a6330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:23.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df9a53640 1 -- 172.21.15.46:0/529549345 shutdown_connections 2024-09-16T04:45:23.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.780+0000 7f6df9a53640 1 -- 172.21.15.46:0/529549345 wait complete. 2024-09-16T04:45:23.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.781+0000 7f6df9a53640 10 monclient: build_initial_monmap 2024-09-16T04:45:23.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.781+0000 7f6df9a53640 10 monclient: monmap: 2024-09-16T04:45:23.793 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:23.793 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:45:23.793 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:45:23.793 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:45:23.793 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:23.794 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.794 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:45:23.794 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:45:23.794 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:45:23.794 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.781+0000 7f6df9a53640 1 Processor -- start 2024-09-16T04:45:23.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.781+0000 7f6df9a53640 1 -- start start 2024-09-16T04:45:23.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.781+0000 7f6df9a53640 10 monclient: init 2024-09-16T04:45:23.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df9a53640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:23.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df9a53640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:45:23.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df9a53640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6df40a3500 0x7f6df41cbc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df9a53640 10 monclient(hunting): picked mon.noname-a con 0x7f6df40a3500 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:45:23.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df9a53640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df4136e50 0x7f6df41cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df9a53640 10 monclient(hunting): picked mon.noname-c con 0x7f6df4136e50 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:23.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df9a53640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df413c1f0 0x7f6df409b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df9a53640 10 monclient(hunting): picked mon.noname-b con 0x7f6df413c1f0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:45:23.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df9a53640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df9a53640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6df40a1b60 con 0x7f6df40a3500 2024-09-16T04:45:23.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df9a53640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df9a53640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6df40a09f0 con 0x7f6df413c1f0 2024-09-16T04:45:23.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df9a53640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df9a53640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f6df40a1ce0 con 0x7f6df4136e50 2024-09-16T04:45:23.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df9a53640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:23.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df9a53640 10 monclient(hunting): authenticate will time out at 1122.197136s 2024-09-16T04:45:23.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6df40a3500 0x7f6df41cbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df27fc640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df4136e50 0x7f6df41cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df27fc640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df4136e50 0x7f6df41cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:34856/0 (socket says 172.21.15.46:34856) 2024-09-16T04:45:23.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6df40a3500 0x7f6df41cbc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32886/0 (socket says 172.21.15.46:32886) 2024-09-16T04:45:23.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df27fc640 1 -- 172.21.15.46:0/3535715671 learned_addr learned my addr 172.21.15.46:0/3535715671 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:23.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df27fc640 10 monclient(hunting): get_auth_request con 0x7f6df4136e50 auth_method 0 2024-09-16T04:45:23.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df2ffd640 10 monclient(hunting): get_auth_request con 0x7f6df40a3500 auth_method 0 2024-09-16T04:45:23.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:23.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:23.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:23.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:23.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:23.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:23.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df37fe640 1 --2- 172.21.15.46:0/3535715671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df413c1f0 0x7f6df409b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df2ffd640 10 monclient(hunting): handle_auth_done global_id 74457 payload 824 2024-09-16T04:45:23.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df2ffd640 1 -- 172.21.15.46:0/3535715671 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df4136e50 msgr2=0x7f6df41cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df2ffd640 1 --2- 172.21.15.46:0/3535715671 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df4136e50 0x7f6df41cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df2ffd640 1 -- 172.21.15.46:0/3535715671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df413c1f0 msgr2=0x7f6df409b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df2ffd640 1 --2- 172.21.15.46:0/3535715671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df413c1f0 0x7f6df409b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df2ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:45:23.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df2ffd640 1 monclient: found mon.noname-a 2024-09-16T04:45:23.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:23.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:23.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df2ffd640 1 -- 172.21.15.46:0/3535715671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df409f1d0 con 0x7f6df40a3500 2024-09-16T04:45:23.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df27fc640 1 --2- 172.21.15.46:0/3535715671 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df4136e50 0x7f6df41cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:45:23.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df2ffd640 1 --2- 172.21.15.46:0/3535715671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6df40a3500 0x7f6df41cbc80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6de400c340 tx=0x7f6de400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:23.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6ddbfff640 1 -- 172.21.15.46:0/3535715671 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de40052d0 con 0x7f6df40a3500 2024-09-16T04:45:23.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6ddbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6ddbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:23.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6ddbfff640 10 monclient: dump: 2024-09-16T04:45:23.798 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.798 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.798 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.798 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.798 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.798 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.799 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.799 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.799 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.799 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6ddbfff640 10 monclient: _finish_auth 0 2024-09-16T04:45:23.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6ddbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:45:23.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6ddbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:23.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df9a53640 5 monclient: authenticate success, global_id 74457 2024-09-16T04:45:23.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df9a53640 10 monclient: _renew_subs 2024-09-16T04:45:23.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6ddbfff640 1 -- 172.21.15.46:0/3535715671 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6de4005490 con 0x7f6df40a3500 2024-09-16T04:45:23.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df9a53640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:23.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df9a53640 1 -- 172.21.15.46:0/3535715671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6df409e000 con 0x7f6df40a3500 2024-09-16T04:45:23.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6ddbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:23.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6ddbfff640 1 -- 172.21.15.46:0/3535715671 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de4007260 con 0x7f6df40a3500 2024-09-16T04:45:23.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6ddbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6ddbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:23.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6ddbfff640 10 monclient: dump: 2024-09-16T04:45:23.800 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.800 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.800 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.800 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.800 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.800 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.800 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.800 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.800 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.800 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.782+0000 7f6df17fa640 4 set_mon_vals no callback set 2024-09-16T04:45:23.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.783+0000 7f6df9a53640 10 monclient: _renew_subs 2024-09-16T04:45:23.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.783+0000 7f6df9a53640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:23.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.783+0000 7f6df9a53640 1 -- 172.21.15.46:0/3535715671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6df40a7280 con 0x7f6df40a3500 2024-09-16T04:45:23.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.783+0000 7f6df9a53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:23.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.783+0000 7f6df9a53640 1 --2- 172.21.15.46:0/3535715671 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df41cc990 0x7f6df402c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.783+0000 7f6df9a53640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.783+0000 7f6df9a53640 1 -- 172.21.15.46:0/3535715671 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f6df413bbd0 con 0x7f6df41cc990 2024-09-16T04:45:23.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.783+0000 7f6ddbfff640 1 -- 172.21.15.46:0/3535715671 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f6de4007400 con 0x7f6df40a3500 2024-09-16T04:45:23.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.783+0000 7f6df27fc640 1 --2- 172.21.15.46:0/3535715671 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df41cc990 0x7f6df402c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.783+0000 7f6df27fc640 10 monclient: get_auth_request con 0x7f6df41cc990 auth_method 0 2024-09-16T04:45:23.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.783+0000 7f6df27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.783+0000 7f6df27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.783+0000 7f6df27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6df27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6df27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6df27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6ddbfff640 1 --2- 172.21.15.46:0/3535715671 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6de00383b0 0x7f6de003a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6ddbfff640 1 -- 172.21.15.46:0/3535715671 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 4346+0+0 (secure 0 0 0) 0x7f6de404aa50 con 0x7f6df40a3500 2024-09-16T04:45:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6df27fc640 10 monclient(hunting): handle_auth_done global_id 84143 payload 824 2024-09-16T04:45:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6df27fc640 1 -- 172.21.15.46:0/3535715671 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6df4137230 con 0x7f6df41cc990 2024-09-16T04:45:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6df27fc640 1 --2- 172.21.15.46:0/3535715671 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df41cc990 0x7f6df402c640 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6de8009720 tx=0x7f6de80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6df37fe640 1 --2- 172.21.15.46:0/3535715671 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6de00383b0 0x7f6de003a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6df37fe640 10 monclient: get_auth_request con 0x7f6de00383b0 auth_method 0 2024-09-16T04:45:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6ddbfff640 1 -- 172.21.15.46:0/3535715671 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de8037040 con 0x7f6df41cc990 2024-09-16T04:45:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6ddbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6ddbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:45:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6ddbfff640 10 monclient: dump: 2024-09-16T04:45:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.803 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.803 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.803 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.803 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.803 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.803 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.803 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.803 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.803 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6ddbfff640 1 -- 172.21.15.46:0/3535715671 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6de8030ad0 con 0x7f6df41cc990 2024-09-16T04:45:23.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6ddbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:23.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6ddbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:45:23.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6ddbfff640 1 -- 172.21.15.46:0/3535715671 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df41cc990 msgr2=0x7f6df402c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6ddbfff640 1 --2- 172.21.15.46:0/3535715671 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df41cc990 0x7f6df402c640 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6de8009720 tx=0x7f6de80046c0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6ddbfff640 1 -- 172.21.15.46:0/3535715671 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df41cc990 msgr2=0x7f6df402c640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:45:23.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6ddbfff640 1 --2- 172.21.15.46:0/3535715671 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df41cc990 0x7f6df402c640 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6de8009720 tx=0x7f6de80046c0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.784+0000 7f6df37fe640 1 --2- 172.21.15.46:0/3535715671 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6de00383b0 0x7f6de003a870 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6dec0012e0 tx=0x7f6dec009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6df9a53640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:45:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6df9a53640 1 --2- 172.21.15.46:0/3535715671 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df402cb80 0x7f6df402cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6df9a53640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6df9a53640 1 -- 172.21.15.46:0/3535715671 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f6df4139190 con 0x7f6df402cb80 2024-09-16T04:45:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6df27fc640 1 --2- 172.21.15.46:0/3535715671 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df402cb80 0x7f6df402cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6df27fc640 10 monclient: get_auth_request con 0x7f6df402cb80 auth_method 0 2024-09-16T04:45:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6df27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6df27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6df27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6df27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6df27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6df27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6df27fc640 10 monclient(hunting): handle_auth_done global_id 84146 payload 824 2024-09-16T04:45:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6df27fc640 1 -- 172.21.15.46:0/3535715671 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6df40a85f0 con 0x7f6df402cb80 2024-09-16T04:45:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6df27fc640 1 --2- 172.21.15.46:0/3535715671 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df402cb80 0x7f6df402cf90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6df4069500 tx=0x7f6de8008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6ddbfff640 1 -- 172.21.15.46:0/3535715671 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de8037030 con 0x7f6df402cb80 2024-09-16T04:45:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6ddbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6ddbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:45:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6ddbfff640 10 monclient: dump: 2024-09-16T04:45:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.808 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6ddbfff640 1 -- 172.21.15.46:0/3535715671 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f6de80039a0 con 0x7f6df402cb80 2024-09-16T04:45:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6ddbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:45:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6ddbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:45:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6ddbfff640 1 -- 172.21.15.46:0/3535715671 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df402cb80 msgr2=0x7f6df402cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6ddbfff640 1 --2- 172.21.15.46:0/3535715671 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df402cb80 0x7f6df402cf90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6df4069500 tx=0x7f6de8008030 comp rx=0 tx=0).stop 2024-09-16T04:45:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6ddbfff640 1 -- 172.21.15.46:0/3535715671 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df402cb80 msgr2=0x7f6df402cf90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:45:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.791+0000 7f6ddbfff640 1 --2- 172.21.15.46:0/3535715671 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f6df402cb80 0x7f6df402cf90 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6df4069500 tx=0x7f6de8008030 comp rx=0 tx=0).stop 2024-09-16T04:45:23.808 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:45:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:45:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:45:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:45:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 78, 2024-09-16T04:45:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 12402, 2024-09-16T04:45:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:45:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:45:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:45:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:45:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 5, 2024-09-16T04:45:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:45:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:45:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:45:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:45:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:45:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:23.812 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.812 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:45:23.812 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:45:23.812 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:45:23.812 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:45:23.812 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:45:23.812 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:45:23.812 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:45:23.812 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:45:23.812 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:45:23.812 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:45:23.812 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:45:23.812 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:45:23.812 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:45:23.813 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:45:23.813 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:45:23.813 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:45:23.813 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:45:23.813 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:23.813 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:23.813 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:23.813 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:23.813 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:23.813 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:23.813 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:23.813 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:23.813 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:23.813 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:23.814 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:23.814 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:23.814 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:45:23.814 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.814 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:45:23.814 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.814 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:45:23.814 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:45:23.814 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:23.814 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:23.814 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.814 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:23.814 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:45:23.814 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:23.815 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.815 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.815 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:23.815 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:45:23.815 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:23.815 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:23.815 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:23.815 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.815 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:45:23.815 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:45:23.815 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:23.815 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:23.815 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:23.815 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.816 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.816 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:45:23.816 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:45:23.816 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:23.816 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:23.816 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.816 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:23.816 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:45:23.816 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:23.816 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.816 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.816 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:23.816 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:45:23.816 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:23.817 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:23.817 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:23.817 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.817 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:45:23.817 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:45:23.817 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:23.817 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:23.817 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:23.817 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.817 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.817 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:45:23.817 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:45:23.817 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:23.817 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:23.818 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.818 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:23.818 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:45:23.818 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:23.818 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.818 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.818 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:23.818 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:45:23.818 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:23.818 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:23.818 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:23.818 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.818 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:45:23.818 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:45:23.818 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:23.819 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:23.819 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:23.819 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:23.819 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:23.819 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.819 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:45:23.819 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:45:23.819 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.819 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:23.819 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:23.819 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:45:23.819 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:23.819 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:23.820 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:45:23.820 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.820 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:23.820 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:23.820 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:45:23.820 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:23.820 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:23.820 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.820 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:45:23.820 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:45:23.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.792+0000 7f6df9a53640 1 -- 172.21.15.46:0/3535715671 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6de00383b0 msgr2=0x7f6de003a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.792+0000 7f6df9a53640 1 --2- 172.21.15.46:0/3535715671 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f6de00383b0 0x7f6de003a870 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6dec0012e0 tx=0x7f6dec009040 comp rx=0 tx=0).stop 2024-09-16T04:45:23.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.792+0000 7f6df9a53640 10 monclient: shutdown 2024-09-16T04:45:23.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.792+0000 7f6df9a53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:23.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.792+0000 7f6df9a53640 1 -- 172.21.15.46:0/3535715671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6df40a3500 msgr2=0x7f6df41cbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.792+0000 7f6df9a53640 1 --2- 172.21.15.46:0/3535715671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6df40a3500 0x7f6df41cbc80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6de400c340 tx=0x7f6de400c800 comp rx=0 tx=0).stop 2024-09-16T04:45:23.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.792+0000 7f6df37fe640 1 -- 172.21.15.46:0/3535715671 reap_dead start 2024-09-16T04:45:23.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.792+0000 7f6df37fe640 1 -- 172.21.15.46:0/3535715671 reap_dead start 2024-09-16T04:45:23.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.792+0000 7f6df9a53640 1 -- 172.21.15.46:0/3535715671 shutdown_connections 2024-09-16T04:45:23.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.792+0000 7f6df9a53640 1 -- 172.21.15.46:0/3535715671 >> 172.21.15.46:0/3535715671 conn(0x7f6df40a5f20 msgr2=0x7f6df413ea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:23.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.792+0000 7f6df9a53640 1 -- 172.21.15.46:0/3535715671 shutdown_connections 2024-09-16T04:45:23.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.792+0000 7f6df9a53640 1 -- 172.21.15.46:0/3535715671 wait complete. 2024-09-16T04:45:23.823 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:45:23.823 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:45:23.823 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:45:23.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.924+0000 7f937fd11640 10 monclient: get_monmap_and_config 2024-09-16T04:45:23.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.924+0000 7f937fd11640 10 monclient: build_initial_monmap 2024-09-16T04:45:23.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.924+0000 7f937fd11640 10 monclient: monmap: 2024-09-16T04:45:23.947 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:23.947 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.947 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:45:23.924690+0000 2024-09-16T04:45:23.947 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:45:23.924690+0000 2024-09-16T04:45:23.947 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:23.947 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.947 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:45:23.947 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:45:23.947 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:45:23.947 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:45:23.947 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:45:23.948 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.924+0000 7f937fd11640 1 Processor -- start 2024-09-16T04:45:23.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937fd11640 1 -- start start 2024-09-16T04:45:23.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937fd11640 10 monclient: init 2024-09-16T04:45:23.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937fd11640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:23.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937fd11640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-16T04:45:23.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937fd11640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f93780a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:45:23.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937fd11640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f93780a3ff0 0x7f93780a4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937fd11640 10 monclient(hunting): picked mon.noname-c con 0x7f93780a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:45:23.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937fd11640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f937813be90 0x7f937813e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937fd11640 10 monclient(hunting): picked mon.noname-b con 0x7f937813be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:23.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937fd11640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f937809ab70 con 0x7f93780a3c10 2024-09-16T04:45:23.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937fd11640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937fd11640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f93780a4d60 con 0x7f93780a3ff0 2024-09-16T04:45:23.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937fd11640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937fd11640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f937809b030 con 0x7f937813be90 2024-09-16T04:45:23.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937fd11640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:23.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937fd11640 10 monclient(hunting): authenticate will time out at 1122.340506s 2024-09-16T04:45:23.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937e2ae640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f937813be90 0x7f937813e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937e2ae640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f937813be90 0x7f937813e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:34894/0 (socket says 172.21.15.46:34894) 2024-09-16T04:45:23.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937e2ae640 1 -- 172.21.15.46:0/4054655694 learned_addr learned my addr 172.21.15.46:0/4054655694 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:23.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937e2ae640 10 monclient(hunting): get_auth_request con 0x7f937813be90 auth_method 0 2024-09-16T04:45:23.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937e2ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937e2ae640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937e2ae640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937e2ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:23.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937e2ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:23.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937e2ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:23.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937d2ac640 1 --2- 172.21.15.46:0/4054655694 >> v2:172.21.15.114:3300/0 conn(0x7f93780a3ff0 0x7f93780a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937d2ac640 10 monclient(hunting): get_auth_request con 0x7f93780a3ff0 auth_method 0 2024-09-16T04:45:23.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937d2ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937d2ac640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937d2ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937e2ae640 10 monclient(hunting): handle_auth_done global_id 84149 payload 274 2024-09-16T04:45:23.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937e2ae640 1 -- 172.21.15.46:0/4054655694 >> v2:172.21.15.114:3300/0 conn(0x7f93780a3ff0 msgr2=0x7f93780a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:45:23.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937e2ae640 1 --2- 172.21.15.46:0/4054655694 >> v2:172.21.15.114:3300/0 conn(0x7f93780a3ff0 0x7f93780a4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937e2ae640 1 -- 172.21.15.46:0/4054655694 >> v1:172.21.15.114:6789/0 conn(0x7f93780a3c10 legacy=0x7f937809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937e2ae640 10 monclient: _finish_hunting 0 2024-09-16T04:45:23.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937e2ae640 1 monclient: found mon.noname-b 2024-09-16T04:45:23.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937e2ae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:23.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937e2ae640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:45:23.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.925+0000 7f937e2ae640 1 -- 172.21.15.46:0/4054655694 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f937809b310 con 0x7f937813be90 2024-09-16T04:45:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.926+0000 7f937e2ae640 1 --2- 172.21.15.46:0/4054655694 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f937813be90 0x7f937813e280 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9370009e50 tx=0x7f937002a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=186b63ec832d4290 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.926+0000 7f937caab640 1 -- 172.21.15.46:0/4054655694 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9370036070 con 0x7f937813be90 2024-09-16T04:45:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.926+0000 7f937caab640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.926+0000 7f937caab640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:45:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.926+0000 7f937caab640 10 monclient: dump: 2024-09-16T04:45:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.962 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.926+0000 7f937caab640 10 monclient: _finish_auth 0 2024-09-16T04:45:23.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.926+0000 7f937caab640 10 monclient: _check_auth_tickets 2024-09-16T04:45:23.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.926+0000 7f937caab640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:23.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.926+0000 7f937caab640 1 -- 172.21.15.46:0/4054655694 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9370031600 con 0x7f937813be90 2024-09-16T04:45:23.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.926+0000 7f937fd11640 5 monclient: authenticate success, global_id 84149 2024-09-16T04:45:23.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.926+0000 7f937fd11640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:45:23.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.926+0000 7f937caab640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:23.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.926+0000 7f937fd11640 10 monclient: get_monmap_and_config success 2024-09-16T04:45:23.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.926+0000 7f937fd11640 4 set_mon_vals no callback set 2024-09-16T04:45:23.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.926+0000 7f937caab640 1 -- 172.21.15.46:0/4054655694 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93700319d0 con 0x7f937813be90 2024-09-16T04:45:23.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.926+0000 7f937caab640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.926+0000 7f937caab640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:45:23.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.926+0000 7f937caab640 10 monclient: dump: 2024-09-16T04:45:23.963 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.963 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.963 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.963 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.963 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.963 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.963 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.963 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.963 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.964 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.927+0000 7f937fd11640 10 monclient: shutdown 2024-09-16T04:45:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.927+0000 7f937fd11640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.927+0000 7f937fd11640 1 -- 172.21.15.46:0/4054655694 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f937813be90 msgr2=0x7f937813e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.927+0000 7f937fd11640 1 --2- 172.21.15.46:0/4054655694 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f937813be90 0x7f937813e280 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9370009e50 tx=0x7f937002a5f0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.927+0000 7f937fd11640 1 -- 172.21.15.46:0/4054655694 shutdown_connections 2024-09-16T04:45:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.927+0000 7f937fd11640 1 --2- 172.21.15.46:0/4054655694 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f937813be90 0x7f937813e280 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.927+0000 7f937fd11640 1 --2- 172.21.15.46:0/4054655694 >> v2:172.21.15.114:3300/0 conn(0x7f93780a3ff0 0x7f93780a4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.927+0000 7f937fd11640 1 -- 172.21.15.46:0/4054655694 >> 172.21.15.46:0/4054655694 conn(0x7f93780a5d10 msgr2=0x7f93780a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.927+0000 7f937fd11640 1 -- 172.21.15.46:0/4054655694 shutdown_connections 2024-09-16T04:45:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.927+0000 7f937fd11640 1 -- 172.21.15.46:0/4054655694 wait complete. 2024-09-16T04:45:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.928+0000 7f937fd11640 10 monclient: build_initial_monmap 2024-09-16T04:45:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.928+0000 7f937fd11640 10 monclient: monmap: 2024-09-16T04:45:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:23.965 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:45:23.965 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:45:23.965 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:45:23.965 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:23.965 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.965 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:45:23.965 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:45:23.965 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:45:23.965 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.928+0000 7f937fd11640 1 Processor -- start 2024-09-16T04:45:23.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.928+0000 7f937fd11640 1 -- start start 2024-09-16T04:45:23.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.928+0000 7f937fd11640 10 monclient: init 2024-09-16T04:45:23.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.928+0000 7f937fd11640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.928+0000 7f937fd11640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:45:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.928+0000 7f937fd11640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93780a3c10 0x7f9378137e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.928+0000 7f937fd11640 10 monclient(hunting): picked mon.noname-a con 0x7f93780a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:45:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.928+0000 7f937fd11640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f93780a3ff0 0x7f93781383b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.928+0000 7f937fd11640 10 monclient(hunting): picked mon.noname-c con 0x7f93780a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.928+0000 7f937fd11640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f937813be90 0x7f93781388f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.928+0000 7f937fd11640 10 monclient(hunting): picked mon.noname-b con 0x7f937813be90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:45:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.928+0000 7f937fd11640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.928+0000 7f937fd11640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f937806e550 con 0x7f93780a3c10 2024-09-16T04:45:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.928+0000 7f937fd11640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.928+0000 7f937fd11640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f937806e3d0 con 0x7f937813be90 2024-09-16T04:45:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.928+0000 7f937fd11640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.928+0000 7f937fd11640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f937813e7c0 con 0x7f93780a3ff0 2024-09-16T04:45:23.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.928+0000 7f937fd11640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:23.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.928+0000 7f937fd11640 10 monclient(hunting): authenticate will time out at 1122.343985s 2024-09-16T04:45:23.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937daad640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93780a3c10 0x7f9378137e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937d2ac640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f93780a3ff0 0x7f93781383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937d2ac640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f93780a3ff0 0x7f93781383b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:34910/0 (socket says 172.21.15.46:34910) 2024-09-16T04:45:23.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937daad640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93780a3c10 0x7f9378137e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32894/0 (socket says 172.21.15.46:32894) 2024-09-16T04:45:23.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937d2ac640 1 -- 172.21.15.46:0/1395237961 learned_addr learned my addr 172.21.15.46:0/1395237961 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:23.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937d2ac640 10 monclient(hunting): get_auth_request con 0x7f93780a3ff0 auth_method 0 2024-09-16T04:45:23.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937d2ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937d2ac640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937d2ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937daad640 10 monclient(hunting): get_auth_request con 0x7f93780a3c10 auth_method 0 2024-09-16T04:45:23.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937daad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937daad640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937daad640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937d2ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:23.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937d2ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:23.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937d2ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:23.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937daad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:23.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937daad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:23.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937daad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:23.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937e2ae640 1 --2- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f937813be90 0x7f93781388f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937d2ac640 10 monclient(hunting): handle_auth_done global_id 84152 payload 824 2024-09-16T04:45:23.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937d2ac640 1 -- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f937813be90 msgr2=0x7f93781388f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937d2ac640 1 --2- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f937813be90 0x7f93781388f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937d2ac640 1 -- 172.21.15.46:0/1395237961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93780a3c10 msgr2=0x7f9378137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937d2ac640 1 --2- 172.21.15.46:0/1395237961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93780a3c10 0x7f9378137e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937d2ac640 10 monclient: _finish_hunting 0 2024-09-16T04:45:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937d2ac640 1 monclient: found mon.noname-c 2024-09-16T04:45:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937d2ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937d2ac640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:45:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937d2ac640 1 -- 172.21.15.46:0/1395237961 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93780a2f40 con 0x7f93780a3ff0 2024-09-16T04:45:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937d2ac640 1 --2- 172.21.15.46:0/1395237961 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f93780a3ff0 0x7f93781383b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f936800d800 tx=0x7f936800dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f936e7fc640 1 -- 172.21.15.46:0/1395237961 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936800b6c0 con 0x7f93780a3ff0 2024-09-16T04:45:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f936e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f936e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:45:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f936e7fc640 10 monclient: dump: 2024-09-16T04:45:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.970 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f936e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:45:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f936e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:45:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f936e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f936e7fc640 1 -- 172.21.15.46:0/1395237961 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9368019070 con 0x7f93780a3ff0 2024-09-16T04:45:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937fd11640 5 monclient: authenticate success, global_id 84152 2024-09-16T04:45:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937fd11640 10 monclient: _renew_subs 2024-09-16T04:45:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937fd11640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:45:23.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937fd11640 1 -- 172.21.15.46:0/1395237961 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93780a0400 con 0x7f93780a3ff0 2024-09-16T04:45:23.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f936e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:23.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f936e7fc640 1 -- 172.21.15.46:0/1395237961 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936800ba80 con 0x7f93780a3ff0 2024-09-16T04:45:23.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f936e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f936e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:45:23.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f936e7fc640 10 monclient: dump: 2024-09-16T04:45:23.971 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.971 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.971 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.971 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.971 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.971 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.971 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.971 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.971 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.972 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f936ffff640 4 set_mon_vals no callback set 2024-09-16T04:45:23.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937fd11640 10 monclient: _renew_subs 2024-09-16T04:45:23.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937fd11640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:45:23.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.929+0000 7f937fd11640 1 -- 172.21.15.46:0/1395237961 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9378139880 con 0x7f93780a3ff0 2024-09-16T04:45:23.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.930+0000 7f937fd11640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:23.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.930+0000 7f936e7fc640 1 -- 172.21.15.46:0/1395237961 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f936800bc20 con 0x7f93780a3ff0 2024-09-16T04:45:23.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.930+0000 7f937fd11640 1 --2- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93781339a0 0x7f9378133d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.930+0000 7f937fd11640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.930+0000 7f937fd11640 1 -- 172.21.15.46:0/1395237961 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f937813a560 con 0x7f93781339a0 2024-09-16T04:45:23.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.930+0000 7f936e7fc640 1 --2- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f935c0383b0 0x7f935c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.930+0000 7f936e7fc640 1 -- 172.21.15.46:0/1395237961 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(254..254 src has 1..254) ==== 4346+0+0 (secure 0 0 0) 0x7f936804bb20 con 0x7f93780a3ff0 2024-09-16T04:45:23.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.930+0000 7f937daad640 1 --2- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93781339a0 0x7f9378133d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.930+0000 7f937daad640 10 monclient: get_auth_request con 0x7f93781339a0 auth_method 0 2024-09-16T04:45:23.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.930+0000 7f937daad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.930+0000 7f937daad640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.930+0000 7f937daad640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.931+0000 7f937daad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:23.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.931+0000 7f937daad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:23.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.931+0000 7f937e2ae640 1 --2- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f935c0383b0 0x7f935c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.931+0000 7f937daad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:23.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.931+0000 7f937e2ae640 10 monclient: get_auth_request con 0x7f935c0383b0 auth_method 0 2024-09-16T04:45:23.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.931+0000 7f937daad640 10 monclient(hunting): handle_auth_done global_id 6532 payload 824 2024-09-16T04:45:23.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.931+0000 7f937daad640 1 -- 172.21.15.46:0/1395237961 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f93780a4400 con 0x7f93781339a0 2024-09-16T04:45:23.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.931+0000 7f937e2ae640 1 --2- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f935c0383b0 0x7f935c03a870 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f9370033000 tx=0x7f9370009000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:23.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.931+0000 7f937daad640 1 --2- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93781339a0 0x7f9378133d80 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f937400f540 tx=0x7f937400fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:23.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.931+0000 7f936e7fc640 1 -- 172.21.15.46:0/1395237961 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9374019070 con 0x7f93781339a0 2024-09-16T04:45:23.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.931+0000 7f936e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.931+0000 7f936e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:45:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.931+0000 7f936e7fc640 10 monclient: dump: 2024-09-16T04:45:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.975 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.931+0000 7f936e7fc640 1 -- 172.21.15.46:0/1395237961 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9374005320 con 0x7f93781339a0 2024-09-16T04:45:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.931+0000 7f936e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.931+0000 7f936e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:45:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.931+0000 7f936e7fc640 1 -- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93781339a0 msgr2=0x7f9378133d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.931+0000 7f936e7fc640 1 --2- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93781339a0 0x7f9378133d80 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f937400f540 tx=0x7f937400fa00 comp rx=0 tx=0).stop 2024-09-16T04:45:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.931+0000 7f936e7fc640 1 -- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93781339a0 msgr2=0x7f9378133d80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:45:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.931+0000 7f936e7fc640 1 --2- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93781339a0 0x7f9378133d80 secure :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f937400f540 tx=0x7f937400fa00 comp rx=0 tx=0).stop 2024-09-16T04:45:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.938+0000 7f937fd11640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:45:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.938+0000 7f937fd11640 1 --2- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93781342c0 0x7f9378134bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.938+0000 7f937fd11640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.938+0000 7f937fd11640 1 -- 172.21.15.46:0/1395237961 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9378069150 con 0x7f93781342c0 2024-09-16T04:45:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.938+0000 7f937daad640 1 --2- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93781342c0 0x7f9378134bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.938+0000 7f937daad640 10 monclient: get_auth_request con 0x7f93781342c0 auth_method 0 2024-09-16T04:45:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.938+0000 7f937daad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.938+0000 7f937daad640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.938+0000 7f937daad640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.938+0000 7f937daad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.938+0000 7f937daad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.938+0000 7f937daad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.938+0000 7f937daad640 10 monclient(hunting): handle_auth_done global_id 6535 payload 824 2024-09-16T04:45:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.938+0000 7f937daad640 1 -- 172.21.15.46:0/1395237961 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f937813b2a0 con 0x7f93781342c0 2024-09-16T04:45:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f937daad640 1 --2- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93781342c0 0x7f9378134bc0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f937400f540 tx=0x7f9374000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f936e7fc640 1 -- 172.21.15.46:0/1395237961 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9374019070 con 0x7f93781342c0 2024-09-16T04:45:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f936e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f936e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:45:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f936e7fc640 10 monclient: dump: 2024-09-16T04:45:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:23.978 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:23.978 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:23.978 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:23.978 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:23.978 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:23.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f936e7fc640 1 -- 172.21.15.46:0/1395237961 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f937401a040 con 0x7f93781342c0 2024-09-16T04:45:23.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f936e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:45:23.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f936e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:45:23.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f936e7fc640 1 -- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93781342c0 msgr2=0x7f9378134bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f936e7fc640 1 --2- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93781342c0 0x7f9378134bc0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f937400f540 tx=0x7f9374000ed0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f936e7fc640 1 -- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93781342c0 msgr2=0x7f9378134bc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:45:23.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f936e7fc640 1 --2- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93781342c0 0x7f9378134bc0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.979 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:45:23.979 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:45:23.979 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:45:23.979 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:45:23.979 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 78, 2024-09-16T04:45:23.979 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 409841, 2024-09-16T04:45:23.979 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:45:23.979 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:45:23.979 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:45:23.979 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:45:23.979 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:23.979 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 5, 2024-09-16T04:45:23.980 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:45:23.980 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:45:23.980 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:45:23.980 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:23.980 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:23.980 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:23.980 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:23.980 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:23.980 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:23.980 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:23.980 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:23.980 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:23.980 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:23.981 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:23.981 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:23.981 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:45:23.981 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:45:23.981 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:23.981 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:23.981 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:23.981 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:23.981 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:23.981 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:23.981 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:23.981 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:23.981 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:23.981 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:23.982 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:23.982 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:23.982 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.982 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:45:23.982 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:45:23.982 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:45:23.982 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:45:23.982 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:45:23.982 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:45:23.982 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:45:23.982 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:45:23.982 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:45:23.982 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:45:23.982 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:45:23.983 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:45:23.983 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:45:23.983 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:45:23.983 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:45:23.983 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:45:23.983 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:45:23.983 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:23.983 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:23.983 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:23.983 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:23.983 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:23.983 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:23.983 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:23.983 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:23.984 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:23.984 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:23.984 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:23.984 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:23.984 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:45:23.984 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.984 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:45:23.984 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.984 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:45:23.984 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:45:23.984 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:23.984 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:23.984 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.984 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:23.985 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:45:23.985 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:23.985 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.985 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.985 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:23.985 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:45:23.985 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:23.985 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:23.985 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:23.985 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.985 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:45:23.985 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:45:23.985 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:23.985 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:23.986 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:23.986 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.986 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.986 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:45:23.986 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:45:23.986 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:23.986 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:23.986 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.986 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:23.986 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:45:23.986 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:23.986 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.986 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.987 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:23.987 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:45:23.987 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:23.987 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:23.987 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:23.987 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.987 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:45:23.987 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:45:23.987 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:23.987 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:23.987 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:23.987 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.987 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.987 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:45:23.987 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:45:23.988 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:23.988 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:23.988 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.988 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:23.988 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:45:23.988 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:23.988 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.988 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.988 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:23.988 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:45:23.988 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:23.988 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:23.988 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:23.989 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.989 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:45:23.989 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:45:23.989 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:23.989 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:23.989 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:23.989 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:23.989 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:23.989 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.989 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:45:23.989 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:45:23.989 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.990 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:23.990 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:23.990 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:45:23.990 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:23.990 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:23.990 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:45:23.990 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.990 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:23.990 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:23.990 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5 2024-09-16T04:45:23.990 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:23.990 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:23.990 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:45:23.990 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.991 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:23.991 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:23.991 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 7 2024-09-16T04:45:23.991 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:23.991 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:23.991 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:45:23.991 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:23.991 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:23.991 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:23.991 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:45:23.991 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:23.992 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:23.992 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:23.992 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:45:23.992 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:45:23.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f937fd11640 1 -- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f935c0383b0 msgr2=0x7f935c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f937fd11640 1 --2- 172.21.15.46:0/1395237961 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f935c0383b0 0x7f935c03a870 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f9370033000 tx=0x7f9370009000 comp rx=0 tx=0).stop 2024-09-16T04:45:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f937fd11640 10 monclient: shutdown 2024-09-16T04:45:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f937fd11640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f937fd11640 1 -- 172.21.15.46:0/1395237961 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f93780a3ff0 msgr2=0x7f93781383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f937fd11640 1 --2- 172.21.15.46:0/1395237961 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f93780a3ff0 0x7f93781383b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f936800d800 tx=0x7f936800dcc0 comp rx=0 tx=0).stop 2024-09-16T04:45:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f937e2ae640 1 -- 172.21.15.46:0/1395237961 reap_dead start 2024-09-16T04:45:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f937e2ae640 1 -- 172.21.15.46:0/1395237961 reap_dead start 2024-09-16T04:45:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f937fd11640 1 -- 172.21.15.46:0/1395237961 shutdown_connections 2024-09-16T04:45:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.939+0000 7f937fd11640 1 -- 172.21.15.46:0/1395237961 >> 172.21.15.46:0/1395237961 conn(0x7f93780a5d10 msgr2=0x7f937809f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.940+0000 7f937fd11640 1 -- 172.21.15.46:0/1395237961 shutdown_connections 2024-09-16T04:45:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:23.940+0000 7f937fd11640 1 -- 172.21.15.46:0/1395237961 wait complete. 2024-09-16T04:45:23.995 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:45:23.995 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T04:45:23.995 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:45:23.995 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T04:45:23.995 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:45:23.996 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:45:23.996 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:45:24.007 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:45:24.007 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:45:24.007 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:45:24.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.147+0000 7f095ded7640 10 monclient: get_monmap_and_config 2024-09-16T04:45:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.147+0000 7f095ded7640 10 monclient: build_initial_monmap 2024-09-16T04:45:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.147+0000 7f095ded7640 10 monclient: monmap: 2024-09-16T04:45:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:45:24.148347+0000 2024-09-16T04:45:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:45:24.148347+0000 2024-09-16T04:45:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:45:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:45:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:45:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:45:24.150 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:45:24.150 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:24.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.148+0000 7f095ded7640 1 Processor -- start 2024-09-16T04:45:24.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.148+0000 7f095ded7640 1 -- start start 2024-09-16T04:45:24.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.148+0000 7f095ded7640 10 monclient: init 2024-09-16T04:45:24.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f095ded7640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:24.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f095ded7640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-16T04:45:24.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f095ded7640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0958133b30 0x7f0958133f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:24.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f095ded7640 10 monclient(hunting): picked mon.noname-c con 0x7f0958133b30 addr v2:172.21.15.114:3300/0 2024-09-16T04:45:24.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f095ded7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f095809aa70 0x7f09580a1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:24.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f095ded7640 10 monclient(hunting): picked mon.noname-a con 0x7f095809aa70 addr v2:172.21.15.46:3300/0 2024-09-16T04:45:24.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f095ded7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09580a1e70 addr v1:172.21.15.46:6789/0 2024-09-16T04:45:24.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f095ded7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f095806c440 con 0x7f09580a1e70 2024-09-16T04:45:24.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f095ded7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:24.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f095ded7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f095809b1c0 con 0x7f095809aa70 2024-09-16T04:45:24.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f095ded7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:24.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f095ded7640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f095809b4a0 con 0x7f0958133b30 2024-09-16T04:45:24.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f095ded7640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:24.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f095ded7640 10 monclient(hunting): authenticate will time out at 1122.564258s 2024-09-16T04:45:24.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f0956ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f095809aa70 0x7f09580a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:24.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f0957fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f09580a1e70 0x7f095813dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45486/0 (socket says 172.21.15.46:45486) 2024-09-16T04:45:24.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f0957fff640 1 -- 172.21.15.46:0/499726682 learned_addr learned my addr 172.21.15.46:0/499726682 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:45:24.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f0956ffd640 10 monclient(hunting): get_auth_request con 0x7f095809aa70 auth_method 0 2024-09-16T04:45:24.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f0956ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:24.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f0956ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:24.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f0956ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:24.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f09577fe640 1 --2- 172.21.15.46:0/499726682 >> v2:172.21.15.114:3300/0 conn(0x7f0958133b30 0x7f0958133f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:24.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f09577fe640 10 monclient(hunting): get_auth_request con 0x7f0958133b30 auth_method 0 2024-09-16T04:45:24.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f09577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:24.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f09577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:24.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f09577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:24.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f0956ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:24.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f0956ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:24.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f0956ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:24.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f09577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:24.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f09577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:24.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f09577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:24.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f0956ffd640 10 monclient(hunting): handle_auth_done global_id 74463 payload 274 2024-09-16T04:45:24.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f0956ffd640 1 -- 172.21.15.46:0/499726682 >> v2:172.21.15.114:3300/0 conn(0x7f0958133b30 msgr2=0x7f0958133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:45:24.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f0956ffd640 1 --2- 172.21.15.46:0/499726682 >> v2:172.21.15.114:3300/0 conn(0x7f0958133b30 0x7f0958133f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:24.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f0956ffd640 1 -- 172.21.15.46:0/499726682 >> v1:172.21.15.46:6789/0 conn(0x7f09580a1e70 legacy=0x7f095813dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:24.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f0956ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:45:24.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f0956ffd640 1 monclient: found mon.noname-a 2024-09-16T04:45:24.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f0956ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:24.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f0956ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:24.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.149+0000 7f0956ffd640 1 -- 172.21.15.46:0/499726682 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f095809b780 con 0x7f095809aa70 2024-09-16T04:45:24.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.150+0000 7f0956ffd640 1 --2- 172.21.15.46:0/499726682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f095809aa70 0x7f09580a1930 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0958073900 tx=0x7f094802a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=837b4ad418f06fb1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:24.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.150+0000 7f09567fc640 1 -- 172.21.15.46:0/499726682 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0948037070 con 0x7f095809aa70 2024-09-16T04:45:24.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.150+0000 7f09567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:24.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.150+0000 7f09567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:24.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.150+0000 7f09567fc640 10 monclient: dump: 2024-09-16T04:45:24.161 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:24.161 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:24.161 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:24.161 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:24.161 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:24.162 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:24.162 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:24.162 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:24.162 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:24.162 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:24.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.150+0000 7f09567fc640 10 monclient: _finish_auth 0 2024-09-16T04:45:24.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.150+0000 7f09567fc640 10 monclient: _check_auth_tickets 2024-09-16T04:45:24.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.150+0000 7f09567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:24.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.150+0000 7f09567fc640 1 -- 172.21.15.46:0/499726682 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f094802add0 con 0x7f095809aa70 2024-09-16T04:45:24.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.150+0000 7f09567fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:24.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.150+0000 7f09567fc640 1 -- 172.21.15.46:0/499726682 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0948032660 con 0x7f095809aa70 2024-09-16T04:45:24.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.150+0000 7f095ded7640 5 monclient: authenticate success, global_id 74463 2024-09-16T04:45:24.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.150+0000 7f095ded7640 10 monclient: get_monmap_and_config success 2024-09-16T04:45:24.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.150+0000 7f095ded7640 4 set_mon_vals no callback set 2024-09-16T04:45:24.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.150+0000 7f09567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:24.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.150+0000 7f09567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:24.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.150+0000 7f09567fc640 10 monclient: dump: 2024-09-16T04:45:24.163 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:24.163 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:24.163 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:24.163 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:24.163 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:24.163 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:24.163 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:24.164 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.151+0000 7f095ded7640 10 monclient: shutdown 2024-09-16T04:45:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.151+0000 7f095ded7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.151+0000 7f095ded7640 1 -- 172.21.15.46:0/499726682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f095809aa70 msgr2=0x7f09580a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.151+0000 7f095ded7640 1 --2- 172.21.15.46:0/499726682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f095809aa70 0x7f09580a1930 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0958073900 tx=0x7f094802a080 comp rx=0 tx=0).stop 2024-09-16T04:45:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.151+0000 7f095ded7640 1 -- 172.21.15.46:0/499726682 shutdown_connections 2024-09-16T04:45:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.151+0000 7f095ded7640 1 --2- 172.21.15.46:0/499726682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f095809aa70 0x7f09580a1930 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.151+0000 7f095ded7640 1 --2- 172.21.15.46:0/499726682 >> v2:172.21.15.114:3300/0 conn(0x7f0958133b30 0x7f0958133f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.151+0000 7f095ded7640 1 -- 172.21.15.46:0/499726682 >> 172.21.15.46:0/499726682 conn(0x7f09580a6630 msgr2=0x7f09580a2a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.151+0000 7f095ded7640 1 -- 172.21.15.46:0/499726682 shutdown_connections 2024-09-16T04:45:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.151+0000 7f095ded7640 1 -- 172.21.15.46:0/499726682 wait complete. 2024-09-16T04:45:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.151+0000 7f095ded7640 10 monclient: build_initial_monmap 2024-09-16T04:45:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.151+0000 7f095ded7640 10 monclient: monmap: 2024-09-16T04:45:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:45:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:45:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:45:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:45:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:45:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:45:24.165 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f095ded7640 1 Processor -- start 2024-09-16T04:45:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f095ded7640 1 -- start start 2024-09-16T04:45:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f095ded7640 10 monclient: init 2024-09-16T04:45:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f095ded7640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f095ded7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:45:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f095ded7640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f095809aa70 0x7f09581cbb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f095ded7640 10 monclient(hunting): picked mon.noname-c con 0x7f095809aa70 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f095ded7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09580a1e70 0x7f09581329d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f095ded7640 10 monclient(hunting): picked mon.noname-b con 0x7f09580a1e70 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:45:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f095ded7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0958133b30 0x7f0958120440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f095ded7640 10 monclient(hunting): picked mon.noname-a con 0x7f0958133b30 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:45:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f095ded7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f095ded7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f09580a33a0 con 0x7f0958133b30 2024-09-16T04:45:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f095ded7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f095ded7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0958134450 con 0x7f09580a1e70 2024-09-16T04:45:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f095ded7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f095ded7640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f09580a3520 con 0x7f095809aa70 2024-09-16T04:45:24.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f095ded7640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:24.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f09577fe640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f095809aa70 msgr2=0x7f09581cbb00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:45:24.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f095ded7640 10 monclient(hunting): authenticate will time out at 1122.567865s 2024-09-16T04:45:24.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f09577fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f095809aa70 0x7f09581cbb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:24.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f0957fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0958133b30 0x7f0958120440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:24.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f0957fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0958133b30 0x7f0958120440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32916/0 (socket says 172.21.15.46:32916) 2024-09-16T04:45:24.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.152+0000 7f0957fff640 1 -- 172.21.15.46:0/2256692366 learned_addr learned my addr 172.21.15.46:0/2256692366 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:24.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0957fff640 10 monclient(hunting): get_auth_request con 0x7f0958133b30 auth_method 0 2024-09-16T04:45:24.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0957fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:24.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0957fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:24.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0957fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:24.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0957fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:24.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0957fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0957fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0956ffd640 1 --2- 172.21.15.46:0/2256692366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09580a1e70 0x7f09581329d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0956ffd640 10 monclient(hunting): get_auth_request con 0x7f09580a1e70 auth_method 0 2024-09-16T04:45:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0956ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0956ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0956ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0957fff640 10 monclient(hunting): handle_auth_done global_id 74469 payload 824 2024-09-16T04:45:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0957fff640 1 -- 172.21.15.46:0/2256692366 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f095809aa70 msgr2=0x7f09581cbb00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:45:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0957fff640 1 --2- 172.21.15.46:0/2256692366 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f095809aa70 0x7f09581cbb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0957fff640 1 -- 172.21.15.46:0/2256692366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09580a1e70 msgr2=0x7f09581329d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0957fff640 1 --2- 172.21.15.46:0/2256692366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09580a1e70 0x7f09581329d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0957fff640 10 monclient: _finish_hunting 0 2024-09-16T04:45:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0957fff640 1 monclient: found mon.noname-a 2024-09-16T04:45:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0957fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0957fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0957fff640 1 -- 172.21.15.46:0/2256692366 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09581cca20 con 0x7f0958133b30 2024-09-16T04:45:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0956ffd640 1 --2- 172.21.15.46:0/2256692366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09580a1e70 0x7f09581329d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:45:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0957fff640 1 --2- 172.21.15.46:0/2256692366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0958133b30 0x7f0958120440 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f095000cbc0 tx=0x7f095000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f094ffff640 1 -- 172.21.15.46:0/2256692366 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950017070 con 0x7f0958133b30 2024-09-16T04:45:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f094ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f094ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f094ffff640 10 monclient: dump: 2024-09-16T04:45:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:24.170 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:24.170 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:24.170 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:24.170 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:24.170 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:24.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f094ffff640 10 monclient: _finish_auth 0 2024-09-16T04:45:24.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f094ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:45:24.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f094ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:24.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f094ffff640 1 -- 172.21.15.46:0/2256692366 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f095000ad40 con 0x7f0958133b30 2024-09-16T04:45:24.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f095ded7640 5 monclient: authenticate success, global_id 74469 2024-09-16T04:45:24.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f095ded7640 10 monclient: _renew_subs 2024-09-16T04:45:24.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f095ded7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:24.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f095ded7640 1 -- 172.21.15.46:0/2256692366 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09581240f0 con 0x7f0958133b30 2024-09-16T04:45:24.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f094ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f094ffff640 1 -- 172.21.15.46:0/2256692366 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950012600 con 0x7f0958133b30 2024-09-16T04:45:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f094ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f094ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f094ffff640 10 monclient: dump: 2024-09-16T04:45:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:24.171 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f0955ffb640 4 set_mon_vals no callback set 2024-09-16T04:45:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f095ded7640 10 monclient: _renew_subs 2024-09-16T04:45:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f095ded7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.153+0000 7f095ded7640 1 -- 172.21.15.46:0/2256692366 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0958134990 con 0x7f0958133b30 2024-09-16T04:45:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.154+0000 7f094ffff640 1 -- 172.21.15.46:0/2256692366 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f095001a070 con 0x7f0958133b30 2024-09-16T04:45:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.154+0000 7f094ffff640 1 --2- 172.21.15.46:0/2256692366 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0938038360 0x7f093803a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.154+0000 7f095ded7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.154+0000 7f095ded7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.154+0000 7f095ded7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.154+0000 7f094ffff640 1 -- 172.21.15.46:0/2256692366 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 4346+0+0 (secure 0 0 0) 0x7f095004b290 con 0x7f0958133b30 2024-09-16T04:45:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.154+0000 7f095ded7640 1 -- 172.21.15.46:0/2256692366 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09580a1520 con 0x7f0958133b30 2024-09-16T04:45:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.155+0000 7f09577fe640 1 --2- 172.21.15.46:0/2256692366 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0938038360 0x7f093803a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.155+0000 7f09577fe640 10 monclient: get_auth_request con 0x7f0938038360 auth_method 0 2024-09-16T04:45:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.155+0000 7f09577fe640 1 --2- 172.21.15.46:0/2256692366 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0938038360 0x7f093803a820 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f09400094f0 tx=0x7f0940009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.159+0000 7f094ffff640 1 -- 172.21.15.46:0/2256692366 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f095001a360 con 0x7f0958133b30 2024-09-16T04:45:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.159+0000 7f094ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.159+0000 7f094ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:45:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.401+0000 7f095ded7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:45:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.401+0000 7f095ded7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:45:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.401+0000 7f095ded7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.401+0000 7f095ded7640 1 -- 172.21.15.46:0/2256692366 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0958133340 con 0x7f0958133b30 2024-09-16T04:45:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.402+0000 7f094ffff640 1 -- 172.21.15.46:0/2256692366 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f095004b080 con 0x7f0958133b30 2024-09-16T04:45:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.402+0000 7f094ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:45:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.402+0000 7f094ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:45:24.404 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":78,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:45:24.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.404+0000 7f095ded7640 1 -- 172.21.15.46:0/2256692366 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0938038360 msgr2=0x7f093803a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:24.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.404+0000 7f095ded7640 1 --2- 172.21.15.46:0/2256692366 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0938038360 0x7f093803a820 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f09400094f0 tx=0x7f0940009190 comp rx=0 tx=0).stop 2024-09-16T04:45:24.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.404+0000 7f095ded7640 10 monclient: shutdown 2024-09-16T04:45:24.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.404+0000 7f095ded7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:24.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.404+0000 7f095ded7640 1 -- 172.21.15.46:0/2256692366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0958133b30 msgr2=0x7f0958120440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:24.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.404+0000 7f095ded7640 1 --2- 172.21.15.46:0/2256692366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0958133b30 0x7f0958120440 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f095000cbc0 tx=0x7f095000a430 comp rx=0 tx=0).stop 2024-09-16T04:45:24.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.404+0000 7f095ded7640 1 -- 172.21.15.46:0/2256692366 shutdown_connections 2024-09-16T04:45:24.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.404+0000 7f095ded7640 1 --2- 172.21.15.46:0/2256692366 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0938038360 0x7f093803a820 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:24.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.404+0000 7f095ded7640 1 --2- 172.21.15.46:0/2256692366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0958133b30 0x7f0958120440 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:24.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.404+0000 7f095ded7640 1 --2- 172.21.15.46:0/2256692366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09580a1e70 0x7f09581329d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:24.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.404+0000 7f095ded7640 1 --2- 172.21.15.46:0/2256692366 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f095809aa70 0x7f09581cbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:24.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.404+0000 7f095ded7640 1 -- 172.21.15.46:0/2256692366 >> 172.21.15.46:0/2256692366 conn(0x7f09580a6630 msgr2=0x7f0958136740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:24.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.404+0000 7f095ded7640 1 -- 172.21.15.46:0/2256692366 shutdown_connections 2024-09-16T04:45:24.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:24.404+0000 7f095ded7640 1 -- 172.21.15.46:0/2256692366 wait complete. 2024-09-16T04:45:27.417 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:45:27.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.529+0000 7f884e503640 10 monclient: get_monmap_and_config 2024-09-16T04:45:27.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.529+0000 7f884e503640 10 monclient: build_initial_monmap 2024-09-16T04:45:27.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.529+0000 7f884e503640 10 monclient: monmap: 2024-09-16T04:45:27.530 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:27.530 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:27.530 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:45:27.529708+0000 2024-09-16T04:45:27.530 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:45:27.529708+0000 2024-09-16T04:45:27.530 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:27.531 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:27.531 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:45:27.531 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:45:27.531 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:45:27.531 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:45:27.531 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:45:27.531 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:27.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f884e503640 1 Processor -- start 2024-09-16T04:45:27.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f884e503640 1 -- start start 2024-09-16T04:45:27.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f884e503640 10 monclient: init 2024-09-16T04:45:27.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f884e503640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:27.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f884e503640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-16T04:45:27.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f884e503640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f884806e160 0x7f88480a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:27.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f884e503640 10 monclient(hunting): picked mon.noname-a con 0x7f884806e160 addr v2:172.21.15.46:3300/0 2024-09-16T04:45:27.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f884e503640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f88480a2ee0 addr v1:172.21.15.114:6789/0 2024-09-16T04:45:27.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f884e503640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8848137050 0x7f884813ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:27.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f884e503640 10 monclient(hunting): picked mon.noname-b con 0x7f8848137050 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:27.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f884e503640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88480a3fd0 con 0x7f88480a2ee0 2024-09-16T04:45:27.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f884e503640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:27.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f884e503640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f88480a3300 con 0x7f884806e160 2024-09-16T04:45:27.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f884e503640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:27.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f884e503640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f884806e540 con 0x7f8848137050 2024-09-16T04:45:27.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f884e503640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:27.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f884e503640 10 monclient(hunting): authenticate will time out at 1125.945603s 2024-09-16T04:45:27.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f884caa0640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8848137050 msgr2=0x7f884813ec80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:45:27.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f884caa0640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8848137050 0x7f884813ec80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:27.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f8847fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f884806e160 0x7f88480a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:27.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f8847fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f884806e160 0x7f88480a29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41954/0 (socket says 172.21.15.46:41954) 2024-09-16T04:45:27.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f8847fff640 1 -- 172.21.15.46:0/1899893647 learned_addr learned my addr 172.21.15.46:0/1899893647 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:27.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f8847fff640 10 monclient(hunting): get_auth_request con 0x7f884806e160 auth_method 0 2024-09-16T04:45:27.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f8847fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:27.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f8847fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:27.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f8847fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:27.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f8847fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:27.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f8847fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:27.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.530+0000 7f8847fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:27.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f8847fff640 10 monclient(hunting): handle_auth_done global_id 74472 payload 274 2024-09-16T04:45:27.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f8847fff640 1 -- 172.21.15.46:0/1899893647 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8848137050 msgr2=0x7f884813ec80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:45:27.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f8847fff640 1 --2- 172.21.15.46:0/1899893647 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8848137050 0x7f884813ec80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:27.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f8847fff640 1 -- 172.21.15.46:0/1899893647 >> v1:172.21.15.114:6789/0 conn(0x7f88480a2ee0 legacy=0x7f8848136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f8847fff640 10 monclient: _finish_hunting 0 2024-09-16T04:45:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f8847fff640 1 monclient: found mon.noname-a 2024-09-16T04:45:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f8847fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f8847fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f8847fff640 1 -- 172.21.15.46:0/1899893647 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88480a0bd0 con 0x7f884806e160 2024-09-16T04:45:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f8847fff640 1 --2- 172.21.15.46:0/1899893647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f884806e160 0x7f88480a29a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f88380098a0 tx=0x7f883802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a605daacfcbac44 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f8846ffd640 1 -- 172.21.15.46:0/1899893647 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8838037070 con 0x7f884806e160 2024-09-16T04:45:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f8846ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f8846ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f8846ffd640 10 monclient: dump: 2024-09-16T04:45:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:27.543 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f8846ffd640 10 monclient: _finish_auth 0 2024-09-16T04:45:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f8846ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:45:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f8846ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f8846ffd640 1 -- 172.21.15.46:0/1899893647 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8838032440 con 0x7f884806e160 2024-09-16T04:45:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f884e503640 5 monclient: authenticate success, global_id 74472 2024-09-16T04:45:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f884e503640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:45:27.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f8846ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:27.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f884e503640 10 monclient: get_monmap_and_config success 2024-09-16T04:45:27.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f8846ffd640 1 -- 172.21.15.46:0/1899893647 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8838032810 con 0x7f884806e160 2024-09-16T04:45:27.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f884e503640 4 set_mon_vals no callback set 2024-09-16T04:45:27.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.531+0000 7f8846ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:27.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.532+0000 7f8846ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:27.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.532+0000 7f8846ffd640 10 monclient: dump: 2024-09-16T04:45:27.544 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:27.544 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:27.544 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:27.544 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:27.544 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:27.545 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:27.545 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:27.545 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:27.545 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:27.545 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:27.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.532+0000 7f884e503640 10 monclient: shutdown 2024-09-16T04:45:27.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.532+0000 7f884e503640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:27.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.532+0000 7f884e503640 1 -- 172.21.15.46:0/1899893647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f884806e160 msgr2=0x7f88480a29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:27.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.532+0000 7f884e503640 1 --2- 172.21.15.46:0/1899893647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f884806e160 0x7f88480a29a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f88380098a0 tx=0x7f883802a180 comp rx=0 tx=0).stop 2024-09-16T04:45:27.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.532+0000 7f884e503640 1 -- 172.21.15.46:0/1899893647 shutdown_connections 2024-09-16T04:45:27.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.532+0000 7f884e503640 1 --2- 172.21.15.46:0/1899893647 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8848137050 0x7f884813ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:27.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.532+0000 7f884e503640 1 --2- 172.21.15.46:0/1899893647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f884806e160 0x7f88480a29a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:27.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.532+0000 7f884e503640 1 -- 172.21.15.46:0/1899893647 >> 172.21.15.46:0/1899893647 conn(0x7f88480a5c10 msgr2=0x7f88480a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:27.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.532+0000 7f884e503640 1 -- 172.21.15.46:0/1899893647 shutdown_connections 2024-09-16T04:45:27.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.532+0000 7f884e503640 1 -- 172.21.15.46:0/1899893647 wait complete. 2024-09-16T04:45:27.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.533+0000 7f884e503640 10 monclient: build_initial_monmap 2024-09-16T04:45:27.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.533+0000 7f884e503640 10 monclient: monmap: 2024-09-16T04:45:27.546 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:27.546 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:45:27.546 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:45:27.546 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:45:27.546 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:27.546 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:27.546 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:45:27.547 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:45:27.547 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:45:27.547 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:27.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.533+0000 7f884e503640 1 Processor -- start 2024-09-16T04:45:27.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.533+0000 7f884e503640 1 -- start start 2024-09-16T04:45:27.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.533+0000 7f884e503640 10 monclient: init 2024-09-16T04:45:27.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f884e503640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:27.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f884e503640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:45:27.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f884e503640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f884806e160 0x7f8848135bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:27.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f884e503640 10 monclient(hunting): picked mon.noname-b con 0x7f884806e160 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:45:27.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f884e503640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88480a2ee0 0x7f8848136110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:27.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f884e503640 10 monclient(hunting): picked mon.noname-a con 0x7f88480a2ee0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:45:27.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f884e503640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8848137050 0x7f884811f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:27.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f884e503640 10 monclient(hunting): picked mon.noname-c con 0x7f8848137050 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:27.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f884e503640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:27.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f884e503640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f88480a34a0 con 0x7f88480a2ee0 2024-09-16T04:45:27.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f884e503640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:27.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f884e503640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f88480a5130 con 0x7f884806e160 2024-09-16T04:45:27.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f884e503640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:27.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f884e503640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f88481369d0 con 0x7f8848137050 2024-09-16T04:45:27.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f884e503640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f884e503640 10 monclient(hunting): authenticate will time out at 1125.949243s 2024-09-16T04:45:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f884caa0640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8848137050 msgr2=0x7f884811f760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:45:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f884caa0640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8848137050 0x7f884811f760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f88477fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88480a2ee0 0x7f8848136110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f88477fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88480a2ee0 0x7f8848136110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41960/0 (socket says 172.21.15.46:41960) 2024-09-16T04:45:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f88477fe640 1 -- 172.21.15.46:0/2122775715 learned_addr learned my addr 172.21.15.46:0/2122775715 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f88477fe640 10 monclient(hunting): get_auth_request con 0x7f88480a2ee0 auth_method 0 2024-09-16T04:45:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f88477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f88477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f88477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f88477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f88477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f88477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f8847fff640 1 --2- 172.21.15.46:0/2122775715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f884806e160 0x7f8848135bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f8847fff640 10 monclient(hunting): get_auth_request con 0x7f884806e160 auth_method 0 2024-09-16T04:45:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f8847fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f8847fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f8847fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f88477fe640 10 monclient(hunting): handle_auth_done global_id 74475 payload 824 2024-09-16T04:45:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f88477fe640 1 -- 172.21.15.46:0/2122775715 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8848137050 msgr2=0x7f884811f760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:45:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f88477fe640 1 --2- 172.21.15.46:0/2122775715 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8848137050 0x7f884811f760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f88477fe640 1 -- 172.21.15.46:0/2122775715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f884806e160 msgr2=0x7f8848135bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f88477fe640 1 --2- 172.21.15.46:0/2122775715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f884806e160 0x7f8848135bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f88477fe640 10 monclient: _finish_hunting 0 2024-09-16T04:45:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f88477fe640 1 monclient: found mon.noname-a 2024-09-16T04:45:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f88477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f88477fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f88477fe640 1 -- 172.21.15.46:0/2122775715 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88481232d0 con 0x7f88480a2ee0 2024-09-16T04:45:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f88477fe640 1 --2- 172.21.15.46:0/2122775715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88480a2ee0 0x7f8848136110 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f883c00c400 tx=0x7f883c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f8844ff9640 1 -- 172.21.15.46:0/2122775715 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f883c0052d0 con 0x7f88480a2ee0 2024-09-16T04:45:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f8844ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:27.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f8844ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:27.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f8844ff9640 10 monclient: dump: 2024-09-16T04:45:27.552 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:27.552 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:27.552 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:27.552 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:27.552 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:27.552 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:27.552 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:27.553 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:27.553 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:27.553 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:27.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f8844ff9640 10 monclient: _finish_auth 0 2024-09-16T04:45:27.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f8844ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:45:27.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f8844ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:27.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f8844ff9640 1 -- 172.21.15.46:0/2122775715 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f883c005490 con 0x7f88480a2ee0 2024-09-16T04:45:27.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f884e503640 5 monclient: authenticate success, global_id 74475 2024-09-16T04:45:27.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f884e503640 10 monclient: _renew_subs 2024-09-16T04:45:27.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.534+0000 7f884e503640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:27.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.535+0000 7f884e503640 1 -- 172.21.15.46:0/2122775715 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88481235b0 con 0x7f88480a2ee0 2024-09-16T04:45:27.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.535+0000 7f8844ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:27.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.535+0000 7f8844ff9640 1 -- 172.21.15.46:0/2122775715 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f883c007260 con 0x7f88480a2ee0 2024-09-16T04:45:27.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.535+0000 7f8844ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:27.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.535+0000 7f8844ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:27.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.535+0000 7f8844ff9640 10 monclient: dump: 2024-09-16T04:45:27.555 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:27.555 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:27.555 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:27.555 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:27.555 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:27.556 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.535+0000 7f88467fc640 4 set_mon_vals no callback set 2024-09-16T04:45:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.535+0000 7f884e503640 10 monclient: _renew_subs 2024-09-16T04:45:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.535+0000 7f884e503640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.535+0000 7f884e503640 1 -- 172.21.15.46:0/2122775715 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8848121ba0 con 0x7f88480a2ee0 2024-09-16T04:45:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.535+0000 7f884e503640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.535+0000 7f884e503640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.535+0000 7f884e503640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.535+0000 7f8844ff9640 1 -- 172.21.15.46:0/2122775715 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f883c007400 con 0x7f88480a2ee0 2024-09-16T04:45:27.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.535+0000 7f884e503640 1 -- 172.21.15.46:0/2122775715 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88481374c0 con 0x7f88480a2ee0 2024-09-16T04:45:27.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.536+0000 7f8844ff9640 1 --2- 172.21.15.46:0/2122775715 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8834038390 0x7f883403a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:27.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.536+0000 7f8844ff9640 1 -- 172.21.15.46:0/2122775715 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 4346+0+0 (secure 0 0 0) 0x7f883c04aaf0 con 0x7f88480a2ee0 2024-09-16T04:45:27.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.536+0000 7f8847fff640 1 --2- 172.21.15.46:0/2122775715 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8834038390 0x7f883403a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:27.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.536+0000 7f8847fff640 10 monclient: get_auth_request con 0x7f8834038390 auth_method 0 2024-09-16T04:45:27.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.537+0000 7f8847fff640 1 --2- 172.21.15.46:0/2122775715 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8834038390 0x7f883403a850 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8838034000 tx=0x7f883802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:27.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.540+0000 7f8844ff9640 1 -- 172.21.15.46:0/2122775715 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f88480a3fd0 con 0x7f88480a2ee0 2024-09-16T04:45:27.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.540+0000 7f8844ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:27.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.540+0000 7f8844ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:45:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.782+0000 7f884e503640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:45:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.782+0000 7f884e503640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:45:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.782+0000 7f884e503640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.782+0000 7f884e503640 1 -- 172.21.15.46:0/2122775715 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8848067e00 con 0x7f88480a2ee0 2024-09-16T04:45:27.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.783+0000 7f8844ff9640 1 -- 172.21.15.46:0/2122775715 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f883c012850 con 0x7f88480a2ee0 2024-09-16T04:45:27.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.783+0000 7f8844ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:45:27.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.783+0000 7f8844ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:45:27.784 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":78,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:45:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.784+0000 7f884e503640 1 -- 172.21.15.46:0/2122775715 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8834038390 msgr2=0x7f883403a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.784+0000 7f884e503640 1 --2- 172.21.15.46:0/2122775715 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8834038390 0x7f883403a850 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8838034000 tx=0x7f883802e000 comp rx=0 tx=0).stop 2024-09-16T04:45:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.784+0000 7f884e503640 10 monclient: shutdown 2024-09-16T04:45:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.784+0000 7f884e503640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.784+0000 7f884e503640 1 -- 172.21.15.46:0/2122775715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88480a2ee0 msgr2=0x7f8848136110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.784+0000 7f884e503640 1 --2- 172.21.15.46:0/2122775715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88480a2ee0 0x7f8848136110 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f883c00c400 tx=0x7f883c00c8c0 comp rx=0 tx=0).stop 2024-09-16T04:45:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.784+0000 7f884e503640 1 -- 172.21.15.46:0/2122775715 shutdown_connections 2024-09-16T04:45:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.785+0000 7f884e503640 1 --2- 172.21.15.46:0/2122775715 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8848137050 0x7f884811f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.785+0000 7f884e503640 1 --2- 172.21.15.46:0/2122775715 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f8834038390 0x7f883403a850 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.785+0000 7f884e503640 1 --2- 172.21.15.46:0/2122775715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88480a2ee0 0x7f8848136110 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.785+0000 7f884e503640 1 --2- 172.21.15.46:0/2122775715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f884806e160 0x7f8848135bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.785+0000 7f884e503640 1 -- 172.21.15.46:0/2122775715 >> 172.21.15.46:0/2122775715 conn(0x7f88480a5c10 msgr2=0x7f884809f520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.785+0000 7f884e503640 1 -- 172.21.15.46:0/2122775715 shutdown_connections 2024-09-16T04:45:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:27.785+0000 7f884e503640 1 -- 172.21.15.46:0/2122775715 wait complete. 2024-09-16T04:45:30.801 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:45:30.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.913+0000 7fb7c3027640 10 monclient: get_monmap_and_config 2024-09-16T04:45:30.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.913+0000 7fb7c3027640 10 monclient: build_initial_monmap 2024-09-16T04:45:30.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.913+0000 7fb7c3027640 10 monclient: monmap: 2024-09-16T04:45:30.915 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:30.915 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:30.915 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:45:30.914148+0000 2024-09-16T04:45:30.915 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:45:30.914148+0000 2024-09-16T04:45:30.915 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:30.915 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:30.915 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:45:30.915 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:45:30.915 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:45:30.915 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:45:30.915 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:45:30.915 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:30.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.914+0000 7fb7c3027640 1 Processor -- start 2024-09-16T04:45:30.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.914+0000 7fb7c3027640 1 -- start start 2024-09-16T04:45:30.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.914+0000 7fb7c3027640 10 monclient: init 2024-09-16T04:45:30.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.914+0000 7fb7c3027640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:30.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.915+0000 7fb7c3027640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-16T04:45:30.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.915+0000 7fb7c3027640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb7bc06e160 addr v1:172.21.15.114:6789/0 2024-09-16T04:45:30.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.915+0000 7fb7c3027640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7bc139c50 0x7fb7bc13c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:30.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.915+0000 7fb7c3027640 10 monclient(hunting): picked mon.noname-b con 0x7fb7bc139c50 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:30.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.915+0000 7fb7c3027640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb7bc13c580 addr v1:172.21.15.46:6789/0 2024-09-16T04:45:30.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.915+0000 7fb7c3027640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7bc0a3fd0 con 0x7fb7bc13c580 2024-09-16T04:45:30.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.915+0000 7fb7c3027640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7bc0a0bd0 con 0x7fb7bc06e160 2024-09-16T04:45:30.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.915+0000 7fb7c3027640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:30.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.915+0000 7fb7c3027640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fb7bc06e540 con 0x7fb7bc139c50 2024-09-16T04:45:30.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.915+0000 7fb7c3027640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:30.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.915+0000 7fb7c3027640 10 monclient(hunting): authenticate will time out at 1129.330122s 2024-09-16T04:45:30.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.915+0000 7fb7b3fff640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7bc139c50 msgr2=0x7fb7bc13c040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:45:30.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.915+0000 7fb7b3fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7bc139c50 0x7fb7bc13c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:30.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.915+0000 7fb7c15c4640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb7bc13c580 0x7fb7bc13e970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43368/0 (socket says 172.21.15.46:43368) 2024-09-16T04:45:30.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:30.915+0000 7fb7c15c4640 1 -- 172.21.15.46:0/1135987635 learned_addr learned my addr 172.21.15.46:0/1135987635 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:45:31.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:31.116+0000 7fb7b3fff640 1 -- 172.21.15.46:0/1135987635 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7bc139c50 msgr2=0x7fb7bc13c040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:45:31.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:31.116+0000 7fb7b3fff640 1 --2- 172.21.15.46:0/1135987635 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7bc139c50 0x7fb7bc13c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:45:31.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:31.517+0000 7fb7b3fff640 1 -- 172.21.15.46:0/1135987635 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7bc139c50 msgr2=0x7fb7bc13c040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:45:31.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:31.517+0000 7fb7b3fff640 1 --2- 172.21.15.46:0/1135987635 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7bc139c50 0x7fb7bc13c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:45:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:32.318+0000 7fb7b3fff640 1 -- 172.21.15.46:0/1135987635 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7bc139c50 msgr2=0x7fb7bc13c040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:45:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:32.318+0000 7fb7b3fff640 1 --2- 172.21.15.46:0/1135987635 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7bc139c50 0x7fb7bc13c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:45:33.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b27fc640 10 monclient(hunting): tick 2024-09-16T04:45:33.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b27fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:45:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b27fc640 1 monclient(hunting): continuing hunt 2024-09-16T04:45:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b27fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:45:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b27fc640 1 -- 172.21.15.46:0/1135987635 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7bc139c50 msgr2=0x7fb7bc13c040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:45:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b27fc640 1 --2- 172.21.15.46:0/1135987635 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7bc139c50 0x7fb7bc13c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b27fc640 1 -- 172.21.15.46:0/1135987635 >> v1:172.21.15.114:6789/0 conn(0x7fb7bc06e160 legacy=0x7fb7bc0a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b27fc640 1 -- 172.21.15.46:0/1135987635 >> v1:172.21.15.46:6789/0 conn(0x7fb7bc13c580 legacy=0x7fb7bc13e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b27fc640 10 monclient: _add_conns ranks=[3,4] 2024-09-16T04:45:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b27fc640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 0x7fb7a8005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b27fc640 10 monclient(hunting): picked mon.noname-a con 0x7fb7a8003710 addr v2:172.21.15.46:3300/0 2024-09-16T04:45:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b27fc640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 0x7fb7a8008720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b27fc640 10 monclient(hunting): picked mon.noname-c con 0x7fb7a8006280 addr v2:172.21.15.114:3300/0 2024-09-16T04:45:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b27fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:33.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b27fc640 1 -- 172.21.15.46:0/1135987635 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb7bc06e540 con 0x7fb7a8003710 2024-09-16T04:45:33.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b27fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:33.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b27fc640 1 -- 172.21.15.46:0/1135987635 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fb7a8008de0 con 0x7fb7a8006280 2024-09-16T04:45:33.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b27fc640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:33.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 0x7fb7a8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:33.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7c0dc3640 10 monclient(hunting): get_auth_request con 0x7fb7a8003710 auth_method 0 2024-09-16T04:45:33.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7c0dc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:33.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7c0dc3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:33.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7c0dc3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:33.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b3fff640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 0x7fb7a8008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:33.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7c0dc3640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 msgr2=0x7fb7a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:45:33.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7c0dc3640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 msgr2=0x7fb7a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:33.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 0x7fb7a8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:33.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b3fff640 10 monclient(hunting): get_auth_request con 0x7fb7a8006280 auth_method 0 2024-09-16T04:45:33.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:33.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:33.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:33.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.915+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 0x7fb7a8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:33.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.916+0000 7fb7b3fff640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 msgr2=0x7fb7a8008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:45:33.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.916+0000 7fb7b3fff640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 msgr2=0x7fb7a8008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:33.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.916+0000 7fb7b3fff640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 0x7fb7a8008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:33.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:33.916+0000 7fb7b3fff640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 0x7fb7a8008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:34.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.116+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 0x7fb7a8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:34.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.116+0000 7fb7c0dc3640 10 monclient(hunting): get_auth_request con 0x7fb7a8003710 auth_method 0 2024-09-16T04:45:34.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.116+0000 7fb7c0dc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:34.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.116+0000 7fb7c0dc3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:34.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.116+0000 7fb7c0dc3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:34.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.116+0000 7fb7c0dc3640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 msgr2=0x7fb7a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:45:34.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.116+0000 7fb7c0dc3640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 msgr2=0x7fb7a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:34.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.116+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 0x7fb7a8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:34.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.116+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 0x7fb7a8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:45:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.117+0000 7fb7b3fff640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 0x7fb7a8008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.117+0000 7fb7b3fff640 10 monclient(hunting): get_auth_request con 0x7fb7a8006280 auth_method 0 2024-09-16T04:45:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.117+0000 7fb7b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.117+0000 7fb7b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.117+0000 7fb7b3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.117+0000 7fb7b3fff640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 msgr2=0x7fb7a8008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:45:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.117+0000 7fb7b3fff640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 msgr2=0x7fb7a8008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.117+0000 7fb7b3fff640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 0x7fb7a8008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.117+0000 7fb7b3fff640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 0x7fb7a8008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:45:34.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.517+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 0x7fb7a8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:34.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.517+0000 7fb7c0dc3640 10 monclient(hunting): get_auth_request con 0x7fb7a8003710 auth_method 0 2024-09-16T04:45:34.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.517+0000 7fb7c0dc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:34.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.517+0000 7fb7c0dc3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:34.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.517+0000 7fb7c0dc3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:34.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.517+0000 7fb7c0dc3640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 msgr2=0x7fb7a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:45:34.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.517+0000 7fb7c0dc3640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 msgr2=0x7fb7a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:34.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.517+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 0x7fb7a8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:34.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.517+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 0x7fb7a8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:45:34.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.518+0000 7fb7b3fff640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 0x7fb7a8008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:34.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.518+0000 7fb7b3fff640 10 monclient(hunting): get_auth_request con 0x7fb7a8006280 auth_method 0 2024-09-16T04:45:34.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.518+0000 7fb7b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:34.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.518+0000 7fb7b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:34.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.518+0000 7fb7b3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:34.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.518+0000 7fb7b3fff640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 msgr2=0x7fb7a8008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:45:34.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.518+0000 7fb7b3fff640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 msgr2=0x7fb7a8008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:34.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.518+0000 7fb7b3fff640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 0x7fb7a8008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:34.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:34.518+0000 7fb7b3fff640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 0x7fb7a8008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:45:35.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:35.318+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 0x7fb7a8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:35.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:35.318+0000 7fb7c0dc3640 10 monclient(hunting): get_auth_request con 0x7fb7a8003710 auth_method 0 2024-09-16T04:45:35.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:35.318+0000 7fb7c0dc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:35.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:35.318+0000 7fb7c0dc3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:35.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:35.318+0000 7fb7c0dc3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:35.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:35.318+0000 7fb7c0dc3640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 msgr2=0x7fb7a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:45:35.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:35.319+0000 7fb7c0dc3640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 msgr2=0x7fb7a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:35.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:35.319+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 0x7fb7a8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:35.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:35.319+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 0x7fb7a8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:45:35.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:35.319+0000 7fb7b3fff640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 0x7fb7a8008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:35.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:35.319+0000 7fb7b3fff640 10 monclient(hunting): get_auth_request con 0x7fb7a8006280 auth_method 0 2024-09-16T04:45:35.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:35.319+0000 7fb7b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:35.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:35.319+0000 7fb7b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:35.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:35.319+0000 7fb7b3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:35.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:35.320+0000 7fb7b3fff640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 msgr2=0x7fb7a8008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:45:35.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:35.320+0000 7fb7b3fff640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 msgr2=0x7fb7a8008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:35.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:35.320+0000 7fb7b3fff640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 0x7fb7a8008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:35.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:35.320+0000 7fb7b3fff640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 0x7fb7a8008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:45:36.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7b27fc640 10 monclient(hunting): tick 2024-09-16T04:45:36.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7b27fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:45:36.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7b27fc640 1 monclient(hunting): continuing hunt 2024-09-16T04:45:36.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7b27fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:45:36.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7b27fc640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 msgr2=0x7fb7a8008720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:45:36.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7b27fc640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8006280 0x7fb7a8008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:36.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7b27fc640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 msgr2=0x7fb7a8005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:45:36.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7b27fc640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8003710 0x7fb7a8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:36.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7b27fc640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-16T04:45:36.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7c15c4640 1 -- 172.21.15.46:0/1135987635 reap_dead start 2024-09-16T04:45:36.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7b27fc640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8009230 0x7fb7a800b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:36.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7b27fc640 10 monclient(hunting): picked mon.noname-c con 0x7fb7a8009230 addr v2:172.21.15.114:3300/0 2024-09-16T04:45:36.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7b27fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb7bc13c580 addr v1:172.21.15.46:6789/0 2024-09-16T04:45:36.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7b27fc640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8008720 0x7fb7a8005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:36.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7b27fc640 10 monclient(hunting): picked mon.noname-a con 0x7fb7a8008720 addr v2:172.21.15.46:3300/0 2024-09-16T04:45:36.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7b27fc640 1 -- 172.21.15.46:0/1135987635 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7bc0a0eb0 con 0x7fb7bc13c580 2024-09-16T04:45:36.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7b27fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:36.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7b27fc640 1 -- 172.21.15.46:0/1135987635 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb7bc06e540 con 0x7fb7a8008720 2024-09-16T04:45:36.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7b27fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:36.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7b27fc640 1 -- 172.21.15.46:0/1135987635 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fb7a8008de0 con 0x7fb7a8009230 2024-09-16T04:45:36.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7b27fc640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:36.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7c15c4640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8008720 0x7fb7a8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:36.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7c15c4640 10 monclient(hunting): get_auth_request con 0x7fb7a8008720 auth_method 0 2024-09-16T04:45:36.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7c15c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:36.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7c15c4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:36.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7c15c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:36.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8009230 0x7fb7a800b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:36.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7c15c4640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8008720 msgr2=0x7fb7a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:45:36.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7c15c4640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8008720 msgr2=0x7fb7a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:36.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7c15c4640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8008720 0x7fb7a8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:36.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7c0dc3640 10 monclient(hunting): get_auth_request con 0x7fb7a8009230 auth_method 0 2024-09-16T04:45:36.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7c0dc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:36.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7c0dc3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:36.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7c0dc3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:36.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7c15c4640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8008720 0x7fb7a8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:36.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7c0dc3640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8009230 msgr2=0x7fb7a800b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:45:36.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7c0dc3640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8009230 msgr2=0x7fb7a800b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:36.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8009230 0x7fb7a800b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:36.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:36.916+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8009230 0x7fb7a800b660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:37.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.117+0000 7fb7c15c4640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8008720 0x7fb7a8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:37.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.117+0000 7fb7c15c4640 10 monclient(hunting): get_auth_request con 0x7fb7a8008720 auth_method 0 2024-09-16T04:45:37.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.117+0000 7fb7c15c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:37.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.117+0000 7fb7c15c4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:37.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.117+0000 7fb7c15c4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:37.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.117+0000 7fb7c15c4640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8008720 msgr2=0x7fb7a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:45:37.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.117+0000 7fb7c15c4640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8008720 msgr2=0x7fb7a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:37.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.117+0000 7fb7c15c4640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8008720 0x7fb7a8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:37.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.117+0000 7fb7c15c4640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8008720 0x7fb7a8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:45:37.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.117+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8009230 0x7fb7a800b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:37.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.117+0000 7fb7c0dc3640 10 monclient(hunting): get_auth_request con 0x7fb7a8009230 auth_method 0 2024-09-16T04:45:37.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.117+0000 7fb7c0dc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:37.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.117+0000 7fb7c0dc3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:37.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.117+0000 7fb7c0dc3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:37.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.117+0000 7fb7c0dc3640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8009230 msgr2=0x7fb7a800b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:45:37.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.117+0000 7fb7c0dc3640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8009230 msgr2=0x7fb7a800b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:37.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.117+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8009230 0x7fb7a800b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:37.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.117+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8009230 0x7fb7a800b660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:45:37.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.518+0000 7fb7c15c4640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8008720 0x7fb7a8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:37.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.518+0000 7fb7c15c4640 10 monclient(hunting): get_auth_request con 0x7fb7a8008720 auth_method 0 2024-09-16T04:45:37.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.518+0000 7fb7c15c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:37.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.518+0000 7fb7c15c4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:37.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.518+0000 7fb7c15c4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:37.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.518+0000 7fb7c15c4640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8008720 msgr2=0x7fb7a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:45:37.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.518+0000 7fb7c15c4640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8008720 msgr2=0x7fb7a8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:37.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.518+0000 7fb7c15c4640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8008720 0x7fb7a8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:37.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.518+0000 7fb7c15c4640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8008720 0x7fb7a8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:45:37.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.518+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8009230 0x7fb7a800b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:37.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.518+0000 7fb7c0dc3640 10 monclient(hunting): get_auth_request con 0x7fb7a8009230 auth_method 0 2024-09-16T04:45:37.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.518+0000 7fb7c0dc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:37.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.518+0000 7fb7c0dc3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:37.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.518+0000 7fb7c0dc3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:37.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.518+0000 7fb7c0dc3640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8009230 msgr2=0x7fb7a800b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:45:37.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.518+0000 7fb7c0dc3640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8009230 msgr2=0x7fb7a800b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:37.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.518+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8009230 0x7fb7a800b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:37.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:37.518+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8009230 0x7fb7a800b660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:45:38.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.038+0000 7fb7b37fe640 1 -- 172.21.15.46:0/1135987635 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2515184812 0 0) 0x7fb7bc0a0eb0 con 0x7fb7bc13c580 2024-09-16T04:45:38.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.038+0000 7fb7b37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:45:38.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.038+0000 7fb7b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:38.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.038+0000 7fb7b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:38.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.038+0000 7fb7b37fe640 10 monclient(hunting): my global_id is 74478 2024-09-16T04:45:38.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.038+0000 7fb7b37fe640 1 -- 172.21.15.46:0/1135987635 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7a4003520 con 0x7fb7bc13c580 2024-09-16T04:45:38.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 1 -- 172.21.15.46:0/1135987635 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2807902643 0 0) 0x7fb7a4003520 con 0x7fb7bc13c580 2024-09-16T04:45:38.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8009230 msgr2=0x7fb7a800b660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:45:38.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.114:3300/0 conn(0x7fb7a8009230 0x7fb7a800b660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 1 -- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8008720 msgr2=0x7fb7a8005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:45:38.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 1 --2- 172.21.15.46:0/1135987635 >> v2:172.21.15.46:3300/0 conn(0x7fb7a8008720 0x7fb7a8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:45:38.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:45:38.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:38.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:45:38.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 1 -- 172.21.15.46:0/1135987635 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7a800e3e0 con 0x7fb7bc13c580 2024-09-16T04:45:38.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 10 monclient: _finish_auth 0 2024-09-16T04:45:38.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:45:38.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:38.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7c3027640 5 monclient: authenticate success, global_id 74478 2024-09-16T04:45:38.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7c3027640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:45:38.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 1 -- 172.21.15.46:0/1135987635 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7fb7ac009830 con 0x7fb7bc13c580 2024-09-16T04:45:38.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:38.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:45:38.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 10 monclient: dump: 2024-09-16T04:45:38.077 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:38.077 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:38.077 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.077 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.077 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:38.077 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.077 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:38.077 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:38.077 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:38.077 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:45:38.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 1 -- 172.21.15.46:0/1135987635 >> v1:172.21.15.46:6789/0 conn(0x7fb7bc13c580 legacy=0x7fb7a800bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:45:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 1 --2- 172.21.15.46:0/1135987635 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7a40039b0 0x7fb7a4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 10 monclient(hunting): picked mon.c con 0x7fb7a40039b0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 1 --2- 172.21.15.46:0/1135987635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7a4004800 0x7fb7a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 10 monclient(hunting): picked mon.a con 0x7fb7a4004800 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:45:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 1 --2- 172.21.15.46:0/1135987635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a40051c0 0x7fb7a400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 10 monclient(hunting): picked mon.b con 0x7fb7a40051c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:45:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 1 -- 172.21.15.46:0/1135987635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc06e540 con 0x7fb7a4004800 2024-09-16T04:45:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 1 -- 172.21.15.46:0/1135987635 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a8008de0 con 0x7fb7a40051c0 2024-09-16T04:45:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 1 -- 172.21.15.46:0/1135987635 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fb7a400c110 con 0x7fb7a40039b0 2024-09-16T04:45:38.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:38.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:45:38.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7c0dc3640 1 -- 172.21.15.46:0/1135987635 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7a40039b0 msgr2=0x7fb7a4003dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:45:38.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7c3027640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:45:38.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/1135987635 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7a40039b0 0x7fb7a4003dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:38.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b3fff640 1 --2- 172.21.15.46:0/1135987635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7a4004800 0x7fb7a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:38.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b3fff640 10 monclient(hunting): get_auth_request con 0x7fb7a4004800 auth_method 0 2024-09-16T04:45:38.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:38.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:38.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:38.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7c15c4640 1 --2- 172.21.15.46:0/1135987635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a40051c0 0x7fb7a400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:38.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7c15c4640 10 monclient(hunting): get_auth_request con 0x7fb7a40051c0 auth_method 0 2024-09-16T04:45:38.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7c15c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:38.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7c15c4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:38.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7c15c4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:45:38.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:38.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:38.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.072+0000 7fb7b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:45:38.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.073+0000 7fb7c15c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:38.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.073+0000 7fb7c15c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:38.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.073+0000 7fb7c15c4640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:45:38.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.073+0000 7fb7b3fff640 10 monclient(hunting): handle_auth_done global_id 74478 payload 293 2024-09-16T04:45:38.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.073+0000 7fb7b3fff640 1 -- 172.21.15.46:0/1135987635 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7a40039b0 msgr2=0x7fb7a4003dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:45:38.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.073+0000 7fb7b3fff640 1 --2- 172.21.15.46:0/1135987635 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7a40039b0 0x7fb7a4003dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.073+0000 7fb7b3fff640 1 -- 172.21.15.46:0/1135987635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a40051c0 msgr2=0x7fb7a400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:45:38.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.073+0000 7fb7b3fff640 1 --2- 172.21.15.46:0/1135987635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a40051c0 0x7fb7a400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.073+0000 7fb7b3fff640 10 monclient: _finish_hunting 0 2024-09-16T04:45:38.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.073+0000 7fb7b3fff640 1 monclient: found mon.a 2024-09-16T04:45:38.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.073+0000 7fb7b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:38.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.073+0000 7fb7b3fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:38.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.073+0000 7fb7c15c4640 1 -- 172.21.15.46:0/1135987635 reap_dead start 2024-09-16T04:45:38.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.073+0000 7fb7b3fff640 1 -- 172.21.15.46:0/1135987635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7a4003520 con 0x7fb7a4004800 2024-09-16T04:45:38.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.073+0000 7fb7b3fff640 10 monclient: _finish_auth 0 2024-09-16T04:45:38.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.073+0000 7fb7b3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:45:38.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.073+0000 7fb7b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:38.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.074+0000 7fb7b3fff640 1 --2- 172.21.15.46:0/1135987635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7a4004800 0x7fb7a4004c80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8008d50 tx=0x7fb7a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba18cdf0fc90a6be server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:38.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.075+0000 7fb7b37fe640 1 -- 172.21.15.46:0/1135987635 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac031070 con 0x7fb7a4004800 2024-09-16T04:45:38.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.075+0000 7fb7b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:38.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.075+0000 7fb7b37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:38.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.075+0000 7fb7b37fe640 10 monclient: dump: 2024-09-16T04:45:38.090 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:38.090 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:38.090 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.090 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.091 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:38.091 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.091 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:38.091 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:38.091 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:38.091 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.075+0000 7fb7c3027640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:45:38.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.075+0000 7fb7b37fe640 1 -- 172.21.15.46:0/1135987635 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7ac035400 con 0x7fb7a4004800 2024-09-16T04:45:38.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.075+0000 7fb7b37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:38.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.075+0000 7fb7b37fe640 1 -- 172.21.15.46:0/1135987635 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac00b410 con 0x7fb7a4004800 2024-09-16T04:45:38.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.075+0000 7fb7c3027640 10 monclient: get_monmap_and_config success 2024-09-16T04:45:38.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.075+0000 7fb7c3027640 4 set_mon_vals no callback set 2024-09-16T04:45:38.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.075+0000 7fb7b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:38.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.075+0000 7fb7b37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:38.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.075+0000 7fb7b37fe640 10 monclient: dump: 2024-09-16T04:45:38.092 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:38.092 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:38.092 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.092 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.092 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:38.092 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.092 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:38.092 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:38.092 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:38.092 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.076+0000 7fb7c3027640 10 monclient: shutdown 2024-09-16T04:45:38.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.076+0000 7fb7c3027640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:38.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.076+0000 7fb7c3027640 1 -- 172.21.15.46:0/1135987635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7a4004800 msgr2=0x7fb7a4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:38.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.076+0000 7fb7c3027640 1 --2- 172.21.15.46:0/1135987635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7a4004800 0x7fb7a4004c80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8008d50 tx=0x7fb7a4005830 comp rx=0 tx=0).stop 2024-09-16T04:45:38.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.076+0000 7fb7c3027640 1 -- 172.21.15.46:0/1135987635 shutdown_connections 2024-09-16T04:45:38.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.076+0000 7fb7c3027640 1 --2- 172.21.15.46:0/1135987635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7a4004800 0x7fb7a4004c80 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.076+0000 7fb7c3027640 1 -- 172.21.15.46:0/1135987635 >> 172.21.15.46:0/1135987635 conn(0x7fb7bc0a5c10 msgr2=0x7fb7bc0a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:38.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.076+0000 7fb7c3027640 1 -- 172.21.15.46:0/1135987635 shutdown_connections 2024-09-16T04:45:38.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.076+0000 7fb7c3027640 1 -- 172.21.15.46:0/1135987635 wait complete. 2024-09-16T04:45:38.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.077+0000 7fb7c3027640 10 monclient: build_initial_monmap 2024-09-16T04:45:38.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.077+0000 7fb7c3027640 10 monclient: monmap: 2024-09-16T04:45:38.093 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:38.093 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:45:38.093 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:45:38.093 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:45:38.093 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:38.094 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.094 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:45:38.094 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:45:38.094 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:45:38.094 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.077+0000 7fb7c3027640 1 Processor -- start 2024-09-16T04:45:38.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.077+0000 7fb7c3027640 1 -- start start 2024-09-16T04:45:38.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.077+0000 7fb7c3027640 10 monclient: init 2024-09-16T04:45:38.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c3027640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:38.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c3027640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:45:38.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c3027640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a4004800 0x7fb7bc09e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c3027640 10 monclient(hunting): picked mon.noname-b con 0x7fb7a4004800 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:45:38.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c3027640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7bc135a10 0x7fb7bc135df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c3027640 10 monclient(hunting): picked mon.noname-c con 0x7fb7bc135a10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:38.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c3027640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7bc11f760 0x7fb7bc11fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c3027640 10 monclient(hunting): picked mon.noname-a con 0x7fb7bc11f760 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:45:38.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c3027640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:38.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c3027640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc0a34a0 con 0x7fb7bc11f760 2024-09-16T04:45:38.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c3027640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:38.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c3027640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc0a11f0 con 0x7fb7a4004800 2024-09-16T04:45:38.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c3027640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:38.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c3027640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fb7bc1223c0 con 0x7fb7bc135a10 2024-09-16T04:45:38.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c3027640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:38.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c3027640 10 monclient(hunting): authenticate will time out at 1136.493581s 2024-09-16T04:45:38.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7b3fff640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7bc135a10 msgr2=0x7fb7bc135df0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:45:38.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7b3fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7bc135a10 0x7fb7bc135df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:38.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c15c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7bc11f760 0x7fb7bc11fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:38.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c15c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7bc11f760 0x7fb7bc11fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55534/0 (socket says 172.21.15.46:55534) 2024-09-16T04:45:38.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c15c4640 1 -- 172.21.15.46:0/3653721671 learned_addr learned my addr 172.21.15.46:0/3653721671 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:38.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c15c4640 10 monclient(hunting): get_auth_request con 0x7fb7bc11f760 auth_method 0 2024-09-16T04:45:38.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c15c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:38.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c15c4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:38.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c15c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:38.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/3653721671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a4004800 0x7fb7bc09e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:38.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c15c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:38.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c15c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:38.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c15c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:38.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c0dc3640 10 monclient(hunting): get_auth_request con 0x7fb7a4004800 auth_method 0 2024-09-16T04:45:38.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c0dc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:38.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c0dc3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:38.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c0dc3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:38.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c0dc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:38.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c0dc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:38.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c0dc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:38.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.078+0000 7fb7c15c4640 10 monclient(hunting): handle_auth_done global_id 74481 payload 824 2024-09-16T04:45:38.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7c15c4640 1 -- 172.21.15.46:0/3653721671 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7bc135a10 msgr2=0x7fb7bc135df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:45:38.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7c15c4640 1 --2- 172.21.15.46:0/3653721671 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7bc135a10 0x7fb7bc135df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7c15c4640 1 -- 172.21.15.46:0/3653721671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a4004800 msgr2=0x7fb7bc09e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:38.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7c15c4640 1 --2- 172.21.15.46:0/3653721671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a4004800 0x7fb7bc09e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7c15c4640 10 monclient: _finish_hunting 0 2024-09-16T04:45:38.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7c15c4640 1 monclient: found mon.noname-a 2024-09-16T04:45:38.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7c15c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:38.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7c15c4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:38.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7c15c4640 1 -- 172.21.15.46:0/3653721671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7bc136330 con 0x7fb7bc11f760 2024-09-16T04:45:38.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7c15c4640 1 --2- 172.21.15.46:0/3653721671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7bc11f760 0x7fb7bc11fbf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc0a5a00 tx=0x7fb7b40078f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:38.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7b17fa640 1 -- 172.21.15.46:0/3653721671 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b401b030 con 0x7fb7bc11f760 2024-09-16T04:45:38.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:38.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7b17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:38.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7b17fa640 10 monclient: dump: 2024-09-16T04:45:38.098 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:38.098 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:38.098 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.098 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.098 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:38.098 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.098 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:38.098 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:38.098 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:38.099 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7b17fa640 10 monclient: _finish_auth 0 2024-09-16T04:45:38.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7b17fa640 10 monclient: _check_auth_tickets 2024-09-16T04:45:38.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:38.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7c3027640 5 monclient: authenticate success, global_id 74481 2024-09-16T04:45:38.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7b17fa640 1 -- 172.21.15.46:0/3653721671 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7b40149c0 con 0x7fb7bc11f760 2024-09-16T04:45:38.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7b17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:38.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7b17fa640 1 -- 172.21.15.46:0/3653721671 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b4014bb0 con 0x7fb7bc11f760 2024-09-16T04:45:38.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7c3027640 10 monclient: _renew_subs 2024-09-16T04:45:38.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7c3027640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:38.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7c3027640 1 -- 172.21.15.46:0/3653721671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7bc1cbd00 con 0x7fb7bc11f760 2024-09-16T04:45:38.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:38.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7b17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:38.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7b17fa640 10 monclient: dump: 2024-09-16T04:45:38.100 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:38.100 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:38.100 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.100 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.100 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:38.100 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.100 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:38.100 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:38.100 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:38.100 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7b2ffd640 4 set_mon_vals no callback set 2024-09-16T04:45:38.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7c3027640 10 monclient: _renew_subs 2024-09-16T04:45:38.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7c3027640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:38.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.079+0000 7fb7c3027640 1 -- 172.21.15.46:0/3653721671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7bc121b20 con 0x7fb7bc11f760 2024-09-16T04:45:38.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.080+0000 7fb7b17fa640 1 -- 172.21.15.46:0/3653721671 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fb7b401e030 con 0x7fb7bc11f760 2024-09-16T04:45:38.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.080+0000 7fb7c3027640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:38.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.080+0000 7fb7c3027640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:38.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.080+0000 7fb7c3027640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:38.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.080+0000 7fb7c3027640 1 -- 172.21.15.46:0/3653721671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7bc0a29a0 con 0x7fb7bc11f760 2024-09-16T04:45:38.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.080+0000 7fb7b17fa640 1 --2- 172.21.15.46:0/3653721671 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb7a8039c20 0x7fb7a803c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.080+0000 7fb7b17fa640 1 -- 172.21.15.46:0/3653721671 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 4346+0+0 (secure 0 0 0) 0x7fb7b404bc90 con 0x7fb7bc11f760 2024-09-16T04:45:38.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.081+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/3653721671 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb7a8039c20 0x7fb7a803c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:38.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.081+0000 7fb7c0dc3640 10 monclient: get_auth_request con 0x7fb7a8039c20 auth_method 0 2024-09-16T04:45:38.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.081+0000 7fb7c0dc3640 1 --2- 172.21.15.46:0/3653721671 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb7a8039c20 0x7fb7a803c0e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8007c30 tx=0x7fb7b8019040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:38.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.086+0000 7fb7b17fa640 1 -- 172.21.15.46:0/3653721671 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb7bc136330 con 0x7fb7bc11f760 2024-09-16T04:45:38.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.086+0000 7fb7b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:38.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.086+0000 7fb7b17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:45:38.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.332+0000 7fb7c3027640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:45:38.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.332+0000 7fb7c3027640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:45:38.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.332+0000 7fb7c3027640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:38.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.332+0000 7fb7c3027640 1 -- 172.21.15.46:0/3653721671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb7bc067e00 con 0x7fb7bc11f760 2024-09-16T04:45:38.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.332+0000 7fb7b17fa640 1 -- 172.21.15.46:0/3653721671 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fb7bc0a3fd0 con 0x7fb7bc11f760 2024-09-16T04:45:38.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.332+0000 7fb7b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:45:38.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.332+0000 7fb7b17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:45:38.334 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":82,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:45:38.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.334+0000 7fb7c3027640 1 -- 172.21.15.46:0/3653721671 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb7a8039c20 msgr2=0x7fb7a803c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:38.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.334+0000 7fb7c3027640 1 --2- 172.21.15.46:0/3653721671 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb7a8039c20 0x7fb7a803c0e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8007c30 tx=0x7fb7b8019040 comp rx=0 tx=0).stop 2024-09-16T04:45:38.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.334+0000 7fb7c3027640 10 monclient: shutdown 2024-09-16T04:45:38.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.334+0000 7fb7c3027640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:38.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.334+0000 7fb7c3027640 1 -- 172.21.15.46:0/3653721671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7bc11f760 msgr2=0x7fb7bc11fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:38.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.334+0000 7fb7c3027640 1 --2- 172.21.15.46:0/3653721671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7bc11f760 0x7fb7bc11fbf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc0a5a00 tx=0x7fb7b40078f0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.335+0000 7fb7c3027640 1 -- 172.21.15.46:0/3653721671 shutdown_connections 2024-09-16T04:45:38.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.335+0000 7fb7c3027640 1 --2- 172.21.15.46:0/3653721671 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb7a8039c20 0x7fb7a803c0e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.335+0000 7fb7c3027640 1 --2- 172.21.15.46:0/3653721671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7bc11f760 0x7fb7bc11fbf0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.335+0000 7fb7c3027640 1 --2- 172.21.15.46:0/3653721671 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fb7bc135a10 0x7fb7bc135df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.335+0000 7fb7c3027640 1 --2- 172.21.15.46:0/3653721671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a4004800 0x7fb7bc09e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.335+0000 7fb7c3027640 1 -- 172.21.15.46:0/3653721671 >> 172.21.15.46:0/3653721671 conn(0x7fb7bc0a5c10 msgr2=0x7fb7bc1392d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:38.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.335+0000 7fb7c3027640 1 -- 172.21.15.46:0/3653721671 shutdown_connections 2024-09-16T04:45:38.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.335+0000 7fb7c3027640 1 -- 172.21.15.46:0/3653721671 wait complete. 2024-09-16T04:45:38.347 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:45:38.347 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:45:38.347 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:45:38.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.448+0000 7f15fe911640 10 monclient: get_monmap_and_config 2024-09-16T04:45:38.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.448+0000 7f15fe911640 10 monclient: build_initial_monmap 2024-09-16T04:45:38.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.448+0000 7f15fe911640 10 monclient: monmap: 2024-09-16T04:45:38.449 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:38.449 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:38.450 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:45:38.448772+0000 2024-09-16T04:45:38.450 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:45:38.448772+0000 2024-09-16T04:45:38.450 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:38.450 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.450 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:45:38.450 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:45:38.450 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:45:38.450 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:45:38.450 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:45:38.450 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.449+0000 7f15fe911640 1 Processor -- start 2024-09-16T04:45:38.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.449+0000 7f15fe911640 1 -- start start 2024-09-16T04:45:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.449+0000 7f15fe911640 10 monclient: init 2024-09-16T04:45:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.449+0000 7f15fe911640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.449+0000 7f15fe911640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-16T04:45:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.449+0000 7f15fe911640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15f80a3c10 0x7f15f809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.449+0000 7f15fe911640 10 monclient(hunting): picked mon.noname-b con 0x7f15f80a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.449+0000 7f15fe911640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f15f80a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:45:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.449+0000 7f15fe911640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f15f813be90 0x7f15f813e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.449+0000 7f15fe911640 10 monclient(hunting): picked mon.noname-c con 0x7f15f813be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:45:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.449+0000 7f15fe911640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15f809b940 con 0x7f15f80a3ff0 2024-09-16T04:45:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.449+0000 7f15fe911640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.449+0000 7f15fe911640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f15f809ab00 con 0x7f15f813be90 2024-09-16T04:45:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.449+0000 7f15fe911640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.449+0000 7f15fe911640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f15f809b030 con 0x7f15f80a3c10 2024-09-16T04:45:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.449+0000 7f15fe911640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.449+0000 7f15fe911640 10 monclient(hunting): authenticate will time out at 1136.864613s 2024-09-16T04:45:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.449+0000 7f15f7fff640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15f80a3c10 msgr2=0x7f15f809bcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:45:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.449+0000 7f15f7fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15f80a3c10 0x7f15f809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.449+0000 7f15fceae640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f15f813be90 0x7f15f813e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.449+0000 7f15f77fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f15f80a3ff0 0x7f15f80a43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.46:33316/0 (socket says 172.21.15.46:33316) 2024-09-16T04:45:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.450+0000 7f15f77fe640 1 -- 172.21.15.46:0/1284665985 learned_addr learned my addr 172.21.15.46:0/1284665985 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:45:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.450+0000 7f15fceae640 1 --2- 172.21.15.46:0/1284665985 >> v2:172.21.15.114:3300/0 conn(0x7f15f813be90 0x7f15f813e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.46:40000/0 (socket says 172.21.15.46:40000) 2024-09-16T04:45:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.450+0000 7f15fceae640 10 monclient(hunting): get_auth_request con 0x7f15f813be90 auth_method 0 2024-09-16T04:45:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.450+0000 7f15fceae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.450+0000 7f15fceae640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.450+0000 7f15fceae640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.450+0000 7f15fceae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.450+0000 7f15fceae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.450+0000 7f15fceae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.450+0000 7f15fceae640 10 monclient(hunting): handle_auth_done global_id 6550 payload 274 2024-09-16T04:45:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.450+0000 7f15fceae640 1 -- 172.21.15.46:0/1284665985 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15f80a3c10 msgr2=0x7f15f809bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:45:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.450+0000 7f15fceae640 1 --2- 172.21.15.46:0/1284665985 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15f80a3c10 0x7f15f809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.450+0000 7f15fceae640 1 -- 172.21.15.46:0/1284665985 >> v1:172.21.15.114:6789/0 conn(0x7f15f80a3ff0 legacy=0x7f15f80a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.450+0000 7f15fceae640 10 monclient: _finish_hunting 0 2024-09-16T04:45:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.450+0000 7f15fceae640 1 monclient: found mon.noname-c 2024-09-16T04:45:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.450+0000 7f15fceae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.450+0000 7f15fceae640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:45:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.450+0000 7f15fceae640 1 -- 172.21.15.46:0/1284665985 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15f809b310 con 0x7f15f813be90 2024-09-16T04:45:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15fceae640 1 --2- 172.21.15.46:0/1284665985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15f813be90 0x7f15f813e280 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009d90 tx=0x7f15f002a560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f824e03c196b5603 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15f6ffd640 1 -- 172.21.15.46:0/1284665985 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f0007740 con 0x7f15f813be90 2024-09-16T04:45:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:45:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15f6ffd640 10 monclient: dump: 2024-09-16T04:45:38.463 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:38.463 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:38.463 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.463 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.463 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:38.463 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.463 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:38.463 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:38.463 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:38.463 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15f6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:45:38.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15f6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:45:38.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:38.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15f6ffd640 1 -- 172.21.15.46:0/1284665985 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15f00078e0 con 0x7f15f813be90 2024-09-16T04:45:38.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15fe911640 5 monclient: authenticate success, global_id 6550 2024-09-16T04:45:38.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15fe911640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:45:38.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15f6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:38.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15fe911640 10 monclient: get_monmap_and_config success 2024-09-16T04:45:38.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15f6ffd640 1 -- 172.21.15.46:0/1284665985 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f0007cb0 con 0x7f15f813be90 2024-09-16T04:45:38.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:38.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15fe911640 4 set_mon_vals no callback set 2024-09-16T04:45:38.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:45:38.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15f6ffd640 10 monclient: dump: 2024-09-16T04:45:38.464 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:38.464 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:38.464 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.464 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.464 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:38.465 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.465 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:38.465 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:38.465 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:38.465 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15fe911640 10 monclient: shutdown 2024-09-16T04:45:38.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15fe911640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:38.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15fe911640 1 -- 172.21.15.46:0/1284665985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15f813be90 msgr2=0x7f15f813e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:38.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15fe911640 1 --2- 172.21.15.46:0/1284665985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15f813be90 0x7f15f813e280 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009d90 tx=0x7f15f002a560 comp rx=0 tx=0).stop 2024-09-16T04:45:38.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15fe911640 1 -- 172.21.15.46:0/1284665985 shutdown_connections 2024-09-16T04:45:38.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15fe911640 1 --2- 172.21.15.46:0/1284665985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15f813be90 0x7f15f813e280 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15fe911640 1 --2- 172.21.15.46:0/1284665985 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15f80a3c10 0x7f15f809bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15fe911640 1 -- 172.21.15.46:0/1284665985 >> 172.21.15.46:0/1284665985 conn(0x7f15f80a6340 msgr2=0x7f15f806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:38.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15fe911640 1 -- 172.21.15.46:0/1284665985 shutdown_connections 2024-09-16T04:45:38.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.451+0000 7f15fe911640 1 -- 172.21.15.46:0/1284665985 wait complete. 2024-09-16T04:45:38.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.452+0000 7f15fe911640 10 monclient: build_initial_monmap 2024-09-16T04:45:38.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.452+0000 7f15fe911640 10 monclient: monmap: 2024-09-16T04:45:38.466 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:38.466 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:45:38.466 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:45:38.466 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:45:38.466 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:38.466 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.466 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:45:38.466 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:45:38.466 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:45:38.466 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.452+0000 7f15fe911640 1 Processor -- start 2024-09-16T04:45:38.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.452+0000 7f15fe911640 1 -- start start 2024-09-16T04:45:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.452+0000 7f15fe911640 10 monclient: init 2024-09-16T04:45:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15fe911640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15fe911640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:45:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15fe911640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15f80a3c10 0x7f15f81cba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15fe911640 10 monclient(hunting): picked mon.noname-b con 0x7f15f80a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:45:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15fe911640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f80a3ff0 0x7f15f81cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15fe911640 10 monclient(hunting): picked mon.noname-a con 0x7f15f80a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:45:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15fe911640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15f813be90 0x7f15f81cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15fe911640 10 monclient(hunting): picked mon.noname-c con 0x7f15f813be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15fe911640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15fe911640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f15f80a6b40 con 0x7f15f80a3ff0 2024-09-16T04:45:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15fe911640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15fe911640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f15f80a69c0 con 0x7f15f80a3c10 2024-09-16T04:45:38.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15fe911640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:38.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15fe911640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f15f813e7c0 con 0x7f15f813be90 2024-09-16T04:45:38.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15fe911640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:38.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15fe911640 10 monclient(hunting): authenticate will time out at 1136.868413s 2024-09-16T04:45:38.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15fceae640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15f813be90 msgr2=0x7f15f81cc4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:45:38.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15fceae640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15f813be90 0x7f15f81cc4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:38.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f80a3ff0 0x7f15f81cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:38.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f80a3ff0 0x7f15f81cbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55546/0 (socket says 172.21.15.46:55546) 2024-09-16T04:45:38.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f77fe640 1 -- 172.21.15.46:0/2571109544 learned_addr learned my addr 172.21.15.46:0/2571109544 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:38.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f77fe640 10 monclient(hunting): get_auth_request con 0x7f15f80a3ff0 auth_method 0 2024-09-16T04:45:38.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:38.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:38.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:38.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:38.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:38.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:38.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f7fff640 1 --2- 172.21.15.46:0/2571109544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15f80a3c10 0x7f15f81cba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:38.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f7fff640 10 monclient(hunting): get_auth_request con 0x7f15f80a3c10 auth_method 0 2024-09-16T04:45:38.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:38.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:38.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:38.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:38.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:38.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:38.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f77fe640 10 monclient(hunting): handle_auth_done global_id 74484 payload 824 2024-09-16T04:45:38.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f77fe640 1 -- 172.21.15.46:0/2571109544 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15f813be90 msgr2=0x7f15f81cc4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:45:38.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f77fe640 1 --2- 172.21.15.46:0/2571109544 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f15f813be90 0x7f15f81cc4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f77fe640 1 -- 172.21.15.46:0/2571109544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15f80a3c10 msgr2=0x7f15f81cba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:38.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f77fe640 1 --2- 172.21.15.46:0/2571109544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15f80a3c10 0x7f15f81cba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:45:38.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f77fe640 1 monclient: found mon.noname-a 2024-09-16T04:45:38.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:38.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:38.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.453+0000 7f15f77fe640 1 -- 172.21.15.46:0/2571109544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15f8139940 con 0x7f15f80a3ff0 2024-09-16T04:45:38.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15f77fe640 1 --2- 172.21.15.46:0/2571109544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f80a3ff0 0x7f15f81cbf90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f15ec00c470 tx=0x7f15ec001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:38.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15f4ff9640 1 -- 172.21.15.46:0/2571109544 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ec017070 con 0x7f15f80a3ff0 2024-09-16T04:45:38.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:38.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15f4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:38.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15f4ff9640 10 monclient: dump: 2024-09-16T04:45:38.470 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:38.470 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:38.470 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.470 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.471 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:38.471 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.471 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:38.471 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:38.471 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:38.471 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15f4ff9640 10 monclient: _finish_auth 0 2024-09-16T04:45:38.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:45:38.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:38.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15f4ff9640 1 -- 172.21.15.46:0/2571109544 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15ec005ce0 con 0x7f15f80a3ff0 2024-09-16T04:45:38.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15fe911640 5 monclient: authenticate success, global_id 74484 2024-09-16T04:45:38.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15fe911640 10 monclient: _renew_subs 2024-09-16T04:45:38.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15fe911640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:38.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15fe911640 1 -- 172.21.15.46:0/2571109544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15f8139c20 con 0x7f15f80a3ff0 2024-09-16T04:45:38.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15f4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15f4ff9640 1 -- 172.21.15.46:0/2571109544 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ec0072f0 con 0x7f15f80a3ff0 2024-09-16T04:45:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15f4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15f4ff9640 10 monclient: dump: 2024-09-16T04:45:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:38.472 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15fe911640 10 monclient: _renew_subs 2024-09-16T04:45:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15fe911640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15f67fc640 4 set_mon_vals no callback set 2024-09-16T04:45:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.454+0000 7f15fe911640 1 -- 172.21.15.46:0/2571109544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15f8139660 con 0x7f15f80a3ff0 2024-09-16T04:45:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.455+0000 7f15f4ff9640 1 -- 172.21.15.46:0/2571109544 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f15ec01a070 con 0x7f15f80a3ff0 2024-09-16T04:45:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.455+0000 7f15fe911640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.455+0000 7f15fe911640 1 --2- 172.21.15.46:0/2571109544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f81339a0 0x7f15f8133d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.455+0000 7f15fe911640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.455+0000 7f15fe911640 1 -- 172.21.15.46:0/2571109544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f15f809bd70 con 0x7f15f81339a0 2024-09-16T04:45:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.455+0000 7f15f7fff640 1 --2- 172.21.15.46:0/2571109544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f81339a0 0x7f15f8133d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.455+0000 7f15f4ff9640 1 --2- 172.21.15.46:0/2571109544 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f15e4038360 0x7f15e403a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.455+0000 7f15f7fff640 10 monclient: get_auth_request con 0x7f15f81339a0 auth_method 0 2024-09-16T04:45:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.455+0000 7f15f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.455+0000 7f15f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:38.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.455+0000 7f15f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:38.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.455+0000 7f15f4ff9640 1 -- 172.21.15.46:0/2571109544 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 4346+0+0 (secure 0 0 0) 0x7f15ec04a990 con 0x7f15f80a3ff0 2024-09-16T04:45:38.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.455+0000 7f15f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:38.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.455+0000 7f15f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:38.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.455+0000 7f15f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:38.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.455+0000 7f15fceae640 1 --2- 172.21.15.46:0/2571109544 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f15e4038360 0x7f15e403a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:38.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.455+0000 7f15fceae640 10 monclient: get_auth_request con 0x7f15e4038360 auth_method 0 2024-09-16T04:45:38.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.455+0000 7f15f7fff640 10 monclient(hunting): handle_auth_done global_id 74487 payload 824 2024-09-16T04:45:38.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.455+0000 7f15f7fff640 1 -- 172.21.15.46:0/2571109544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f15f80a0400 con 0x7f15f81339a0 2024-09-16T04:45:38.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.455+0000 7f15f7fff640 1 --2- 172.21.15.46:0/2571109544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f81339a0 0x7f15f8133d80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f15e8003980 tx=0x7f15e8009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:38.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.456+0000 7f15f4ff9640 1 -- 172.21.15.46:0/2571109544 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e8016070 con 0x7f15f81339a0 2024-09-16T04:45:38.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.456+0000 7f15f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:38.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.456+0000 7f15f4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:38.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.456+0000 7f15f4ff9640 10 monclient: dump: 2024-09-16T04:45:38.475 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:38.475 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:38.475 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.475 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.475 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:38.475 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.475 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:38.475 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:38.475 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:38.475 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.456+0000 7f15fceae640 1 --2- 172.21.15.46:0/2571109544 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f15e4038360 0x7f15e403a820 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f15f0034000 tx=0x7f15f002f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:38.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.456+0000 7f15f4ff9640 1 -- 172.21.15.46:0/2571109544 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f15e8012600 con 0x7f15f81339a0 2024-09-16T04:45:38.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.456+0000 7f15f4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:38.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.456+0000 7f15f4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:45:38.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.456+0000 7f15f4ff9640 1 -- 172.21.15.46:0/2571109544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f81339a0 msgr2=0x7f15f8133d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:38.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.456+0000 7f15f4ff9640 1 --2- 172.21.15.46:0/2571109544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f81339a0 0x7f15f8133d80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f15e8003980 tx=0x7f15e8009b00 comp rx=0 tx=0).stop 2024-09-16T04:45:38.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.456+0000 7f15f4ff9640 1 -- 172.21.15.46:0/2571109544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f81339a0 msgr2=0x7f15f8133d80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:45:38.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.456+0000 7f15f4ff9640 1 --2- 172.21.15.46:0/2571109544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f81339a0 0x7f15f8133d80 secure :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f15e8003980 tx=0x7f15e8009b00 comp rx=0 tx=0).stop 2024-09-16T04:45:38.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.462+0000 7f15fe911640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:45:38.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.462+0000 7f15fe911640 1 --2- 172.21.15.46:0/2571109544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f81342c0 0x7f15f8134bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.462+0000 7f15fe911640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:38.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.462+0000 7f15fe911640 1 -- 172.21.15.46:0/2571109544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f15f81384a0 con 0x7f15f81342c0 2024-09-16T04:45:38.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.462+0000 7f15f7fff640 1 --2- 172.21.15.46:0/2571109544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f81342c0 0x7f15f8134bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:38.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.462+0000 7f15f7fff640 10 monclient: get_auth_request con 0x7f15f81342c0 auth_method 0 2024-09-16T04:45:38.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.462+0000 7f15f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:38.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.462+0000 7f15f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:38.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.462+0000 7f15f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:38.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.462+0000 7f15f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:38.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.462+0000 7f15f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:38.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.462+0000 7f15f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:38.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.462+0000 7f15f7fff640 10 monclient(hunting): handle_auth_done global_id 74490 payload 824 2024-09-16T04:45:38.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.462+0000 7f15f7fff640 1 -- 172.21.15.46:0/2571109544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f15f8067e20 con 0x7f15f81342c0 2024-09-16T04:45:38.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15f7fff640 1 --2- 172.21.15.46:0/2571109544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f81342c0 0x7f15f8134bc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f15e8000c00 tx=0x7f15e80059d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:38.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15f4ff9640 1 -- 172.21.15.46:0/2571109544 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e8016040 con 0x7f15f81342c0 2024-09-16T04:45:38.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:38.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15f4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:38.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15f4ff9640 10 monclient: dump: 2024-09-16T04:45:38.479 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:38.479 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:38.479 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.479 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.479 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:38.479 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.479 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:38.480 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:38.480 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:38.480 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15f4ff9640 1 -- 172.21.15.46:0/2571109544 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4702 (secure 0 0 0) 0x7f15e801dac0 con 0x7f15f81342c0 2024-09-16T04:45:38.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15f4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:45:38.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15f4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:45:38.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15f4ff9640 1 -- 172.21.15.46:0/2571109544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f81342c0 msgr2=0x7f15f8134bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:38.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15f4ff9640 1 --2- 172.21.15.46:0/2571109544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f81342c0 0x7f15f8134bc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f15e8000c00 tx=0x7f15e80059d0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15f4ff9640 1 -- 172.21.15.46:0/2571109544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f81342c0 msgr2=0x7f15f8134bc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:45:38.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15f4ff9640 1 --2- 172.21.15.46:0/2571109544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f81342c0 0x7f15f8134bc0 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f15e8000c00 tx=0x7f15e80059d0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.480 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:45:38.481 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:45:38.481 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:45:38.481 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:45:38.481 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 82, 2024-09-16T04:45:38.481 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 78459, 2024-09-16T04:45:38.481 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:45:38.481 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:45:38.481 INFO:teuthology.orchestra.run.smithi046.stdout: 1 2024-09-16T04:45:38.481 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:38.481 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:45:38.481 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:45:38.481 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:45:38.481 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:45:38.482 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:38.482 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:38.482 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:38.482 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:38.482 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:38.482 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:38.482 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:38.482 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:38.482 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:38.482 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:38.482 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:38.482 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:38.482 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:45:38.482 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:45:38.483 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:38.483 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:38.483 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:38.483 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:38.483 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:38.483 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:38.483 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:38.483 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:38.483 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:38.483 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:38.483 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:38.483 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:38.483 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.483 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:45:38.484 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:45:38.484 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:45:38.484 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:45:38.484 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:45:38.484 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:45:38.484 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:45:38.484 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:45:38.484 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:45:38.484 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:45:38.484 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:45:38.484 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:45:38.484 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:45:38.484 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:45:38.484 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:45:38.485 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:45:38.485 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:45:38.485 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:38.485 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:38.485 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:38.485 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:38.485 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:38.485 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:38.485 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:38.485 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:38.485 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:38.485 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:38.485 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:38.486 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:38.486 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:45:38.486 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.486 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:45:38.486 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.486 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:45:38.486 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:45:38.486 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:38.486 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:38.486 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.486 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:38.486 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:45:38.486 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:38.486 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.487 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.487 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:38.487 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:45:38.487 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:38.487 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:38.487 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:38.487 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.487 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:45:38.487 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:45:38.487 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:38.487 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:38.487 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:38.487 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.487 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.488 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:45:38.488 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:45:38.488 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:38.488 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:38.488 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.488 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:38.488 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:45:38.488 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:38.488 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.488 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.488 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:38.488 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:45:38.488 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:38.489 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:38.489 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:38.489 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.489 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:45:38.489 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:45:38.489 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:38.489 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:38.489 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:38.489 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.489 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.489 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:45:38.489 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:45:38.489 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:38.489 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:38.489 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.490 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:38.490 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:45:38.490 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:38.490 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.490 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.490 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:38.490 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:45:38.490 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:38.490 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:38.490 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:38.490 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.490 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:45:38.490 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:45:38.490 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:38.491 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:38.491 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:38.491 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:38.491 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:38.491 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.491 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:45:38.491 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:45:38.491 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.491 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:38.491 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:38.491 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:45:38.491 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:38.491 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:38.491 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:45:38.492 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.492 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:38.492 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:38.492 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:45:38.492 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:38.492 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:38.492 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:45:38.492 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.492 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:38.492 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:38.492 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:45:38.492 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:38.492 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:38.492 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.493 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:45:38.493 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:45:38.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15fe911640 1 -- 172.21.15.46:0/2571109544 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f15e4038360 msgr2=0x7f15e403a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:38.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15fe911640 1 --2- 172.21.15.46:0/2571109544 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f15e4038360 0x7f15e403a820 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f15f0034000 tx=0x7f15f002f000 comp rx=0 tx=0).stop 2024-09-16T04:45:38.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15fe911640 10 monclient: shutdown 2024-09-16T04:45:38.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15fe911640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:38.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15fe911640 1 -- 172.21.15.46:0/2571109544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f80a3ff0 msgr2=0x7f15f81cbf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:38.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15fe911640 1 --2- 172.21.15.46:0/2571109544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f80a3ff0 0x7f15f81cbf90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f15ec00c470 tx=0x7f15ec001e70 comp rx=0 tx=0).stop 2024-09-16T04:45:38.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15fceae640 1 -- 172.21.15.46:0/2571109544 reap_dead start 2024-09-16T04:45:38.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15fe911640 1 -- 172.21.15.46:0/2571109544 shutdown_connections 2024-09-16T04:45:38.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15fe911640 1 --2- 172.21.15.46:0/2571109544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f80a3ff0 0x7f15f81cbf90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15fe911640 1 -- 172.21.15.46:0/2571109544 >> 172.21.15.46:0/2571109544 conn(0x7f15f80a6340 msgr2=0x7f15f813b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:38.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15fe911640 1 -- 172.21.15.46:0/2571109544 shutdown_connections 2024-09-16T04:45:38.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.463+0000 7f15fe911640 1 -- 172.21.15.46:0/2571109544 wait complete. 2024-09-16T04:45:38.497 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:45:38.497 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:45:38.498 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:45:38.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.599+0000 7ff6c65fd640 10 monclient: get_monmap_and_config 2024-09-16T04:45:38.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.599+0000 7ff6c65fd640 10 monclient: build_initial_monmap 2024-09-16T04:45:38.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.599+0000 7ff6c65fd640 10 monclient: monmap: 2024-09-16T04:45:38.600 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:38.600 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:38.600 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:45:38.599958+0000 2024-09-16T04:45:38.601 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:45:38.599958+0000 2024-09-16T04:45:38.601 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:38.601 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.601 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:45:38.601 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:45:38.601 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:45:38.601 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:45:38.601 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:45:38.601 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6c65fd640 1 Processor -- start 2024-09-16T04:45:38.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6c65fd640 1 -- start start 2024-09-16T04:45:38.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6c65fd640 10 monclient: init 2024-09-16T04:45:38.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6c65fd640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:38.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6c65fd640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-16T04:45:38.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6c65fd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff6c006e160 0x7ff6c00a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6c65fd640 10 monclient(hunting): picked mon.noname-b con 0x7ff6c006e160 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:38.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6c65fd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff6c00a2ee0 0x7ff6c0136940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6c65fd640 10 monclient(hunting): picked mon.noname-a con 0x7ff6c00a2ee0 addr v2:172.21.15.46:3300/0 2024-09-16T04:45:38.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6c65fd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff6c0136e80 0x7ff6c0137330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6c65fd640 10 monclient(hunting): picked mon.noname-c con 0x7ff6c0136e80 addr v2:172.21.15.114:3300/0 2024-09-16T04:45:38.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6c65fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:38.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6c65fd640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff6c00a3300 con 0x7ff6c00a2ee0 2024-09-16T04:45:38.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6c65fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:38.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6c65fd640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ff6c006e540 con 0x7ff6c0136e80 2024-09-16T04:45:38.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6c65fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:38.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6c65fd640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7ff6c00a3fd0 con 0x7ff6c006e160 2024-09-16T04:45:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6c65fd640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6c65fd640 10 monclient(hunting): authenticate will time out at 1137.015800s 2024-09-16T04:45:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6bffff640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff6c006e160 msgr2=0x7ff6c00a29a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:45:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6bffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff6c006e160 0x7ff6c00a29a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6bf7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff6c00a2ee0 0x7ff6c0136940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6bf7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff6c00a2ee0 0x7ff6c0136940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55574/0 (socket says 172.21.15.46:55574) 2024-09-16T04:45:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6bf7fe640 1 -- 172.21.15.46:0/1542567155 learned_addr learned my addr 172.21.15.46:0/1542567155 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6bf7fe640 10 monclient(hunting): get_auth_request con 0x7ff6c00a2ee0 auth_method 0 2024-09-16T04:45:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.600+0000 7ff6bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.601+0000 7ff6bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.601+0000 7ff6bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.601+0000 7ff6bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.601+0000 7ff6c4b9a640 1 --2- 172.21.15.46:0/1542567155 >> v2:172.21.15.114:3300/0 conn(0x7ff6c0136e80 0x7ff6c0137330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.601+0000 7ff6bf7fe640 10 monclient(hunting): handle_auth_done global_id 74493 payload 274 2024-09-16T04:45:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.601+0000 7ff6bf7fe640 1 -- 172.21.15.46:0/1542567155 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff6c006e160 msgr2=0x7ff6c00a29a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:45:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.601+0000 7ff6bf7fe640 1 --2- 172.21.15.46:0/1542567155 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff6c006e160 0x7ff6c00a29a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.601+0000 7ff6bf7fe640 1 -- 172.21.15.46:0/1542567155 >> v2:172.21.15.114:3300/0 conn(0x7ff6c0136e80 msgr2=0x7ff6c0137330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:45:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.601+0000 7ff6bf7fe640 1 --2- 172.21.15.46:0/1542567155 >> v2:172.21.15.114:3300/0 conn(0x7ff6c0136e80 0x7ff6c0137330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.601+0000 7ff6bf7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:45:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.601+0000 7ff6bf7fe640 1 monclient: found mon.noname-a 2024-09-16T04:45:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.601+0000 7ff6bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.601+0000 7ff6bf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.601+0000 7ff6bf7fe640 1 -- 172.21.15.46:0/1542567155 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6c00a0bd0 con 0x7ff6c00a2ee0 2024-09-16T04:45:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.601+0000 7ff6c4b9a640 10 monclient: get_auth_request con 0x7ff6c0136e80 auth_method 0 2024-09-16T04:45:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.601+0000 7ff6c4b9a640 1 --2- 172.21.15.46:0/1542567155 >> v2:172.21.15.114:3300/0 conn(0x7ff6c0136e80 0x7ff6c0137330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:45:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.601+0000 7ff6bf7fe640 1 --2- 172.21.15.46:0/1542567155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6c00a2ee0 0x7ff6c0136940 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0068140 tx=0x7ff6b402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3469f4615eaf41bf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6beffd640 1 -- 172.21.15.46:0/1542567155 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b4037070 con 0x7ff6c00a2ee0 2024-09-16T04:45:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6beffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6beffd640 10 monclient: dump: 2024-09-16T04:45:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:38.615 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6beffd640 10 monclient: _finish_auth 0 2024-09-16T04:45:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6beffd640 10 monclient: _check_auth_tickets 2024-09-16T04:45:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6c65fd640 5 monclient: authenticate success, global_id 74493 2024-09-16T04:45:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6beffd640 1 -- 172.21.15.46:0/1542567155 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6b402ae20 con 0x7ff6c00a2ee0 2024-09-16T04:45:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6c65fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:45:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6beffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6c65fd640 10 monclient: get_monmap_and_config success 2024-09-16T04:45:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6c65fd640 4 set_mon_vals no callback set 2024-09-16T04:45:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6beffd640 1 -- 172.21.15.46:0/1542567155 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b4032660 con 0x7ff6c00a2ee0 2024-09-16T04:45:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6beffd640 10 monclient: dump: 2024-09-16T04:45:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:38.616 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6c65fd640 10 monclient: shutdown 2024-09-16T04:45:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6c65fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6c65fd640 1 -- 172.21.15.46:0/1542567155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6c00a2ee0 msgr2=0x7ff6c0136940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6c65fd640 1 --2- 172.21.15.46:0/1542567155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6c00a2ee0 0x7ff6c0136940 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0068140 tx=0x7ff6b402a110 comp rx=0 tx=0).stop 2024-09-16T04:45:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6c65fd640 1 -- 172.21.15.46:0/1542567155 shutdown_connections 2024-09-16T04:45:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6c65fd640 1 --2- 172.21.15.46:0/1542567155 >> v2:172.21.15.114:3300/0 conn(0x7ff6c0136e80 0x7ff6c0137330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6c65fd640 1 --2- 172.21.15.46:0/1542567155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6c00a2ee0 0x7ff6c0136940 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6c65fd640 1 --2- 172.21.15.46:0/1542567155 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff6c006e160 0x7ff6c00a29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6c65fd640 1 -- 172.21.15.46:0/1542567155 >> 172.21.15.46:0/1542567155 conn(0x7ff6c00a5c10 msgr2=0x7ff6c00a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6c65fd640 1 -- 172.21.15.46:0/1542567155 shutdown_connections 2024-09-16T04:45:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.602+0000 7ff6c65fd640 1 -- 172.21.15.46:0/1542567155 wait complete. 2024-09-16T04:45:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.603+0000 7ff6c65fd640 10 monclient: build_initial_monmap 2024-09-16T04:45:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.603+0000 7ff6c65fd640 10 monclient: monmap: 2024-09-16T04:45:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:45:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:45:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:45:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:45:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:45:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:45:38.618 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.603+0000 7ff6c65fd640 1 Processor -- start 2024-09-16T04:45:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.603+0000 7ff6c65fd640 1 -- start start 2024-09-16T04:45:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.603+0000 7ff6c65fd640 10 monclient: init 2024-09-16T04:45:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c65fd640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c65fd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:45:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c65fd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff6c006e160 0x7ff6c01cbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c65fd640 10 monclient(hunting): picked mon.noname-c con 0x7ff6c006e160 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c65fd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6c00a2ee0 0x7ff6c0135d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c65fd640 10 monclient(hunting): picked mon.noname-b con 0x7ff6c00a2ee0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:45:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c65fd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6c0136e80 0x7ff6c01362d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c65fd640 10 monclient(hunting): picked mon.noname-a con 0x7ff6c0136e80 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:45:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c65fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c65fd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c00a5130 con 0x7ff6c0136e80 2024-09-16T04:45:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c65fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c65fd640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c00a1190 con 0x7ff6c00a2ee0 2024-09-16T04:45:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6bffff640 1 -- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff6c006e160 msgr2=0x7ff6c01cbbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3301/0 2024-09-16T04:45:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c65fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c65fd640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7ff6c00a34a0 con 0x7ff6c006e160 2024-09-16T04:45:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c65fd640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c65fd640 10 monclient(hunting): authenticate will time out at 1137.019301s 2024-09-16T04:45:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6bffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff6c006e160 0x7ff6c01cbbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c4b9a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6c0136e80 0x7ff6c01362d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c4b9a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6c0136e80 0x7ff6c01362d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55586/0 (socket says 172.21.15.46:55586) 2024-09-16T04:45:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c4b9a640 1 -- 172.21.15.46:0/3543434177 learned_addr learned my addr 172.21.15.46:0/3543434177 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c4b9a640 10 monclient(hunting): get_auth_request con 0x7ff6c0136e80 auth_method 0 2024-09-16T04:45:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c4b9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c4b9a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c4b9a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c4b9a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c4b9a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6bf7fe640 1 --2- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6c00a2ee0 0x7ff6c0135d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c4b9a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6bf7fe640 10 monclient(hunting): get_auth_request con 0x7ff6c00a2ee0 auth_method 0 2024-09-16T04:45:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c4b9a640 10 monclient(hunting): handle_auth_done global_id 74496 payload 824 2024-09-16T04:45:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c4b9a640 1 -- 172.21.15.46:0/3543434177 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff6c006e160 msgr2=0x7ff6c01cbbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:45:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c4b9a640 1 --2- 172.21.15.46:0/3543434177 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff6c006e160 0x7ff6c01cbbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c4b9a640 1 -- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6c00a2ee0 msgr2=0x7ff6c0135d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c4b9a640 1 --2- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6c00a2ee0 0x7ff6c0135d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c4b9a640 10 monclient: _finish_hunting 0 2024-09-16T04:45:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c4b9a640 1 monclient: found mon.noname-a 2024-09-16T04:45:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c4b9a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c4b9a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c4b9a640 1 -- 172.21.15.46:0/3543434177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6c0127790 con 0x7ff6c0136e80 2024-09-16T04:45:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.604+0000 7ff6c4b9a640 1 --2- 172.21.15.46:0/3543434177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6c0136e80 0x7ff6c01362d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff6b800cf20 tx=0x7ff6b800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.605+0000 7ff6bcff9640 1 -- 172.21.15.46:0/3543434177 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b801a070 con 0x7ff6c0136e80 2024-09-16T04:45:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.605+0000 7ff6bcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.605+0000 7ff6bcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.605+0000 7ff6bcff9640 10 monclient: dump: 2024-09-16T04:45:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.623 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.623 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:38.623 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.623 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:38.623 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:38.623 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:38.623 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.605+0000 7ff6bcff9640 10 monclient: _finish_auth 0 2024-09-16T04:45:38.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.605+0000 7ff6bcff9640 10 monclient: _check_auth_tickets 2024-09-16T04:45:38.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.605+0000 7ff6bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:38.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.605+0000 7ff6bcff9640 1 -- 172.21.15.46:0/3543434177 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6b800d840 con 0x7ff6c0136e80 2024-09-16T04:45:38.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.605+0000 7ff6bcff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:38.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.605+0000 7ff6bcff9640 1 -- 172.21.15.46:0/3543434177 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b800dc30 con 0x7ff6c0136e80 2024-09-16T04:45:38.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.605+0000 7ff6c65fd640 5 monclient: authenticate success, global_id 74496 2024-09-16T04:45:38.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.605+0000 7ff6c65fd640 10 monclient: _renew_subs 2024-09-16T04:45:38.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.605+0000 7ff6c65fd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:38.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.605+0000 7ff6c65fd640 1 -- 172.21.15.46:0/3543434177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6c0125940 con 0x7ff6c0136e80 2024-09-16T04:45:38.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.605+0000 7ff6bcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:38.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.605+0000 7ff6bcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:38.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.605+0000 7ff6bcff9640 10 monclient: dump: 2024-09-16T04:45:38.624 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:38.624 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:38.624 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.624 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.624 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:38.624 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.624 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:38.625 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.605+0000 7ff6be7fc640 4 set_mon_vals no callback set 2024-09-16T04:45:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.605+0000 7ff6c65fd640 10 monclient: _renew_subs 2024-09-16T04:45:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.605+0000 7ff6c65fd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.605+0000 7ff6c65fd640 1 -- 172.21.15.46:0/3543434177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6c00a0300 con 0x7ff6c0136e80 2024-09-16T04:45:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.606+0000 7ff6c65fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.606+0000 7ff6bcff9640 1 -- 172.21.15.46:0/3543434177 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7ff6b801d070 con 0x7ff6c0136e80 2024-09-16T04:45:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.606+0000 7ff6c65fd640 1 --2- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6c0123800 0x7ff6c0123be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.606+0000 7ff6c65fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.606+0000 7ff6c65fd640 1 -- 172.21.15.46:0/3543434177 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c00a4e20 con 0x7ff6c0123800 2024-09-16T04:45:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.606+0000 7ff6bcff9640 1 --2- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff6ac038360 0x7ff6ac03a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.606+0000 7ff6bcff9640 1 -- 172.21.15.46:0/3543434177 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 4346+0+0 (secure 0 0 0) 0x7ff6b804bdd0 con 0x7ff6c0136e80 2024-09-16T04:45:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.606+0000 7ff6bffff640 1 --2- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6c0123800 0x7ff6c0123be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.606+0000 7ff6bffff640 10 monclient: get_auth_request con 0x7ff6c0123800 auth_method 0 2024-09-16T04:45:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.606+0000 7ff6bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.606+0000 7ff6bffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.606+0000 7ff6bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.606+0000 7ff6bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.606+0000 7ff6bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.606+0000 7ff6bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.606+0000 7ff6bf7fe640 1 --2- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff6ac038360 0x7ff6ac03a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.606+0000 7ff6bf7fe640 10 monclient: get_auth_request con 0x7ff6ac038360 auth_method 0 2024-09-16T04:45:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.606+0000 7ff6bffff640 10 monclient(hunting): handle_auth_done global_id 6562 payload 824 2024-09-16T04:45:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.606+0000 7ff6bffff640 1 -- 172.21.15.46:0/3543434177 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff6c009f670 con 0x7ff6c0123800 2024-09-16T04:45:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.606+0000 7ff6bffff640 1 --2- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6c0123800 0x7ff6c0123be0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff6b000b450 tx=0x7ff6b000b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.607+0000 7ff6bf7fe640 1 --2- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff6ac038360 0x7ff6ac03a820 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4034000 tx=0x7ff6b402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.607+0000 7ff6bcff9640 1 -- 172.21.15.46:0/3543434177 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0003dc0 con 0x7ff6c0123800 2024-09-16T04:45:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.607+0000 7ff6bcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.607+0000 7ff6bcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:45:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.607+0000 7ff6bcff9640 10 monclient: dump: 2024-09-16T04:45:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:38.628 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.607+0000 7ff6bcff9640 1 -- 172.21.15.46:0/3543434177 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff6b0015070 con 0x7ff6c0123800 2024-09-16T04:45:38.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.607+0000 7ff6bcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:38.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.607+0000 7ff6bcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:45:38.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.607+0000 7ff6bcff9640 1 -- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6c0123800 msgr2=0x7ff6c0123be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:38.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.607+0000 7ff6bcff9640 1 --2- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6c0123800 0x7ff6c0123be0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff6b000b450 tx=0x7ff6b000b910 comp rx=0 tx=0).stop 2024-09-16T04:45:38.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.607+0000 7ff6bcff9640 1 -- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6c0123800 msgr2=0x7ff6c0123be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:45:38.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.607+0000 7ff6bcff9640 1 --2- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6c0123800 0x7ff6c0123be0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.613+0000 7ff6c65fd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:45:38.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.613+0000 7ff6c65fd640 1 --2- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6c0124120 0x7ff6c011f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:38.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.613+0000 7ff6c65fd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:38.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.613+0000 7ff6c65fd640 1 -- 172.21.15.46:0/3543434177 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c00a8050 con 0x7ff6c0124120 2024-09-16T04:45:38.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.613+0000 7ff6bffff640 1 --2- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6c0124120 0x7ff6c011f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:38.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.613+0000 7ff6bffff640 10 monclient: get_auth_request con 0x7ff6c0124120 auth_method 0 2024-09-16T04:45:38.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.613+0000 7ff6bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.613+0000 7ff6bffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.613+0000 7ff6bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.614+0000 7ff6bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.614+0000 7ff6bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.614+0000 7ff6bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.614+0000 7ff6bffff640 10 monclient(hunting): handle_auth_done global_id 6565 payload 824 2024-09-16T04:45:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.614+0000 7ff6bffff640 1 -- 172.21.15.46:0/3543434177 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff6c0067e00 con 0x7ff6c0124120 2024-09-16T04:45:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.614+0000 7ff6bffff640 1 --2- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6c0124120 0x7ff6c011f760 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7ff6b000b450 tx=0x7ff6b00059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.614+0000 7ff6bcff9640 1 -- 172.21.15.46:0/3543434177 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0023410 con 0x7ff6c0124120 2024-09-16T04:45:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.614+0000 7ff6bcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.614+0000 7ff6bcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:45:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.614+0000 7ff6bcff9640 10 monclient: dump: 2024-09-16T04:45:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:38.632 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.614+0000 7ff6bcff9640 1 -- 172.21.15.46:0/3543434177 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4868 (secure 0 0 0) 0x7ff6b0010400 con 0x7ff6c0124120 2024-09-16T04:45:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.614+0000 7ff6bcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:45:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.614+0000 7ff6bcff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:45:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.614+0000 7ff6bcff9640 1 -- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6c0124120 msgr2=0x7ff6c011f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.614+0000 7ff6bcff9640 1 --2- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6c0124120 0x7ff6c011f760 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7ff6b000b450 tx=0x7ff6b00059d0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.614+0000 7ff6bcff9640 1 -- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6c0124120 msgr2=0x7ff6c011f760 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:45:38.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.614+0000 7ff6bcff9640 1 --2- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6c0124120 0x7ff6c011f760 secure :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7ff6b000b450 tx=0x7ff6b00059d0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.633 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:45:38.633 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:45:38.633 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:45:38.633 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:45:38.633 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 82, 2024-09-16T04:45:38.633 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 424516, 2024-09-16T04:45:38.633 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:45:38.633 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:45:38.634 INFO:teuthology.orchestra.run.smithi046.stdout: 1 2024-09-16T04:45:38.634 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:38.634 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:45:38.634 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:45:38.634 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:45:38.634 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:45:38.634 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:38.634 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:38.634 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:38.634 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:38.634 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:38.634 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:38.634 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:38.634 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:38.635 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:38.635 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:38.635 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:38.635 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:38.635 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:45:38.635 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:45:38.635 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:38.635 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:38.635 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:38.635 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:38.635 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:38.635 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:38.635 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:38.635 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:38.636 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:38.636 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:38.636 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:38.636 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:38.636 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.636 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:45:38.636 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:45:38.636 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:45:38.636 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:45:38.636 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:45:38.636 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:45:38.636 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:45:38.636 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:45:38.637 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:45:38.637 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:45:38.637 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:45:38.637 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:45:38.637 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:45:38.637 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:45:38.637 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:45:38.637 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:45:38.637 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:45:38.637 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:45:38.637 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:45:38.637 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:45:38.637 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:45:38.637 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:45:38.638 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:45:38.638 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:45:38.638 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:45:38.638 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:45:38.638 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:45:38.638 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:45:38.638 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:38.638 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:45:38.638 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.638 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:45:38.638 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.638 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:45:38.638 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:45:38.638 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:38.639 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:38.639 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.639 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:38.639 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:45:38.639 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:38.639 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.639 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.639 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:38.639 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:45:38.639 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:38.639 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:38.639 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:38.639 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.639 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:45:38.640 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:45:38.640 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:38.640 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:38.640 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:38.640 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.640 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.640 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:45:38.640 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:45:38.640 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:38.640 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:38.640 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.640 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:38.640 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:45:38.641 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:38.641 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.641 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.641 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:38.641 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:45:38.641 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:38.641 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:38.641 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:38.641 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.641 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:45:38.641 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:45:38.641 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:38.641 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:38.641 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:38.642 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.642 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.642 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:45:38.642 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:45:38.642 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:45:38.642 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:45:38.642 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.642 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:45:38.642 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:45:38.642 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:38.642 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.642 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.642 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:45:38.642 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:45:38.643 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:45:38.643 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:38.643 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:38.643 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.643 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:45:38.643 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:45:38.643 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:45:38.643 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:45:38.643 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:45:38.643 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:38.643 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:38.643 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.643 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:45:38.643 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:45:38.644 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.644 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:38.644 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:38.644 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:45:38.644 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:38.644 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:38.644 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:45:38.644 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.644 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:38.644 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:38.644 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5 2024-09-16T04:45:38.644 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:38.644 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:38.644 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:45:38.645 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.645 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:38.645 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:38.645 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 7 2024-09-16T04:45:38.645 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:38.645 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:45:38.645 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:45:38.645 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:45:38.645 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:45:38.645 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:45:38.645 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:45:38.645 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:45:38.645 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:45:38.645 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:45:38.646 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:45:38.646 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:45:38.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.615+0000 7ff6c65fd640 1 -- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff6ac038360 msgr2=0x7ff6ac03a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:38.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.615+0000 7ff6c65fd640 1 --2- 172.21.15.46:0/3543434177 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff6ac038360 0x7ff6ac03a820 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4034000 tx=0x7ff6b402e000 comp rx=0 tx=0).stop 2024-09-16T04:45:38.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.615+0000 7ff6c65fd640 10 monclient: shutdown 2024-09-16T04:45:38.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.615+0000 7ff6c65fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:38.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.615+0000 7ff6c65fd640 1 -- 172.21.15.46:0/3543434177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6c0136e80 msgr2=0x7ff6c01362d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:38.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.615+0000 7ff6c4b9a640 1 -- 172.21.15.46:0/3543434177 reap_dead start 2024-09-16T04:45:38.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.615+0000 7ff6c65fd640 1 --2- 172.21.15.46:0/3543434177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6c0136e80 0x7ff6c01362d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff6b800cf20 tx=0x7ff6b800a7d0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.615+0000 7ff6c65fd640 1 -- 172.21.15.46:0/3543434177 shutdown_connections 2024-09-16T04:45:38.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.615+0000 7ff6c65fd640 1 --2- 172.21.15.46:0/3543434177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6c0136e80 0x7ff6c01362d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:38.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.615+0000 7ff6c65fd640 1 -- 172.21.15.46:0/3543434177 >> 172.21.15.46:0/3543434177 conn(0x7ff6c00a5c10 msgr2=0x7ff6c00a77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:38.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.615+0000 7ff6c65fd640 1 -- 172.21.15.46:0/3543434177 shutdown_connections 2024-09-16T04:45:38.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:38.615+0000 7ff6c65fd640 1 -- 172.21.15.46:0/3543434177 wait complete. 2024-09-16T04:45:38.648 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:45:38.648 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:45:40.146 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39406]: starting. 2024-09-16T04:45:40.147 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39406]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-39405 2024-09-16T04:45:40.147 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39406]: created object 0... 2024-09-16T04:45:40.147 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39406]: created object 25... 2024-09-16T04:45:40.147 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39406]: created object 49... 2024-09-16T04:45:40.147 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39406]: finishing. 2024-09-16T04:45:40.147 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39406]: shutting down. 2024-09-16T04:45:41.059 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39407]: starting. 2024-09-16T04:45:41.059 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39407]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-39405 2024-09-16T04:45:41.059 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39407]: shutting down. 2024-09-16T04:45:43.155 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:43.155 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39440]: starting. 2024-09-16T04:45:43.155 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39440]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-39405 2024-09-16T04:45:43.155 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39440]: created object 0... 2024-09-16T04:45:43.155 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39440]: created object 25... 2024-09-16T04:45:43.155 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39440]: created object 49... 2024-09-16T04:45:43.155 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39440]: finishing. 2024-09-16T04:45:43.155 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39440]: shutting down. 2024-09-16T04:45:43.167 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:43.167 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39442]: starting. 2024-09-16T04:45:43.167 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39442]: listing objects. 2024-09-16T04:45:43.168 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39442]: listed object 0... 2024-09-16T04:45:43.168 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39442]: listed object 25... 2024-09-16T04:45:43.168 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39442]: saw 50 objects 2024-09-16T04:45:43.168 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39442]: shutting down. 2024-09-16T04:45:44.071 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:44.071 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39441]: starting. 2024-09-16T04:45:44.071 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39441]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-39405 2024-09-16T04:45:44.071 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39441]: shutting down. 2024-09-16T04:45:44.073 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:44.073 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:44.073 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:45:44.076 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:45:46.112 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39509]: starting. 2024-09-16T04:45:46.113 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39509]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-39508 2024-09-16T04:45:46.113 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39509]: created object 0... 2024-09-16T04:45:46.113 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39509]: created object 25... 2024-09-16T04:45:46.113 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39509]: created object 49... 2024-09-16T04:45:46.113 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39509]: finishing. 2024-09-16T04:45:46.113 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39509]: shutting down. 2024-09-16T04:45:47.032 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39510]: starting. 2024-09-16T04:45:47.032 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39510]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-39508 2024-09-16T04:45:47.033 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39510]: shutting down. 2024-09-16T04:45:49.134 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:49.134 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39543]: starting. 2024-09-16T04:45:49.134 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39543]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-39508 2024-09-16T04:45:49.134 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39543]: created object 0... 2024-09-16T04:45:49.135 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39543]: created object 25... 2024-09-16T04:45:49.135 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39543]: created object 49... 2024-09-16T04:45:49.135 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39543]: finishing. 2024-09-16T04:45:49.135 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39543]: shutting down. 2024-09-16T04:45:49.150 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:49.150 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39545]: starting. 2024-09-16T04:45:49.150 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39545]: listing objects. 2024-09-16T04:45:49.150 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39545]: listed object 0... 2024-09-16T04:45:49.150 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39545]: listed object 25... 2024-09-16T04:45:49.151 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39545]: saw 50 objects 2024-09-16T04:45:49.151 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39545]: shutting down. 2024-09-16T04:45:50.052 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:50.053 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39544]: starting. 2024-09-16T04:45:50.053 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39544]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-39508 2024-09-16T04:45:50.053 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39544]: shutting down. 2024-09-16T04:45:50.054 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:50.054 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:50.054 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:45:50.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:45:52.145 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39612]: starting. 2024-09-16T04:45:52.145 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39612]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-39611 2024-09-16T04:45:52.145 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39612]: created object 0... 2024-09-16T04:45:52.145 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39612]: created object 25... 2024-09-16T04:45:52.145 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39612]: created object 49... 2024-09-16T04:45:52.145 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39612]: finishing. 2024-09-16T04:45:52.145 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39612]: shutting down. 2024-09-16T04:45:53.067 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39613]: starting. 2024-09-16T04:45:53.067 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39613]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-39611 2024-09-16T04:45:53.067 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39613]: shutting down. 2024-09-16T04:45:55.356 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:55.356 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39646]: starting. 2024-09-16T04:45:55.356 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39646]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-39611 2024-09-16T04:45:55.356 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39646]: created object 0... 2024-09-16T04:45:55.356 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39646]: created object 25... 2024-09-16T04:45:55.356 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39646]: created object 49... 2024-09-16T04:45:55.356 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39646]: finishing. 2024-09-16T04:45:55.356 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39646]: shutting down. 2024-09-16T04:45:55.370 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:55.370 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39648]: starting. 2024-09-16T04:45:55.370 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39648]: listing objects. 2024-09-16T04:45:55.370 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39648]: listed object 0... 2024-09-16T04:45:55.370 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39648]: listed object 25... 2024-09-16T04:45:55.370 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39648]: saw 50 objects 2024-09-16T04:45:55.370 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39648]: shutting down. 2024-09-16T04:45:56.033 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:56.033 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39647]: starting. 2024-09-16T04:45:56.033 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39647]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-39611 2024-09-16T04:45:56.033 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39647]: shutting down. 2024-09-16T04:45:56.035 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:56.035 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:45:56.035 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:45:56.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:45:58.164 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39715]: starting. 2024-09-16T04:45:58.165 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39715]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-39714 2024-09-16T04:45:58.165 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39715]: created object 0... 2024-09-16T04:45:58.165 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39715]: created object 25... 2024-09-16T04:45:58.165 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39715]: created object 49... 2024-09-16T04:45:58.165 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39715]: finishing. 2024-09-16T04:45:58.165 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39715]: shutting down. 2024-09-16T04:45:58.649 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:45:58.649 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T04:45:58.649 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:45:58.649 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:45:58.652 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:45:58.652 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:45:58.653 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:45:58.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.765+0000 7fa3af4b7640 10 monclient: get_monmap_and_config 2024-09-16T04:45:58.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.765+0000 7fa3af4b7640 10 monclient: build_initial_monmap 2024-09-16T04:45:58.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.765+0000 7fa3af4b7640 10 monclient: monmap: 2024-09-16T04:45:58.767 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:58.767 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:58.767 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:45:58.766566+0000 2024-09-16T04:45:58.767 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:45:58.766566+0000 2024-09-16T04:45:58.767 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:58.767 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:58.767 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:45:58.768 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:45:58.768 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:45:58.768 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:45:58.768 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:45:58.768 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:58.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.765+0000 7fa3af4b7640 1 Processor -- start 2024-09-16T04:45:58.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.765+0000 7fa3af4b7640 1 -- start start 2024-09-16T04:45:58.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.765+0000 7fa3af4b7640 10 monclient: init 2024-09-16T04:45:58.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3af4b7640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:58.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3af4b7640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-16T04:45:58.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3af4b7640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3a80a3c10 0x7fa3a809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:58.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3af4b7640 10 monclient(hunting): picked mon.noname-b con 0x7fa3a80a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:58.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3af4b7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3a80a3ff0 0x7fa3a80a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:58.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3af4b7640 10 monclient(hunting): picked mon.noname-a con 0x7fa3a80a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:45:58.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3af4b7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa3a81369d0 addr v1:172.21.15.46:6789/0 2024-09-16T04:45:58.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3af4b7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3a809ab70 con 0x7fa3a81369d0 2024-09-16T04:45:58.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3af4b7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:58.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3af4b7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa3a80a4d60 con 0x7fa3a80a3ff0 2024-09-16T04:45:58.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3af4b7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:58.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3af4b7640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fa3a809b030 con 0x7fa3a80a3c10 2024-09-16T04:45:58.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3af4b7640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:58.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3af4b7640 10 monclient(hunting): authenticate will time out at 1157.182455s 2024-09-16T04:45:58.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3aca52640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3a80a3ff0 0x7fa3a80a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:58.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3ad253640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3a80a3c10 0x7fa3a809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:58.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3aca52640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3a80a3ff0 0x7fa3a80a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54970/0 (socket says 172.21.15.46:54970) 2024-09-16T04:45:58.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3ad253640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3a80a3c10 0x7fa3a809bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:51028/0 (socket says 172.21.15.46:51028) 2024-09-16T04:45:58.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3ada54640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa3a81369d0 0x7fa3a8136e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57930/0 (socket says 172.21.15.46:57930) 2024-09-16T04:45:58.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3aca52640 1 -- 172.21.15.46:0/2815196035 learned_addr learned my addr 172.21.15.46:0/2815196035 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:58.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3aca52640 10 monclient(hunting): get_auth_request con 0x7fa3a80a3ff0 auth_method 0 2024-09-16T04:45:58.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3aca52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:58.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3aca52640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:58.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3aca52640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:58.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3ad253640 10 monclient(hunting): get_auth_request con 0x7fa3a80a3c10 auth_method 0 2024-09-16T04:45:58.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3ad253640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:58.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3ad253640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:58.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3ad253640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:58.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3aca52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:58.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3aca52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:58.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3aca52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:58.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3ad253640 1 -- 172.21.15.46:0/2815196035 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3a80a3c10 msgr2=0x7fa3a809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:45:58.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3ad253640 1 -- 172.21.15.46:0/2815196035 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3a80a3c10 msgr2=0x7fa3a809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:45:58.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3ad253640 1 --2- 172.21.15.46:0/2815196035 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3a80a3c10 0x7fa3a809bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:58.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3ad253640 1 --2- 172.21.15.46:0/2815196035 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3a80a3c10 0x7fa3a809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:58.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3aca52640 10 monclient(hunting): handle_auth_done global_id 74595 payload 274 2024-09-16T04:45:58.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3aca52640 1 -- 172.21.15.46:0/2815196035 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3a80a3c10 msgr2=0x7fa3a809bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:45:58.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3aca52640 1 --2- 172.21.15.46:0/2815196035 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3a80a3c10 0x7fa3a809bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:58.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa3aca52640 1 -- 172.21.15.46:0/2815196035 >> v1:172.21.15.46:6789/0 conn(0x7fa3a81369d0 legacy=0x7fa3a8136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:58.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa3aca52640 10 monclient: _finish_hunting 0 2024-09-16T04:45:58.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa3aca52640 1 monclient: found mon.noname-a 2024-09-16T04:45:58.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa3aca52640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:58.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa3aca52640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:58.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa3aca52640 1 -- 172.21.15.46:0/2815196035 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a809b310 con 0x7fa3a80a3ff0 2024-09-16T04:45:58.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.766+0000 7fa39ffff640 1 -- 172.21.15.46:0/2815196035 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 762759344 0 0) 0x7fa3a809ab70 con 0x7fa3a81369d0 2024-09-16T04:45:58.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa39ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:45:58.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa3aca52640 1 --2- 172.21.15.46:0/2815196035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a80a3ff0 0x7fa3a80a43d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009870 tx=0x7fa3a402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc98379da882188 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:58.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa39ffff640 1 -- 172.21.15.46:0/2815196035 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4037070 con 0x7fa3a80a3ff0 2024-09-16T04:45:58.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa39ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:58.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa39ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:58.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa39ffff640 10 monclient: dump: 2024-09-16T04:45:58.781 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:58.781 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:58.781 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:58.781 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:58.781 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:58.781 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:58.781 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:58.782 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:58.782 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:58.782 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:58.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa39ffff640 10 monclient: _finish_auth 0 2024-09-16T04:45:58.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa39ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:45:58.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa39ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:58.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa3af4b7640 5 monclient: authenticate success, global_id 74595 2024-09-16T04:45:58.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa3af4b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:45:58.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa39ffff640 1 -- 172.21.15.46:0/2815196035 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3a4032440 con 0x7fa3a80a3ff0 2024-09-16T04:45:58.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa39ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:58.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa3af4b7640 10 monclient: get_monmap_and_config success 2024-09-16T04:45:58.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa3af4b7640 4 set_mon_vals no callback set 2024-09-16T04:45:58.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa39ffff640 1 -- 172.21.15.46:0/2815196035 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4032810 con 0x7fa3a80a3ff0 2024-09-16T04:45:58.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa39ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:58.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa39ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:58.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.767+0000 7fa39ffff640 10 monclient: dump: 2024-09-16T04:45:58.783 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:58.783 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:58.783 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:58.783 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:58.783 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:58.783 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:58.783 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:58.783 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:58.783 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:58.783 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:58.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.768+0000 7fa3af4b7640 10 monclient: shutdown 2024-09-16T04:45:58.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.768+0000 7fa3af4b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:58.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.768+0000 7fa3af4b7640 1 -- 172.21.15.46:0/2815196035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a80a3ff0 msgr2=0x7fa3a80a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:58.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.768+0000 7fa3af4b7640 1 --2- 172.21.15.46:0/2815196035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a80a3ff0 0x7fa3a80a43d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009870 tx=0x7fa3a402a1c0 comp rx=0 tx=0).stop 2024-09-16T04:45:58.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.768+0000 7fa3af4b7640 1 -- 172.21.15.46:0/2815196035 shutdown_connections 2024-09-16T04:45:58.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.768+0000 7fa3af4b7640 1 --2- 172.21.15.46:0/2815196035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a80a3ff0 0x7fa3a80a43d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:58.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.768+0000 7fa3af4b7640 1 --2- 172.21.15.46:0/2815196035 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3a80a3c10 0x7fa3a809bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:58.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.768+0000 7fa3af4b7640 1 -- 172.21.15.46:0/2815196035 >> 172.21.15.46:0/2815196035 conn(0x7fa3a80a5d10 msgr2=0x7fa3a80a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:58.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.768+0000 7fa3af4b7640 1 -- 172.21.15.46:0/2815196035 shutdown_connections 2024-09-16T04:45:58.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.768+0000 7fa3af4b7640 1 -- 172.21.15.46:0/2815196035 wait complete. 2024-09-16T04:45:58.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.769+0000 7fa3af4b7640 10 monclient: build_initial_monmap 2024-09-16T04:45:58.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.769+0000 7fa3af4b7640 10 monclient: monmap: 2024-09-16T04:45:58.784 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:45:58.784 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:45:58.784 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:45:58.784 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:45:58.784 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:45:58.785 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:58.785 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:45:58.785 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:45:58.785 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:45:58.785 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:58.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.769+0000 7fa3af4b7640 1 Processor -- start 2024-09-16T04:45:58.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.769+0000 7fa3af4b7640 1 -- start start 2024-09-16T04:45:58.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.769+0000 7fa3af4b7640 10 monclient: init 2024-09-16T04:45:58.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3af4b7640 10 monclient: _reopen_session rank -1 2024-09-16T04:45:58.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3af4b7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:45:58.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3af4b7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3a80a3c10 0x7fa3a81cff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:58.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3af4b7640 10 monclient(hunting): picked mon.noname-b con 0x7fa3a80a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:45:58.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3af4b7640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3a80a3ff0 0x7fa3a81d0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:58.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3af4b7640 10 monclient(hunting): picked mon.noname-c con 0x7fa3a80a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:45:58.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3af4b7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a81369d0 0x7fa3a806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:58.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3af4b7640 10 monclient(hunting): picked mon.noname-a con 0x7fa3a81369d0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:45:58.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3af4b7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:58.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3af4b7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a80a4910 con 0x7fa3a81369d0 2024-09-16T04:45:58.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3af4b7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:58.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3af4b7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a806e3d0 con 0x7fa3a80a3c10 2024-09-16T04:45:58.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3af4b7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:45:58.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3af4b7640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fa3a81362f0 con 0x7fa3a80a3ff0 2024-09-16T04:45:58.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3af4b7640 10 monclient(hunting): _renew_subs 2024-09-16T04:45:58.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3af4b7640 10 monclient(hunting): authenticate will time out at 1157.186299s 2024-09-16T04:45:58.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3aca52640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3a80a3ff0 0x7fa3a81d0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:58.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3aca52640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3a80a3ff0 0x7fa3a81d0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:51044/0 (socket says 172.21.15.46:51044) 2024-09-16T04:45:58.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3aca52640 1 -- 172.21.15.46:0/2738273538 learned_addr learned my addr 172.21.15.46:0/2738273538 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:45:58.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3aca52640 10 monclient(hunting): get_auth_request con 0x7fa3a80a3ff0 auth_method 0 2024-09-16T04:45:58.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3aca52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:58.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3aca52640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:58.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3aca52640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:58.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ada54640 1 --2- 172.21.15.46:0/2738273538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a81369d0 0x7fa3a806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:58.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ad253640 1 --2- 172.21.15.46:0/2738273538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3a80a3c10 0x7fa3a81cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:58.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ada54640 10 monclient(hunting): get_auth_request con 0x7fa3a81369d0 auth_method 0 2024-09-16T04:45:58.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ada54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:58.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ada54640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:58.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ada54640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:58.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ad253640 10 monclient(hunting): get_auth_request con 0x7fa3a80a3c10 auth_method 0 2024-09-16T04:45:58.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ad253640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:45:58.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ad253640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:45:58.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ad253640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:45:58.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3aca52640 1 -- 172.21.15.46:0/2738273538 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3a80a3ff0 msgr2=0x7fa3a81d0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:45:58.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ada54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:58.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3aca52640 1 -- 172.21.15.46:0/2738273538 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3a80a3ff0 msgr2=0x7fa3a81d0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:45:58.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ada54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:58.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3aca52640 1 --2- 172.21.15.46:0/2738273538 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3a80a3ff0 0x7fa3a81d0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:45:58.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ada54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:58.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ad253640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:45:58.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ad253640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:45:58.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ad253640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:45:58.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3aca52640 1 --2- 172.21.15.46:0/2738273538 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3a80a3ff0 0x7fa3a81d0460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:45:58.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ada54640 10 monclient(hunting): handle_auth_done global_id 74601 payload 824 2024-09-16T04:45:58.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ada54640 1 -- 172.21.15.46:0/2738273538 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3a80a3ff0 msgr2=0x7fa3a81d0460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:45:58.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ada54640 1 --2- 172.21.15.46:0/2738273538 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3a80a3ff0 0x7fa3a81d0460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:58.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ada54640 1 -- 172.21.15.46:0/2738273538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3a80a3c10 msgr2=0x7fa3a81cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:58.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ada54640 1 --2- 172.21.15.46:0/2738273538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3a80a3c10 0x7fa3a81cff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:58.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ada54640 10 monclient: _finish_hunting 0 2024-09-16T04:45:58.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ada54640 1 monclient: found mon.noname-a 2024-09-16T04:45:58.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ada54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:45:58.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ada54640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:45:58.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ada54640 1 -- 172.21.15.46:0/2738273538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a8072930 con 0x7fa3a81369d0 2024-09-16T04:45:58.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.770+0000 7fa3ad253640 1 --2- 172.21.15.46:0/2738273538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3a80a3c10 0x7fa3a81cff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:45:58.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa3ada54640 1 --2- 172.21.15.46:0/2738273538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a81369d0 0x7fa3a806ef60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0000f80 tx=0x7fa3a000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:58.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa39dffb640 1 -- 172.21.15.46:0/2738273538 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0018070 con 0x7fa3a81369d0 2024-09-16T04:45:58.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa39dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:58.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa39dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:45:58.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa39dffb640 10 monclient: dump: 2024-09-16T04:45:58.789 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:58.789 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:58.789 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:58.789 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:58.790 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:58.790 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:58.790 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:58.790 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:58.790 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:58.790 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:58.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa39dffb640 10 monclient: _finish_auth 0 2024-09-16T04:45:58.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa39dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:45:58.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa39dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:45:58.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa3af4b7640 5 monclient: authenticate success, global_id 74601 2024-09-16T04:45:58.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa39dffb640 1 -- 172.21.15.46:0/2738273538 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3a0004d60 con 0x7fa3a81369d0 2024-09-16T04:45:58.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa3af4b7640 10 monclient: _renew_subs 2024-09-16T04:45:58.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa3af4b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:58.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa3af4b7640 1 -- 172.21.15.46:0/2738273538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3a80a0400 con 0x7fa3a81369d0 2024-09-16T04:45:58.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa39dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:45:58.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa39dffb640 1 -- 172.21.15.46:0/2738273538 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a000a860 con 0x7fa3a81369d0 2024-09-16T04:45:58.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa39dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:45:58.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa39dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:45:58.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa39dffb640 10 monclient: dump: 2024-09-16T04:45:58.791 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:45:58.791 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:45:58.791 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:58.791 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:45:58.791 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:45:58.791 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:45:58.791 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:45:58.791 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:45:58.791 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:45:58.791 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:45:58.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa3af4b7640 10 monclient: _renew_subs 2024-09-16T04:45:58.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa3af4b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:58.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa39f7fe640 4 set_mon_vals no callback set 2024-09-16T04:45:58.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.771+0000 7fa3af4b7640 1 -- 172.21.15.46:0/2738273538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3a806d930 con 0x7fa3a81369d0 2024-09-16T04:45:58.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.772+0000 7fa39dffb640 1 -- 172.21.15.46:0/2738273538 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fa3a001b070 con 0x7fa3a81369d0 2024-09-16T04:45:58.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.772+0000 7fa39dffb640 1 --2- 172.21.15.46:0/2738273538 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa38c0383e0 0x7fa38c03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:45:58.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.772+0000 7fa39dffb640 1 -- 172.21.15.46:0/2738273538 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(274..274 src has 1..274) ==== 4738+0+0 (secure 0 0 0) 0x7fa3a004ac20 con 0x7fa3a81369d0 2024-09-16T04:45:58.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.772+0000 7fa3af4b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:58.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.772+0000 7fa3af4b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:58.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.772+0000 7fa3af4b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:58.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.772+0000 7fa3af4b7640 1 -- 172.21.15.46:0/2738273538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3a8137580 con 0x7fa3a81369d0 2024-09-16T04:45:58.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.772+0000 7fa3ad253640 1 --2- 172.21.15.46:0/2738273538 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa38c0383e0 0x7fa38c03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:45:58.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.772+0000 7fa3ad253640 10 monclient: get_auth_request con 0x7fa38c0383e0 auth_method 0 2024-09-16T04:45:58.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.773+0000 7fa3ad253640 1 --2- 172.21.15.46:0/2738273538 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa38c0383e0 0x7fa38c03a8a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa398006fd0 tx=0x7fa398008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:45:58.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.777+0000 7fa39dffb640 1 -- 172.21.15.46:0/2738273538 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa3a001b980 con 0x7fa3a81369d0 2024-09-16T04:45:58.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.777+0000 7fa39dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:45:58.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:58.777+0000 7fa39dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:45:59.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:59.022+0000 7fa3af4b7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:45:59.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:59.022+0000 7fa3af4b7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:45:59.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:59.022+0000 7fa3af4b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:45:59.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:59.022+0000 7fa3af4b7640 1 -- 172.21.15.46:0/2738273538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa3a8069150 con 0x7fa3a81369d0 2024-09-16T04:45:59.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:59.022+0000 7fa39dffb640 1 -- 172.21.15.46:0/2738273538 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fa3a001bbe0 con 0x7fa3a81369d0 2024-09-16T04:45:59.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:59.022+0000 7fa39dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:45:59.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:59.022+0000 7fa39dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:45:59.025 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":82,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:45:59.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:59.025+0000 7fa3af4b7640 1 -- 172.21.15.46:0/2738273538 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa38c0383e0 msgr2=0x7fa38c03a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:59.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:59.025+0000 7fa3af4b7640 1 --2- 172.21.15.46:0/2738273538 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa38c0383e0 0x7fa38c03a8a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa398006fd0 tx=0x7fa398008040 comp rx=0 tx=0).stop 2024-09-16T04:45:59.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:59.025+0000 7fa3af4b7640 10 monclient: shutdown 2024-09-16T04:45:59.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:59.025+0000 7fa3af4b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:45:59.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:59.025+0000 7fa3af4b7640 1 -- 172.21.15.46:0/2738273538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a81369d0 msgr2=0x7fa3a806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:45:59.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:59.025+0000 7fa3af4b7640 1 --2- 172.21.15.46:0/2738273538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a81369d0 0x7fa3a806ef60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0000f80 tx=0x7fa3a000cbe0 comp rx=0 tx=0).stop 2024-09-16T04:45:59.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:59.025+0000 7fa3af4b7640 1 -- 172.21.15.46:0/2738273538 shutdown_connections 2024-09-16T04:45:59.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:59.025+0000 7fa3af4b7640 1 --2- 172.21.15.46:0/2738273538 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fa38c0383e0 0x7fa38c03a8a0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:59.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:59.025+0000 7fa3af4b7640 1 --2- 172.21.15.46:0/2738273538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a81369d0 0x7fa3a806ef60 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:59.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:59.025+0000 7fa3af4b7640 1 --2- 172.21.15.46:0/2738273538 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fa3a80a3ff0 0x7fa3a81d0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:59.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:59.025+0000 7fa3af4b7640 1 --2- 172.21.15.46:0/2738273538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3a80a3c10 0x7fa3a81cff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:45:59.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:59.025+0000 7fa3af4b7640 1 -- 172.21.15.46:0/2738273538 >> 172.21.15.46:0/2738273538 conn(0x7fa3a80a5d10 msgr2=0x7fa3a80a8770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:45:59.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:59.025+0000 7fa3af4b7640 1 -- 172.21.15.46:0/2738273538 shutdown_connections 2024-09-16T04:45:59.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:45:59.025+0000 7fa3af4b7640 1 -- 172.21.15.46:0/2738273538 wait complete. 2024-09-16T04:45:59.084 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39716]: starting. 2024-09-16T04:45:59.084 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39716]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-39714 2024-09-16T04:45:59.084 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39716]: shutting down. 2024-09-16T04:46:02.038 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:46:02.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.149+0000 7f0fc9cc9640 10 monclient: get_monmap_and_config 2024-09-16T04:46:02.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.149+0000 7f0fc9cc9640 10 monclient: build_initial_monmap 2024-09-16T04:46:02.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.149+0000 7f0fc9cc9640 10 monclient: monmap: 2024-09-16T04:46:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:02.150965+0000 2024-09-16T04:46:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:02.150965+0000 2024-09-16T04:46:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:02.153 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:02.153 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:02.153 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:02.153 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:02.153 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:02.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.150+0000 7f0fc9cc9640 1 Processor -- start 2024-09-16T04:46:02.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.150+0000 7f0fc9cc9640 1 -- start start 2024-09-16T04:46:02.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.150+0000 7f0fc9cc9640 10 monclient: init 2024-09-16T04:46:02.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.150+0000 7f0fc9cc9640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.150+0000 7f0fc9cc9640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-16T04:46:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.150+0000 7f0fc9cc9640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 0x7f0fc409bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.150+0000 7f0fc9cc9640 10 monclient(hunting): picked mon.noname-c con 0x7f0fc40a3ba0 addr v2:172.21.15.114:3300/0 2024-09-16T04:46:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.150+0000 7f0fc9cc9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 0x7f0fc40a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.150+0000 7f0fc9cc9640 10 monclient(hunting): picked mon.noname-a con 0x7f0fc409c1c0 addr v2:172.21.15.46:3300/0 2024-09-16T04:46:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.150+0000 7f0fc9cc9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0fc40a44c0 addr v1:172.21.15.114:6789/0 2024-09-16T04:46:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.150+0000 7f0fc9cc9640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fc409afc0 con 0x7f0fc40a44c0 2024-09-16T04:46:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.150+0000 7f0fc9cc9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.150+0000 7f0fc9cc9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0fc409b860 con 0x7f0fc409c1c0 2024-09-16T04:46:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.150+0000 7f0fc9cc9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.150+0000 7f0fc9cc9640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f0fc409b2a0 con 0x7f0fc40a3ba0 2024-09-16T04:46:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.150+0000 7f0fc9cc9640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.150+0000 7f0fc9cc9640 10 monclient(hunting): authenticate will time out at 1160.566869s 2024-09-16T04:46:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.151+0000 7f0fc2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 0x7f0fc40a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.151+0000 7f0fc2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 0x7f0fc40a3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55036/0 (socket says 172.21.15.46:55036) 2024-09-16T04:46:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.151+0000 7f0fc2ffd640 1 -- 172.21.15.46:0/2971904955 learned_addr learned my addr 172.21.15.46:0/2971904955 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.151+0000 7f0fc2ffd640 10 monclient(hunting): get_auth_request con 0x7f0fc409c1c0 auth_method 0 2024-09-16T04:46:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.151+0000 7f0fc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.151+0000 7f0fc2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:02.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.151+0000 7f0fc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:02.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.151+0000 7f0fc2ffd640 1 -- 172.21.15.46:0/2971904955 >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 msgr2=0x7f0fc40a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:46:02.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.151+0000 7f0fc2ffd640 1 -- 172.21.15.46:0/2971904955 >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 msgr2=0x7f0fc40a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:46:02.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.151+0000 7f0fc2ffd640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 0x7f0fc40a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:02.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.151+0000 7f0fc2ffd640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 0x7f0fc40a3f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:02.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.151+0000 7f0fc37fe640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 0x7f0fc409bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.151+0000 7f0fc37fe640 10 monclient(hunting): get_auth_request con 0x7f0fc40a3ba0 auth_method 0 2024-09-16T04:46:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.151+0000 7f0fc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.151+0000 7f0fc37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.151+0000 7f0fc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.151+0000 7f0fc37fe640 1 -- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 msgr2=0x7f0fc409bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:46:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.151+0000 7f0fc37fe640 1 -- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 msgr2=0x7f0fc409bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:46:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.151+0000 7f0fc37fe640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 0x7f0fc409bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.151+0000 7f0fc37fe640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 0x7f0fc409bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:02.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.352+0000 7f0fc2ffd640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 0x7f0fc40a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:02.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.352+0000 7f0fc2ffd640 10 monclient(hunting): get_auth_request con 0x7f0fc409c1c0 auth_method 0 2024-09-16T04:46:02.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.352+0000 7f0fc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:02.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.352+0000 7f0fc2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:02.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.352+0000 7f0fc2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:46:02.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.352+0000 7f0fc2ffd640 1 -- 172.21.15.46:0/2971904955 >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 msgr2=0x7f0fc40a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:46:02.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.352+0000 7f0fc2ffd640 1 -- 172.21.15.46:0/2971904955 >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 msgr2=0x7f0fc40a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:46:02.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.352+0000 7f0fc2ffd640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 0x7f0fc40a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:02.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.352+0000 7f0fc2ffd640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 0x7f0fc40a3f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:46:02.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.352+0000 7f0fc37fe640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 0x7f0fc409bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:02.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.352+0000 7f0fc37fe640 10 monclient(hunting): get_auth_request con 0x7f0fc40a3ba0 auth_method 0 2024-09-16T04:46:02.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.352+0000 7f0fc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:02.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.352+0000 7f0fc37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:02.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.352+0000 7f0fc37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:46:02.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.352+0000 7f0fc37fe640 1 -- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 msgr2=0x7f0fc409bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:46:02.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.352+0000 7f0fc37fe640 1 -- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 msgr2=0x7f0fc409bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:46:02.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.352+0000 7f0fc37fe640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 0x7f0fc409bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:02.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.352+0000 7f0fc37fe640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 0x7f0fc409bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:46:02.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.753+0000 7f0fc2ffd640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 0x7f0fc40a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:02.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.753+0000 7f0fc2ffd640 10 monclient(hunting): get_auth_request con 0x7f0fc409c1c0 auth_method 0 2024-09-16T04:46:02.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.753+0000 7f0fc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:02.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.753+0000 7f0fc2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:02.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.753+0000 7f0fc2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:46:02.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.753+0000 7f0fc2ffd640 1 -- 172.21.15.46:0/2971904955 >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 msgr2=0x7f0fc40a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:46:02.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.753+0000 7f0fc2ffd640 1 -- 172.21.15.46:0/2971904955 >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 msgr2=0x7f0fc40a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:46:02.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.753+0000 7f0fc2ffd640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 0x7f0fc40a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:02.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.753+0000 7f0fc2ffd640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 0x7f0fc40a3f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:46:02.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.753+0000 7f0fc37fe640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 0x7f0fc409bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:02.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.753+0000 7f0fc37fe640 10 monclient(hunting): get_auth_request con 0x7f0fc40a3ba0 auth_method 0 2024-09-16T04:46:02.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.753+0000 7f0fc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:02.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.753+0000 7f0fc37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:02.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.753+0000 7f0fc37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:46:02.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.753+0000 7f0fc37fe640 1 -- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 msgr2=0x7f0fc409bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:46:02.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.753+0000 7f0fc37fe640 1 -- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 msgr2=0x7f0fc409bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:46:02.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.753+0000 7f0fc37fe640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 0x7f0fc409bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:02.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:02.753+0000 7f0fc37fe640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 0x7f0fc409bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:46:03.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:03.554+0000 7f0fc2ffd640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 0x7f0fc40a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:03.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:03.554+0000 7f0fc2ffd640 10 monclient(hunting): get_auth_request con 0x7f0fc409c1c0 auth_method 0 2024-09-16T04:46:03.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:03.554+0000 7f0fc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:03.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:03.554+0000 7f0fc2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:03.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:03.554+0000 7f0fc2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:46:03.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:03.554+0000 7f0fc2ffd640 1 -- 172.21.15.46:0/2971904955 >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 msgr2=0x7f0fc40a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:46:03.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:03.554+0000 7f0fc2ffd640 1 -- 172.21.15.46:0/2971904955 >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 msgr2=0x7f0fc40a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:46:03.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:03.554+0000 7f0fc2ffd640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 0x7f0fc40a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:03.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:03.554+0000 7f0fc2ffd640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 0x7f0fc40a3f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:46:03.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:03.554+0000 7f0fc37fe640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 0x7f0fc409bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:03.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:03.554+0000 7f0fc37fe640 10 monclient(hunting): get_auth_request con 0x7f0fc40a3ba0 auth_method 0 2024-09-16T04:46:03.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:03.554+0000 7f0fc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:03.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:03.554+0000 7f0fc37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:03.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:03.554+0000 7f0fc37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:46:03.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:03.555+0000 7f0fc37fe640 1 -- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 msgr2=0x7f0fc409bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:46:03.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:03.555+0000 7f0fc37fe640 1 -- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 msgr2=0x7f0fc409bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:46:03.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:03.555+0000 7f0fc37fe640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 0x7f0fc409bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:03.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:03.555+0000 7f0fc37fe640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 0x7f0fc409bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:46:04.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.371+0000 7f0fc27fc640 1 -- 172.21.15.46:0/2971904955 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1665999108 0 0) 0x7f0fc409afc0 con 0x7f0fc40a44c0 2024-09-16T04:46:04.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.371+0000 7f0fc27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:46:04.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.371+0000 7f0fc27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.371+0000 7f0fc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.371+0000 7f0fc27fc640 10 monclient(hunting): my global_id is 6682 2024-09-16T04:46:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.371+0000 7f0fc27fc640 1 -- 172.21.15.46:0/2971904955 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fac003520 con 0x7f0fc40a44c0 2024-09-16T04:46:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.371+0000 7f0fc27fc640 1 -- 172.21.15.46:0/2971904955 <== mon.1 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1443163089 0 0) 0x7f0fac003520 con 0x7f0fc40a44c0 2024-09-16T04:46:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 1 -- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 msgr2=0x7f0fc409bc80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:46:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.114:3300/0 conn(0x7f0fc40a3ba0 0x7f0fc409bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 1 -- 172.21.15.46:0/2971904955 >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 msgr2=0x7f0fc40a3f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:46:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 1 --2- 172.21.15.46:0/2971904955 >> v2:172.21.15.46:3300/0 conn(0x7f0fc409c1c0 0x7f0fc40a3f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 10 monclient: _finish_hunting 0 2024-09-16T04:46:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 1 monclient: found mon.noname-c-legacy 2024-09-16T04:46:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.114:6789/0 2024-09-16T04:46:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 1 -- 172.21.15.46:0/2971904955 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc409b580 con 0x7f0fc40a44c0 2024-09-16T04:46:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 10 monclient: _finish_auth 0 2024-09-16T04:46:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:46:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 1 -- 172.21.15.46:0/2971904955 <== mon.1 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f0fbc0019f0 con 0x7f0fc40a44c0 2024-09-16T04:46:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc9cc9640 5 monclient: authenticate success, global_id 6682 2024-09-16T04:46:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc9cc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T04:46:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 10 monclient: dump: 2024-09-16T04:46:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:04.380 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:04.380 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:04.380 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:04.380 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:04.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T04:46:04.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:04.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 1 -- 172.21.15.46:0/2971904955 >> v1:172.21.15.114:6789/0 conn(0x7f0fc40a44c0 legacy=0x7f0fc4135950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:04.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:46:04.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 1 --2- 172.21.15.46:0/2971904955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fac003a00 0x7f0fac003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 10 monclient(hunting): picked mon.a con 0x7f0fac003a00 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:04.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 1 --2- 172.21.15.46:0/2971904955 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0fac004800 0x7f0fac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 10 monclient(hunting): picked mon.c con 0x7f0fac004800 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:04.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 1 --2- 172.21.15.46:0/2971904955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fac005210 0x7f0fac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 10 monclient(hunting): picked mon.b con 0x7f0fac005210 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:04.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:04.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 1 -- 172.21.15.46:0/2971904955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc409b860 con 0x7f0fac003a00 2024-09-16T04:46:04.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:04.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 1 -- 172.21.15.46:0/2971904955 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc409b2a0 con 0x7f0fac005210 2024-09-16T04:46:04.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:04.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 1 -- 172.21.15.46:0/2971904955 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f0fac00c1c0 con 0x7f0fac004800 2024-09-16T04:46:04.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:04.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:46:04.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc9cc9640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:04.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc2ffd640 1 --2- 172.21.15.46:0/2971904955 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0fac004800 0x7f0fac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc2ffd640 10 monclient(hunting): get_auth_request con 0x7f0fac004800 auth_method 0 2024-09-16T04:46:04.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:04.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:04.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:46:04.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc3fff640 1 --2- 172.21.15.46:0/2971904955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fac005210 0x7f0fac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc3fff640 10 monclient(hunting): get_auth_request con 0x7f0fac005210 auth_method 0 2024-09-16T04:46:04.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:04.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:04.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:46:04.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:04.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:04.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:46:04.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:04.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:04.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:46:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc3fff640 10 monclient(hunting): handle_auth_done global_id 6682 payload 293 2024-09-16T04:46:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc3fff640 1 -- 172.21.15.46:0/2971904955 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0fac004800 msgr2=0x7f0fac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc3fff640 1 --2- 172.21.15.46:0/2971904955 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0fac004800 0x7f0fac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc3fff640 1 -- 172.21.15.46:0/2971904955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fac003a00 msgr2=0x7f0fac003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc3fff640 1 --2- 172.21.15.46:0/2971904955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fac003a00 0x7f0fac003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc3fff640 10 monclient: _finish_hunting 0 2024-09-16T04:46:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.372+0000 7f0fc3fff640 1 monclient: found mon.b 2024-09-16T04:46:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.373+0000 7f0fc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.373+0000 7f0fc3fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:46:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.373+0000 7f0fc3fff640 1 -- 172.21.15.46:0/2971904955 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fac003520 con 0x7f0fac005210 2024-09-16T04:46:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.373+0000 7f0fc3fff640 10 monclient: _finish_auth 0 2024-09-16T04:46:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.373+0000 7f0fc3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:46:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.373+0000 7f0fc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.373+0000 7f0fc3fff640 1 -- 172.21.15.46:0/2971904955 reap_dead start 2024-09-16T04:46:04.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.374+0000 7f0fc3fff640 1 --2- 172.21.15.46:0/2971904955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fac005210 0x7f0fac00b9c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc00c490 tx=0x7f0fac005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c182fa837610c9d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:04.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.374+0000 7f0fc27fc640 1 -- 172.21.15.46:0/2971904955 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc00c0e0 con 0x7f0fac005210 2024-09-16T04:46:04.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.374+0000 7f0fc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:04.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.374+0000 7f0fc27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:46:04.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.374+0000 7f0fc27fc640 10 monclient: dump: 2024-09-16T04:46:04.384 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:04.384 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:04.384 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.384 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.384 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:04.384 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:04.384 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:04.384 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:04.384 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:04.385 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:04.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.374+0000 7f0fc27fc640 1 -- 172.21.15.46:0/2971904955 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fbc02f640 con 0x7f0fac005210 2024-09-16T04:46:04.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.374+0000 7f0fc27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:04.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.374+0000 7f0fc27fc640 1 -- 172.21.15.46:0/2971904955 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc02fa30 con 0x7f0fac005210 2024-09-16T04:46:04.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.374+0000 7f0fc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:04.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.374+0000 7f0fc27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:46:04.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.374+0000 7f0fc27fc640 10 monclient: dump: 2024-09-16T04:46:04.385 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:04.385 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:04.385 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.385 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.385 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:04.385 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:04.385 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:04.386 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:04.386 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:04.386 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:04.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.374+0000 7f0fc9cc9640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:04.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.374+0000 7f0fc9cc9640 4 set_mon_vals no callback set 2024-09-16T04:46:04.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.375+0000 7f0fc9cc9640 10 monclient: shutdown 2024-09-16T04:46:04.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.375+0000 7f0fc9cc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:04.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.375+0000 7f0fc9cc9640 1 -- 172.21.15.46:0/2971904955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fac005210 msgr2=0x7f0fac00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:04.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.375+0000 7f0fc9cc9640 1 --2- 172.21.15.46:0/2971904955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fac005210 0x7f0fac00b9c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc00c490 tx=0x7f0fac005830 comp rx=0 tx=0).stop 2024-09-16T04:46:04.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.375+0000 7f0fc9cc9640 1 -- 172.21.15.46:0/2971904955 shutdown_connections 2024-09-16T04:46:04.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.375+0000 7f0fc9cc9640 1 --2- 172.21.15.46:0/2971904955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fac005210 0x7f0fac00b9c0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.375+0000 7f0fc9cc9640 1 -- 172.21.15.46:0/2971904955 >> 172.21.15.46:0/2971904955 conn(0x7f0fc40a60c0 msgr2=0x7f0fc40a64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:04.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.375+0000 7f0fc9cc9640 1 -- 172.21.15.46:0/2971904955 shutdown_connections 2024-09-16T04:46:04.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.375+0000 7f0fc9cc9640 1 -- 172.21.15.46:0/2971904955 wait complete. 2024-09-16T04:46:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.377+0000 7f0fc9cc9640 10 monclient: build_initial_monmap 2024-09-16T04:46:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.377+0000 7f0fc9cc9640 10 monclient: monmap: 2024-09-16T04:46:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:04.393 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:04.393 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:04.393 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:04.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.378+0000 7f0fc9cc9640 1 Processor -- start 2024-09-16T04:46:04.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.379+0000 7f0fc9cc9640 1 -- start start 2024-09-16T04:46:04.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.379+0000 7f0fc9cc9640 10 monclient: init 2024-09-16T04:46:04.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc9cc9640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:04.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc9cc9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:46:04.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc9cc9640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0fac005210 0x7f0fc4136cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc9cc9640 10 monclient(hunting): picked mon.noname-c con 0x7f0fac005210 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:04.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc9cc9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc41cb8a0 0x7f0fc41371f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc9cc9640 10 monclient(hunting): picked mon.noname-a con 0x7f0fc41cb8a0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:04.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc9cc9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc41206e0 0x7f0fc4120b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc9cc9640 10 monclient(hunting): picked mon.noname-b con 0x7f0fc41206e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc9cc9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc9cc9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc40a5540 con 0x7f0fc41cb8a0 2024-09-16T04:46:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc9cc9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc9cc9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc40a53c0 con 0x7f0fc41206e0 2024-09-16T04:46:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc9cc9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc9cc9640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc40a3f80 con 0x7f0fac005210 2024-09-16T04:46:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc37fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0fac005210 0x7f0fc4136cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc37fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0fac005210 0x7f0fc4136cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:43946/0 (socket says 172.21.15.46:43946) 2024-09-16T04:46:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc37fe640 1 -- 172.21.15.46:0/3268811116 learned_addr learned my addr 172.21.15.46:0/3268811116 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc3fff640 1 --2- 172.21.15.46:0/3268811116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc41206e0 0x7f0fc4120b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc9cc9640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc2ffd640 1 --2- 172.21.15.46:0/3268811116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc41cb8a0 0x7f0fc41371f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc9cc9640 10 monclient(hunting): authenticate will time out at 1162.796813s 2024-09-16T04:46:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc37fe640 10 monclient(hunting): get_auth_request con 0x7f0fac005210 auth_method 0 2024-09-16T04:46:04.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:04.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:04.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:04.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc3fff640 10 monclient(hunting): get_auth_request con 0x7f0fc41206e0 auth_method 0 2024-09-16T04:46:04.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:04.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:04.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:04.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc2ffd640 10 monclient(hunting): get_auth_request con 0x7f0fc41cb8a0 auth_method 0 2024-09-16T04:46:04.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:04.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:04.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.380+0000 7f0fc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:04.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.381+0000 7f0fc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:04.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.381+0000 7f0fc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:04.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.381+0000 7f0fc37fe640 1 -- 172.21.15.46:0/3268811116 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0fac005210 msgr2=0x7f0fc4136cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:46:04.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.381+0000 7f0fc37fe640 1 -- 172.21.15.46:0/3268811116 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0fac005210 msgr2=0x7f0fc4136cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:46:04.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.381+0000 7f0fc37fe640 1 --2- 172.21.15.46:0/3268811116 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0fac005210 0x7f0fc4136cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:04.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.381+0000 7f0fc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:04.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.381+0000 7f0fc37fe640 1 --2- 172.21.15.46:0/3268811116 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0fac005210 0x7f0fc4136cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:04.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.381+0000 7f0fc2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:04.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.381+0000 7f0fc2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:04.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.381+0000 7f0fc2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:04.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.381+0000 7f0fc3fff640 10 monclient(hunting): handle_auth_done global_id 6685 payload 824 2024-09-16T04:46:04.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.381+0000 7f0fc3fff640 1 -- 172.21.15.46:0/3268811116 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0fac005210 msgr2=0x7f0fc4136cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:46:04.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.381+0000 7f0fc3fff640 1 --2- 172.21.15.46:0/3268811116 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0fac005210 0x7f0fc4136cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.381+0000 7f0fc3fff640 1 -- 172.21.15.46:0/3268811116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc41cb8a0 msgr2=0x7f0fc41371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:04.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.381+0000 7f0fc3fff640 1 --2- 172.21.15.46:0/3268811116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc41cb8a0 0x7f0fc41371f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.381+0000 7f0fc3fff640 10 monclient: _finish_hunting 0 2024-09-16T04:46:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.381+0000 7f0fc3fff640 1 monclient: found mon.noname-b 2024-09-16T04:46:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.381+0000 7f0fc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.381+0000 7f0fc3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3300/0 2024-09-16T04:46:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.381+0000 7f0fc3fff640 1 -- 172.21.15.46:0/3268811116 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc41240b0 con 0x7f0fc41206e0 2024-09-16T04:46:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fc3fff640 1 --2- 172.21.15.46:0/3268811116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc41206e0 0x7f0fc4120b60 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f0fc409ccf0 tx=0x7f0fc409b750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fabfff640 1 -- 172.21.15.46:0/3268811116 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc03d070 con 0x7f0fc41206e0 2024-09-16T04:46:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fabfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:46:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fabfff640 10 monclient: dump: 2024-09-16T04:46:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:04.398 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fabfff640 10 monclient: _finish_auth 0 2024-09-16T04:46:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fabfff640 10 monclient: _check_auth_tickets 2024-09-16T04:46:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fabfff640 1 -- 172.21.15.46:0/3268811116 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fbc006d60 con 0x7f0fc41206e0 2024-09-16T04:46:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fc9cc9640 5 monclient: authenticate success, global_id 6685 2024-09-16T04:46:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fc9cc9640 10 monclient: _renew_subs 2024-09-16T04:46:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fc9cc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:46:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fc9cc9640 1 -- 172.21.15.46:0/3268811116 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fc4122ee0 con 0x7f0fc41206e0 2024-09-16T04:46:04.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fabfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:04.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fabfff640 1 -- 172.21.15.46:0/3268811116 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc02fa30 con 0x7f0fc41206e0 2024-09-16T04:46:04.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:04.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fabfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:46:04.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fabfff640 10 monclient: dump: 2024-09-16T04:46:04.399 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:04.399 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:04.399 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.399 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.399 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:04.399 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:04.399 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:04.399 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:04.399 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:04.399 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:04.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fc1ffb640 4 set_mon_vals no callback set 2024-09-16T04:46:04.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fc9cc9640 10 monclient: _renew_subs 2024-09-16T04:46:04.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fc9cc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:46:04.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.382+0000 7f0fc9cc9640 1 -- 172.21.15.46:0/3268811116 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fc4122c60 con 0x7f0fc41206e0 2024-09-16T04:46:04.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.383+0000 7f0fabfff640 1 -- 172.21.15.46:0/3268811116 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f0fbc040070 con 0x7f0fc41206e0 2024-09-16T04:46:04.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.383+0000 7f0fc9cc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:04.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.383+0000 7f0fc9cc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:04.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.383+0000 7f0fc9cc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:46:04.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.383+0000 7f0fc9cc9640 1 -- 172.21.15.46:0/3268811116 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fc409bc80 con 0x7f0fc41206e0 2024-09-16T04:46:04.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.384+0000 7f0fabfff640 1 --2- 172.21.15.46:0/3268811116 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0fb00383e0 0x7f0fb003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.384+0000 7f0fabfff640 1 -- 172.21.15.46:0/3268811116 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 4346+0+0 (secure 0 0 0) 0x7f0fbc06aa90 con 0x7f0fc41206e0 2024-09-16T04:46:04.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.384+0000 7f0fc37fe640 1 --2- 172.21.15.46:0/3268811116 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0fb00383e0 0x7f0fb003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.384+0000 7f0fc37fe640 10 monclient: get_auth_request con 0x7f0fb00383e0 auth_method 0 2024-09-16T04:46:04.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.384+0000 7f0fc37fe640 1 --2- 172.21.15.46:0/3268811116 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0fb00383e0 0x7f0fb003a8a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4006c00 tx=0x7f0fb40112a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:04.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.389+0000 7f0fabfff640 1 -- 172.21.15.46:0/3268811116 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0fbc033030 con 0x7f0fc41206e0 2024-09-16T04:46:04.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.389+0000 7f0fabfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:04.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.389+0000 7f0fabfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.642+0000 7f0fc9cc9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:46:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.642+0000 7f0fc9cc9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.642+0000 7f0fc9cc9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:46:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.642+0000 7f0fc9cc9640 1 -- 172.21.15.46:0/3268811116 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0fc409d920 con 0x7f0fc41206e0 2024-09-16T04:46:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.643+0000 7f0fabfff640 1 -- 172.21.15.46:0/3268811116 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f0fbc0379d0 con 0x7f0fc41206e0 2024-09-16T04:46:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.643+0000 7f0fabfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.643+0000 7f0fabfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:04.645 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:46:04.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.645+0000 7f0fc9cc9640 1 -- 172.21.15.46:0/3268811116 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0fb00383e0 msgr2=0x7f0fb003a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:04.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.645+0000 7f0fc9cc9640 1 --2- 172.21.15.46:0/3268811116 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0fb00383e0 0x7f0fb003a8a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4006c00 tx=0x7f0fb40112a0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.645+0000 7f0fc9cc9640 10 monclient: shutdown 2024-09-16T04:46:04.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.645+0000 7f0fc9cc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:04.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.645+0000 7f0fc9cc9640 1 -- 172.21.15.46:0/3268811116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc41206e0 msgr2=0x7f0fc4120b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:04.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.645+0000 7f0fc9cc9640 1 --2- 172.21.15.46:0/3268811116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc41206e0 0x7f0fc4120b60 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f0fc409ccf0 tx=0x7f0fc409b750 comp rx=0 tx=0).stop 2024-09-16T04:46:04.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.645+0000 7f0fc9cc9640 1 -- 172.21.15.46:0/3268811116 shutdown_connections 2024-09-16T04:46:04.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.645+0000 7f0fc9cc9640 1 --2- 172.21.15.46:0/3268811116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc41206e0 0x7f0fc4120b60 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.645+0000 7f0fc9cc9640 1 --2- 172.21.15.46:0/3268811116 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0fb00383e0 0x7f0fb003a8a0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.645+0000 7f0fc9cc9640 1 --2- 172.21.15.46:0/3268811116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc41cb8a0 0x7f0fc41371f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.645+0000 7f0fc9cc9640 1 --2- 172.21.15.46:0/3268811116 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0fac005210 0x7f0fc4136cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.645+0000 7f0fc9cc9640 1 -- 172.21.15.46:0/3268811116 >> 172.21.15.46:0/3268811116 conn(0x7f0fc40a60c0 msgr2=0x7f0fc409dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:04.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.645+0000 7f0fc9cc9640 1 -- 172.21.15.46:0/3268811116 shutdown_connections 2024-09-16T04:46:04.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.645+0000 7f0fc9cc9640 1 -- 172.21.15.46:0/3268811116 wait complete. 2024-09-16T04:46:04.658 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:46:04.658 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:46:04.658 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:46:04.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.758+0000 7f5766c27640 10 monclient: get_monmap_and_config 2024-09-16T04:46:04.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.758+0000 7f5766c27640 10 monclient: build_initial_monmap 2024-09-16T04:46:04.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.758+0000 7f5766c27640 10 monclient: monmap: 2024-09-16T04:46:04.761 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:04.761 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:04.761 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:04.760086+0000 2024-09-16T04:46:04.761 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:04.760086+0000 2024-09-16T04:46:04.761 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:04.761 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:04.761 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:04.761 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:04.761 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:04.761 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:04.761 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:04.761 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:04.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.759+0000 7f5766c27640 1 Processor -- start 2024-09-16T04:46:04.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.759+0000 7f5766c27640 1 -- start start 2024-09-16T04:46:04.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.759+0000 7f5766c27640 10 monclient: init 2024-09-16T04:46:04.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.759+0000 7f5766c27640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:04.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.759+0000 7f5766c27640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-16T04:46:04.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.759+0000 7f5766c27640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f576006e160 addr v1:172.21.15.114:6789/0 2024-09-16T04:46:04.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.759+0000 7f5766c27640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5760139c50 0x7f576013c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.759+0000 7f5766c27640 10 monclient(hunting): picked mon.noname-c con 0x7f5760139c50 addr v2:172.21.15.114:3300/0 2024-09-16T04:46:04.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.759+0000 7f5766c27640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f576013c580 0x7f576013e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.759+0000 7f5766c27640 10 monclient(hunting): picked mon.noname-b con 0x7f576013c580 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:04.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5766c27640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57600a3fd0 con 0x7f576006e160 2024-09-16T04:46:04.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5766c27640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:04.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5766c27640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f576006e540 con 0x7f5760139c50 2024-09-16T04:46:04.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5766c27640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:04.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5766c27640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f57600a0bd0 con 0x7f576013c580 2024-09-16T04:46:04.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5766c27640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:04.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5766c27640 10 monclient(hunting): authenticate will time out at 1163.176057s 2024-09-16T04:46:04.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f57651c4640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f576013c580 0x7f576013e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f57651c4640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f576013c580 0x7f576013e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:43958/0 (socket says 172.21.15.46:43958) 2024-09-16T04:46:04.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f57651c4640 1 -- 172.21.15.46:0/3616313353 learned_addr learned my addr 172.21.15.46:0/3616313353 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:04.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f57651c4640 10 monclient(hunting): get_auth_request con 0x7f576013c580 auth_method 0 2024-09-16T04:46:04.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f57651c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:04.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f57651c4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:04.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f57651c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:04.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f57651c4640 1 -- 172.21.15.46:0/3616313353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f576013c580 msgr2=0x7f576013e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:46:04.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f57651c4640 1 -- 172.21.15.46:0/3616313353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f576013c580 msgr2=0x7f576013e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:46:04.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f57651c4640 1 --2- 172.21.15.46:0/3616313353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f576013c580 0x7f576013e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:04.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5757fff640 1 --2- 172.21.15.46:0/3616313353 >> v2:172.21.15.114:3300/0 conn(0x7f5760139c50 0x7f576013c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5757fff640 10 monclient(hunting): get_auth_request con 0x7f5760139c50 auth_method 0 2024-09-16T04:46:04.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5757fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:04.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5757fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:04.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5757fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:04.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f57651c4640 1 --2- 172.21.15.46:0/3616313353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f576013c580 0x7f576013e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:04.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5757fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:04.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5757fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:04.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5757fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:04.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5757fff640 10 monclient(hunting): handle_auth_done global_id 6688 payload 274 2024-09-16T04:46:04.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5757fff640 1 -- 172.21.15.46:0/3616313353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f576013c580 msgr2=0x7f576013e970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:46:04.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5757fff640 1 --2- 172.21.15.46:0/3616313353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f576013c580 0x7f576013e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5757fff640 1 -- 172.21.15.46:0/3616313353 >> v1:172.21.15.114:6789/0 conn(0x7f576006e160 legacy=0x7f57600a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:04.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5757fff640 10 monclient: _finish_hunting 0 2024-09-16T04:46:04.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5757fff640 1 monclient: found mon.noname-c 2024-09-16T04:46:04.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5757fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:04.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5757fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:46:04.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f5757fff640 1 -- 172.21.15.46:0/3616313353 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57600a0eb0 con 0x7f5760139c50 2024-09-16T04:46:04.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f57577fe640 1 -- 172.21.15.46:0/3616313353 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 277911585 0 0) 0x7f575c001110 con 0x7f576006e160 2024-09-16T04:46:04.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.760+0000 7f57577fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:46:04.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.761+0000 7f5757fff640 1 --2- 172.21.15.46:0/3616313353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5760139c50 0x7f576013c040 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f57500097a0 tx=0x7f575002a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d707e8e43479ac9c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:04.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.761+0000 7f57577fe640 1 -- 172.21.15.46:0/3616313353 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5750037070 con 0x7f5760139c50 2024-09-16T04:46:04.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.761+0000 7f57577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:04.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.761+0000 7f57577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:04.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.761+0000 7f57577fe640 10 monclient: dump: 2024-09-16T04:46:04.775 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:04.775 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:04.775 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.775 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.775 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:04.775 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:04.775 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:04.775 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:04.775 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:04.776 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:04.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.761+0000 7f57577fe640 10 monclient: _finish_auth 0 2024-09-16T04:46:04.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.761+0000 7f57577fe640 10 monclient: _check_auth_tickets 2024-09-16T04:46:04.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.761+0000 7f57577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:04.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.761+0000 7f57577fe640 1 -- 172.21.15.46:0/3616313353 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f575002adb0 con 0x7f5760139c50 2024-09-16T04:46:04.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.761+0000 7f5766c27640 5 monclient: authenticate success, global_id 6688 2024-09-16T04:46:04.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.761+0000 7f5766c27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:04.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.761+0000 7f57577fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:04.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.761+0000 7f5766c27640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:04.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.761+0000 7f5766c27640 4 set_mon_vals no callback set 2024-09-16T04:46:04.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.761+0000 7f57577fe640 1 -- 172.21.15.46:0/3616313353 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5750032660 con 0x7f5760139c50 2024-09-16T04:46:04.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.761+0000 7f57577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:04.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.761+0000 7f57577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:46:04.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.761+0000 7f57577fe640 10 monclient: dump: 2024-09-16T04:46:04.777 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:04.777 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:04.777 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.777 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.777 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:04.777 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:04.777 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:04.777 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:04.777 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:04.778 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:04.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.762+0000 7f5766c27640 10 monclient: shutdown 2024-09-16T04:46:04.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.762+0000 7f5766c27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:04.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.762+0000 7f5766c27640 1 -- 172.21.15.46:0/3616313353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5760139c50 msgr2=0x7f576013c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:04.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.762+0000 7f5766c27640 1 --2- 172.21.15.46:0/3616313353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5760139c50 0x7f576013c040 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f57500097a0 tx=0x7f575002a060 comp rx=0 tx=0).stop 2024-09-16T04:46:04.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.762+0000 7f5766c27640 1 -- 172.21.15.46:0/3616313353 shutdown_connections 2024-09-16T04:46:04.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.762+0000 7f5766c27640 1 --2- 172.21.15.46:0/3616313353 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f576013c580 0x7f576013e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.762+0000 7f5766c27640 1 --2- 172.21.15.46:0/3616313353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5760139c50 0x7f576013c040 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.762+0000 7f5766c27640 1 -- 172.21.15.46:0/3616313353 >> 172.21.15.46:0/3616313353 conn(0x7f57600a5c10 msgr2=0x7f57600a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:04.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.762+0000 7f5766c27640 1 -- 172.21.15.46:0/3616313353 shutdown_connections 2024-09-16T04:46:04.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.762+0000 7f5766c27640 1 -- 172.21.15.46:0/3616313353 wait complete. 2024-09-16T04:46:04.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.762+0000 7f5766c27640 10 monclient: build_initial_monmap 2024-09-16T04:46:04.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.762+0000 7f5766c27640 10 monclient: monmap: 2024-09-16T04:46:04.778 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:04.779 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:04.779 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:04.779 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:04.779 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:04.779 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:04.779 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:04.779 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:04.779 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:04.779 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:04.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f5766c27640 1 Processor -- start 2024-09-16T04:46:04.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f5766c27640 1 -- start start 2024-09-16T04:46:04.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f5766c27640 10 monclient: init 2024-09-16T04:46:04.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f5766c27640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:04.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f5766c27640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:46:04.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f5766c27640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f576006e160 0x7f576009e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f5766c27640 10 monclient(hunting): picked mon.noname-c con 0x7f576006e160 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:04.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f5766c27640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5760139c50 0x7f576009e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f5766c27640 10 monclient(hunting): picked mon.noname-b con 0x7f5760139c50 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:04.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f5766c27640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f576013c580 0x7f576009ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f5766c27640 10 monclient(hunting): picked mon.noname-a con 0x7f576013c580 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:04.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f5766c27640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:04.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f5766c27640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f57600a34a0 con 0x7f576013c580 2024-09-16T04:46:04.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f5766c27640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:04.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f5766c27640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f57600a4d90 con 0x7f5760139c50 2024-09-16T04:46:04.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f5766c27640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:04.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f5766c27640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f57601205d0 con 0x7f576006e160 2024-09-16T04:46:04.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f5766c27640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:04.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f5766c27640 10 monclient(hunting): authenticate will time out at 1163.179748s 2024-09-16T04:46:04.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f57649c3640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f576006e160 0x7f576009e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f57649c3640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f576006e160 0x7f576009e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:43962/0 (socket says 172.21.15.46:43962) 2024-09-16T04:46:04.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f57649c3640 1 -- 172.21.15.46:0/4214498239 learned_addr learned my addr 172.21.15.46:0/4214498239 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:04.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f57649c3640 10 monclient(hunting): get_auth_request con 0x7f576006e160 auth_method 0 2024-09-16T04:46:04.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f57649c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:04.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f57649c3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:04.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f57649c3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:04.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f57651c4640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f576013c580 0x7f576009ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f57649c3640 1 -- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f576006e160 msgr2=0x7f576009e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:46:04.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f57651c4640 10 monclient(hunting): get_auth_request con 0x7f576013c580 auth_method 0 2024-09-16T04:46:04.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f57649c3640 1 -- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f576006e160 msgr2=0x7f576009e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:46:04.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f57651c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:04.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f57651c4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:04.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f57651c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:04.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f57649c3640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f576006e160 0x7f576009e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:04.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.763+0000 7f57649c3640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f576006e160 0x7f576009e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:04.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57651c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:04.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57651c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:04.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57651c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:04.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f5757fff640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5760139c50 0x7f576009e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f5757fff640 10 monclient(hunting): get_auth_request con 0x7f5760139c50 auth_method 0 2024-09-16T04:46:04.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f5757fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:04.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f5757fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:04.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f5757fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:04.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f5757fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:04.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f5757fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:04.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f5757fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:04.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57651c4640 10 monclient(hunting): handle_auth_done global_id 74628 payload 824 2024-09-16T04:46:04.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57651c4640 1 -- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f576006e160 msgr2=0x7f576009e270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:46:04.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57651c4640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f576006e160 0x7f576009e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57651c4640 1 -- 172.21.15.46:0/4214498239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5760139c50 msgr2=0x7f576009e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:04.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57651c4640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5760139c50 0x7f576009e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57651c4640 10 monclient: _finish_hunting 0 2024-09-16T04:46:04.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57651c4640 1 monclient: found mon.noname-a 2024-09-16T04:46:04.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57651c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:04.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57651c4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:46:04.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57651c4640 1 -- 172.21.15.46:0/4214498239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5760121860 con 0x7f576013c580 2024-09-16T04:46:04.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57651c4640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f576013c580 0x7f576009ecf0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5760067c70 tx=0x7f5758007e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:04.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57557fa640 1 -- 172.21.15.46:0/4214498239 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575801a070 con 0x7f576013c580 2024-09-16T04:46:04.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:04.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57557fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:46:04.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57557fa640 10 monclient: dump: 2024-09-16T04:46:04.784 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:04.784 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:04.784 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.784 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.784 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:04.784 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:04.784 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:04.784 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:04.784 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:04.785 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:04.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57557fa640 10 monclient: _finish_auth 0 2024-09-16T04:46:04.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57557fa640 10 monclient: _check_auth_tickets 2024-09-16T04:46:04.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:04.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57557fa640 1 -- 172.21.15.46:0/4214498239 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f575800db50 con 0x7f576013c580 2024-09-16T04:46:04.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57557fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:04.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57557fa640 1 -- 172.21.15.46:0/4214498239 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5758015400 con 0x7f576013c580 2024-09-16T04:46:04.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:04.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:04.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f57557fa640 10 monclient: dump: 2024-09-16T04:46:04.785 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:04.785 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:04.785 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.785 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.786 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:04.786 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:04.786 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:04.786 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:04.786 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:04.786 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:04.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f5766c27640 5 monclient: authenticate success, global_id 74628 2024-09-16T04:46:04.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f5766c27640 10 monclient: _renew_subs 2024-09-16T04:46:04.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f5766c27640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:04.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f5766c27640 1 -- 172.21.15.46:0/4214498239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5760121b40 con 0x7f576013c580 2024-09-16T04:46:04.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f5756ffd640 4 set_mon_vals no callback set 2024-09-16T04:46:04.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f5766c27640 10 monclient: _renew_subs 2024-09-16T04:46:04.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f5766c27640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:04.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.764+0000 7f5766c27640 1 -- 172.21.15.46:0/4214498239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57600a0300 con 0x7f576013c580 2024-09-16T04:46:04.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.765+0000 7f5766c27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:04.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.765+0000 7f5766c27640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5760135950 0x7f5760135d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.765+0000 7f5766c27640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:04.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.765+0000 7f5766c27640 1 -- 172.21.15.46:0/4214498239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f57600a3620 con 0x7f5760135950 2024-09-16T04:46:04.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.765+0000 7f57557fa640 1 -- 172.21.15.46:0/4214498239 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f575801d070 con 0x7f576013c580 2024-09-16T04:46:04.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.765+0000 7f57649c3640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5760135950 0x7f5760135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.765+0000 7f57649c3640 10 monclient: get_auth_request con 0x7f5760135950 auth_method 0 2024-09-16T04:46:04.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.765+0000 7f57649c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:04.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.765+0000 7f57649c3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:04.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.765+0000 7f57649c3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:04.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.765+0000 7f57649c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:04.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.765+0000 7f57649c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:04.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.765+0000 7f57649c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:04.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.765+0000 7f57557fa640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f574c0383b0 0x7f574c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.765+0000 7f57557fa640 1 -- 172.21.15.46:0/4214498239 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 4346+0+0 (secure 0 0 0) 0x7f575804b370 con 0x7f576013c580 2024-09-16T04:46:04.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.766+0000 7f57649c3640 10 monclient(hunting): handle_auth_done global_id 74631 payload 824 2024-09-16T04:46:04.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.766+0000 7f57649c3640 1 -- 172.21.15.46:0/4214498239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f57600a5ff0 con 0x7f5760135950 2024-09-16T04:46:04.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.766+0000 7f5757fff640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f574c0383b0 0x7f574c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.766+0000 7f5757fff640 10 monclient: get_auth_request con 0x7f574c0383b0 auth_method 0 2024-09-16T04:46:04.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.766+0000 7f57649c3640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5760135950 0x7f5760135d30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f575c007ce0 tx=0x7f575c00e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:04.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.766+0000 7f57557fa640 1 -- 172.21.15.46:0/4214498239 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c0052e0 con 0x7f5760135950 2024-09-16T04:46:04.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.766+0000 7f57557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:04.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.766+0000 7f57557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:04.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.766+0000 7f57557fa640 10 monclient: dump: 2024-09-16T04:46:04.788 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:04.788 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:04.788 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.788 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:04.789 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.766+0000 7f57557fa640 1 -- 172.21.15.46:0/4214498239 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f575c018070 con 0x7f5760135950 2024-09-16T04:46:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.766+0000 7f57557fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.766+0000 7f57557fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.766+0000 7f57557fa640 1 -- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5760135950 msgr2=0x7f5760135d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.766+0000 7f57557fa640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5760135950 0x7f5760135d30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f575c007ce0 tx=0x7f575c00e960 comp rx=0 tx=0).stop 2024-09-16T04:46:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.766+0000 7f57557fa640 1 -- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5760135950 msgr2=0x7f5760135d30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.766+0000 7f57557fa640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5760135950 0x7f5760135d30 secure :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f575c007ce0 tx=0x7f575c00e960 comp rx=0 tx=0).stop 2024-09-16T04:46:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.766+0000 7f5757fff640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f574c0383b0 0x7f574c03a870 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5750034000 tx=0x7f575002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:04.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.772+0000 7f5766c27640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:46:04.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.772+0000 7f5766c27640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57601392b0 0x7f5760139690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.772+0000 7f5766c27640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:04.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.772+0000 7f5766c27640 1 -- 172.21.15.46:0/4214498239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f57600a3140 con 0x7f57601392b0 2024-09-16T04:46:04.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.772+0000 7f57649c3640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57601392b0 0x7f5760139690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.772+0000 7f57649c3640 10 monclient: get_auth_request con 0x7f57601392b0 auth_method 0 2024-09-16T04:46:04.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.772+0000 7f57649c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:04.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.772+0000 7f57649c3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:04.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.772+0000 7f57649c3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:04.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.773+0000 7f57649c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:04.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.773+0000 7f57649c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:04.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.773+0000 7f57649c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:04.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.773+0000 7f57649c3640 10 monclient(hunting): handle_auth_done global_id 74634 payload 824 2024-09-16T04:46:04.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.773+0000 7f57649c3640 1 -- 172.21.15.46:0/4214498239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5760120130 con 0x7f57601392b0 2024-09-16T04:46:04.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.773+0000 7f57649c3640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57601392b0 0x7f5760139690 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f575c007ce0 tx=0x7f575c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:04.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.773+0000 7f57557fa640 1 -- 172.21.15.46:0/4214498239 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c0035f0 con 0x7f57601392b0 2024-09-16T04:46:04.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.773+0000 7f57557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:04.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.773+0000 7f57557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:04.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.773+0000 7f57557fa640 10 monclient: dump: 2024-09-16T04:46:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:04.793 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.773+0000 7f57557fa640 1 -- 172.21.15.46:0/4214498239 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7f575c0035f0 con 0x7f57601392b0 2024-09-16T04:46:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.773+0000 7f57557fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:46:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.773+0000 7f57557fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.773+0000 7f57557fa640 1 -- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57601392b0 msgr2=0x7f5760139690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:04.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.773+0000 7f57557fa640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57601392b0 0x7f5760139690 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f575c007ce0 tx=0x7f575c000ed0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.773+0000 7f57557fa640 1 -- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57601392b0 msgr2=0x7f5760139690 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:04.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.773+0000 7f57557fa640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57601392b0 0x7f5760139690 secure :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f575c007ce0 tx=0x7f575c000ed0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.794 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:46:04.794 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:46:04.794 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:46:04.794 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:46:04.795 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 88, 2024-09-16T04:46:04.795 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 104769, 2024-09-16T04:46:04.795 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:46:04.795 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:46:04.795 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:46:04.795 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:46:04.795 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:04.795 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:46:04.795 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:04.795 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:46:04.795 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:46:04.796 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:04.796 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:04.796 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:04.796 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:04.796 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:04.796 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:04.796 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:04.796 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:04.796 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:04.796 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:04.797 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:04.797 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:04.797 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:46:04.797 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:46:04.797 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:04.797 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:04.797 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:04.797 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:04.797 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:04.797 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:04.797 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:04.797 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:04.798 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:04.798 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:04.798 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:04.798 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:04.798 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:04.798 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:46:04.798 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:46:04.798 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:46:04.798 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:46:04.799 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:46:04.799 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:46:04.799 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:04.799 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:04.799 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:46:04.799 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:46:04.799 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:46:04.799 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:46:04.799 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:46:04.799 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:46:04.799 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:46:04.799 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:04.799 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:46:04.800 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:04.800 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:04.800 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:04.800 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:04.800 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:04.800 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:04.800 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:04.800 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:04.800 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:04.800 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:04.800 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:04.800 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:04.800 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:46:04.801 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:04.801 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:46:04.801 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:04.801 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:46:04.801 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:46:04.801 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:04.801 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:04.801 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:04.801 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:04.801 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:46:04.801 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:04.801 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:04.801 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:04.802 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:04.802 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:46:04.802 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:04.802 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:04.802 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:04.802 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:04.802 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:46:04.802 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:46:04.802 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:04.802 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:04.802 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:04.802 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:04.802 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:04.802 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:46:04.803 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:46:04.803 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:04.803 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:04.803 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:04.803 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:04.803 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:46:04.803 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:04.803 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:04.803 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:04.803 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:04.803 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:46:04.803 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:04.803 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:04.803 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:04.804 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:04.804 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:46:04.804 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:46:04.804 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:04.804 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:04.804 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:04.804 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:04.804 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:04.804 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:46:04.804 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:46:04.804 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:04.804 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:04.805 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:04.805 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:04.805 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:46:04.805 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:04.805 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:04.805 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:04.805 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:04.805 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:46:04.805 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:04.805 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:04.805 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:04.805 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:04.805 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:46:04.806 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:46:04.806 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:04.806 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:04.806 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:04.806 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:04.806 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:04.806 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:04.806 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:46:04.806 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:46:04.806 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:04.806 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:04.806 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:04.806 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:04.806 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:04.806 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:04.807 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:46:04.807 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:04.807 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:04.807 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:04.807 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:04.807 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:04.807 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:04.807 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:46:04.807 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:04.807 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:04.807 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:04.807 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4 2024-09-16T04:46:04.808 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:04.808 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:04.808 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:04.808 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:46:04.808 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:46:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.773+0000 7f5766c27640 1 -- 172.21.15.46:0/4214498239 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f574c0383b0 msgr2=0x7f574c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.774+0000 7f5766c27640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f574c0383b0 0x7f574c03a870 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5750034000 tx=0x7f575002e000 comp rx=0 tx=0).stop 2024-09-16T04:46:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.774+0000 7f5766c27640 10 monclient: shutdown 2024-09-16T04:46:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.774+0000 7f5766c27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.774+0000 7f5766c27640 1 -- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f576013c580 msgr2=0x7f576009ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.774+0000 7f5766c27640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f576013c580 0x7f576009ecf0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5760067c70 tx=0x7f5758007e30 comp rx=0 tx=0).stop 2024-09-16T04:46:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.774+0000 7f57651c4640 1 -- 172.21.15.46:0/4214498239 reap_dead start 2024-09-16T04:46:04.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.774+0000 7f5766c27640 1 -- 172.21.15.46:0/4214498239 shutdown_connections 2024-09-16T04:46:04.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.774+0000 7f5766c27640 1 --2- 172.21.15.46:0/4214498239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f576013c580 0x7f576009ecf0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.774+0000 7f5766c27640 1 -- 172.21.15.46:0/4214498239 >> 172.21.15.46:0/4214498239 conn(0x7f57600a5c10 msgr2=0x7f57600a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:04.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.774+0000 7f5766c27640 1 -- 172.21.15.46:0/4214498239 shutdown_connections 2024-09-16T04:46:04.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.774+0000 7f5766c27640 1 -- 172.21.15.46:0/4214498239 wait complete. 2024-09-16T04:46:04.810 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:46:04.810 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:46:04.810 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:46:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.910+0000 7fe74e245640 10 monclient: get_monmap_and_config 2024-09-16T04:46:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.910+0000 7fe74e245640 10 monclient: build_initial_monmap 2024-09-16T04:46:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.910+0000 7fe74e245640 10 monclient: monmap: 2024-09-16T04:46:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:04.911935+0000 2024-09-16T04:46:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:04.911935+0000 2024-09-16T04:46:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:04.913 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:04.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe74e245640 1 Processor -- start 2024-09-16T04:46:04.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe74e245640 1 -- start start 2024-09-16T04:46:04.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe74e245640 10 monclient: init 2024-09-16T04:46:04.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe74e245640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:04.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe74e245640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-16T04:46:04.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe74e245640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe7480a2890 0x7fe748134c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe74e245640 10 monclient(hunting): picked mon.noname-a con 0x7fe7480a2890 addr v2:172.21.15.46:3300/0 2024-09-16T04:46:04.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe74e245640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe74809e980 0x7fe748135150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe74e245640 10 monclient(hunting): picked mon.noname-c con 0x7fe74809e980 addr v2:172.21.15.114:3300/0 2024-09-16T04:46:04.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe74e245640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74809fc70 0x7fe7480a00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe74e245640 10 monclient(hunting): picked mon.noname-b con 0x7fe74809fc70 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:04.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe74e245640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:04.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe74e245640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe7480a2c70 con 0x7fe7480a2890 2024-09-16T04:46:04.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe74e245640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:04.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe74e245640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fe7481347c0 con 0x7fe74809e980 2024-09-16T04:46:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe74e245640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe74e245640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe7480a1050 con 0x7fe74809fc70 2024-09-16T04:46:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe74e245640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe74e245640 10 monclient(hunting): authenticate will time out at 1163.327768s 2024-09-16T04:46:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe747fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74809fc70 0x7fe7480a00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe7477fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe7480a2890 0x7fe748134c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe747fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74809fc70 0x7fe7480a00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:43966/0 (socket says 172.21.15.46:43966) 2024-09-16T04:46:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe747fff640 1 -- 172.21.15.46:0/277498514 learned_addr learned my addr 172.21.15.46:0/277498514 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe747fff640 10 monclient(hunting): get_auth_request con 0x7fe74809fc70 auth_method 0 2024-09-16T04:46:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe747fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe747fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe747fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe7477fe640 10 monclient(hunting): get_auth_request con 0x7fe7480a2890 auth_method 0 2024-09-16T04:46:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe7477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe7477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.911+0000 7fe7477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe7477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe7477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe747fff640 1 -- 172.21.15.46:0/277498514 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74809fc70 msgr2=0x7fe7480a00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:46:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe747fff640 1 -- 172.21.15.46:0/277498514 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74809fc70 msgr2=0x7fe7480a00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:46:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe747fff640 1 --2- 172.21.15.46:0/277498514 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74809fc70 0x7fe7480a00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe7477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe747fff640 1 --2- 172.21.15.46:0/277498514 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74809fc70 0x7fe7480a00e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe746ffd640 1 --2- 172.21.15.46:0/277498514 >> v2:172.21.15.114:3300/0 conn(0x7fe74809e980 0x7fe748135150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe746ffd640 10 monclient(hunting): get_auth_request con 0x7fe74809e980 auth_method 0 2024-09-16T04:46:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe746ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe7477fe640 10 monclient(hunting): handle_auth_done global_id 74637 payload 274 2024-09-16T04:46:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe7477fe640 1 -- 172.21.15.46:0/277498514 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74809fc70 msgr2=0x7fe7480a00e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:46:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe7477fe640 1 --2- 172.21.15.46:0/277498514 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74809fc70 0x7fe7480a00e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe7477fe640 1 -- 172.21.15.46:0/277498514 >> v2:172.21.15.114:3300/0 conn(0x7fe74809e980 msgr2=0x7fe748135150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe7477fe640 1 --2- 172.21.15.46:0/277498514 >> v2:172.21.15.114:3300/0 conn(0x7fe74809e980 0x7fe748135150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe7477fe640 10 monclient: _finish_hunting 0 2024-09-16T04:46:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe7477fe640 1 monclient: found mon.noname-a 2024-09-16T04:46:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe7477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe7477fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:46:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe7477fe640 1 -- 172.21.15.46:0/277498514 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7480a1330 con 0x7fe7480a2890 2024-09-16T04:46:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.912+0000 7fe7477fe640 1 --2- 172.21.15.46:0/277498514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7480a2890 0x7fe748134c10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe74806db50 tx=0x7fe73802a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11ab99b06b40124c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe7467fc640 1 -- 172.21.15.46:0/277498514 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738037070 con 0x7fe7480a2890 2024-09-16T04:46:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe7467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe7467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:46:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe7467fc640 10 monclient: dump: 2024-09-16T04:46:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:04.927 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe7467fc640 10 monclient: _finish_auth 0 2024-09-16T04:46:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe7467fc640 10 monclient: _check_auth_tickets 2024-09-16T04:46:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe7467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe7467fc640 1 -- 172.21.15.46:0/277498514 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe73802add0 con 0x7fe7480a2890 2024-09-16T04:46:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe74e245640 5 monclient: authenticate success, global_id 74637 2024-09-16T04:46:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe74e245640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe7467fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe74e245640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe7467fc640 1 -- 172.21.15.46:0/277498514 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738032660 con 0x7fe7480a2890 2024-09-16T04:46:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe74e245640 4 set_mon_vals no callback set 2024-09-16T04:46:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe7467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe7467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe7467fc640 10 monclient: dump: 2024-09-16T04:46:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:04.928 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe74e245640 10 monclient: shutdown 2024-09-16T04:46:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe74e245640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe74e245640 1 -- 172.21.15.46:0/277498514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7480a2890 msgr2=0x7fe748134c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe74e245640 1 --2- 172.21.15.46:0/277498514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7480a2890 0x7fe748134c10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe74806db50 tx=0x7fe73802a080 comp rx=0 tx=0).stop 2024-09-16T04:46:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe74e245640 1 -- 172.21.15.46:0/277498514 shutdown_connections 2024-09-16T04:46:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe74e245640 1 --2- 172.21.15.46:0/277498514 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74809fc70 0x7fe7480a00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe74e245640 1 --2- 172.21.15.46:0/277498514 >> v2:172.21.15.114:3300/0 conn(0x7fe74809e980 0x7fe748135150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe74e245640 1 --2- 172.21.15.46:0/277498514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7480a2890 0x7fe748134c10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe74e245640 1 -- 172.21.15.46:0/277498514 >> 172.21.15.46:0/277498514 conn(0x7fe7480a1d70 msgr2=0x7fe7480a2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe74e245640 1 -- 172.21.15.46:0/277498514 shutdown_connections 2024-09-16T04:46:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.913+0000 7fe74e245640 1 -- 172.21.15.46:0/277498514 wait complete. 2024-09-16T04:46:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.914+0000 7fe74e245640 10 monclient: build_initial_monmap 2024-09-16T04:46:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.914+0000 7fe74e245640 10 monclient: monmap: 2024-09-16T04:46:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:04.930 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.914+0000 7fe74e245640 1 Processor -- start 2024-09-16T04:46:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.914+0000 7fe74e245640 1 -- start start 2024-09-16T04:46:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.914+0000 7fe74e245640 10 monclient: init 2024-09-16T04:46:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe74e245640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe74e245640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:46:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe74e245640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe74809e980 0x7fe7481c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe74e245640 10 monclient(hunting): picked mon.noname-a con 0x7fe74809e980 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe74e245640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74809fc70 0x7fe74812db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe74e245640 10 monclient(hunting): picked mon.noname-c con 0x7fe74809fc70 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe74e245640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7480a2890 0x7fe748071d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe74e245640 10 monclient(hunting): picked mon.noname-b con 0x7fe7480a2890 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe74e245640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe74e245640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe7480a07a0 con 0x7fe74809e980 2024-09-16T04:46:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe74e245640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe74e245640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe7480a0620 con 0x7fe7480a2890 2024-09-16T04:46:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe74e245640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe74e245640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe7480a0920 con 0x7fe74809fc70 2024-09-16T04:46:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe74e245640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe74e245640 10 monclient(hunting): authenticate will time out at 1163.331297s 2024-09-16T04:46:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe746ffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74809fc70 0x7fe74812db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe7477fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe74809e980 0x7fe7481c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe746ffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74809fc70 0x7fe74812db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:43976/0 (socket says 172.21.15.46:43976) 2024-09-16T04:46:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe746ffd640 1 -- 172.21.15.46:0/2795135357 learned_addr learned my addr 172.21.15.46:0/2795135357 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe746ffd640 10 monclient(hunting): get_auth_request con 0x7fe74809fc70 auth_method 0 2024-09-16T04:46:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe746ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe7477fe640 10 monclient(hunting): get_auth_request con 0x7fe74809e980 auth_method 0 2024-09-16T04:46:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe7477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe7477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe7477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe746ffd640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74809fc70 msgr2=0x7fe74812db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:46:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe746ffd640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74809fc70 msgr2=0x7fe74812db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:46:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe746ffd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74809fc70 0x7fe74812db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe7477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe7477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe7477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe746ffd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74809fc70 0x7fe74812db30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe747fff640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7480a2890 0x7fe748071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe747fff640 10 monclient(hunting): get_auth_request con 0x7fe7480a2890 auth_method 0 2024-09-16T04:46:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe747fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe747fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe747fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe7477fe640 10 monclient(hunting): handle_auth_done global_id 74640 payload 824 2024-09-16T04:46:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe7477fe640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74809fc70 msgr2=0x7fe74812db30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:46:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe7477fe640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74809fc70 0x7fe74812db30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe7477fe640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7480a2890 msgr2=0x7fe748071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe7477fe640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7480a2890 0x7fe748071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe7477fe640 10 monclient: _finish_hunting 0 2024-09-16T04:46:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe7477fe640 1 monclient: found mon.noname-a 2024-09-16T04:46:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe7477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe7477fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:46:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe7477fe640 1 -- 172.21.15.46:0/2795135357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe748073e00 con 0x7fe74809e980 2024-09-16T04:46:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe747fff640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7480a2890 0x7fe748071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:46:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.915+0000 7fe7477fe640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe74809e980 0x7fe7481c75e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe74809c430 tx=0x7fe7380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe72ffff640 1 -- 172.21.15.46:0/2795135357 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738037040 con 0x7fe74809e980 2024-09-16T04:46:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe72ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe72ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:46:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe72ffff640 10 monclient: dump: 2024-09-16T04:46:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:04.935 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe72ffff640 10 monclient: _finish_auth 0 2024-09-16T04:46:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe72ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:46:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe72ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe72ffff640 1 -- 172.21.15.46:0/2795135357 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe73802abd0 con 0x7fe74809e980 2024-09-16T04:46:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe74e245640 5 monclient: authenticate success, global_id 74640 2024-09-16T04:46:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe72ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe72ffff640 1 -- 172.21.15.46:0/2795135357 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73803b670 con 0x7fe74809e980 2024-09-16T04:46:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe74e245640 10 monclient: _renew_subs 2024-09-16T04:46:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe74e245640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe74e245640 1 -- 172.21.15.46:0/2795135357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe74809c460 con 0x7fe74809e980 2024-09-16T04:46:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe72ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe72ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe72ffff640 10 monclient: dump: 2024-09-16T04:46:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:04.937 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe745ffb640 4 set_mon_vals no callback set 2024-09-16T04:46:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe74e245640 10 monclient: _renew_subs 2024-09-16T04:46:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe74e245640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe74e245640 1 -- 172.21.15.46:0/2795135357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe74811d230 con 0x7fe74809e980 2024-09-16T04:46:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.916+0000 7fe74e245640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.917+0000 7fe74e245640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 0x7fe748072250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.917+0000 7fe74e245640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.917+0000 7fe74e245640 1 -- 172.21.15.46:0/2795135357 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe74809e6e0 con 0x7fe74812e070 2024-09-16T04:46:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.917+0000 7fe72ffff640 1 -- 172.21.15.46:0/2795135357 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fe738044020 con 0x7fe74809e980 2024-09-16T04:46:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.917+0000 7fe746ffd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 0x7fe748072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.917+0000 7fe746ffd640 10 monclient: get_auth_request con 0x7fe74812e070 auth_method 0 2024-09-16T04:46:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.917+0000 7fe746ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.917+0000 7fe746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.917+0000 7fe746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.917+0000 7fe746ffd640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 msgr2=0x7fe748072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:46:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.917+0000 7fe746ffd640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 msgr2=0x7fe748072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:46:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.917+0000 7fe746ffd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 0x7fe748072250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.917+0000 7fe746ffd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 0x7fe748072250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.917+0000 7fe72ffff640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe7340383b0 0x7fe73403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.917+0000 7fe72ffff640 1 -- 172.21.15.46:0/2795135357 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 4346+0+0 (secure 0 0 0) 0x7fe73806b560 con 0x7fe74809e980 2024-09-16T04:46:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.917+0000 7fe747fff640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe7340383b0 0x7fe73403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.917+0000 7fe747fff640 10 monclient: get_auth_request con 0x7fe7340383b0 auth_method 0 2024-09-16T04:46:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:04.918+0000 7fe747fff640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe7340383b0 0x7fe73403a870 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe74811cd20 tx=0x7fe740009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:05.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:05.118+0000 7fe746ffd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 0x7fe748072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:05.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:05.118+0000 7fe746ffd640 10 monclient: get_auth_request con 0x7fe74812e070 auth_method 0 2024-09-16T04:46:05.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:05.118+0000 7fe746ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:05.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:05.118+0000 7fe746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:05.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:05.118+0000 7fe746ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:46:05.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:05.118+0000 7fe746ffd640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 msgr2=0x7fe748072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:46:05.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:05.118+0000 7fe746ffd640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 msgr2=0x7fe748072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:46:05.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:05.118+0000 7fe746ffd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 0x7fe748072250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:05.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:05.118+0000 7fe746ffd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 0x7fe748072250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:46:05.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:05.519+0000 7fe746ffd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 0x7fe748072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:05.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:05.519+0000 7fe746ffd640 10 monclient: get_auth_request con 0x7fe74812e070 auth_method 0 2024-09-16T04:46:05.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:05.519+0000 7fe746ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:05.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:05.519+0000 7fe746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:05.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:05.519+0000 7fe746ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:46:05.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:05.519+0000 7fe746ffd640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 msgr2=0x7fe748072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:46:05.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:05.519+0000 7fe746ffd640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 msgr2=0x7fe748072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:46:05.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:05.519+0000 7fe746ffd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 0x7fe748072250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:05.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:05.519+0000 7fe746ffd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 0x7fe748072250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:46:06.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:06.320+0000 7fe746ffd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 0x7fe748072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:06.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:06.320+0000 7fe746ffd640 10 monclient: get_auth_request con 0x7fe74812e070 auth_method 0 2024-09-16T04:46:06.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:06.320+0000 7fe746ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:06.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:06.320+0000 7fe746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:06.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:06.320+0000 7fe746ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:46:06.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:06.320+0000 7fe746ffd640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 msgr2=0x7fe748072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:46:06.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:06.320+0000 7fe746ffd640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 msgr2=0x7fe748072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:46:06.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:06.320+0000 7fe746ffd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 0x7fe748072250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:06.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:06.320+0000 7fe746ffd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 0x7fe748072250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:46:06.536 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:06.536 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39749]: starting. 2024-09-16T04:46:06.536 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39749]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-39714 2024-09-16T04:46:06.536 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39749]: created object 0... 2024-09-16T04:46:06.536 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39749]: created object 25... 2024-09-16T04:46:06.537 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39749]: created object 49... 2024-09-16T04:46:06.537 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39749]: finishing. 2024-09-16T04:46:06.537 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39749]: shutting down. 2024-09-16T04:46:06.549 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:06.549 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39751]: starting. 2024-09-16T04:46:06.549 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39751]: listing objects. 2024-09-16T04:46:06.549 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39751]: listed object 0... 2024-09-16T04:46:06.549 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39751]: listed object 25... 2024-09-16T04:46:06.549 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39751]: saw 50 objects 2024-09-16T04:46:06.549 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39751]: shutting down. 2024-09-16T04:46:07.443 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:07.444 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39750]: starting. 2024-09-16T04:46:07.444 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39750]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-39714 2024-09-16T04:46:07.444 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39750]: shutting down. 2024-09-16T04:46:07.445 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:07.445 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:07.445 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:46:07.448 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:46:07.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:07.915+0000 7fe72effd640 10 monclient: tick 2024-09-16T04:46:07.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:07.915+0000 7fe72effd640 10 monclient: _check_auth_tickets 2024-09-16T04:46:07.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:07.915+0000 7fe72effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:07.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:07.915+0000 7fe72effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:07.921+0000 7fe746ffd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 0x7fe748072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:07.921+0000 7fe746ffd640 10 monclient: get_auth_request con 0x7fe74812e070 auth_method 0 2024-09-16T04:46:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:07.921+0000 7fe746ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:07.921+0000 7fe746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:07.921+0000 7fe746ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:46:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:07.921+0000 7fe746ffd640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 msgr2=0x7fe748072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:46:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:07.921+0000 7fe746ffd640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 msgr2=0x7fe748072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:46:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:07.921+0000 7fe746ffd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 0x7fe748072250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:07.921+0000 7fe746ffd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 0x7fe748072250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-09-16T04:46:08.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.915+0000 7fe72effd640 10 monclient: tick 2024-09-16T04:46:08.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.915+0000 7fe72effd640 10 monclient: _check_auth_tickets 2024-09-16T04:46:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.915+0000 7fe72effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.915+0000 7fe72effd640 5 monclient: _check_tell_commands timeout tell command 1 2024-09-16T04:46:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.915+0000 7fe72effd640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 msgr2=0x7fe748072250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:46:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.915+0000 7fe72effd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 0x7fe748072250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.915+0000 7fe72effd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe720003540 0x7fe7200059d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.915+0000 7fe72effd640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 msgr2=0x7fe748072250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.915+0000 7fe72effd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74812e070 0x7fe748072250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.915+0000 7fe72effd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.915+0000 7fe72effd640 1 -- 172.21.15.46:0/2795135357 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe74809e6e0 con 0x7fe720003540 2024-09-16T04:46:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.915+0000 7fe746ffd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe720003540 0x7fe7200059d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.915+0000 7fe746ffd640 10 monclient: get_auth_request con 0x7fe720003540 auth_method 0 2024-09-16T04:46:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.915+0000 7fe746ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.915+0000 7fe746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.915+0000 7fe746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.916+0000 7fe746ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.916+0000 7fe746ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.916+0000 7fe746ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.916+0000 7fe746ffd640 10 monclient(hunting): handle_auth_done global_id 94100 payload 824 2024-09-16T04:46:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.916+0000 7fe746ffd640 1 -- 172.21.15.46:0/2795135357 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe7200060b0 con 0x7fe720003540 2024-09-16T04:46:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.916+0000 7fe746ffd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe720003540 0x7fe7200059d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe73c006fd0 tx=0x7fe73c01b720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.916+0000 7fe72ffff640 1 -- 172.21.15.46:0/2795135357 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c01be40 con 0x7fe720003540 2024-09-16T04:46:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.916+0000 7fe72ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.916+0000 7fe72ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.916+0000 7fe72ffff640 10 monclient: dump: 2024-09-16T04:46:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:08.921 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.917+0000 7fe72ffff640 1 -- 172.21.15.46:0/2795135357 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe73c01be40 con 0x7fe720003540 2024-09-16T04:46:08.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.917+0000 7fe72ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:08.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.917+0000 7fe72ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:08.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.917+0000 7fe72ffff640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe720003540 msgr2=0x7fe7200059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:08.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.917+0000 7fe72ffff640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe720003540 0x7fe7200059d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe73c006fd0 tx=0x7fe73c01b720 comp rx=0 tx=0).stop 2024-09-16T04:46:08.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.917+0000 7fe72ffff640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe720003540 msgr2=0x7fe7200059d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:08.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.917+0000 7fe72ffff640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe720003540 0x7fe7200059d0 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe73c006fd0 tx=0x7fe73c01b720 comp rx=0 tx=0).stop 2024-09-16T04:46:08.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.924+0000 7fe74e245640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:46:08.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.924+0000 7fe74e245640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe748072790 0x7fe74809c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:08.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.924+0000 7fe74e245640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:08.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.924+0000 7fe74e245640 1 -- 172.21.15.46:0/2795135357 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe748072ba0 con 0x7fe748072790 2024-09-16T04:46:08.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.924+0000 7fe746ffd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe748072790 0x7fe74809c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.924+0000 7fe746ffd640 10 monclient: get_auth_request con 0x7fe748072790 auth_method 0 2024-09-16T04:46:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.924+0000 7fe746ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.924+0000 7fe746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.924+0000 7fe746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.924+0000 7fe746ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.924+0000 7fe746ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.924+0000 7fe746ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.924+0000 7fe746ffd640 10 monclient(hunting): handle_auth_done global_id 94103 payload 824 2024-09-16T04:46:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.924+0000 7fe746ffd640 1 -- 172.21.15.46:0/2795135357 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe7481c8470 con 0x7fe748072790 2024-09-16T04:46:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.925+0000 7fe746ffd640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe748072790 0x7fe74809c990 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe73c006fd0 tx=0x7fe73c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.925+0000 7fe72ffff640 1 -- 172.21.15.46:0/2795135357 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c02e770 con 0x7fe748072790 2024-09-16T04:46:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.925+0000 7fe72ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.925+0000 7fe72ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:08.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.925+0000 7fe72ffff640 10 monclient: dump: 2024-09-16T04:46:08.929 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:08.929 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:08.929 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:08.929 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:08.929 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:08.929 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:08.929 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:08.929 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:08.929 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:08.929 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:08.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.925+0000 7fe72ffff640 1 -- 172.21.15.46:0/2795135357 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7fe73c02e770 con 0x7fe748072790 2024-09-16T04:46:08.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.925+0000 7fe72ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:46:08.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.925+0000 7fe72ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:08.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.925+0000 7fe72ffff640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe748072790 msgr2=0x7fe74809c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:08.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.925+0000 7fe72ffff640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe748072790 0x7fe74809c990 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe73c006fd0 tx=0x7fe73c004730 comp rx=0 tx=0).stop 2024-09-16T04:46:08.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.925+0000 7fe72ffff640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe748072790 msgr2=0x7fe74809c990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:08.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.925+0000 7fe72ffff640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe748072790 0x7fe74809c990 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe73c006fd0 tx=0x7fe73c004730 comp rx=0 tx=0).stop 2024-09-16T04:46:08.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.925+0000 7fe747fff640 1 -- 172.21.15.46:0/2795135357 reap_dead start 2024-09-16T04:46:08.930 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:46:08.930 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:46:08.930 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:46:08.930 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:46:08.931 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 88, 2024-09-16T04:46:08.931 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 10180, 2024-09-16T04:46:08.931 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:46:08.931 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:46:08.931 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:46:08.931 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:46:08.931 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:08.931 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 4, 2024-09-16T04:46:08.931 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:08.931 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:46:08.931 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:46:08.931 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:08.931 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:08.931 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:08.932 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:08.932 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:08.932 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:08.932 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:08.932 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:08.932 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:08.932 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:08.932 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:08.932 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:08.932 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:46:08.932 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:46:08.932 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:08.932 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:08.932 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:08.933 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:08.933 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:08.933 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:08.933 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:08.933 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:08.933 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:08.933 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:08.933 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:08.933 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:08.933 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:08.933 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:46:08.933 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:46:08.933 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:46:08.933 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:46:08.934 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:46:08.934 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:46:08.934 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:08.934 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:08.934 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:46:08.934 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:46:08.934 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:46:08.934 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:46:08.934 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:46:08.934 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:46:08.934 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:46:08.934 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:08.934 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:46:08.934 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:08.935 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:08.935 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:08.935 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:08.935 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:08.935 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:08.935 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:08.935 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:08.935 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:08.935 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:08.936 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:08.936 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:08.936 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:46:08.936 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:08.936 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:46:08.936 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:08.936 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:46:08.936 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:46:08.936 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:08.936 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:08.936 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:08.936 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:08.936 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:46:08.937 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:08.937 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:08.937 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:08.937 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:08.937 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:46:08.937 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:08.937 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:08.937 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:08.937 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:08.937 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:46:08.937 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:46:08.937 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:08.937 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:08.937 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:08.938 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:08.938 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:08.938 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:46:08.938 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:46:08.938 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:08.938 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:08.938 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:08.938 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:08.938 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:46:08.938 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:08.938 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:08.938 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:08.938 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:08.938 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:46:08.939 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:08.939 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:08.939 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:08.939 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:08.939 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:46:08.939 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:46:08.939 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:08.939 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:08.939 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:08.939 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:08.939 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:08.939 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:46:08.939 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:46:08.939 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:08.939 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:08.940 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:08.940 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:08.940 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:46:08.940 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:08.940 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:08.940 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:08.940 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:08.940 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:46:08.940 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:08.940 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:08.940 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:08.940 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:08.940 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:46:08.941 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:46:08.941 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:08.941 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:08.941 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:08.941 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:08.941 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:08.941 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:08.941 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:46:08.941 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:46:08.941 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:08.942 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:08.942 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:08.942 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:08.942 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:08.942 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:08.942 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:46:08.942 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:08.942 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:08.942 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:08.942 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:08.942 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:08.942 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:08.942 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:08.942 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:46:08.942 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:46:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.926+0000 7fe74e245640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe7340383b0 msgr2=0x7fe73403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.926+0000 7fe74e245640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe7340383b0 0x7fe73403a870 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe74811cd20 tx=0x7fe740009040 comp rx=0 tx=0).stop 2024-09-16T04:46:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.926+0000 7fe74e245640 10 monclient: shutdown 2024-09-16T04:46:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.926+0000 7fe74e245640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.926+0000 7fe74e245640 1 -- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe74809e980 msgr2=0x7fe7481c75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.926+0000 7fe74e245640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe74809e980 0x7fe7481c75e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe74809c430 tx=0x7fe7380046c0 comp rx=0 tx=0).stop 2024-09-16T04:46:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.926+0000 7fe74e245640 1 -- 172.21.15.46:0/2795135357 shutdown_connections 2024-09-16T04:46:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.926+0000 7fe74e245640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe7340383b0 0x7fe73403a870 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.926+0000 7fe74e245640 1 --2- 172.21.15.46:0/2795135357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe74809e980 0x7fe7481c75e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.926+0000 7fe74e245640 1 -- 172.21.15.46:0/2795135357 >> 172.21.15.46:0/2795135357 conn(0x7fe7480a1d70 msgr2=0x7fe748130fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.926+0000 7fe74e245640 1 -- 172.21.15.46:0/2795135357 shutdown_connections 2024-09-16T04:46:08.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:08.926+0000 7fe74e245640 1 -- 172.21.15.46:0/2795135357 wait complete. 2024-09-16T04:46:08.945 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:46:08.945 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:46:08.945 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:46:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.054+0000 7f2da6f98640 10 monclient: get_monmap_and_config 2024-09-16T04:46:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.054+0000 7f2da6f98640 10 monclient: build_initial_monmap 2024-09-16T04:46:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.055+0000 7f2da6f98640 10 monclient: monmap: 2024-09-16T04:46:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:09.056465+0000 2024-09-16T04:46:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:09.056465+0000 2024-09-16T04:46:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:09.058 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:09.058 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:09.058 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:09.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.055+0000 7f2da6f98640 1 Processor -- start 2024-09-16T04:46:09.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.055+0000 7f2da6f98640 1 -- start start 2024-09-16T04:46:09.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.055+0000 7f2da6f98640 10 monclient: init 2024-09-16T04:46:09.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da6f98640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:09.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da6f98640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-16T04:46:09.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da6f98640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2da00a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:46:09.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da6f98640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2da00a3ff0 0x7f2da00a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:09.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da6f98640 10 monclient(hunting): picked mon.noname-a con 0x7f2da00a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:46:09.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da6f98640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2da013be90 0x7f2da013e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:09.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da6f98640 10 monclient(hunting): picked mon.noname-b con 0x7f2da013be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:09.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da6f98640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2da009b940 con 0x7f2da00a3c10 2024-09-16T04:46:09.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da6f98640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:09.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da6f98640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2da009ab00 con 0x7f2da00a3ff0 2024-09-16T04:46:09.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da6f98640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:09.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da6f98640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f2da009b030 con 0x7f2da013be90 2024-09-16T04:46:09.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da6f98640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:09.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da6f98640 10 monclient(hunting): authenticate will time out at 1167.472362s 2024-09-16T04:46:09.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da5535640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2da013be90 0x7f2da013e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:09.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2d9ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2da00a3ff0 0x7f2da00a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:09.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da5535640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2da013be90 0x7f2da013e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:44050/0 (socket says 172.21.15.46:44050) 2024-09-16T04:46:09.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da5535640 1 -- 172.21.15.46:0/1790101646 learned_addr learned my addr 172.21.15.46:0/1790101646 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:09.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da5535640 10 monclient(hunting): get_auth_request con 0x7f2da013be90 auth_method 0 2024-09-16T04:46:09.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2d9ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2da00a3ff0 0x7f2da00a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33374/0 (socket says 172.21.15.46:33374) 2024-09-16T04:46:09.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da5535640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:09.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da5535640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:09.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da5535640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:09.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2d9ffff640 10 monclient(hunting): get_auth_request con 0x7f2da00a3ff0 auth_method 0 2024-09-16T04:46:09.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2d9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:09.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2d9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:09.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2d9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:09.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da5535640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:09.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da5535640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:09.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da5535640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:09.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2d9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:09.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2d9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:09.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2d9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:09.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da5535640 10 monclient(hunting): handle_auth_done global_id 94106 payload 274 2024-09-16T04:46:09.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da5535640 1 -- 172.21.15.46:0/1790101646 >> v2:172.21.15.46:3300/0 conn(0x7f2da00a3ff0 msgr2=0x7f2da00a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:09.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da5535640 1 --2- 172.21.15.46:0/1790101646 >> v2:172.21.15.46:3300/0 conn(0x7f2da00a3ff0 0x7f2da00a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:09.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da5535640 1 -- 172.21.15.46:0/1790101646 >> v1:172.21.15.114:6789/0 conn(0x7f2da00a3c10 legacy=0x7f2da009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:09.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da5535640 10 monclient: _finish_hunting 0 2024-09-16T04:46:09.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da5535640 1 monclient: found mon.noname-b 2024-09-16T04:46:09.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da5535640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:09.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da5535640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:46:09.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2da5535640 1 -- 172.21.15.46:0/1790101646 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da009b310 con 0x7f2da013be90 2024-09-16T04:46:09.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.056+0000 7f2d9ffff640 1 --2- 172.21.15.46:0/1790101646 >> v2:172.21.15.46:3300/0 conn(0x7f2da00a3ff0 0x7f2da00a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:09.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.057+0000 7f2da5535640 1 --2- 172.21.15.46:0/1790101646 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2da013be90 0x7f2da013e280 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2d98009e50 tx=0x7f2d9802a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=702a47eb0f07c63a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:09.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.057+0000 7f2d9f7fe640 1 -- 172.21.15.46:0/1790101646 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d98036070 con 0x7f2da013be90 2024-09-16T04:46:09.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.057+0000 7f2d9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:09.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.057+0000 7f2d9f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:46:09.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.057+0000 7f2d9f7fe640 10 monclient: dump: 2024-09-16T04:46:09.071 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:09.071 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:09.072 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:09.072 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:09.072 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:09.072 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:09.072 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:09.072 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:09.072 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:09.072 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:09.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.057+0000 7f2d9f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:46:09.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.057+0000 7f2d9f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:46:09.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.057+0000 7f2d9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:09.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.057+0000 7f2da6f98640 5 monclient: authenticate success, global_id 94106 2024-09-16T04:46:09.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.057+0000 7f2da6f98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:09.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.057+0000 7f2d9f7fe640 1 -- 172.21.15.46:0/1790101646 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d98031600 con 0x7f2da013be90 2024-09-16T04:46:09.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.057+0000 7f2d9f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:09.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.057+0000 7f2da6f98640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:09.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.057+0000 7f2da6f98640 4 set_mon_vals no callback set 2024-09-16T04:46:09.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.057+0000 7f2d9f7fe640 1 -- 172.21.15.46:0/1790101646 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d980319d0 con 0x7f2da013be90 2024-09-16T04:46:09.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.057+0000 7f2d9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:09.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.057+0000 7f2d9f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:09.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.057+0000 7f2d9f7fe640 10 monclient: dump: 2024-09-16T04:46:09.073 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:09.073 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:09.074 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:09.074 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:09.074 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:09.074 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:09.074 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:09.074 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:09.074 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:09.074 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:09.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.058+0000 7f2da6f98640 10 monclient: shutdown 2024-09-16T04:46:09.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.058+0000 7f2da6f98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:09.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.058+0000 7f2da6f98640 1 -- 172.21.15.46:0/1790101646 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2da013be90 msgr2=0x7f2da013e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:09.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.058+0000 7f2da6f98640 1 --2- 172.21.15.46:0/1790101646 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2da013be90 0x7f2da013e280 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2d98009e50 tx=0x7f2d9802a680 comp rx=0 tx=0).stop 2024-09-16T04:46:09.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.058+0000 7f2da6f98640 1 -- 172.21.15.46:0/1790101646 shutdown_connections 2024-09-16T04:46:09.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.058+0000 7f2da6f98640 1 --2- 172.21.15.46:0/1790101646 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2da013be90 0x7f2da013e280 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:09.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.058+0000 7f2da6f98640 1 --2- 172.21.15.46:0/1790101646 >> v2:172.21.15.46:3300/0 conn(0x7f2da00a3ff0 0x7f2da00a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:09.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.058+0000 7f2da6f98640 1 -- 172.21.15.46:0/1790101646 >> 172.21.15.46:0/1790101646 conn(0x7f2da00a6340 msgr2=0x7f2da006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:09.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.058+0000 7f2da6f98640 1 -- 172.21.15.46:0/1790101646 shutdown_connections 2024-09-16T04:46:09.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.058+0000 7f2da6f98640 1 -- 172.21.15.46:0/1790101646 wait complete. 2024-09-16T04:46:09.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.058+0000 7f2da6f98640 10 monclient: build_initial_monmap 2024-09-16T04:46:09.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.058+0000 7f2da6f98640 10 monclient: monmap: 2024-09-16T04:46:09.075 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:09.075 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:09.075 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:09.075 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:09.075 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:09.076 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:09.076 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:09.076 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:09.076 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:09.076 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:09.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.059+0000 7f2da6f98640 1 Processor -- start 2024-09-16T04:46:09.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.059+0000 7f2da6f98640 1 -- start start 2024-09-16T04:46:09.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.059+0000 7f2da6f98640 10 monclient: init 2024-09-16T04:46:09.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.059+0000 7f2da6f98640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:09.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.059+0000 7f2da6f98640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:46:09.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.059+0000 7f2da6f98640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da00a3c10 0x7f2da0133cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:09.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.059+0000 7f2da6f98640 10 monclient(hunting): picked mon.noname-b con 0x7f2da00a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:09.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.059+0000 7f2da6f98640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2da00a3ff0 0x7f2da0134230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:09.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.059+0000 7f2da6f98640 10 monclient(hunting): picked mon.noname-c con 0x7f2da00a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:09.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.059+0000 7f2da6f98640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da013be90 0x7f2da01cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:09.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.059+0000 7f2da6f98640 10 monclient(hunting): picked mon.noname-a con 0x7f2da013be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:09.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.059+0000 7f2da6f98640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:09.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.059+0000 7f2da6f98640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2da00a6b40 con 0x7f2da013be90 2024-09-16T04:46:09.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.059+0000 7f2da6f98640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:09.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.059+0000 7f2da6f98640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2da00a69c0 con 0x7f2da00a3c10 2024-09-16T04:46:09.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.059+0000 7f2da6f98640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:09.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.059+0000 7f2da6f98640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f2da013e7c0 con 0x7f2da00a3ff0 2024-09-16T04:46:09.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.059+0000 7f2da6f98640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:09.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.059+0000 7f2da6f98640 10 monclient(hunting): authenticate will time out at 1167.475970s 2024-09-16T04:46:09.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da5535640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da013be90 0x7f2da01cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:09.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9ffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2da00a3ff0 0x7f2da0134230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da5535640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da013be90 0x7f2da01cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33382/0 (socket says 172.21.15.46:33382) 2024-09-16T04:46:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da5535640 1 -- 172.21.15.46:0/1499914383 learned_addr learned my addr 172.21.15.46:0/1499914383 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da5535640 10 monclient(hunting): get_auth_request con 0x7f2da013be90 auth_method 0 2024-09-16T04:46:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da5535640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da5535640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da5535640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da4d34640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da00a3c10 0x7f2da0133cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9ffff640 10 monclient(hunting): get_auth_request con 0x7f2da00a3ff0 auth_method 0 2024-09-16T04:46:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da4d34640 10 monclient(hunting): get_auth_request con 0x7f2da00a3c10 auth_method 0 2024-09-16T04:46:09.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da4d34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da4d34640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da4d34640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da5535640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da5535640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da5535640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da4d34640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da4d34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da4d34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da5535640 10 monclient(hunting): handle_auth_done global_id 74661 payload 824 2024-09-16T04:46:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da5535640 1 -- 172.21.15.46:0/1499914383 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2da00a3ff0 msgr2=0x7f2da0134230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:09.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da5535640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2da00a3ff0 0x7f2da0134230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:09.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da5535640 1 -- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da00a3c10 msgr2=0x7f2da0133cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:09.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da5535640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da00a3c10 0x7f2da0133cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:09.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da5535640 10 monclient: _finish_hunting 0 2024-09-16T04:46:09.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da5535640 1 monclient: found mon.noname-a 2024-09-16T04:46:09.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da5535640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:09.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da5535640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:46:09.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da5535640 1 -- 172.21.15.46:0/1499914383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da013a4c0 con 0x7f2da013be90 2024-09-16T04:46:09.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9ffff640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2da00a3ff0 0x7f2da0134230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:09.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da5535640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da013be90 0x7f2da01cb9e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2d98009e50 tx=0x7f2d98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:09.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9d7fa640 1 -- 172.21.15.46:0/1499914383 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d98036050 con 0x7f2da013be90 2024-09-16T04:46:09.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:09.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:46:09.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9d7fa640 10 monclient: dump: 2024-09-16T04:46:09.081 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:09.081 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:09.081 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:09.081 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:09.081 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:09.081 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:09.081 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:09.081 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:09.081 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:09.081 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:09.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:46:09.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:46:09.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:09.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9d7fa640 1 -- 172.21.15.46:0/1499914383 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d98042070 con 0x7f2da013be90 2024-09-16T04:46:09.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da6f98640 5 monclient: authenticate success, global_id 74661 2024-09-16T04:46:09.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da6f98640 10 monclient: _renew_subs 2024-09-16T04:46:09.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da6f98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:09.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da6f98640 1 -- 172.21.15.46:0/1499914383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2da00a0400 con 0x7f2da013be90 2024-09-16T04:46:09.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:09.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9d7fa640 1 -- 172.21.15.46:0/1499914383 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d98009510 con 0x7f2da013be90 2024-09-16T04:46:09.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:09.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:09.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9d7fa640 10 monclient: dump: 2024-09-16T04:46:09.082 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:09.083 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:09.083 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:09.083 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:09.083 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:09.083 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:09.083 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:09.083 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:09.083 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:09.083 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:09.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da6f98640 10 monclient: _renew_subs 2024-09-16T04:46:09.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2d9effd640 4 set_mon_vals no callback set 2024-09-16T04:46:09.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da6f98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:09.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.060+0000 7f2da6f98640 1 -- 172.21.15.46:0/1499914383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2da013a1e0 con 0x7f2da013be90 2024-09-16T04:46:09.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.061+0000 7f2da6f98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:09.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.061+0000 7f2da6f98640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da01cbf20 0x7f2da01cc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:09.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.061+0000 7f2da6f98640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:09.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.061+0000 7f2da6f98640 1 -- 172.21.15.46:0/1499914383 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2da01cc840 con 0x7f2da01cbf20 2024-09-16T04:46:09.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.061+0000 7f2d9d7fa640 1 -- 172.21.15.46:0/1499914383 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f2d98007bf0 con 0x7f2da013be90 2024-09-16T04:46:09.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.061+0000 7f2da4d34640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da01cbf20 0x7f2da01cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:09.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.061+0000 7f2da4d34640 10 monclient: get_auth_request con 0x7f2da01cbf20 auth_method 0 2024-09-16T04:46:09.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.061+0000 7f2da4d34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:09.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.061+0000 7f2da4d34640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:09.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.061+0000 7f2da4d34640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:09.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.061+0000 7f2da4d34640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:09.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.061+0000 7f2da4d34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:09.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.061+0000 7f2da4d34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:09.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.062+0000 7f2d9d7fa640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2d8c0383b0 0x7f2d8c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.062+0000 7f2da4d34640 10 monclient(hunting): handle_auth_done global_id 6721 payload 824 2024-09-16T04:46:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.062+0000 7f2d9d7fa640 1 -- 172.21.15.46:0/1499914383 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(279..279 src has 1..279) ==== 4738+0+0 (secure 0 0 0) 0x7f2d9806c400 con 0x7f2da013be90 2024-09-16T04:46:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.062+0000 7f2da4d34640 1 -- 172.21.15.46:0/1499914383 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2da00a43d0 con 0x7f2da01cbf20 2024-09-16T04:46:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.062+0000 7f2d9ffff640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2d8c0383b0 0x7f2d8c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.062+0000 7f2d9ffff640 10 monclient: get_auth_request con 0x7f2d8c0383b0 auth_method 0 2024-09-16T04:46:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.062+0000 7f2da4d34640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da01cbf20 0x7f2da01cc300 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2d9000f770 tx=0x7f2d9000fc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.062+0000 7f2d9d7fa640 1 -- 172.21.15.46:0/1499914383 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d900050d0 con 0x7f2da01cbf20 2024-09-16T04:46:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.062+0000 7f2d9d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.062+0000 7f2d9d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:46:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.062+0000 7f2d9d7fa640 10 monclient: dump: 2024-09-16T04:46:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:09.086 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.062+0000 7f2d9ffff640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2d8c0383b0 0x7f2d8c03a870 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2da00a8b40 tx=0x7f2d94009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.062+0000 7f2d9d7fa640 1 -- 172.21.15.46:0/1499914383 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2d90018070 con 0x7f2da01cbf20 2024-09-16T04:46:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.062+0000 7f2d9d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.062+0000 7f2d9d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.062+0000 7f2d9d7fa640 1 -- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da01cbf20 msgr2=0x7f2da01cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.062+0000 7f2d9d7fa640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da01cbf20 0x7f2da01cc300 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2d9000f770 tx=0x7f2d9000fc30 comp rx=0 tx=0).stop 2024-09-16T04:46:09.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.062+0000 7f2d9d7fa640 1 -- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da01cbf20 msgr2=0x7f2da01cc300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:09.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.062+0000 7f2d9d7fa640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da01cbf20 0x7f2da01cc300 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:09.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.068+0000 7f2da6f98640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:46:09.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.069+0000 7f2da6f98640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da009d110 0x7f2da009d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:09.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.069+0000 7f2da6f98640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:09.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.069+0000 7f2da6f98640 1 -- 172.21.15.46:0/1499914383 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2da0139010 con 0x7f2da009d110 2024-09-16T04:46:09.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.069+0000 7f2da4d34640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da009d110 0x7f2da009d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.069+0000 7f2da4d34640 10 monclient: get_auth_request con 0x7f2da009d110 auth_method 0 2024-09-16T04:46:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.069+0000 7f2da4d34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.069+0000 7f2da4d34640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.069+0000 7f2da4d34640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.069+0000 7f2da4d34640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.069+0000 7f2da4d34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.069+0000 7f2da4d34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.069+0000 7f2da4d34640 10 monclient(hunting): handle_auth_done global_id 6724 payload 824 2024-09-16T04:46:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.069+0000 7f2da4d34640 1 -- 172.21.15.46:0/1499914383 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2da0067e20 con 0x7f2da009d110 2024-09-16T04:46:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.069+0000 7f2da4d34640 1 -- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da009d110 msgr2=0x7f2da009d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:46:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.069+0000 7f2da4d34640 1 -- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da009d110 msgr2=0x7f2da009d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:46:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.069+0000 7f2da4d34640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da009d110 0x7f2da009d4f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f2d9000f770 tx=0x7f2d90001120 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:09.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.069+0000 7f2da4d34640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da009d110 0x7f2da009d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:09.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.270+0000 7f2da4d34640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da009d110 0x7f2da009d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:09.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.270+0000 7f2da4d34640 10 monclient: get_auth_request con 0x7f2da009d110 auth_method 0 2024-09-16T04:46:09.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.270+0000 7f2da4d34640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.270+0000 7f2da4d34640 10 monclient: _init_auth method 2 2024-09-16T04:46:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.270+0000 7f2da4d34640 10 monclient: _init_auth already have auth, reseting 2024-09-16T04:46:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.270+0000 7f2da4d34640 10 monclient: handle_auth_reply_more payload 9 2024-09-16T04:46:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.270+0000 7f2da4d34640 10 monclient: handle_auth_reply_more payload_len 9 2024-09-16T04:46:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.270+0000 7f2da4d34640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-16T04:46:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.270+0000 7f2da4d34640 10 monclient: handle_auth_done global_id 6724 payload 293 2024-09-16T04:46:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.271+0000 7f2da4d34640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da009d110 0x7f2da009d4f0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f2d9000f770 tx=0x7f2d90005000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.271+0000 7f2d9d7fa640 1 -- 172.21.15.46:0/1499914383 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9000d8b0 con 0x7f2da009d110 2024-09-16T04:46:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.271+0000 7f2d9d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.271+0000 7f2d9d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:46:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.271+0000 7f2d9d7fa640 10 monclient: dump: 2024-09-16T04:46:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:09.275 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.271+0000 7f2d9d7fa640 1 -- 172.21.15.46:0/1499914383 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f2d9000d810 con 0x7f2da009d110 2024-09-16T04:46:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.271+0000 7f2d9d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:46:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.271+0000 7f2d9d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.271+0000 7f2d9d7fa640 1 -- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da009d110 msgr2=0x7f2da009d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.271+0000 7f2d9d7fa640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da009d110 0x7f2da009d4f0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f2d9000f770 tx=0x7f2d90005000 comp rx=0 tx=0).stop 2024-09-16T04:46:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.271+0000 7f2d9d7fa640 1 -- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da009d110 msgr2=0x7f2da009d4f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.271+0000 7f2d9d7fa640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2da009d110 0x7f2da009d4f0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:09.276 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:46:09.276 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:46:09.276 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:46:09.276 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:46:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 88, 2024-09-16T04:46:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 455174, 2024-09-16T04:46:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:46:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:46:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:46:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:46:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 4, 2024-09-16T04:46:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:46:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:46:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:09.277 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:09.278 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:09.278 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:09.278 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:09.278 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:09.278 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:09.278 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:09.278 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:09.278 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:09.278 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:09.278 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:46:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:46:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:46:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:46:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:46:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:46:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:46:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:46:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:46:09.280 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:46:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:46:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:46:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:46:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:46:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:46:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:46:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:09.281 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:46:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:09.282 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:46:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:46:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:46:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:46:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:09.283 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:46:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:46:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:46:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:46:09.284 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:46:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:46:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:46:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:09.285 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:46:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:46:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:46:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:46:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:09.286 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:46:09.287 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:09.287 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:09.287 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:09.287 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:09.287 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:46:09.287 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:09.287 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:09.287 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:09.287 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:46:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:46:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:46:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:46:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:09.288 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:46:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5 2024-09-16T04:46:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:09.289 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:46:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 7 2024-09-16T04:46:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:46:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:09.290 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:09.291 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:09.291 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:46:09.291 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:46:09.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.271+0000 7f2da6f98640 1 -- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2d8c0383b0 msgr2=0x7f2d8c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:09.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.271+0000 7f2da6f98640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2d8c0383b0 0x7f2d8c03a870 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2da00a8b40 tx=0x7f2d94009190 comp rx=0 tx=0).stop 2024-09-16T04:46:09.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.271+0000 7f2da6f98640 10 monclient: shutdown 2024-09-16T04:46:09.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.271+0000 7f2da6f98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:09.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.271+0000 7f2da6f98640 1 -- 172.21.15.46:0/1499914383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da013be90 msgr2=0x7f2da01cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:09.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.272+0000 7f2da6f98640 1 --2- 172.21.15.46:0/1499914383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da013be90 0x7f2da01cb9e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2d98009e50 tx=0x7f2d98004a40 comp rx=0 tx=0).stop 2024-09-16T04:46:09.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.272+0000 7f2da5535640 1 -- 172.21.15.46:0/1499914383 reap_dead start 2024-09-16T04:46:09.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.272+0000 7f2da5535640 1 -- 172.21.15.46:0/1499914383 reap_dead start 2024-09-16T04:46:09.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.272+0000 7f2da6f98640 1 -- 172.21.15.46:0/1499914383 shutdown_connections 2024-09-16T04:46:09.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.272+0000 7f2da6f98640 1 -- 172.21.15.46:0/1499914383 >> 172.21.15.46:0/1499914383 conn(0x7f2da00a6340 msgr2=0x7f2da0136c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:09.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.272+0000 7f2da6f98640 1 -- 172.21.15.46:0/1499914383 shutdown_connections 2024-09-16T04:46:09.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.272+0000 7f2da6f98640 1 -- 172.21.15.46:0/1499914383 wait complete. 2024-09-16T04:46:09.293 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:46:09.293 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:46:09.293 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:46:09.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.394+0000 7fd82de65640 10 monclient: get_monmap_and_config 2024-09-16T04:46:09.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.394+0000 7fd82de65640 10 monclient: build_initial_monmap 2024-09-16T04:46:09.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.394+0000 7fd82de65640 10 monclient: monmap: 2024-09-16T04:46:09.397 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:09.397 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:09.397 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:09.396237+0000 2024-09-16T04:46:09.397 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:09.396237+0000 2024-09-16T04:46:09.397 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:09.397 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:09.397 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:09.397 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:09.397 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:09.397 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:09.397 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:09.397 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:09.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.395+0000 7fd82de65640 1 Processor -- start 2024-09-16T04:46:09.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.395+0000 7fd82de65640 1 -- start start 2024-09-16T04:46:09.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.395+0000 7fd82de65640 10 monclient: init 2024-09-16T04:46:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.395+0000 7fd82de65640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd82de65640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-16T04:46:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd82de65640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd82806e160 0x7fd8280a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd82de65640 10 monclient(hunting): picked mon.noname-c con 0x7fd82806e160 addr v2:172.21.15.114:3300/0 2024-09-16T04:46:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd82de65640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd8280a2ee0 0x7fd828136940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd82de65640 10 monclient(hunting): picked mon.noname-b con 0x7fd8280a2ee0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd82de65640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd828136e80 0x7fd828137330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd82de65640 10 monclient(hunting): picked mon.noname-a con 0x7fd828136e80 addr v2:172.21.15.46:3300/0 2024-09-16T04:46:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd82de65640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd82de65640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd8280a3300 con 0x7fd828136e80 2024-09-16T04:46:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd82de65640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd82de65640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fd82806e540 con 0x7fd82806e160 2024-09-16T04:46:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd82de65640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd82de65640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd8280a3fd0 con 0x7fd8280a2ee0 2024-09-16T04:46:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd82de65640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd82de65640 10 monclient(hunting): authenticate will time out at 1167.812086s 2024-09-16T04:46:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd827fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd828136e80 0x7fd828137330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd826ffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd8280a2ee0 0x7fd828136940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd827fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd828136e80 0x7fd828137330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33394/0 (socket says 172.21.15.46:33394) 2024-09-16T04:46:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd827fff640 1 -- 172.21.15.46:0/3964968081 learned_addr learned my addr 172.21.15.46:0/3964968081 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd827fff640 10 monclient(hunting): get_auth_request con 0x7fd828136e80 auth_method 0 2024-09-16T04:46:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd827fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd827fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd827fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd826ffd640 10 monclient(hunting): get_auth_request con 0x7fd8280a2ee0 auth_method 0 2024-09-16T04:46:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd826ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd826ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd826ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd827fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd827fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd827fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd826ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd826ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd826ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd8277fe640 1 --2- 172.21.15.46:0/3964968081 >> v2:172.21.15.114:3300/0 conn(0x7fd82806e160 0x7fd8280a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd827fff640 10 monclient(hunting): handle_auth_done global_id 74664 payload 274 2024-09-16T04:46:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd827fff640 1 -- 172.21.15.46:0/3964968081 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd8280a2ee0 msgr2=0x7fd828136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd827fff640 1 --2- 172.21.15.46:0/3964968081 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd8280a2ee0 0x7fd828136940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd827fff640 1 -- 172.21.15.46:0/3964968081 >> v2:172.21.15.114:3300/0 conn(0x7fd82806e160 msgr2=0x7fd8280a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd827fff640 1 --2- 172.21.15.46:0/3964968081 >> v2:172.21.15.114:3300/0 conn(0x7fd82806e160 0x7fd8280a29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd827fff640 10 monclient: _finish_hunting 0 2024-09-16T04:46:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd827fff640 1 monclient: found mon.noname-a 2024-09-16T04:46:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd827fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd827fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:46:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd827fff640 1 -- 172.21.15.46:0/3964968081 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8280a0bd0 con 0x7fd828136e80 2024-09-16T04:46:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd8277fe640 10 monclient: get_auth_request con 0x7fd82806e160 auth_method 0 2024-09-16T04:46:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd826ffd640 1 --2- 172.21.15.46:0/3964968081 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd8280a2ee0 0x7fd828136940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.396+0000 7fd8277fe640 1 --2- 172.21.15.46:0/3964968081 >> v2:172.21.15.114:3300/0 conn(0x7fd82806e160 0x7fd8280a29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:46:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd827fff640 1 --2- 172.21.15.46:0/3964968081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd828136e80 0x7fd828137330 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd828068700 tx=0x7fd82002a5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69b796af813225c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd8267fc640 1 -- 172.21.15.46:0/3964968081 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd820036070 con 0x7fd828136e80 2024-09-16T04:46:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd8267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd8267fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:46:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd8267fc640 10 monclient: dump: 2024-09-16T04:46:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:09.410 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd8267fc640 10 monclient: _finish_auth 0 2024-09-16T04:46:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd8267fc640 10 monclient: _check_auth_tickets 2024-09-16T04:46:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd8267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd82de65640 5 monclient: authenticate success, global_id 74664 2024-09-16T04:46:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd8267fc640 1 -- 172.21.15.46:0/3964968081 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd820031600 con 0x7fd828136e80 2024-09-16T04:46:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd82de65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd8267fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd82de65640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd82de65640 4 set_mon_vals no callback set 2024-09-16T04:46:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd8267fc640 1 -- 172.21.15.46:0/3964968081 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8200319d0 con 0x7fd828136e80 2024-09-16T04:46:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd8267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd8267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd8267fc640 10 monclient: dump: 2024-09-16T04:46:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:09.412 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:09.412 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:09.412 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:09.412 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:09.412 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:09.412 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:09.412 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:09.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd82de65640 10 monclient: shutdown 2024-09-16T04:46:09.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd82de65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:09.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd82de65640 1 -- 172.21.15.46:0/3964968081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd828136e80 msgr2=0x7fd828137330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:09.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.397+0000 7fd82de65640 1 --2- 172.21.15.46:0/3964968081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd828136e80 0x7fd828137330 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd828068700 tx=0x7fd82002a5b0 comp rx=0 tx=0).stop 2024-09-16T04:46:09.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.398+0000 7fd82de65640 1 -- 172.21.15.46:0/3964968081 shutdown_connections 2024-09-16T04:46:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.398+0000 7fd82de65640 1 --2- 172.21.15.46:0/3964968081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd828136e80 0x7fd828137330 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.398+0000 7fd82de65640 1 --2- 172.21.15.46:0/3964968081 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd8280a2ee0 0x7fd828136940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.398+0000 7fd82de65640 1 --2- 172.21.15.46:0/3964968081 >> v2:172.21.15.114:3300/0 conn(0x7fd82806e160 0x7fd8280a29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.398+0000 7fd82de65640 1 -- 172.21.15.46:0/3964968081 >> 172.21.15.46:0/3964968081 conn(0x7fd8280a5c10 msgr2=0x7fd8280a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.398+0000 7fd82de65640 1 -- 172.21.15.46:0/3964968081 shutdown_connections 2024-09-16T04:46:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.398+0000 7fd82de65640 1 -- 172.21.15.46:0/3964968081 wait complete. 2024-09-16T04:46:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.398+0000 7fd82de65640 10 monclient: build_initial_monmap 2024-09-16T04:46:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.398+0000 7fd82de65640 10 monclient: monmap: 2024-09-16T04:46:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:09.414 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:09.414 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:09.414 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:09.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.398+0000 7fd82de65640 1 Processor -- start 2024-09-16T04:46:09.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd82de65640 1 -- start start 2024-09-16T04:46:09.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd82de65640 10 monclient: init 2024-09-16T04:46:09.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd82de65640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:09.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd82de65640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:46:09.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd82de65640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd82806e160 0x7fd8281cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:09.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd82de65640 10 monclient(hunting): picked mon.noname-c con 0x7fd82806e160 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:09.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd82de65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8280a2ee0 0x7fd8281cc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:09.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd82de65640 10 monclient(hunting): picked mon.noname-a con 0x7fd8280a2ee0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:09.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd82de65640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd828136e80 0x7fd8281234d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd82de65640 10 monclient(hunting): picked mon.noname-b con 0x7fd828136e80 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd82de65640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd82de65640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd8280a5130 con 0x7fd8280a2ee0 2024-09-16T04:46:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd82de65640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd82de65640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd8280a1190 con 0x7fd828136e80 2024-09-16T04:46:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd82de65640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd82de65640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd8280a34a0 con 0x7fd82806e160 2024-09-16T04:46:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd82de65640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd82de65640 10 monclient(hunting): authenticate will time out at 1167.815616s 2024-09-16T04:46:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd8277fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd82806e160 0x7fd8281cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd826ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8280a2ee0 0x7fd8281cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd8277fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd82806e160 0x7fd8281cbb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:44064/0 (socket says 172.21.15.46:44064) 2024-09-16T04:46:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd8277fe640 1 -- 172.21.15.46:0/2678647519 learned_addr learned my addr 172.21.15.46:0/2678647519 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd8277fe640 10 monclient(hunting): get_auth_request con 0x7fd82806e160 auth_method 0 2024-09-16T04:46:09.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd8277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:09.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd8277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:09.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd8277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:09.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd826ffd640 10 monclient(hunting): get_auth_request con 0x7fd8280a2ee0 auth_method 0 2024-09-16T04:46:09.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd826ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:09.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd826ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:09.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd826ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:09.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd826ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:09.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd826ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:09.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd826ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:09.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd8277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:09.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd8277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:09.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd8277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:09.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd827fff640 1 --2- 172.21.15.46:0/2678647519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd828136e80 0x7fd8281234d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:09.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.399+0000 7fd8277fe640 10 monclient(hunting): handle_auth_done global_id 94115 payload 824 2024-09-16T04:46:09.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd8277fe640 1 -- 172.21.15.46:0/2678647519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd828136e80 msgr2=0x7fd8281234d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:09.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd8277fe640 1 --2- 172.21.15.46:0/2678647519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd828136e80 0x7fd8281234d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:09.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd8277fe640 1 -- 172.21.15.46:0/2678647519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8280a2ee0 msgr2=0x7fd8281cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:09.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd8277fe640 1 --2- 172.21.15.46:0/2678647519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8280a2ee0 0x7fd8281cc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:09.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd8277fe640 10 monclient: _finish_hunting 0 2024-09-16T04:46:09.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd8277fe640 1 monclient: found mon.noname-c 2024-09-16T04:46:09.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd8277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:09.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd8277fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:46:09.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd8277fe640 1 -- 172.21.15.46:0/2678647519 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd828125c90 con 0x7fd82806e160 2024-09-16T04:46:09.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd826ffd640 1 --2- 172.21.15.46:0/2678647519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8280a2ee0 0x7fd8281cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:09.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd8277fe640 1 --2- 172.21.15.46:0/2678647519 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd82806e160 0x7fd8281cbb30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd81800c820 tx=0x7fd81800cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:09.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd80ffff640 1 -- 172.21.15.46:0/2678647519 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd818015070 con 0x7fd82806e160 2024-09-16T04:46:09.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd80ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:09.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd80ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:09.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd80ffff640 10 monclient: dump: 2024-09-16T04:46:09.418 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:09.418 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:09.418 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:09.418 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:09.418 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:09.418 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:09.418 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:09.418 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:09.418 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:09.418 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd80ffff640 10 monclient: _finish_auth 0 2024-09-16T04:46:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd80ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:46:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd80ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd80ffff640 1 -- 172.21.15.46:0/2678647519 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd818007a30 con 0x7fd82806e160 2024-09-16T04:46:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd82de65640 5 monclient: authenticate success, global_id 94115 2024-09-16T04:46:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd82de65640 10 monclient: _renew_subs 2024-09-16T04:46:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd82de65640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd82de65640 1 -- 172.21.15.46:0/2678647519 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8281255d0 con 0x7fd82806e160 2024-09-16T04:46:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd80ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd80ffff640 1 -- 172.21.15.46:0/2678647519 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd818007de0 con 0x7fd82806e160 2024-09-16T04:46:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd80ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd80ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd80ffff640 10 monclient: dump: 2024-09-16T04:46:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:09.420 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd82de65640 10 monclient: _renew_subs 2024-09-16T04:46:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd82de65640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd82de65640 1 -- 172.21.15.46:0/2678647519 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8280a8370 con 0x7fd82806e160 2024-09-16T04:46:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.400+0000 7fd825ffb640 4 set_mon_vals no callback set 2024-09-16T04:46:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.401+0000 7fd80ffff640 1 -- 172.21.15.46:0/2678647519 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fd818018070 con 0x7fd82806e160 2024-09-16T04:46:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.401+0000 7fd82de65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.401+0000 7fd82de65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.401+0000 7fd82de65640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.401+0000 7fd82de65640 1 -- 172.21.15.46:0/2678647519 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8280a4e20 con 0x7fd82806e160 2024-09-16T04:46:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.401+0000 7fd80ffff640 1 --2- 172.21.15.46:0/2678647519 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd8140383b0 0x7fd81403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.401+0000 7fd80ffff640 1 -- 172.21.15.46:0/2678647519 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(279..279 src has 1..279) ==== 4738+0+0 (secure 0 0 0) 0x7fd81804b740 con 0x7fd82806e160 2024-09-16T04:46:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.401+0000 7fd826ffd640 1 --2- 172.21.15.46:0/2678647519 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd8140383b0 0x7fd81403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.401+0000 7fd826ffd640 10 monclient: get_auth_request con 0x7fd8140383b0 auth_method 0 2024-09-16T04:46:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.402+0000 7fd826ffd640 1 --2- 172.21.15.46:0/2678647519 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd8140383b0 0x7fd81403a870 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd81c009560 tx=0x7fd81c009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.406+0000 7fd80ffff640 1 -- 172.21.15.46:0/2678647519 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd81800ce10 con 0x7fd82806e160 2024-09-16T04:46:09.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.406+0000 7fd80ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:09.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.406+0000 7fd80ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:09.550 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39818]: starting. 2024-09-16T04:46:09.550 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39818]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-39817 2024-09-16T04:46:09.550 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39818]: created object 0... 2024-09-16T04:46:09.550 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39818]: created object 25... 2024-09-16T04:46:09.550 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39818]: created object 49... 2024-09-16T04:46:09.551 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39818]: finishing. 2024-09-16T04:46:09.551 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39818]: shutting down. 2024-09-16T04:46:09.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.665+0000 7fd82de65640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:46:09.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.665+0000 7fd82de65640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:46:09.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.665+0000 7fd82de65640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:09.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.665+0000 7fd82de65640 1 -- 172.21.15.46:0/2678647519 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fd828067e00 con 0x7fd82806e160 2024-09-16T04:46:09.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.666+0000 7fd80ffff640 1 -- 172.21.15.46:0/2678647519 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fd818010b80 con 0x7fd82806e160 2024-09-16T04:46:09.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.666+0000 7fd80ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:46:09.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.666+0000 7fd80ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:09.669 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.667+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.669 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.667+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=11,kv=10,logm=61} crc {auth=1581326169,config=3541143443,health=2684431194,kv=4218700806,logm=3511591773}) 2024-09-16T04:46:09.669 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.667+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=2,health=12,kv=11,logm=59} crc {auth=1581326169,config=3541143443,health=2301855438,kv=2804613839,logm=4239949188}) 2024-09-16T04:46:09.669 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.667+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.669 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.667+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=11,kv=10,logm=61} crc {auth=1581326169,config=3541143443,health=2684431194,kv=4218700806,logm=3511591773}) 2024-09-16T04:46:09.669 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.667+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=15,config=2,health=12,kv=12,logm=59} crc {auth=2707793131,config=3541143443,health=2254079205,kv=4017288519,logm=3111550797}) 2024-09-16T04:46:09.670 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.667+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.670 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.667+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=809459765}) 2024-09-16T04:46:09.670 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.667+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2739423300}) 2024-09-16T04:46:09.670 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.667+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.670 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.667+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=809459765}) 2024-09-16T04:46:09.670 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.667+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4123464335}) 2024-09-16T04:46:09.670 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.668+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.671 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.668+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2653081396}) 2024-09-16T04:46:09.671 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.668+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2972971771}) 2024-09-16T04:46:09.671 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.668+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.671 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.668+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2653081396}) 2024-09-16T04:46:09.671 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.668+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1410334656}) 2024-09-16T04:46:09.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.669+0000 7fd82de65640 1 -- 172.21.15.46:0/2678647519 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd8140383b0 msgr2=0x7fd81403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.669+0000 7fd82de65640 1 --2- 172.21.15.46:0/2678647519 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd8140383b0 0x7fd81403a870 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd81c009560 tx=0x7fd81c009190 comp rx=0 tx=0).stop 2024-09-16T04:46:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.669+0000 7fd82de65640 10 monclient: shutdown 2024-09-16T04:46:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.669+0000 7fd82de65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.669+0000 7fd82de65640 1 -- 172.21.15.46:0/2678647519 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd82806e160 msgr2=0x7fd8281cbb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.669+0000 7fd82de65640 1 --2- 172.21.15.46:0/2678647519 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd82806e160 0x7fd8281cbb30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd81800c820 tx=0x7fd81800cce0 comp rx=0 tx=0).stop 2024-09-16T04:46:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.669+0000 7fd82de65640 1 -- 172.21.15.46:0/2678647519 shutdown_connections 2024-09-16T04:46:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.669+0000 7fd82de65640 1 --2- 172.21.15.46:0/2678647519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd828136e80 0x7fd8281234d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.669+0000 7fd82de65640 1 --2- 172.21.15.46:0/2678647519 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd8140383b0 0x7fd81403a870 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.669+0000 7fd82de65640 1 --2- 172.21.15.46:0/2678647519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8280a2ee0 0x7fd8281cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.669+0000 7fd82de65640 1 --2- 172.21.15.46:0/2678647519 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd82806e160 0x7fd8281cbb30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.669+0000 7fd82de65640 1 -- 172.21.15.46:0/2678647519 >> 172.21.15.46:0/2678647519 conn(0x7fd8280a5c10 msgr2=0x7fd8280a5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:09.673 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.669+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.673 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.669+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4191328852}) 2024-09-16T04:46:09.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.669+0000 7fd82de65640 1 -- 172.21.15.46:0/2678647519 shutdown_connections 2024-09-16T04:46:09.673 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.669+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1116937907}) 2024-09-16T04:46:09.673 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.669+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:09.669+0000 7fd82de65640 1 -- 172.21.15.46:0/2678647519 wait complete. 2024-09-16T04:46:09.676 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.669+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4191328852}) 2024-09-16T04:46:09.676 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.669+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1215932038}) 2024-09-16T04:46:09.676 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.670+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.676 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.670+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=599330793}) 2024-09-16T04:46:09.676 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.670+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3136813019}) 2024-09-16T04:46:09.676 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.670+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.676 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.670+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=599330793}) 2024-09-16T04:46:09.676 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.670+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=945433073}) 2024-09-16T04:46:09.676 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.670+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.677 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.670+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2882319475}) 2024-09-16T04:46:09.677 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.670+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1428631635}) 2024-09-16T04:46:09.677 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.670+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.677 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.670+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2882319475}) 2024-09-16T04:46:09.677 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.670+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=935648889}) 2024-09-16T04:46:09.677 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.671+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.677 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.671+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2793484814}) 2024-09-16T04:46:09.677 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.671+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=94538684}) 2024-09-16T04:46:09.677 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.671+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.677 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.671+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2793484814}) 2024-09-16T04:46:09.677 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.671+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3219617075}) 2024-09-16T04:46:09.677 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.672+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.677 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.672+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=441916353}) 2024-09-16T04:46:09.677 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.672+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3183519382}) 2024-09-16T04:46:09.677 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.672+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.678 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.672+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=441916353}) 2024-09-16T04:46:09.678 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.672+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2269461264}) 2024-09-16T04:46:09.678 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.673+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.678 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.673+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4098409322}) 2024-09-16T04:46:09.678 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.673+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3070205997}) 2024-09-16T04:46:09.678 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.673+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.678 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.674+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4098409322}) 2024-09-16T04:46:09.678 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.674+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3238980854}) 2024-09-16T04:46:09.680 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.674+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.681 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.675+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068076114}) 2024-09-16T04:46:09.681 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.675+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=757561485}) 2024-09-16T04:46:09.681 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.675+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.681 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.675+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068076114}) 2024-09-16T04:46:09.681 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.675+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4110683706}) 2024-09-16T04:46:09.681 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.675+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.681 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.675+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4014267148}) 2024-09-16T04:46:09.681 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.675+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1680584390}) 2024-09-16T04:46:09.681 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.675+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.681 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.676+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4014267148}) 2024-09-16T04:46:09.681 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.676+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3402402554}) 2024-09-16T04:46:09.681 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.677+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.681 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.677+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3096284968}) 2024-09-16T04:46:09.681 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.677+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=489580433}) 2024-09-16T04:46:09.682 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.677+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.682 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.677+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3096284968}) 2024-09-16T04:46:09.682 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.677+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1898298285}) 2024-09-16T04:46:09.682 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.678+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.682 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.678+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=210290355}) 2024-09-16T04:46:09.682 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.678+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3311630341}) 2024-09-16T04:46:09.682 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.678+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.682 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.678+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=210290355}) 2024-09-16T04:46:09.682 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.678+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3749212896}) 2024-09-16T04:46:09.682 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.678+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.682 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.679+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1011118368}) 2024-09-16T04:46:09.682 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.679+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3515768923}) 2024-09-16T04:46:09.682 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.679+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.682 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.679+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1011118368}) 2024-09-16T04:46:09.683 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.679+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4140451652}) 2024-09-16T04:46:09.685 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.679+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.685 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.679+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2373623329}) 2024-09-16T04:46:09.685 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.679+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2168893033}) 2024-09-16T04:46:09.685 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.679+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.685 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.679+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2373623329}) 2024-09-16T04:46:09.685 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.679+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1472774363}) 2024-09-16T04:46:09.685 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.680+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.685 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.680+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1927195261}) 2024-09-16T04:46:09.686 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.680+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2834589048}) 2024-09-16T04:46:09.686 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.680+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.686 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.680+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1927195261}) 2024-09-16T04:46:09.686 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.680+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=156605016}) 2024-09-16T04:46:09.686 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.681+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.686 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.681+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3717360251}) 2024-09-16T04:46:09.686 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.681+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1974586464}) 2024-09-16T04:46:09.686 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.681+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.686 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.681+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3717360251}) 2024-09-16T04:46:09.686 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.681+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2665557580}) 2024-09-16T04:46:09.686 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.682+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.686 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.682+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=162320917}) 2024-09-16T04:46:09.686 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.682+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2210853173}) 2024-09-16T04:46:09.686 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.682+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.687 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.682+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=162320917}) 2024-09-16T04:46:09.687 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.682+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1467272922}) 2024-09-16T04:46:09.687 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.683+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.687 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.683+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3532042580}) 2024-09-16T04:46:09.687 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.683+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=224645924}) 2024-09-16T04:46:09.687 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.683+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.687 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.683+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3532042580}) 2024-09-16T04:46:09.687 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.683+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1140973350}) 2024-09-16T04:46:09.688 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:46:09.690 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.684+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.690 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.684+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2844016974}) 2024-09-16T04:46:09.690 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.684+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3998425299}) 2024-09-16T04:46:09.690 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.684+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.691 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.684+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2844016974}) 2024-09-16T04:46:09.691 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.684+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3415805802}) 2024-09-16T04:46:09.691 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.684+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.691 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.684+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4178812775}) 2024-09-16T04:46:09.691 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.684+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3242325607}) 2024-09-16T04:46:09.691 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.684+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.691 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.684+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4178812775}) 2024-09-16T04:46:09.691 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.684+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3636797275}) 2024-09-16T04:46:09.691 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.685+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.691 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.685+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1449851137}) 2024-09-16T04:46:09.691 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.685+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1571952717}) 2024-09-16T04:46:09.691 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.685+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.691 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.685+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1449851137}) 2024-09-16T04:46:09.691 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.685+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3799494407}) 2024-09-16T04:46:09.692 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.686+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.692 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.686+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=36,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=16} crc {logm=192942467,mdsmap=3956149183,mgr=2343726890,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=1307993747,mon_config_key=1385660123,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=2257708948,osd_snap=723727393,osdmap=1308311135}) 2024-09-16T04:46:09.692 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.686+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=37,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=19} crc {logm=1686102244,mdsmap=3956149183,mgr=1081907435,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=883986065,mon_config_key=265800298,monmap=3088247691,osd_metadata=1392435868,osd_pg_creating=2257708948,osd_snap=723727393,osdmap=3691009640}) 2024-09-16T04:46:09.692 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.686+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.692 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.686+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=36,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=16} crc {logm=192942467,mdsmap=3956149183,mgr=2343726890,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=1307993747,mon_config_key=1385660123,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=2257708948,osd_snap=723727393,osdmap=1308311135}) 2024-09-16T04:46:09.692 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.686+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=38,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=15} crc {logm=3425204237,mdsmap=3956149183,mgr=1081907436,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=2434468698,mon_config_key=124642177,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=2257708948,osd_snap=723727393,osdmap=3374199333}) 2024-09-16T04:46:09.692 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.686+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.692 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.686+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1945587835}) 2024-09-16T04:46:09.692 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.686+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=857300018}) 2024-09-16T04:46:09.692 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.686+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.692 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.686+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1945587835}) 2024-09-16T04:46:09.692 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.686+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2425079353}) 2024-09-16T04:46:09.692 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.687+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.693 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.687+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3680995051}) 2024-09-16T04:46:09.693 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.687+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2423153035}) 2024-09-16T04:46:09.693 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.687+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.693 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.687+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3680995051}) 2024-09-16T04:46:09.693 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.687+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3794970345}) 2024-09-16T04:46:09.693 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.688+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.693 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.688+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3594595492}) 2024-09-16T04:46:09.693 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.688+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2269488112}) 2024-09-16T04:46:09.693 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.688+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.693 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.688+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3594595492}) 2024-09-16T04:46:09.693 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.688+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2525170759}) 2024-09-16T04:46:09.694 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.690+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.694 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.690+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1645712839}) 2024-09-16T04:46:09.694 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.690+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2684133670}) 2024-09-16T04:46:09.694 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.690+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:09.694 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.690+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1645712839}) 2024-09-16T04:46:09.694 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:09.690+0000 7f6377ffa640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2928974847}) 2024-09-16T04:46:10.032 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39819]: starting. 2024-09-16T04:46:10.032 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39819]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-39817 2024-09-16T04:46:10.032 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39819]: shutting down. 2024-09-16T04:46:10.689 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:46:10.689 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:46:10.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.800+0000 7ff543c8c640 10 monclient: get_monmap_and_config 2024-09-16T04:46:10.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.800+0000 7ff543c8c640 10 monclient: build_initial_monmap 2024-09-16T04:46:10.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.800+0000 7ff543c8c640 10 monclient: monmap: 2024-09-16T04:46:10.802 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:10.802 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:10.802 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:10.801890+0000 2024-09-16T04:46:10.803 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:10.801890+0000 2024-09-16T04:46:10.803 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:10.803 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:10.803 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:10.803 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:10.803 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:10.803 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:10.803 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:10.803 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:10.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.801+0000 7ff543c8c640 1 Processor -- start 2024-09-16T04:46:10.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.801+0000 7ff543c8c640 1 -- start start 2024-09-16T04:46:10.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.801+0000 7ff543c8c640 10 monclient: init 2024-09-16T04:46:10.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.801+0000 7ff543c8c640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:10.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.801+0000 7ff543c8c640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-16T04:46:10.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.801+0000 7ff543c8c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff53c073760 addr v1:172.21.15.114:6789/0 2024-09-16T04:46:10.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.801+0000 7ff543c8c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff53c139be0 0x7ff53c13bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:10.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.801+0000 7ff543c8c640 10 monclient(hunting): picked mon.noname-c con 0x7ff53c139be0 addr v2:172.21.15.114:3300/0 2024-09-16T04:46:10.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.801+0000 7ff543c8c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff53c13c510 0x7ff53c13e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:10.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.801+0000 7ff543c8c640 10 monclient(hunting): picked mon.noname-a con 0x7ff53c13c510 addr v2:172.21.15.46:3300/0 2024-09-16T04:46:10.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.801+0000 7ff543c8c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff53c074250 con 0x7ff53c073760 2024-09-16T04:46:10.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.801+0000 7ff543c8c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:10.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.801+0000 7ff543c8c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff53c06e070 con 0x7ff53c13c510 2024-09-16T04:46:10.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.801+0000 7ff543c8c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:10.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.801+0000 7ff543c8c640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ff53c09e960 con 0x7ff53c139be0 2024-09-16T04:46:10.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.801+0000 7ff543c8c640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:10.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.801+0000 7ff543c8c640 10 monclient(hunting): authenticate will time out at 1169.217838s 2024-09-16T04:46:10.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff542229640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff53c13c510 0x7ff53c13e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:10.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff542229640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff53c13c510 0x7ff53c13e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33410/0 (socket says 172.21.15.46:33410) 2024-09-16T04:46:10.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff542229640 1 -- 172.21.15.46:0/3477587887 learned_addr learned my addr 172.21.15.46:0/3477587887 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:10.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff542229640 10 monclient(hunting): get_auth_request con 0x7ff53c13c510 auth_method 0 2024-09-16T04:46:10.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff542229640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:10.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff542229640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:10.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff542229640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:10.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff542229640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:10.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff542229640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:10.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff542229640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:10.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff541227640 1 --2- 172.21.15.46:0/3477587887 >> v2:172.21.15.114:3300/0 conn(0x7ff53c139be0 0x7ff53c13bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:10.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff541227640 10 monclient(hunting): get_auth_request con 0x7ff53c139be0 auth_method 0 2024-09-16T04:46:10.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff541227640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:10.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff541227640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:10.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff541227640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:10.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff542229640 10 monclient(hunting): handle_auth_done global_id 74688 payload 274 2024-09-16T04:46:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff542229640 1 -- 172.21.15.46:0/3477587887 >> v2:172.21.15.114:3300/0 conn(0x7ff53c139be0 msgr2=0x7ff53c13bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff542229640 1 --2- 172.21.15.46:0/3477587887 >> v2:172.21.15.114:3300/0 conn(0x7ff53c139be0 0x7ff53c13bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff542229640 1 -- 172.21.15.46:0/3477587887 >> v1:172.21.15.114:6789/0 conn(0x7ff53c073760 legacy=0x7ff53c073b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff542229640 10 monclient: _finish_hunting 0 2024-09-16T04:46:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff542229640 1 monclient: found mon.noname-a 2024-09-16T04:46:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff542229640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff542229640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:46:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.802+0000 7ff542229640 1 -- 172.21.15.46:0/3477587887 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff53c09ef20 con 0x7ff53c13c510 2024-09-16T04:46:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff542229640 1 --2- 172.21.15.46:0/3477587887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff53c13c510 0x7ff53c13e900 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0a4ba0 tx=0x7ff53402a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b42ed6d40d7b5b4b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff540a26640 1 -- 172.21.15.46:0/3477587887 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff534037070 con 0x7ff53c13c510 2024-09-16T04:46:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff540a26640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff540a26640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:46:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff540a26640 10 monclient: dump: 2024-09-16T04:46:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:10.815 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff540a26640 10 monclient: _finish_auth 0 2024-09-16T04:46:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff540a26640 10 monclient: _check_auth_tickets 2024-09-16T04:46:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff540a26640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff540a26640 1 -- 172.21.15.46:0/3477587887 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff534032440 con 0x7ff53c13c510 2024-09-16T04:46:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff543c8c640 5 monclient: authenticate success, global_id 74688 2024-09-16T04:46:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff543c8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff540a26640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff543c8c640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff540a26640 1 -- 172.21.15.46:0/3477587887 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff534032810 con 0x7ff53c13c510 2024-09-16T04:46:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff543c8c640 4 set_mon_vals no callback set 2024-09-16T04:46:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff540a26640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff540a26640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff540a26640 10 monclient: dump: 2024-09-16T04:46:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:10.817 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff543c8c640 10 monclient: shutdown 2024-09-16T04:46:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff543c8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff543c8c640 1 -- 172.21.15.46:0/3477587887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff53c13c510 msgr2=0x7ff53c13e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff543c8c640 1 --2- 172.21.15.46:0/3477587887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff53c13c510 0x7ff53c13e900 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0a4ba0 tx=0x7ff53402a150 comp rx=0 tx=0).stop 2024-09-16T04:46:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff543c8c640 1 -- 172.21.15.46:0/3477587887 shutdown_connections 2024-09-16T04:46:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff543c8c640 1 --2- 172.21.15.46:0/3477587887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff53c13c510 0x7ff53c13e900 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff543c8c640 1 --2- 172.21.15.46:0/3477587887 >> v2:172.21.15.114:3300/0 conn(0x7ff53c139be0 0x7ff53c13bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.803+0000 7ff543c8c640 1 -- 172.21.15.46:0/3477587887 >> 172.21.15.46:0/3477587887 conn(0x7ff53c0a3b10 msgr2=0x7ff53c0a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.804+0000 7ff543c8c640 1 -- 172.21.15.46:0/3477587887 shutdown_connections 2024-09-16T04:46:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.804+0000 7ff543c8c640 1 -- 172.21.15.46:0/3477587887 wait complete. 2024-09-16T04:46:10.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.804+0000 7ff543c8c640 10 monclient: build_initial_monmap 2024-09-16T04:46:10.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.804+0000 7ff543c8c640 10 monclient: monmap: 2024-09-16T04:46:10.818 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:10.818 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:10.818 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:10.818 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:10.818 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:10.818 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:10.818 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:10.818 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:10.818 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:10.818 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:10.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.804+0000 7ff543c8c640 1 Processor -- start 2024-09-16T04:46:10.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff543c8c640 1 -- start start 2024-09-16T04:46:10.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff543c8c640 10 monclient: init 2024-09-16T04:46:10.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff543c8c640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:10.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff543c8c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:46:10.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff543c8c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff53c073760 0x7ff53c09cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:10.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff543c8c640 10 monclient(hunting): picked mon.noname-a con 0x7ff53c073760 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:10.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff543c8c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff53c139be0 0x7ff53c137f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:10.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff543c8c640 10 monclient(hunting): picked mon.noname-c con 0x7ff53c139be0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:10.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff543c8c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff53c13c510 0x7ff53c1384d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:10.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff543c8c640 10 monclient(hunting): picked mon.noname-b con 0x7ff53c13c510 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:10.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff543c8c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:10.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff543c8c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff53c09df80 con 0x7ff53c073760 2024-09-16T04:46:10.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff543c8c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:10.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff543c8c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff53c09e6c0 con 0x7ff53c13c510 2024-09-16T04:46:10.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff543c8c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff543c8c640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7ff53c0a2c90 con 0x7ff53c139be0 2024-09-16T04:46:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff543c8c640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff543c8c640 10 monclient(hunting): authenticate will time out at 1169.221487s 2024-09-16T04:46:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541a28640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff53c073760 0x7ff53c09cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541227640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff53c139be0 0x7ff53c137f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541a28640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff53c073760 0x7ff53c09cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33422/0 (socket says 172.21.15.46:33422) 2024-09-16T04:46:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541227640 10 monclient(hunting): get_auth_request con 0x7ff53c139be0 auth_method 0 2024-09-16T04:46:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541a28640 1 -- 172.21.15.46:0/3331528483 learned_addr learned my addr 172.21.15.46:0/3331528483 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541227640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541227640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541227640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541a28640 10 monclient(hunting): get_auth_request con 0x7ff53c073760 auth_method 0 2024-09-16T04:46:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541a28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541a28640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541a28640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541a28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541a28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541a28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541227640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541227640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff542229640 1 --2- 172.21.15.46:0/3331528483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff53c13c510 0x7ff53c1384d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541227640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff542229640 10 monclient(hunting): get_auth_request con 0x7ff53c13c510 auth_method 0 2024-09-16T04:46:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff542229640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff542229640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff542229640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff542229640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff542229640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff542229640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541a28640 10 monclient(hunting): handle_auth_done global_id 74691 payload 824 2024-09-16T04:46:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541a28640 1 -- 172.21.15.46:0/3331528483 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff53c139be0 msgr2=0x7ff53c137f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541a28640 1 --2- 172.21.15.46:0/3331528483 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff53c139be0 0x7ff53c137f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541a28640 1 -- 172.21.15.46:0/3331528483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff53c13c510 msgr2=0x7ff53c1384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541a28640 1 --2- 172.21.15.46:0/3331528483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff53c13c510 0x7ff53c1384d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541a28640 10 monclient: _finish_hunting 0 2024-09-16T04:46:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541a28640 1 monclient: found mon.noname-a 2024-09-16T04:46:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541a28640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541a28640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:46:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.805+0000 7ff541a28640 1 -- 172.21.15.46:0/3331528483 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff53c09e200 con 0x7ff53c073760 2024-09-16T04:46:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff541227640 1 --2- 172.21.15.46:0/3331528483 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff53c139be0 0x7ff53c137f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:10.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff541a28640 1 --2- 172.21.15.46:0/3331528483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff53c073760 0x7ff53c09cbf0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff52c00c790 tx=0x7ff52c00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:10.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff53a7fc640 1 -- 172.21.15.46:0/3331528483 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff52c016020 con 0x7ff53c073760 2024-09-16T04:46:10.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff53a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:10.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff53a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:46:10.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff53a7fc640 10 monclient: dump: 2024-09-16T04:46:10.823 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:10.823 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:10.823 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:10.823 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:10.823 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:10.823 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:10.823 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:10.823 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:10.824 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff53a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:46:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff53a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:46:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff53a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff53a7fc640 1 -- 172.21.15.46:0/3331528483 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff52c006d50 con 0x7ff53c073760 2024-09-16T04:46:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff543c8c640 5 monclient: authenticate success, global_id 74691 2024-09-16T04:46:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff53a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff53a7fc640 1 -- 172.21.15.46:0/3331528483 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff52c003bb0 con 0x7ff53c073760 2024-09-16T04:46:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff543c8c640 10 monclient: _renew_subs 2024-09-16T04:46:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff543c8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff543c8c640 1 -- 172.21.15.46:0/3331528483 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff53c124660 con 0x7ff53c073760 2024-09-16T04:46:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff53a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff53a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff53a7fc640 10 monclient: dump: 2024-09-16T04:46:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:10.825 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff53b7fe640 4 set_mon_vals no callback set 2024-09-16T04:46:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff543c8c640 10 monclient: _renew_subs 2024-09-16T04:46:10.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff543c8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:10.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.806+0000 7ff543c8c640 1 -- 172.21.15.46:0/3331528483 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff53c0a5030 con 0x7ff53c073760 2024-09-16T04:46:10.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.807+0000 7ff53a7fc640 1 -- 172.21.15.46:0/3331528483 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7ff52c00a850 con 0x7ff53c073760 2024-09-16T04:46:10.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.807+0000 7ff543c8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:10.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.807+0000 7ff543c8c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:10.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.807+0000 7ff543c8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:10.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.807+0000 7ff543c8c640 1 -- 172.21.15.46:0/3331528483 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff53c09f900 con 0x7ff53c073760 2024-09-16T04:46:10.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.807+0000 7ff53a7fc640 1 --2- 172.21.15.46:0/3331528483 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff5280383b0 0x7ff52803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:10.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.807+0000 7ff53a7fc640 1 -- 172.21.15.46:0/3331528483 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(281..281 src has 1..281) ==== 4346+0+0 (secure 0 0 0) 0x7ff52c04b2a0 con 0x7ff53c073760 2024-09-16T04:46:10.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.807+0000 7ff541227640 1 --2- 172.21.15.46:0/3331528483 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff5280383b0 0x7ff52803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:10.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.808+0000 7ff541227640 10 monclient: get_auth_request con 0x7ff5280383b0 auth_method 0 2024-09-16T04:46:10.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.808+0000 7ff541227640 1 --2- 172.21.15.46:0/3331528483 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff5280383b0 0x7ff52803a870 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff53c09d570 tx=0x7ff530008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:10.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.812+0000 7ff53a7fc640 1 -- 172.21.15.46:0/3331528483 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff52c020020 con 0x7ff53c073760 2024-09-16T04:46:10.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.812+0000 7ff53a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:10.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:10.812+0000 7ff53a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:11.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.061+0000 7ff543c8c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:46:11.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.061+0000 7ff543c8c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:11.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.061+0000 7ff543c8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:11.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.061+0000 7ff543c8c640 1 -- 172.21.15.46:0/3331528483 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff53c1cbe50 con 0x7ff53c073760 2024-09-16T04:46:11.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.062+0000 7ff53a7fc640 1 -- 172.21.15.46:0/3331528483 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7ff53c09e200 con 0x7ff53c073760 2024-09-16T04:46:11.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.062+0000 7ff53a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:11.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.062+0000 7ff53a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:11.064 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:46:11.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.064+0000 7ff543c8c640 1 -- 172.21.15.46:0/3331528483 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff5280383b0 msgr2=0x7ff52803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.064+0000 7ff543c8c640 1 --2- 172.21.15.46:0/3331528483 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff5280383b0 0x7ff52803a870 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff53c09d570 tx=0x7ff530008040 comp rx=0 tx=0).stop 2024-09-16T04:46:11.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.064+0000 7ff543c8c640 10 monclient: shutdown 2024-09-16T04:46:11.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.064+0000 7ff543c8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:11.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.064+0000 7ff543c8c640 1 -- 172.21.15.46:0/3331528483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff53c073760 msgr2=0x7ff53c09cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.064+0000 7ff543c8c640 1 --2- 172.21.15.46:0/3331528483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff53c073760 0x7ff53c09cbf0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff52c00c790 tx=0x7ff52c00cc50 comp rx=0 tx=0).stop 2024-09-16T04:46:11.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.064+0000 7ff543c8c640 1 -- 172.21.15.46:0/3331528483 shutdown_connections 2024-09-16T04:46:11.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.064+0000 7ff543c8c640 1 --2- 172.21.15.46:0/3331528483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff53c13c510 0x7ff53c1384d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.064+0000 7ff543c8c640 1 --2- 172.21.15.46:0/3331528483 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff53c139be0 0x7ff53c137f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.064+0000 7ff543c8c640 1 --2- 172.21.15.46:0/3331528483 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff5280383b0 0x7ff52803a870 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.064+0000 7ff543c8c640 1 --2- 172.21.15.46:0/3331528483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff53c073760 0x7ff53c09cbf0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.064+0000 7ff543c8c640 1 -- 172.21.15.46:0/3331528483 >> 172.21.15.46:0/3331528483 conn(0x7ff53c0a3b10 msgr2=0x7ff53c138df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:11.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.064+0000 7ff543c8c640 1 -- 172.21.15.46:0/3331528483 shutdown_connections 2024-09-16T04:46:11.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.064+0000 7ff543c8c640 1 -- 172.21.15.46:0/3331528483 wait complete. 2024-09-16T04:46:11.076 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:46:11.076 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:46:11.076 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:46:11.077 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:46:11.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.176+0000 7f7246349640 10 monclient: get_monmap_and_config 2024-09-16T04:46:11.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.176+0000 7f7246349640 10 monclient: build_initial_monmap 2024-09-16T04:46:11.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.176+0000 7f7246349640 10 monclient: monmap: 2024-09-16T04:46:11.178 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:11.178 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.178 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:11.177819+0000 2024-09-16T04:46:11.178 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:11.177819+0000 2024-09-16T04:46:11.179 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:11.179 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.179 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:11.179 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:11.179 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:11.179 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:11.179 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:11.179 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f7246349640 1 Processor -- start 2024-09-16T04:46:11.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f7246349640 1 -- start start 2024-09-16T04:46:11.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f7246349640 10 monclient: init 2024-09-16T04:46:11.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f7246349640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:11.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f7246349640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-16T04:46:11.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f7246349640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f724006e160 addr v1:172.21.15.114:6789/0 2024-09-16T04:46:11.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f7246349640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7240139c50 0x7f724013c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f7246349640 10 monclient(hunting): picked mon.noname-b con 0x7f7240139c50 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:11.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f7246349640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f724013c580 0x7f724013e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f7246349640 10 monclient(hunting): picked mon.noname-a con 0x7f724013c580 addr v2:172.21.15.46:3300/0 2024-09-16T04:46:11.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f7246349640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72400a3fd0 con 0x7f724006e160 2024-09-16T04:46:11.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f7246349640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f7246349640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f724006e540 con 0x7f724013c580 2024-09-16T04:46:11.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f7246349640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f7246349640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f72400a0bd0 con 0x7f7240139c50 2024-09-16T04:46:11.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f7246349640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:11.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f7246349640 10 monclient(hunting): authenticate will time out at 1169.593682s 2024-09-16T04:46:11.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f72448e6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f724013c580 0x7f724013e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f723f7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7240139c50 0x7f724013c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f72448e6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f724013c580 0x7f724013e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33424/0 (socket says 172.21.15.46:33424) 2024-09-16T04:46:11.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f723f7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7240139c50 0x7f724013c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:44082/0 (socket says 172.21.15.46:44082) 2024-09-16T04:46:11.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f72448e6640 1 -- 172.21.15.46:0/2767128178 learned_addr learned my addr 172.21.15.46:0/2767128178 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:11.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f72448e6640 10 monclient(hunting): get_auth_request con 0x7f724013c580 auth_method 0 2024-09-16T04:46:11.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f72448e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f72448e6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f72448e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f723f7fe640 10 monclient(hunting): get_auth_request con 0x7f7240139c50 auth_method 0 2024-09-16T04:46:11.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f723f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f723f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f723f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f72448e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:11.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f72448e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:11.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f72448e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:11.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f723f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:11.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f723f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:11.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.177+0000 7f723f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:11.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.178+0000 7f723f7fe640 10 monclient(hunting): handle_auth_done global_id 94139 payload 274 2024-09-16T04:46:11.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.178+0000 7f723f7fe640 1 -- 172.21.15.46:0/2767128178 >> v2:172.21.15.46:3300/0 conn(0x7f724013c580 msgr2=0x7f724013e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:11.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.178+0000 7f723f7fe640 1 --2- 172.21.15.46:0/2767128178 >> v2:172.21.15.46:3300/0 conn(0x7f724013c580 0x7f724013e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.178+0000 7f723f7fe640 1 -- 172.21.15.46:0/2767128178 >> v1:172.21.15.114:6789/0 conn(0x7f724006e160 legacy=0x7f72400a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:11.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.178+0000 7f723f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:46:11.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.178+0000 7f723f7fe640 1 monclient: found mon.noname-b 2024-09-16T04:46:11.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.178+0000 7f723f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:11.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.178+0000 7f723f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:46:11.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.178+0000 7f723f7fe640 1 -- 172.21.15.46:0/2767128178 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72400a0eb0 con 0x7f7240139c50 2024-09-16T04:46:11.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.178+0000 7f72448e6640 1 --2- 172.21.15.46:0/2767128178 >> v2:172.21.15.46:3300/0 conn(0x7f724013c580 0x7f724013e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:11.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.178+0000 7f723f7fe640 1 --2- 172.21.15.46:0/2767128178 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7240139c50 0x7f724013c040 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f72340097a0 tx=0x7f723402a010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7d3d7f40ac218e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:11.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.178+0000 7f723effd640 1 -- 172.21.15.46:0/2767128178 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7234037070 con 0x7f7240139c50 2024-09-16T04:46:11.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.178+0000 7f723effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.178+0000 7f723effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:46:11.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.178+0000 7f723effd640 10 monclient: dump: 2024-09-16T04:46:11.192 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.192 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.192 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.192 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.192 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.192 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.192 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.192 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.192 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.192 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f723effd640 10 monclient: _finish_auth 0 2024-09-16T04:46:11.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f723effd640 10 monclient: _check_auth_tickets 2024-09-16T04:46:11.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f723effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:11.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f723effd640 1 -- 172.21.15.46:0/2767128178 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f723402ad20 con 0x7f7240139c50 2024-09-16T04:46:11.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f7246349640 5 monclient: authenticate success, global_id 94139 2024-09-16T04:46:11.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f723effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:11.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f7246349640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:11.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f7246349640 4 set_mon_vals no callback set 2024-09-16T04:46:11.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f723effd640 1 -- 172.21.15.46:0/2767128178 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7234032660 con 0x7f7240139c50 2024-09-16T04:46:11.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f723effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f723effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:11.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f723effd640 10 monclient: dump: 2024-09-16T04:46:11.193 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.193 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.194 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.194 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.194 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.194 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.194 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.194 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.194 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.194 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f7246349640 10 monclient: shutdown 2024-09-16T04:46:11.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f7246349640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:11.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f7246349640 1 -- 172.21.15.46:0/2767128178 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7240139c50 msgr2=0x7f724013c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f7246349640 1 --2- 172.21.15.46:0/2767128178 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7240139c50 0x7f724013c040 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f72340097a0 tx=0x7f723402a010 comp rx=0 tx=0).stop 2024-09-16T04:46:11.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f7246349640 1 -- 172.21.15.46:0/2767128178 shutdown_connections 2024-09-16T04:46:11.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f7246349640 1 --2- 172.21.15.46:0/2767128178 >> v2:172.21.15.46:3300/0 conn(0x7f724013c580 0x7f724013e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f7246349640 1 --2- 172.21.15.46:0/2767128178 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7240139c50 0x7f724013c040 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f7246349640 1 -- 172.21.15.46:0/2767128178 >> 172.21.15.46:0/2767128178 conn(0x7f72400a5c10 msgr2=0x7f72400a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:11.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f7246349640 1 -- 172.21.15.46:0/2767128178 shutdown_connections 2024-09-16T04:46:11.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.179+0000 7f7246349640 1 -- 172.21.15.46:0/2767128178 wait complete. 2024-09-16T04:46:11.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.180+0000 7f7246349640 10 monclient: build_initial_monmap 2024-09-16T04:46:11.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.180+0000 7f7246349640 10 monclient: monmap: 2024-09-16T04:46:11.195 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:11.195 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:11.195 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:11.195 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:11.195 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:11.195 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.195 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:11.195 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:11.196 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:11.196 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.180+0000 7f7246349640 1 Processor -- start 2024-09-16T04:46:11.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.180+0000 7f7246349640 1 -- start start 2024-09-16T04:46:11.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.180+0000 7f7246349640 10 monclient: init 2024-09-16T04:46:11.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f7246349640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:11.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f7246349640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:46:11.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f7246349640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f724006e160 0x7f724009f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f7246349640 10 monclient(hunting): picked mon.noname-a con 0x7f724006e160 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:11.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f7246349640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7240139c50 0x7f7240136090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f7246349640 10 monclient(hunting): picked mon.noname-c con 0x7f7240139c50 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:11.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f7246349640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f724013c580 0x7f72401cbc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f7246349640 10 monclient(hunting): picked mon.noname-b con 0x7f724013c580 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:11.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f7246349640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f7246349640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f72400a34a0 con 0x7f724006e160 2024-09-16T04:46:11.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f7246349640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f7246349640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f72400a4d90 con 0x7f724013c580 2024-09-16T04:46:11.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f7246349640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f7246349640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f7240122510 con 0x7f7240139c50 2024-09-16T04:46:11.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f7246349640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:11.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f7246349640 10 monclient(hunting): authenticate will time out at 1169.597210s 2024-09-16T04:46:11.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723f7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7240139c50 0x7f7240136090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f724006e160 0x7f724009f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723f7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7240139c50 0x7f7240136090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:44096/0 (socket says 172.21.15.46:44096) 2024-09-16T04:46:11.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723f7fe640 1 -- 172.21.15.46:0/2536149654 learned_addr learned my addr 172.21.15.46:0/2536149654 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:11.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723f7fe640 10 monclient(hunting): get_auth_request con 0x7f7240139c50 auth_method 0 2024-09-16T04:46:11.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723ffff640 10 monclient(hunting): get_auth_request con 0x7f724006e160 auth_method 0 2024-09-16T04:46:11.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:11.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:11.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:11.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:11.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:11.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:11.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f72448e6640 1 --2- 172.21.15.46:0/2536149654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f724013c580 0x7f72401cbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f72448e6640 10 monclient(hunting): get_auth_request con 0x7f724013c580 auth_method 0 2024-09-16T04:46:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f72448e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f72448e6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f72448e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723ffff640 10 monclient(hunting): handle_auth_done global_id 74697 payload 824 2024-09-16T04:46:11.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723ffff640 1 -- 172.21.15.46:0/2536149654 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7240139c50 msgr2=0x7f7240136090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723ffff640 1 --2- 172.21.15.46:0/2536149654 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7240139c50 0x7f7240136090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723ffff640 1 -- 172.21.15.46:0/2536149654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f724013c580 msgr2=0x7f72401cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723ffff640 1 --2- 172.21.15.46:0/2536149654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f724013c580 0x7f72401cbc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:46:11.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723ffff640 1 monclient: found mon.noname-a 2024-09-16T04:46:11.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:11.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:46:11.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723ffff640 1 -- 172.21.15.46:0/2536149654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7240120d10 con 0x7f724006e160 2024-09-16T04:46:11.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f72448e6640 1 --2- 172.21.15.46:0/2536149654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f724013c580 0x7f72401cbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:46:11.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723f7fe640 1 --2- 172.21.15.46:0/2536149654 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7240139c50 0x7f7240136090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:11.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723ffff640 1 --2- 172.21.15.46:0/2536149654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f724006e160 0x7f724009f080 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f723000b3e0 tx=0x7f723000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:11.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723cff9640 1 -- 172.21.15.46:0/2536149654 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7230015030 con 0x7f724006e160 2024-09-16T04:46:11.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:46:11.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723cff9640 10 monclient: dump: 2024-09-16T04:46:11.201 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.201 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.201 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.201 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.201 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.201 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.201 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.202 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723cff9640 10 monclient: _finish_auth 0 2024-09-16T04:46:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723cff9640 10 monclient: _check_auth_tickets 2024-09-16T04:46:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f7246349640 5 monclient: authenticate success, global_id 74697 2024-09-16T04:46:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723cff9640 1 -- 172.21.15.46:0/2536149654 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7230004020 con 0x7f724006e160 2024-09-16T04:46:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f7246349640 10 monclient: _renew_subs 2024-09-16T04:46:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723cff9640 1 -- 172.21.15.46:0/2536149654 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7230004450 con 0x7f724006e160 2024-09-16T04:46:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f7246349640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f7246349640 1 -- 172.21.15.46:0/2536149654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7240122a40 con 0x7f724006e160 2024-09-16T04:46:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.182+0000 7f723cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.182+0000 7f723cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.182+0000 7f723cff9640 10 monclient: dump: 2024-09-16T04:46:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.203 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.181+0000 7f723e7fc640 4 set_mon_vals no callback set 2024-09-16T04:46:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.182+0000 7f7246349640 10 monclient: _renew_subs 2024-09-16T04:46:11.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.182+0000 7f7246349640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:11.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.182+0000 7f7246349640 1 -- 172.21.15.46:0/2536149654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f724011fc10 con 0x7f724006e160 2024-09-16T04:46:11.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.182+0000 7f7246349640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:11.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.182+0000 7f7246349640 1 --2- 172.21.15.46:0/2536149654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72401365d0 0x7f72401cc140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.182+0000 7f7246349640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.182+0000 7f7246349640 1 -- 172.21.15.46:0/2536149654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f72400a3620 con 0x7f72401365d0 2024-09-16T04:46:11.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.182+0000 7f723cff9640 1 -- 172.21.15.46:0/2536149654 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f7230018070 con 0x7f724006e160 2024-09-16T04:46:11.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.182+0000 7f723f7fe640 1 --2- 172.21.15.46:0/2536149654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72401365d0 0x7f72401cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.183+0000 7f723f7fe640 10 monclient: get_auth_request con 0x7f72401365d0 auth_method 0 2024-09-16T04:46:11.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.183+0000 7f723f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.183+0000 7f723f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.183+0000 7f723f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.183+0000 7f723f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:11.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.183+0000 7f723f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.183+0000 7f723f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.183+0000 7f723cff9640 1 --2- 172.21.15.46:0/2536149654 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f722c0383b0 0x7f722c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.183+0000 7f723cff9640 1 -- 172.21.15.46:0/2536149654 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(282..282 src has 1..282) ==== 4738+0+0 (secure 0 0 0) 0x7f723004b1d0 con 0x7f724006e160 2024-09-16T04:46:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.183+0000 7f723f7fe640 10 monclient(hunting): handle_auth_done global_id 74700 payload 824 2024-09-16T04:46:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.183+0000 7f723f7fe640 1 -- 172.21.15.46:0/2536149654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f72400a8120 con 0x7f72401365d0 2024-09-16T04:46:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.183+0000 7f72448e6640 1 --2- 172.21.15.46:0/2536149654 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f722c0383b0 0x7f722c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.183+0000 7f72448e6640 10 monclient: get_auth_request con 0x7f722c0383b0 auth_method 0 2024-09-16T04:46:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.183+0000 7f723f7fe640 1 --2- 172.21.15.46:0/2536149654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72401365d0 0x7f72401cc140 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f72340097a0 tx=0x7f72340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.183+0000 7f723cff9640 1 -- 172.21.15.46:0/2536149654 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7234043070 con 0x7f72401365d0 2024-09-16T04:46:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.183+0000 7f723cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.183+0000 7f723cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.183+0000 7f723cff9640 10 monclient: dump: 2024-09-16T04:46:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.206 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.183+0000 7f72448e6640 1 --2- 172.21.15.46:0/2536149654 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f722c0383b0 0x7f722c03a870 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f72400a0340 tx=0x7f723800b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.184+0000 7f723cff9640 1 -- 172.21.15.46:0/2536149654 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f723403c640 con 0x7f72401365d0 2024-09-16T04:46:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.184+0000 7f723cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.184+0000 7f723cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:11.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.184+0000 7f723cff9640 1 -- 172.21.15.46:0/2536149654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72401365d0 msgr2=0x7f72401cc140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.184+0000 7f723cff9640 1 --2- 172.21.15.46:0/2536149654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72401365d0 0x7f72401cc140 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f72340097a0 tx=0x7f72340046c0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.184+0000 7f723cff9640 1 -- 172.21.15.46:0/2536149654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72401365d0 msgr2=0x7f72401cc140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:11.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.184+0000 7f723cff9640 1 --2- 172.21.15.46:0/2536149654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72401365d0 0x7f72401cc140 secure :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f72340097a0 tx=0x7f72340046c0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f7246349640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:46:11.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f7246349640 1 --2- 172.21.15.46:0/2536149654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72401cc680 0x7f724009aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f7246349640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f7246349640 1 -- 172.21.15.46:0/2536149654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f72401cca90 con 0x7f72401cc680 2024-09-16T04:46:11.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723f7fe640 1 --2- 172.21.15.46:0/2536149654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72401cc680 0x7f724009aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723f7fe640 10 monclient: get_auth_request con 0x7f72401cc680 auth_method 0 2024-09-16T04:46:11.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:11.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:11.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:11.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723f7fe640 10 monclient(hunting): handle_auth_done global_id 74703 payload 824 2024-09-16T04:46:11.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723f7fe640 1 -- 172.21.15.46:0/2536149654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f72401222c0 con 0x7f72401cc680 2024-09-16T04:46:11.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723f7fe640 1 --2- 172.21.15.46:0/2536149654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72401cc680 0x7f724009aa70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f72400a6ba0 tx=0x7f7234009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:11.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723cff9640 1 -- 172.21.15.46:0/2536149654 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7234037040 con 0x7f72401cc680 2024-09-16T04:46:11.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:11.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723cff9640 10 monclient: dump: 2024-09-16T04:46:11.210 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.210 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.210 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.210 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.210 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.210 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.210 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.210 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.210 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.211 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723cff9640 1 -- 172.21.15.46:0/2536149654 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7f723403b3a0 con 0x7f72401cc680 2024-09-16T04:46:11.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:46:11.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:11.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723cff9640 1 -- 172.21.15.46:0/2536149654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72401cc680 msgr2=0x7f724009aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723cff9640 1 --2- 172.21.15.46:0/2536149654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72401cc680 0x7f724009aa70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f72400a6ba0 tx=0x7f7234009190 comp rx=0 tx=0).stop 2024-09-16T04:46:11.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723cff9640 1 -- 172.21.15.46:0/2536149654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72401cc680 msgr2=0x7f724009aa70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:11.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.190+0000 7f723cff9640 1 --2- 172.21.15.46:0/2536149654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72401cc680 0x7f724009aa70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.211 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:46:11.211 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:46:11.212 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:46:11.212 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:46:11.212 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 88, 2024-09-16T04:46:11.212 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 111186, 2024-09-16T04:46:11.212 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:46:11.212 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:46:11.212 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:46:11.212 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:46:11.212 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:11.212 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 6, 2024-09-16T04:46:11.212 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:11.212 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:46:11.212 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:46:11.213 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:11.213 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:11.213 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:11.213 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:11.213 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:11.213 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:11.213 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:11.213 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:11.213 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:11.213 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:11.213 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:11.213 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:11.213 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:46:11.213 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:46:11.214 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:11.214 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:11.214 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:11.214 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:11.214 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:11.214 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:11.214 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:11.214 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:11.214 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:11.214 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:11.214 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:11.214 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:11.214 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.214 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:46:11.215 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:46:11.215 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:46:11.215 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:46:11.215 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:46:11.215 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:46:11.215 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:11.215 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:11.215 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:46:11.215 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:46:11.215 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:46:11.215 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:46:11.215 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:46:11.215 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:46:11.215 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:46:11.216 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:11.216 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:46:11.216 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:11.216 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:11.216 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:11.216 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:11.216 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:11.216 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:11.216 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:11.216 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:11.216 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:11.216 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:11.216 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:11.216 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:11.217 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:46:11.217 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.217 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:46:11.217 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.217 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:46:11.217 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:46:11.217 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:11.217 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:11.217 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.217 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:11.217 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:46:11.217 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:11.217 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.217 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.218 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:11.218 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:46:11.218 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:11.218 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:11.218 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:11.218 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.218 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:46:11.218 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:46:11.218 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:11.218 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:11.218 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:11.218 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.218 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.218 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:46:11.219 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:46:11.219 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:11.219 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:11.219 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.219 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:11.219 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:46:11.219 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:11.219 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.219 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.219 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:11.219 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:46:11.219 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:11.219 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:11.219 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:11.220 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.220 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:46:11.220 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:46:11.220 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:11.220 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:11.220 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:11.220 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.220 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.220 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:46:11.220 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:46:11.220 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:11.220 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:11.220 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.220 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:11.221 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:46:11.221 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:11.221 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.221 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.221 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:11.221 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:46:11.221 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:11.221 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:11.221 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:11.221 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.221 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:46:11.221 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:46:11.221 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:11.221 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:11.222 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:11.222 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:11.222 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:11.222 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.222 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:46:11.222 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:46:11.222 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.222 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:11.222 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:11.222 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:11.222 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:11.222 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:11.222 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:46:11.223 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.223 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:11.223 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:11.223 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:11.223 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:11.223 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:11.223 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:46:11.223 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.223 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:11.223 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:11.223 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:46:11.223 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:11.223 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:11.223 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.223 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:46:11.224 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:46:11.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.191+0000 7f7246349640 1 -- 172.21.15.46:0/2536149654 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f722c0383b0 msgr2=0x7f722c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.191+0000 7f7246349640 1 --2- 172.21.15.46:0/2536149654 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f722c0383b0 0x7f722c03a870 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f72400a0340 tx=0x7f723800b040 comp rx=0 tx=0).stop 2024-09-16T04:46:11.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.191+0000 7f7246349640 10 monclient: shutdown 2024-09-16T04:46:11.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.191+0000 7f7246349640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:11.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.191+0000 7f7246349640 1 -- 172.21.15.46:0/2536149654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f724006e160 msgr2=0x7f724009f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.191+0000 7f7246349640 1 --2- 172.21.15.46:0/2536149654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f724006e160 0x7f724009f080 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f723000b3e0 tx=0x7f723000b8a0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.191+0000 7f72448e6640 1 -- 172.21.15.46:0/2536149654 reap_dead start 2024-09-16T04:46:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.191+0000 7f72448e6640 1 -- 172.21.15.46:0/2536149654 reap_dead start 2024-09-16T04:46:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.191+0000 7f7246349640 1 -- 172.21.15.46:0/2536149654 shutdown_connections 2024-09-16T04:46:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.191+0000 7f7246349640 1 -- 172.21.15.46:0/2536149654 >> 172.21.15.46:0/2536149654 conn(0x7f72400a5c10 msgr2=0x7f72400a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.191+0000 7f7246349640 1 -- 172.21.15.46:0/2536149654 shutdown_connections 2024-09-16T04:46:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.191+0000 7f7246349640 1 -- 172.21.15.46:0/2536149654 wait complete. 2024-09-16T04:46:11.226 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:46:11.226 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:46:11.226 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:46:11.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.326+0000 7f11b8786640 10 monclient: get_monmap_and_config 2024-09-16T04:46:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.326+0000 7f11b8786640 10 monclient: build_initial_monmap 2024-09-16T04:46:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.326+0000 7f11b8786640 10 monclient: monmap: 2024-09-16T04:46:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:11.328433+0000 2024-09-16T04:46:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:11.328433+0000 2024-09-16T04:46:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:11.329 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:11.330 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:11.330 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.327+0000 7f11b8786640 1 Processor -- start 2024-09-16T04:46:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.327+0000 7f11b8786640 1 -- start start 2024-09-16T04:46:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.327+0000 7f11b8786640 10 monclient: init 2024-09-16T04:46:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b8786640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b8786640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-16T04:46:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b8786640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b0136e50 0x7f11b0137230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b8786640 10 monclient(hunting): picked mon.noname-b con 0x7f11b0136e50 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b8786640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f11b00a3500 addr v1:172.21.15.46:6789/0 2024-09-16T04:46:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b8786640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f11b00a4350 0x7f11b00a47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b8786640 10 monclient(hunting): picked mon.noname-a con 0x7f11b00a4350 addr v2:172.21.15.46:3300/0 2024-09-16T04:46:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b8786640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11b00a4e90 con 0x7f11b00a3500 2024-09-16T04:46:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b8786640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b8786640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f11b00a5410 con 0x7f11b00a4350 2024-09-16T04:46:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b8786640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b8786640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f11b00a0dc0 con 0x7f11b0136e50 2024-09-16T04:46:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b8786640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b8786640 10 monclient(hunting): authenticate will time out at 1169.744288s 2024-09-16T04:46:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6522640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b0136e50 0x7f11b0137230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6d23640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f11b00a4350 0x7f11b00a47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6522640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b0136e50 0x7f11b0137230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:44108/0 (socket says 172.21.15.46:44108) 2024-09-16T04:46:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6522640 1 -- 172.21.15.46:0/2984759035 learned_addr learned my addr 172.21.15.46:0/2984759035 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6d23640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f11b00a4350 0x7f11b00a47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33470/0 (socket says 172.21.15.46:33470) 2024-09-16T04:46:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6d23640 10 monclient(hunting): get_auth_request con 0x7f11b00a4350 auth_method 0 2024-09-16T04:46:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6d23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6d23640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6d23640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6522640 10 monclient(hunting): get_auth_request con 0x7f11b0136e50 auth_method 0 2024-09-16T04:46:11.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6522640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6522640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6522640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6d23640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:11.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6d23640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:11.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6d23640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:11.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6522640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:11.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6522640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:11.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6522640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:11.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6522640 10 monclient(hunting): handle_auth_done global_id 94145 payload 274 2024-09-16T04:46:11.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6522640 1 -- 172.21.15.46:0/2984759035 >> v2:172.21.15.46:3300/0 conn(0x7f11b00a4350 msgr2=0x7f11b00a47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:11.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6522640 1 --2- 172.21.15.46:0/2984759035 >> v2:172.21.15.46:3300/0 conn(0x7f11b00a4350 0x7f11b00a47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6522640 1 -- 172.21.15.46:0/2984759035 >> v1:172.21.15.46:6789/0 conn(0x7f11b00a3500 legacy=0x7f11b00a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6522640 10 monclient: _finish_hunting 0 2024-09-16T04:46:11.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6522640 1 monclient: found mon.noname-b 2024-09-16T04:46:11.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6522640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:11.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6522640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:46:11.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6522640 1 -- 172.21.15.46:0/2984759035 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b00a10a0 con 0x7f11b0136e50 2024-09-16T04:46:11.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.328+0000 7f11b6d23640 1 --2- 172.21.15.46:0/2984759035 >> v2:172.21.15.46:3300/0 conn(0x7f11b00a4350 0x7f11b00a47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:11.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.329+0000 7f11b6522640 1 --2- 172.21.15.46:0/2984759035 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b0136e50 0x7f11b0137230 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f11ac009870 tx=0x7f11ac02a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac625e172a97b5a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:11.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.329+0000 7f11b5520640 1 -- 172.21.15.46:0/2984759035 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ac037070 con 0x7f11b0136e50 2024-09-16T04:46:11.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.329+0000 7f11b5520640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.329+0000 7f11b5520640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:46:11.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.329+0000 7f11b5520640 10 monclient: dump: 2024-09-16T04:46:11.341 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.341 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.342 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.342 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.342 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.342 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.342 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.342 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.342 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.342 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.329+0000 7f11b5520640 10 monclient: _finish_auth 0 2024-09-16T04:46:11.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.329+0000 7f11b5520640 10 monclient: _check_auth_tickets 2024-09-16T04:46:11.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.329+0000 7f11b5520640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:11.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.329+0000 7f11b5520640 1 -- 172.21.15.46:0/2984759035 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11ac032440 con 0x7f11b0136e50 2024-09-16T04:46:11.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.329+0000 7f11b8786640 5 monclient: authenticate success, global_id 94145 2024-09-16T04:46:11.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.329+0000 7f11b8786640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.329+0000 7f11b5520640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.329+0000 7f11b8786640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.329+0000 7f11b5520640 1 -- 172.21.15.46:0/2984759035 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ac032810 con 0x7f11b0136e50 2024-09-16T04:46:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.329+0000 7f11b8786640 4 set_mon_vals no callback set 2024-09-16T04:46:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.329+0000 7f11b5520640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.329+0000 7f11b5520640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.329+0000 7f11b5520640 10 monclient: dump: 2024-09-16T04:46:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.344 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.330+0000 7f11b8786640 10 monclient: shutdown 2024-09-16T04:46:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.330+0000 7f11b8786640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.330+0000 7f11b8786640 1 -- 172.21.15.46:0/2984759035 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b0136e50 msgr2=0x7f11b0137230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.330+0000 7f11b8786640 1 --2- 172.21.15.46:0/2984759035 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b0136e50 0x7f11b0137230 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f11ac009870 tx=0x7f11ac02a130 comp rx=0 tx=0).stop 2024-09-16T04:46:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.330+0000 7f11b8786640 1 -- 172.21.15.46:0/2984759035 shutdown_connections 2024-09-16T04:46:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.330+0000 7f11b8786640 1 --2- 172.21.15.46:0/2984759035 >> v2:172.21.15.46:3300/0 conn(0x7f11b00a4350 0x7f11b00a47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.330+0000 7f11b8786640 1 --2- 172.21.15.46:0/2984759035 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b0136e50 0x7f11b0137230 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.330+0000 7f11b8786640 1 -- 172.21.15.46:0/2984759035 >> 172.21.15.46:0/2984759035 conn(0x7f11b00a5f20 msgr2=0x7f11b00a6330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.330+0000 7f11b8786640 1 -- 172.21.15.46:0/2984759035 shutdown_connections 2024-09-16T04:46:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.330+0000 7f11b8786640 1 -- 172.21.15.46:0/2984759035 wait complete. 2024-09-16T04:46:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.330+0000 7f11b8786640 10 monclient: build_initial_monmap 2024-09-16T04:46:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.330+0000 7f11b8786640 10 monclient: monmap: 2024-09-16T04:46:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:11.346 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b8786640 1 Processor -- start 2024-09-16T04:46:11.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b8786640 1 -- start start 2024-09-16T04:46:11.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b8786640 10 monclient: init 2024-09-16T04:46:11.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b8786640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:11.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b8786640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:46:11.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b8786640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11b00a3500 0x7f11b01cb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b8786640 10 monclient(hunting): picked mon.noname-a con 0x7f11b00a3500 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:11.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b8786640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b00a4350 0x7f11b0135bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b8786640 10 monclient(hunting): picked mon.noname-c con 0x7f11b00a4350 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:11.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b8786640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11b0136e50 0x7f11b0136100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b8786640 10 monclient(hunting): picked mon.noname-b con 0x7f11b0136e50 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:11.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b8786640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b8786640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f11b00a0b30 con 0x7f11b00a3500 2024-09-16T04:46:11.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b8786640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b8786640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f11b00a09b0 con 0x7f11b0136e50 2024-09-16T04:46:11.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b8786640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b8786640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f11b00a1b60 con 0x7f11b00a4350 2024-09-16T04:46:11.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b8786640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:11.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b8786640 10 monclient(hunting): authenticate will time out at 1169.747617s 2024-09-16T04:46:11.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b6522640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11b00a3500 0x7f11b01cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b5d21640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b00a4350 0x7f11b0135bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b6522640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11b00a3500 0x7f11b01cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33486/0 (socket says 172.21.15.46:33486) 2024-09-16T04:46:11.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b6522640 1 -- 172.21.15.46:0/1631353640 learned_addr learned my addr 172.21.15.46:0/1631353640 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:11.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b6522640 10 monclient(hunting): get_auth_request con 0x7f11b00a3500 auth_method 0 2024-09-16T04:46:11.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b6522640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b6522640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b6522640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b5d21640 10 monclient(hunting): get_auth_request con 0x7f11b00a4350 auth_method 0 2024-09-16T04:46:11.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b5d21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b5d21640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b5d21640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b6522640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:11.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b6522640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:11.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b6522640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:11.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b5d21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:11.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b5d21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:11.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b5d21640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:11.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b6d23640 1 --2- 172.21.15.46:0/1631353640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11b0136e50 0x7f11b0136100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b6d23640 10 monclient(hunting): get_auth_request con 0x7f11b0136e50 auth_method 0 2024-09-16T04:46:11.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b6d23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b6d23640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b6d23640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.331+0000 7f11b5d21640 10 monclient(hunting): handle_auth_done global_id 94148 payload 824 2024-09-16T04:46:11.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11b5d21640 1 -- 172.21.15.46:0/1631353640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11b0136e50 msgr2=0x7f11b0136100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11b5d21640 1 --2- 172.21.15.46:0/1631353640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11b0136e50 0x7f11b0136100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11b5d21640 1 -- 172.21.15.46:0/1631353640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11b00a3500 msgr2=0x7f11b01cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11b5d21640 1 --2- 172.21.15.46:0/1631353640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11b00a3500 0x7f11b01cb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11b5d21640 10 monclient: _finish_hunting 0 2024-09-16T04:46:11.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11b5d21640 1 monclient: found mon.noname-c 2024-09-16T04:46:11.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11b5d21640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:11.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11b5d21640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:46:11.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11b5d21640 1 -- 172.21.15.46:0/1631353640 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b0127000 con 0x7f11b00a4350 2024-09-16T04:46:11.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11b6522640 1 --2- 172.21.15.46:0/1631353640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11b00a3500 0x7f11b01cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:11.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11b5d21640 1 --2- 172.21.15.46:0/1631353640 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b00a4350 0x7f11b0135bc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f11a400c310 tx=0x7f11a400c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:11.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11a2ffd640 1 -- 172.21.15.46:0/1631353640 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a40052d0 con 0x7f11b00a4350 2024-09-16T04:46:11.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11a2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:11.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11a2ffd640 10 monclient: dump: 2024-09-16T04:46:11.350 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.350 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.350 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.350 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.350 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.350 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.350 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.350 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.350 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.351 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11a2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:46:11.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11a2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:46:11.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:11.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11a2ffd640 1 -- 172.21.15.46:0/1631353640 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11a4005490 con 0x7f11b00a4350 2024-09-16T04:46:11.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11b8786640 5 monclient: authenticate success, global_id 94148 2024-09-16T04:46:11.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11b8786640 10 monclient: _renew_subs 2024-09-16T04:46:11.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11b8786640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:11.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11b8786640 1 -- 172.21.15.46:0/1631353640 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11b01cbec0 con 0x7f11b00a4350 2024-09-16T04:46:11.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11a2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:11.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11a2ffd640 1 -- 172.21.15.46:0/1631353640 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a4007260 con 0x7f11b00a4350 2024-09-16T04:46:11.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:11.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11a2ffd640 10 monclient: dump: 2024-09-16T04:46:11.352 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.352 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.352 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.352 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.352 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.352 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.352 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.352 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.352 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.352 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11b8786640 10 monclient: _renew_subs 2024-09-16T04:46:11.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11b8786640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:11.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11b8786640 1 -- 172.21.15.46:0/1631353640 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f11b00a2ab0 con 0x7f11b00a4350 2024-09-16T04:46:11.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.332+0000 7f11a3fff640 4 set_mon_vals no callback set 2024-09-16T04:46:11.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.333+0000 7f11b8786640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:11.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.333+0000 7f11b8786640 1 --2- 172.21.15.46:0/1631353640 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b0123680 0x7f11b0123a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.333+0000 7f11b8786640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.333+0000 7f11b8786640 1 -- 172.21.15.46:0/1631353640 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f11b0125770 con 0x7f11b0123680 2024-09-16T04:46:11.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.333+0000 7f11a2ffd640 1 -- 172.21.15.46:0/1631353640 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f11a4007400 con 0x7f11b00a4350 2024-09-16T04:46:11.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.333+0000 7f11b6522640 1 --2- 172.21.15.46:0/1631353640 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b0123680 0x7f11b0123a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.333+0000 7f11b6522640 10 monclient: get_auth_request con 0x7f11b0123680 auth_method 0 2024-09-16T04:46:11.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.333+0000 7f11b6522640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.333+0000 7f11b6522640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.333+0000 7f11b6522640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.333+0000 7f11b6522640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:11.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.333+0000 7f11b6522640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:11.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.333+0000 7f11b6522640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:11.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.333+0000 7f11a2ffd640 1 --2- 172.21.15.46:0/1631353640 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f11940383e0 0x7f119403a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.333+0000 7f11b6522640 10 monclient(hunting): handle_auth_done global_id 94151 payload 824 2024-09-16T04:46:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.333+0000 7f11a2ffd640 1 -- 172.21.15.46:0/1631353640 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(282..282 src has 1..282) ==== 4738+0+0 (secure 0 0 0) 0x7f11a404ab80 con 0x7f11b00a4350 2024-09-16T04:46:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.333+0000 7f11b6522640 1 -- 172.21.15.46:0/1631353640 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f11b00a47c0 con 0x7f11b0123680 2024-09-16T04:46:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.333+0000 7f11b6d23640 1 --2- 172.21.15.46:0/1631353640 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f11940383e0 0x7f119403a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.333+0000 7f11b6522640 1 --2- 172.21.15.46:0/1631353640 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b0123680 0x7f11b0123a60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f11ac035040 tx=0x7f11ac0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.333+0000 7f11b6d23640 10 monclient: get_auth_request con 0x7f11940383e0 auth_method 0 2024-09-16T04:46:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.334+0000 7f11a2ffd640 1 -- 172.21.15.46:0/1631353640 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ac043070 con 0x7f11b0123680 2024-09-16T04:46:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.334+0000 7f11a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.334+0000 7f11a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.334+0000 7f11a2ffd640 10 monclient: dump: 2024-09-16T04:46:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.355 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.334+0000 7f11a2ffd640 1 -- 172.21.15.46:0/1631353640 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f11ac032440 con 0x7f11b0123680 2024-09-16T04:46:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.334+0000 7f11a2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.334+0000 7f11a2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.334+0000 7f11a2ffd640 1 -- 172.21.15.46:0/1631353640 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b0123680 msgr2=0x7f11b0123a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.334+0000 7f11a2ffd640 1 --2- 172.21.15.46:0/1631353640 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b0123680 0x7f11b0123a60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f11ac035040 tx=0x7f11ac0046c0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.334+0000 7f11a2ffd640 1 -- 172.21.15.46:0/1631353640 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b0123680 msgr2=0x7f11b0123a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.334+0000 7f11a2ffd640 1 --2- 172.21.15.46:0/1631353640 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b0123680 0x7f11b0123a60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.334+0000 7f11b6d23640 1 --2- 172.21.15.46:0/1631353640 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f11940383e0 0x7f119403a8a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f11b0126bc0 tx=0x7f11a800b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.340+0000 7f11b8786640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:46:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.340+0000 7f11b8786640 1 --2- 172.21.15.46:0/1631353640 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b0123fa0 0x7f11b011f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.340+0000 7f11b8786640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.340+0000 7f11b8786640 1 -- 172.21.15.46:0/1631353640 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f11b0067e40 con 0x7f11b0123fa0 2024-09-16T04:46:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.340+0000 7f11b6522640 1 --2- 172.21.15.46:0/1631353640 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b0123fa0 0x7f11b011f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.340+0000 7f11b6522640 10 monclient: get_auth_request con 0x7f11b0123fa0 auth_method 0 2024-09-16T04:46:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.340+0000 7f11b6522640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.340+0000 7f11b6522640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.340+0000 7f11b6522640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.340+0000 7f11b6522640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.340+0000 7f11b6522640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.340+0000 7f11b6522640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.340+0000 7f11b6522640 10 monclient(hunting): handle_auth_done global_id 94154 payload 824 2024-09-16T04:46:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.340+0000 7f11b6522640 1 -- 172.21.15.46:0/1631353640 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f11b0126d20 con 0x7f11b0123fa0 2024-09-16T04:46:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.340+0000 7f11b6522640 1 --2- 172.21.15.46:0/1631353640 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b0123fa0 0x7f11b011f890 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f11b01258f0 tx=0x7f11ac009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11a2ffd640 1 -- 172.21.15.46:0/1631353640 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ac037040 con 0x7f11b0123fa0 2024-09-16T04:46:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11a2ffd640 10 monclient: dump: 2024-09-16T04:46:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.359 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11a2ffd640 1 -- 172.21.15.46:0/1631353640 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f11ac003c80 con 0x7f11b0123fa0 2024-09-16T04:46:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11a2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:46:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11a2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11a2ffd640 1 -- 172.21.15.46:0/1631353640 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b0123fa0 msgr2=0x7f11b011f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11a2ffd640 1 --2- 172.21.15.46:0/1631353640 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b0123fa0 0x7f11b011f890 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f11b01258f0 tx=0x7f11ac009240 comp rx=0 tx=0).stop 2024-09-16T04:46:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11a2ffd640 1 -- 172.21.15.46:0/1631353640 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b0123fa0 msgr2=0x7f11b011f890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11a2ffd640 1 --2- 172.21.15.46:0/1631353640 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b0123fa0 0x7f11b011f890 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f11b01258f0 tx=0x7f11ac009240 comp rx=0 tx=0).stop 2024-09-16T04:46:11.360 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:46:11.360 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:46:11.360 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:46:11.360 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:46:11.360 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 88, 2024-09-16T04:46:11.360 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 12596, 2024-09-16T04:46:11.360 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:46:11.361 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:46:11.361 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:46:11.361 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:46:11.361 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:11.361 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 6, 2024-09-16T04:46:11.361 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:11.361 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:46:11.361 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:46:11.361 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:11.361 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:11.361 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:11.361 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:11.361 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:11.361 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:11.362 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:11.362 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:11.362 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:11.362 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:11.362 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:11.362 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:11.362 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:46:11.362 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:46:11.362 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:11.362 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:11.362 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:11.362 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:11.362 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:11.363 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:11.363 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:11.363 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:11.363 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:11.363 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:11.363 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:11.363 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:11.363 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.363 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:46:11.363 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:46:11.363 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:46:11.363 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:46:11.363 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:46:11.363 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:46:11.363 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:11.364 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:11.364 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:46:11.364 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:46:11.364 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:46:11.364 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:46:11.364 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:46:11.364 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:46:11.364 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:46:11.364 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:11.364 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:46:11.364 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:11.364 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:11.364 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:11.364 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:11.364 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:11.365 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:11.365 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:11.365 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:11.365 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:11.365 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:11.365 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:11.365 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:11.365 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:46:11.365 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.366 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:46:11.366 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.366 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:46:11.366 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:46:11.366 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:11.366 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:11.366 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.366 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:11.366 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:46:11.366 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:11.366 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.366 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.366 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:11.366 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:46:11.366 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:11.367 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:11.367 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:11.367 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.367 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:46:11.367 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:46:11.367 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:11.367 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:11.367 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:11.367 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.367 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.367 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:46:11.367 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:46:11.367 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:11.367 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:11.367 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.368 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:11.368 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:46:11.368 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:11.368 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.368 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.368 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:11.368 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:46:11.368 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:11.368 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:11.368 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:11.368 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.368 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:46:11.368 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:46:11.369 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:11.369 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:11.369 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:11.369 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.369 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.369 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:46:11.369 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:46:11.369 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:11.369 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:11.369 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.369 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:11.369 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:46:11.369 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:11.369 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.370 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.370 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:11.370 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:46:11.370 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:11.370 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:11.370 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:11.370 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.370 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:46:11.370 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:46:11.370 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:11.370 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:11.370 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:11.370 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:11.370 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:11.371 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.371 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:46:11.371 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:46:11.371 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.371 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:11.371 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:11.371 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:11.371 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:11.371 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:11.371 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:46:11.371 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.371 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:11.371 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:11.371 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5 2024-09-16T04:46:11.372 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:11.372 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:11.372 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.372 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:46:11.372 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:46:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11b8786640 1 -- 172.21.15.46:0/1631353640 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f11940383e0 msgr2=0x7f119403a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11b8786640 1 --2- 172.21.15.46:0/1631353640 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f11940383e0 0x7f119403a8a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f11b0126bc0 tx=0x7f11a800b040 comp rx=0 tx=0).stop 2024-09-16T04:46:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11b8786640 10 monclient: shutdown 2024-09-16T04:46:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11b8786640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11b8786640 1 -- 172.21.15.46:0/1631353640 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b00a4350 msgr2=0x7f11b0135bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11b8786640 1 --2- 172.21.15.46:0/1631353640 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b00a4350 0x7f11b0135bc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f11a400c310 tx=0x7f11a400c7d0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11b6d23640 1 -- 172.21.15.46:0/1631353640 reap_dead start 2024-09-16T04:46:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11b8786640 1 -- 172.21.15.46:0/1631353640 shutdown_connections 2024-09-16T04:46:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11b8786640 1 --2- 172.21.15.46:0/1631353640 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f11b00a4350 0x7f11b0135bc0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11b8786640 1 -- 172.21.15.46:0/1631353640 >> 172.21.15.46:0/1631353640 conn(0x7f11b00a5f20 msgr2=0x7f11b00a7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11b8786640 1 -- 172.21.15.46:0/1631353640 shutdown_connections 2024-09-16T04:46:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.341+0000 7f11b8786640 1 -- 172.21.15.46:0/1631353640 wait complete. 2024-09-16T04:46:11.374 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:46:11.374 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:46:11.374 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:46:11.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.475+0000 7f2d6254b640 10 monclient: get_monmap_and_config 2024-09-16T04:46:11.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.475+0000 7f2d6254b640 10 monclient: build_initial_monmap 2024-09-16T04:46:11.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.475+0000 7f2d6254b640 10 monclient: monmap: 2024-09-16T04:46:11.477 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:11.477 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.477 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:11.476566+0000 2024-09-16T04:46:11.477 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:11.476566+0000 2024-09-16T04:46:11.477 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:11.477 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.477 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:11.477 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:11.478 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:11.478 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:11.478 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:11.478 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.475+0000 7f2d6254b640 1 Processor -- start 2024-09-16T04:46:11.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.475+0000 7f2d6254b640 1 -- start start 2024-09-16T04:46:11.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.475+0000 7f2d6254b640 10 monclient: init 2024-09-16T04:46:11.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d6254b640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:11.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d6254b640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-16T04:46:11.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d6254b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2d5c0a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:46:11.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d6254b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2d5c0a3ff0 0x7f2d5c0a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d6254b640 10 monclient(hunting): picked mon.noname-b con 0x7f2d5c0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d6254b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2d5c13be90 0x7f2d5c13e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d6254b640 10 monclient(hunting): picked mon.noname-a con 0x7f2d5c13be90 addr v2:172.21.15.46:3300/0 2024-09-16T04:46:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d6254b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d5c09ab70 con 0x7f2d5c0a3c10 2024-09-16T04:46:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d6254b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d6254b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2d5c0a4d60 con 0x7f2d5c13be90 2024-09-16T04:46:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d6254b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d6254b640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f2d5c09b030 con 0x7f2d5c0a3ff0 2024-09-16T04:46:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d6254b640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d6254b640 10 monclient(hunting): authenticate will time out at 1169.892376s 2024-09-16T04:46:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d5b7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2d5c0a3ff0 0x7f2d5c0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d60ae8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2d5c13be90 0x7f2d5c13e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d5bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2d5c0a3c10 0x7f2d5c09bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37888/0 (socket says 172.21.15.46:37888) 2024-09-16T04:46:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d5bfff640 1 -- 172.21.15.46:0/1164848371 learned_addr learned my addr 172.21.15.46:0/1164848371 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:46:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d5b7fe640 10 monclient(hunting): get_auth_request con 0x7f2d5c0a3ff0 auth_method 0 2024-09-16T04:46:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d60ae8640 10 monclient(hunting): get_auth_request con 0x7f2d5c13be90 auth_method 0 2024-09-16T04:46:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d60ae8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d60ae8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d60ae8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d60ae8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d60ae8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d60ae8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d60ae8640 10 monclient(hunting): handle_auth_done global_id 74712 payload 274 2024-09-16T04:46:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d60ae8640 1 -- 172.21.15.46:0/1164848371 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2d5c0a3ff0 msgr2=0x7f2d5c0a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d60ae8640 1 --2- 172.21.15.46:0/1164848371 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2d5c0a3ff0 0x7f2d5c0a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d60ae8640 1 -- 172.21.15.46:0/1164848371 >> v1:172.21.15.46:6789/0 conn(0x7f2d5c0a3c10 legacy=0x7f2d5c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d60ae8640 10 monclient: _finish_hunting 0 2024-09-16T04:46:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d60ae8640 1 monclient: found mon.noname-a 2024-09-16T04:46:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d60ae8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d60ae8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:46:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d60ae8640 1 -- 172.21.15.46:0/1164848371 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d5c09b310 con 0x7f2d5c13be90 2024-09-16T04:46:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.476+0000 7f2d5b7fe640 1 --2- 172.21.15.46:0/1164848371 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2d5c0a3ff0 0x7f2d5c0a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.477+0000 7f2d60ae8640 1 --2- 172.21.15.46:0/1164848371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d5c13be90 0x7f2d5c13e280 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2d54009d90 tx=0x7f2d5402a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5caa0dafce2a6a86 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.477+0000 7f2d5affd640 1 -- 172.21.15.46:0/1164848371 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d54007740 con 0x7f2d5c13be90 2024-09-16T04:46:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.477+0000 7f2d5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.477+0000 7f2d5affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:46:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.477+0000 7f2d5affd640 10 monclient: dump: 2024-09-16T04:46:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.489 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.477+0000 7f2d5affd640 10 monclient: _finish_auth 0 2024-09-16T04:46:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.477+0000 7f2d5affd640 10 monclient: _check_auth_tickets 2024-09-16T04:46:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.477+0000 7f2d5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.477+0000 7f2d5affd640 1 -- 172.21.15.46:0/1164848371 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d540078e0 con 0x7f2d5c13be90 2024-09-16T04:46:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.477+0000 7f2d6254b640 5 monclient: authenticate success, global_id 74712 2024-09-16T04:46:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.477+0000 7f2d6254b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.477+0000 7f2d5affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.477+0000 7f2d6254b640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.477+0000 7f2d5affd640 1 -- 172.21.15.46:0/1164848371 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d54007cb0 con 0x7f2d5c13be90 2024-09-16T04:46:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.477+0000 7f2d5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.477+0000 7f2d6254b640 4 set_mon_vals no callback set 2024-09-16T04:46:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.477+0000 7f2d5affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.477+0000 7f2d5affd640 10 monclient: dump: 2024-09-16T04:46:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.491 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.478+0000 7f2d6254b640 10 monclient: shutdown 2024-09-16T04:46:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.478+0000 7f2d6254b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.478+0000 7f2d6254b640 1 -- 172.21.15.46:0/1164848371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d5c13be90 msgr2=0x7f2d5c13e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.478+0000 7f2d6254b640 1 --2- 172.21.15.46:0/1164848371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d5c13be90 0x7f2d5c13e280 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2d54009d90 tx=0x7f2d5402a530 comp rx=0 tx=0).stop 2024-09-16T04:46:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.478+0000 7f2d6254b640 1 -- 172.21.15.46:0/1164848371 shutdown_connections 2024-09-16T04:46:11.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.478+0000 7f2d6254b640 1 --2- 172.21.15.46:0/1164848371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d5c13be90 0x7f2d5c13e280 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.478+0000 7f2d6254b640 1 --2- 172.21.15.46:0/1164848371 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2d5c0a3ff0 0x7f2d5c0a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.478+0000 7f2d6254b640 1 -- 172.21.15.46:0/1164848371 >> 172.21.15.46:0/1164848371 conn(0x7f2d5c0a5d10 msgr2=0x7f2d5c0a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:11.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.478+0000 7f2d6254b640 1 -- 172.21.15.46:0/1164848371 shutdown_connections 2024-09-16T04:46:11.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.478+0000 7f2d6254b640 1 -- 172.21.15.46:0/1164848371 wait complete. 2024-09-16T04:46:11.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.478+0000 7f2d6254b640 10 monclient: build_initial_monmap 2024-09-16T04:46:11.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.478+0000 7f2d6254b640 10 monclient: monmap: 2024-09-16T04:46:11.492 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:11.492 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:11.492 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:11.492 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:11.492 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:11.492 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.492 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:11.493 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:11.493 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:11.493 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d6254b640 1 Processor -- start 2024-09-16T04:46:11.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d6254b640 1 -- start start 2024-09-16T04:46:11.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d6254b640 10 monclient: init 2024-09-16T04:46:11.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d6254b640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:11.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d6254b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:46:11.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d6254b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d5c0a3c10 0x7f2d5c1cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d6254b640 10 monclient(hunting): picked mon.noname-a con 0x7f2d5c0a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:11.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d6254b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2d5c0a3ff0 0x7f2d5c137140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d6254b640 10 monclient(hunting): picked mon.noname-c con 0x7f2d5c0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:11.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d6254b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5c13be90 0x7f2d5c139810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d6254b640 10 monclient(hunting): picked mon.noname-b con 0x7f2d5c13be90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:11.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d6254b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d6254b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5c06e550 con 0x7f2d5c0a3c10 2024-09-16T04:46:11.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d6254b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d6254b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5c06e3d0 con 0x7f2d5c13be90 2024-09-16T04:46:11.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d6254b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d6254b640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f2d5c13e7c0 con 0x7f2d5c0a3ff0 2024-09-16T04:46:11.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d6254b640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:11.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d6254b640 10 monclient(hunting): authenticate will time out at 1169.895812s 2024-09-16T04:46:11.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d5bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d5c0a3c10 0x7f2d5c1cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d5b7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2d5c0a3ff0 0x7f2d5c137140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d5bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d5c0a3c10 0x7f2d5c1cbbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33498/0 (socket says 172.21.15.46:33498) 2024-09-16T04:46:11.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d5bfff640 1 -- 172.21.15.46:0/2087286529 learned_addr learned my addr 172.21.15.46:0/2087286529 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d5bfff640 10 monclient(hunting): get_auth_request con 0x7f2d5c0a3c10 auth_method 0 2024-09-16T04:46:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d5b7fe640 10 monclient(hunting): get_auth_request con 0x7f2d5c0a3ff0 auth_method 0 2024-09-16T04:46:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.479+0000 7f2d5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:11.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d60ae8640 1 --2- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5c13be90 0x7f2d5c139810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d5b7fe640 10 monclient(hunting): handle_auth_done global_id 94160 payload 824 2024-09-16T04:46:11.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d5b7fe640 1 -- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5c13be90 msgr2=0x7f2d5c139810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d5b7fe640 1 --2- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5c13be90 0x7f2d5c139810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d5b7fe640 1 -- 172.21.15.46:0/2087286529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d5c0a3c10 msgr2=0x7f2d5c1cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d5b7fe640 1 --2- 172.21.15.46:0/2087286529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d5c0a3c10 0x7f2d5c1cbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d5b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:46:11.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d5b7fe640 1 monclient: found mon.noname-c 2024-09-16T04:46:11.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:11.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d5b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:46:11.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d5b7fe640 1 -- 172.21.15.46:0/2087286529 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d5c121ad0 con 0x7f2d5c0a3ff0 2024-09-16T04:46:11.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d60ae8640 10 monclient: get_auth_request con 0x7f2d5c13be90 auth_method 0 2024-09-16T04:46:11.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d60ae8640 1 --2- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5c13be90 0x7f2d5c139810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:46:11.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d5bfff640 1 --2- 172.21.15.46:0/2087286529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d5c0a3c10 0x7f2d5c1cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:11.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d5b7fe640 1 --2- 172.21.15.46:0/2087286529 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2d5c0a3ff0 0x7f2d5c137140 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2d5000e900 tx=0x7f2d5000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:11.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d58ff9640 1 -- 172.21.15.46:0/2087286529 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d50017070 con 0x7f2d5c0a3ff0 2024-09-16T04:46:11.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d58ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d58ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:11.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d58ff9640 10 monclient: dump: 2024-09-16T04:46:11.497 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.497 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.497 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.497 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.497 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.497 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.497 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.497 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.498 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.498 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d58ff9640 10 monclient: _finish_auth 0 2024-09-16T04:46:11.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d58ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:46:11.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d58ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:11.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d58ff9640 1 -- 172.21.15.46:0/2087286529 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d5000ca00 con 0x7f2d5c0a3ff0 2024-09-16T04:46:11.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d6254b640 5 monclient: authenticate success, global_id 94160 2024-09-16T04:46:11.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d6254b640 10 monclient: _renew_subs 2024-09-16T04:46:11.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d6254b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:11.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d6254b640 1 -- 172.21.15.46:0/2087286529 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d5c1200c0 con 0x7f2d5c0a3ff0 2024-09-16T04:46:11.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d58ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:11.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d58ff9640 1 -- 172.21.15.46:0/2087286529 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5000cdf0 con 0x7f2d5c0a3ff0 2024-09-16T04:46:11.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d58ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d58ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:11.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d58ff9640 10 monclient: dump: 2024-09-16T04:46:11.499 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.499 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.499 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.499 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.499 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.499 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.499 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.499 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.499 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.499 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d6254b640 10 monclient: _renew_subs 2024-09-16T04:46:11.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d6254b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:11.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d5a7fc640 4 set_mon_vals no callback set 2024-09-16T04:46:11.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.480+0000 7f2d6254b640 1 -- 172.21.15.46:0/2087286529 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d5c13c8a0 con 0x7f2d5c0a3ff0 2024-09-16T04:46:11.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.481+0000 7f2d6254b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:11.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.481+0000 7f2d58ff9640 1 -- 172.21.15.46:0/2087286529 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f2d5001a070 con 0x7f2d5c0a3ff0 2024-09-16T04:46:11.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.481+0000 7f2d6254b640 1 --2- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5c137680 0x7f2d5c139d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.481+0000 7f2d6254b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.481+0000 7f2d6254b640 1 -- 172.21.15.46:0/2087286529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5c0a43d0 con 0x7f2d5c137680 2024-09-16T04:46:11.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.481+0000 7f2d58ff9640 1 --2- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2d480383b0 0x7f2d4803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.481+0000 7f2d58ff9640 1 -- 172.21.15.46:0/2087286529 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(282..282 src has 1..282) ==== 4738+0+0 (secure 0 0 0) 0x7f2d5004b620 con 0x7f2d5c0a3ff0 2024-09-16T04:46:11.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.481+0000 7f2d5bfff640 1 --2- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5c137680 0x7f2d5c139d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.481+0000 7f2d5bfff640 10 monclient: get_auth_request con 0x7f2d5c137680 auth_method 0 2024-09-16T04:46:11.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.481+0000 7f2d5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.481+0000 7f2d5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.481+0000 7f2d5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.481+0000 7f2d5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:11.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.481+0000 7f2d5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:11.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.481+0000 7f2d5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:11.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.481+0000 7f2d60ae8640 1 --2- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2d480383b0 0x7f2d4803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.481+0000 7f2d60ae8640 10 monclient: get_auth_request con 0x7f2d480383b0 auth_method 0 2024-09-16T04:46:11.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.482+0000 7f2d5bfff640 10 monclient(hunting): handle_auth_done global_id 6760 payload 824 2024-09-16T04:46:11.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.482+0000 7f2d5bfff640 1 -- 172.21.15.46:0/2087286529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d5c069150 con 0x7f2d5c137680 2024-09-16T04:46:11.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.482+0000 7f2d5bfff640 1 --2- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5c137680 0x7f2d5c139d50 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c001850 tx=0x7f2d4c001d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:11.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.482+0000 7f2d60ae8640 1 --2- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2d480383b0 0x7f2d4803a870 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f2d54034000 tx=0x7f2d5402f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:11.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.482+0000 7f2d58ff9640 1 -- 172.21.15.46:0/2087286529 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c00f9e0 con 0x7f2d5c137680 2024-09-16T04:46:11.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.482+0000 7f2d58ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.482+0000 7f2d58ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:46:11.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.482+0000 7f2d58ff9640 10 monclient: dump: 2024-09-16T04:46:11.501 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.501 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.502 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.502 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.502 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.502 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.502 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.502 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.502 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.502 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.482+0000 7f2d58ff9640 1 -- 172.21.15.46:0/2087286529 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2d4c018070 con 0x7f2d5c137680 2024-09-16T04:46:11.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.482+0000 7f2d58ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:11.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.482+0000 7f2d58ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:11.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.482+0000 7f2d58ff9640 1 -- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5c137680 msgr2=0x7f2d5c139d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.482+0000 7f2d58ff9640 1 --2- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5c137680 0x7f2d5c139d50 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c001850 tx=0x7f2d4c001d10 comp rx=0 tx=0).stop 2024-09-16T04:46:11.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.482+0000 7f2d58ff9640 1 -- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5c137680 msgr2=0x7f2d5c139d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:11.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.482+0000 7f2d58ff9640 1 --2- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5c137680 0x7f2d5c139d50 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.488+0000 7f2d6254b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:46:11.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.488+0000 7f2d6254b640 1 --2- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5c13a290 0x7f2d5c0a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.488+0000 7f2d6254b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.488+0000 7f2d6254b640 1 -- 172.21.15.46:0/2087286529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5c13a6a0 con 0x7f2d5c13a290 2024-09-16T04:46:11.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.489+0000 7f2d5bfff640 1 --2- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5c13a290 0x7f2d5c0a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.489+0000 7f2d5bfff640 10 monclient: get_auth_request con 0x7f2d5c13a290 auth_method 0 2024-09-16T04:46:11.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.489+0000 7f2d5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.489+0000 7f2d5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.489+0000 7f2d5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.489+0000 7f2d5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.489+0000 7f2d5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.489+0000 7f2d5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.489+0000 7f2d5bfff640 10 monclient(hunting): handle_auth_done global_id 6763 payload 824 2024-09-16T04:46:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.489+0000 7f2d5bfff640 1 -- 172.21.15.46:0/2087286529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2d5c0008d0 con 0x7f2d5c13a290 2024-09-16T04:46:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.489+0000 7f2d5bfff640 1 --2- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5c13a290 0x7f2d5c0a0610 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c001850 tx=0x7f2d4c000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d58ff9640 1 -- 172.21.15.46:0/2087286529 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c013600 con 0x7f2d5c13a290 2024-09-16T04:46:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d58ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d58ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:46:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d58ff9640 10 monclient: dump: 2024-09-16T04:46:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.506 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d58ff9640 1 -- 172.21.15.46:0/2087286529 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f2d4c013600 con 0x7f2d5c13a290 2024-09-16T04:46:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d58ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:46:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d58ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d58ff9640 1 -- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5c13a290 msgr2=0x7f2d5c0a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d58ff9640 1 --2- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5c13a290 0x7f2d5c0a0610 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c001850 tx=0x7f2d4c000ed0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d58ff9640 1 -- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5c13a290 msgr2=0x7f2d5c0a0610 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:11.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d58ff9640 1 --2- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5c13a290 0x7f2d5c0a0610 secure :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c001850 tx=0x7f2d4c000ed0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.507 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:46:11.507 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:46:11.507 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:46:11.507 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:46:11.507 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 88, 2024-09-16T04:46:11.507 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 457392, 2024-09-16T04:46:11.507 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:46:11.507 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:46:11.507 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:46:11.508 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:46:11.508 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:11.508 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 7, 2024-09-16T04:46:11.508 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:11.508 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:46:11.508 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:46:11.508 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:11.508 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:11.508 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:11.508 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:11.508 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:11.508 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:11.508 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:11.508 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:11.509 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:11.509 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:11.509 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:11.509 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:11.509 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:46:11.509 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:46:11.509 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:11.509 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:11.509 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:11.509 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:11.509 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:11.509 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:11.509 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:11.509 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:11.510 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:11.510 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:11.510 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:11.510 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:11.510 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.510 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:46:11.510 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:46:11.510 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:46:11.510 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:46:11.510 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:46:11.510 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:46:11.510 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:11.510 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:11.510 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:46:11.511 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:46:11.511 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:46:11.511 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:46:11.511 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:46:11.511 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:46:11.511 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:46:11.511 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:11.511 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:46:11.511 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:11.511 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:11.511 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:11.511 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:11.511 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:11.512 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:11.512 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:11.512 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:11.512 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:11.512 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:11.512 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:11.512 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:11.512 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:46:11.512 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.512 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:46:11.512 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.512 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:46:11.512 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:46:11.512 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:11.513 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:11.513 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.513 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:11.513 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:46:11.513 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:11.513 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.513 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.513 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:11.513 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:46:11.513 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:11.513 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:11.513 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:11.513 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.513 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:46:11.514 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:46:11.514 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:11.514 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:11.514 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:11.514 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.514 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.514 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:46:11.514 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:46:11.514 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:11.514 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:11.514 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.514 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:11.514 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:46:11.514 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:11.515 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.515 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.515 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:11.515 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:46:11.515 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:11.515 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:11.515 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:11.515 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.515 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:46:11.515 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:46:11.515 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:11.515 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:11.515 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:11.515 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.516 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.516 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:46:11.516 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:46:11.516 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:11.516 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:11.516 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.516 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:11.516 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:46:11.516 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:11.516 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.516 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.516 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:11.516 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:46:11.516 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:11.517 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:11.517 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:11.517 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.517 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:46:11.517 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:46:11.517 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:11.517 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:11.517 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:11.517 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:11.517 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:11.517 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.517 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:46:11.517 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:46:11.517 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.518 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:11.518 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:11.518 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:11.518 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:11.518 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:11.518 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:46:11.518 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.518 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:11.518 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:11.518 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5 2024-09-16T04:46:11.518 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:11.518 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:11.518 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:46:11.518 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.518 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:11.519 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:11.519 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5 2024-09-16T04:46:11.519 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:11.519 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:11.519 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:46:11.519 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:11.519 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:11.519 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:11.519 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:11.519 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:11.519 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:11.519 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:11.519 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:46:11.519 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:46:11.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d6254b640 1 -- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2d480383b0 msgr2=0x7f2d4803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d6254b640 1 --2- 172.21.15.46:0/2087286529 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f2d480383b0 0x7f2d4803a870 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f2d54034000 tx=0x7f2d5402f000 comp rx=0 tx=0).stop 2024-09-16T04:46:11.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d6254b640 10 monclient: shutdown 2024-09-16T04:46:11.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d6254b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:11.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d6254b640 1 -- 172.21.15.46:0/2087286529 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2d5c0a3ff0 msgr2=0x7f2d5c137140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d6254b640 1 --2- 172.21.15.46:0/2087286529 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2d5c0a3ff0 0x7f2d5c137140 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2d5000e900 tx=0x7f2d5000edc0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d60ae8640 1 -- 172.21.15.46:0/2087286529 reap_dead start 2024-09-16T04:46:11.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d60ae8640 1 -- 172.21.15.46:0/2087286529 reap_dead start 2024-09-16T04:46:11.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d6254b640 1 -- 172.21.15.46:0/2087286529 shutdown_connections 2024-09-16T04:46:11.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d6254b640 1 -- 172.21.15.46:0/2087286529 >> 172.21.15.46:0/2087286529 conn(0x7f2d5c0a5d10 msgr2=0x7f2d5c13e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:11.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d6254b640 1 -- 172.21.15.46:0/2087286529 shutdown_connections 2024-09-16T04:46:11.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.490+0000 7f2d6254b640 1 -- 172.21.15.46:0/2087286529 wait complete. 2024-09-16T04:46:11.522 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:46:11.522 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T04:46:11.522 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:46:11.522 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T04:46:11.522 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:46:11.523 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T04:46:11.523 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:46:11.534 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T04:46:11.534 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:46:11.534 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:46:11.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.672+0000 7f122c7bd640 10 monclient: get_monmap_and_config 2024-09-16T04:46:11.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.672+0000 7f122c7bd640 10 monclient: build_initial_monmap 2024-09-16T04:46:11.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.672+0000 7f122c7bd640 10 monclient: monmap: 2024-09-16T04:46:11.674 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:11.674 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.674 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:11.673569+0000 2024-09-16T04:46:11.674 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:11.673569+0000 2024-09-16T04:46:11.674 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:11.674 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.674 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:11.674 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:11.675 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:11.675 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:11.675 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:11.675 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.672+0000 7f122c7bd640 1 Processor -- start 2024-09-16T04:46:11.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.672+0000 7f122c7bd640 1 -- start start 2024-09-16T04:46:11.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.672+0000 7f122c7bd640 10 monclient: init 2024-09-16T04:46:11.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f122c7bd640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:11.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f122c7bd640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-16T04:46:11.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f122c7bd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f12240a3c10 0x7f122409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f122c7bd640 10 monclient(hunting): picked mon.noname-c con 0x7f12240a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:46:11.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f122c7bd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12240a3ff0 0x7f12240a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f122c7bd640 10 monclient(hunting): picked mon.noname-b con 0x7f12240a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:11.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f122c7bd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f12241369d0 0x7f1224136e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f122c7bd640 10 monclient(hunting): picked mon.noname-a con 0x7f12241369d0 addr v2:172.21.15.46:3300/0 2024-09-16T04:46:11.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f122c7bd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f122c7bd640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f122409ab70 con 0x7f12241369d0 2024-09-16T04:46:11.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f122c7bd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f122c7bd640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f12240a4d60 con 0x7f12240a3c10 2024-09-16T04:46:11.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f122c7bd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f122c7bd640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f122409b030 con 0x7f12240a3ff0 2024-09-16T04:46:11.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f122c7bd640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:11.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f122c7bd640 10 monclient(hunting): authenticate will time out at 1170.089516s 2024-09-16T04:46:11.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f122ad5a640 1 -- >> v2:172.21.15.46:3300/0 conn(0x7f12241369d0 msgr2=0x7f1224136e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:46:11.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f122ad5a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f12241369d0 0x7f1224136e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:11.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f1229d58640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12240a3ff0 0x7f12240a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f1229d58640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12240a3ff0 0x7f12240a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:44168/0 (socket says 172.21.15.46:44168) 2024-09-16T04:46:11.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f1229d58640 1 -- 172.21.15.46:0/2927783967 learned_addr learned my addr 172.21.15.46:0/2927783967 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:11.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f1229d58640 10 monclient(hunting): get_auth_request con 0x7f12240a3ff0 auth_method 0 2024-09-16T04:46:11.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f1229d58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f1229d58640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f1229d58640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f1229d58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:11.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f1229d58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:11.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f1229d58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:11.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f122a559640 1 --2- 172.21.15.46:0/2927783967 >> v2:172.21.15.114:3300/0 conn(0x7f12240a3c10 0x7f122409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f1229d58640 10 monclient(hunting): handle_auth_done global_id 94163 payload 274 2024-09-16T04:46:11.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f1229d58640 1 -- 172.21.15.46:0/2927783967 >> v2:172.21.15.114:3300/0 conn(0x7f12240a3c10 msgr2=0x7f122409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:11.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f1229d58640 1 --2- 172.21.15.46:0/2927783967 >> v2:172.21.15.114:3300/0 conn(0x7f12240a3c10 0x7f122409bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f1229d58640 1 -- 172.21.15.46:0/2927783967 >> v2:172.21.15.46:3300/0 conn(0x7f12241369d0 msgr2=0x7f1224136e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:46:11.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f1229d58640 1 --2- 172.21.15.46:0/2927783967 >> v2:172.21.15.46:3300/0 conn(0x7f12241369d0 0x7f1224136e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f1229d58640 10 monclient: _finish_hunting 0 2024-09-16T04:46:11.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f1229d58640 1 monclient: found mon.noname-b 2024-09-16T04:46:11.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f1229d58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:11.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f1229d58640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:46:11.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f1229d58640 1 -- 172.21.15.46:0/2927783967 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122409b310 con 0x7f12240a3ff0 2024-09-16T04:46:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f122a559640 10 monclient: get_auth_request con 0x7f12240a3c10 auth_method 0 2024-09-16T04:46:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.673+0000 7f122a559640 1 --2- 172.21.15.46:0/2927783967 >> v2:172.21.15.114:3300/0 conn(0x7f12240a3c10 0x7f122409bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:46:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.674+0000 7f1229d58640 1 --2- 172.21.15.46:0/2927783967 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12240a3ff0 0x7f12240a43d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f122406be80 tx=0x7f121402a160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a8dca55db872dcf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.674+0000 7f1229557640 1 -- 172.21.15.46:0/2927783967 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214037070 con 0x7f12240a3ff0 2024-09-16T04:46:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.674+0000 7f1229557640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.674+0000 7f1229557640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:46:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.674+0000 7f1229557640 10 monclient: dump: 2024-09-16T04:46:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.686 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.686 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.686 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.686 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.674+0000 7f1229557640 10 monclient: _finish_auth 0 2024-09-16T04:46:11.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.674+0000 7f1229557640 10 monclient: _check_auth_tickets 2024-09-16T04:46:11.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.674+0000 7f1229557640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:11.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.674+0000 7f1229557640 1 -- 172.21.15.46:0/2927783967 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1214032440 con 0x7f12240a3ff0 2024-09-16T04:46:11.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.674+0000 7f122c7bd640 5 monclient: authenticate success, global_id 94163 2024-09-16T04:46:11.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.674+0000 7f122c7bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:11.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.674+0000 7f1229557640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:11.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.674+0000 7f122c7bd640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:11.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.674+0000 7f1229557640 1 -- 172.21.15.46:0/2927783967 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214032810 con 0x7f12240a3ff0 2024-09-16T04:46:11.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.674+0000 7f122c7bd640 4 set_mon_vals no callback set 2024-09-16T04:46:11.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.674+0000 7f1229557640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.674+0000 7f1229557640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:11.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.674+0000 7f1229557640 10 monclient: dump: 2024-09-16T04:46:11.687 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.687 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.687 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.687 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.687 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.687 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.687 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.687 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.687 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.687 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.675+0000 7f122c7bd640 10 monclient: shutdown 2024-09-16T04:46:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.675+0000 7f122c7bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.675+0000 7f122c7bd640 1 -- 172.21.15.46:0/2927783967 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12240a3ff0 msgr2=0x7f12240a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.675+0000 7f122c7bd640 1 --2- 172.21.15.46:0/2927783967 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12240a3ff0 0x7f12240a43d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f122406be80 tx=0x7f121402a160 comp rx=0 tx=0).stop 2024-09-16T04:46:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.675+0000 7f122c7bd640 1 -- 172.21.15.46:0/2927783967 shutdown_connections 2024-09-16T04:46:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.675+0000 7f122c7bd640 1 --2- 172.21.15.46:0/2927783967 >> v2:172.21.15.46:3300/0 conn(0x7f12241369d0 0x7f1224136e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.675+0000 7f122c7bd640 1 --2- 172.21.15.46:0/2927783967 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12240a3ff0 0x7f12240a43d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.675+0000 7f122c7bd640 1 --2- 172.21.15.46:0/2927783967 >> v2:172.21.15.114:3300/0 conn(0x7f12240a3c10 0x7f122409bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.675+0000 7f122c7bd640 1 -- 172.21.15.46:0/2927783967 >> 172.21.15.46:0/2927783967 conn(0x7f12240a5d10 msgr2=0x7f12240a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.675+0000 7f122c7bd640 1 -- 172.21.15.46:0/2927783967 shutdown_connections 2024-09-16T04:46:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.675+0000 7f122c7bd640 1 -- 172.21.15.46:0/2927783967 wait complete. 2024-09-16T04:46:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.676+0000 7f122c7bd640 10 monclient: build_initial_monmap 2024-09-16T04:46:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.676+0000 7f122c7bd640 10 monclient: monmap: 2024-09-16T04:46:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:11.689 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.676+0000 7f122c7bd640 1 Processor -- start 2024-09-16T04:46:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.676+0000 7f122c7bd640 1 -- start start 2024-09-16T04:46:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.676+0000 7f122c7bd640 10 monclient: init 2024-09-16T04:46:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.676+0000 7f122c7bd640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.676+0000 7f122c7bd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:46:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.676+0000 7f122c7bd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12240a3c10 0x7f12241cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.676+0000 7f122c7bd640 10 monclient(hunting): picked mon.noname-b con 0x7f12240a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.676+0000 7f122c7bd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12240a3ff0 0x7f12241cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f122c7bd640 10 monclient(hunting): picked mon.noname-c con 0x7f12240a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f122c7bd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12241369d0 0x7f1224123420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f122c7bd640 10 monclient(hunting): picked mon.noname-a con 0x7f12241369d0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f122c7bd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f122c7bd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f12240a4910 con 0x7f12241369d0 2024-09-16T04:46:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f122c7bd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f122c7bd640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f122406e3d0 con 0x7f12240a3c10 2024-09-16T04:46:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f122c7bd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f122c7bd640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f1224127d40 con 0x7f12240a3ff0 2024-09-16T04:46:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f122c7bd640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f122c7bd640 10 monclient(hunting): authenticate will time out at 1170.093060s 2024-09-16T04:46:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f122ad5a640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12241369d0 msgr2=0x7f1224123420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:46:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f122ad5a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12241369d0 0x7f1224123420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1229d58640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12240a3ff0 0x7f12241cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1229d58640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12240a3ff0 0x7f12241cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:44178/0 (socket says 172.21.15.46:44178) 2024-09-16T04:46:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1229d58640 1 -- 172.21.15.46:0/839298858 learned_addr learned my addr 172.21.15.46:0/839298858 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1229d58640 10 monclient(hunting): get_auth_request con 0x7f12240a3ff0 auth_method 0 2024-09-16T04:46:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1229d58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1229d58640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1229d58640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1229d58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1229d58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1229d58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1229d58640 10 monclient(hunting): handle_auth_done global_id 94166 payload 824 2024-09-16T04:46:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1229d58640 1 -- 172.21.15.46:0/839298858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12240a3c10 msgr2=0x7f12241cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1229d58640 1 --2- 172.21.15.46:0/839298858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12240a3c10 0x7f12241cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1229d58640 1 -- 172.21.15.46:0/839298858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12241369d0 msgr2=0x7f1224123420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:46:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1229d58640 1 --2- 172.21.15.46:0/839298858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12241369d0 0x7f1224123420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1229d58640 10 monclient: _finish_hunting 0 2024-09-16T04:46:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1229d58640 1 monclient: found mon.noname-c 2024-09-16T04:46:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1229d58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1229d58640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:46:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1229d58640 1 -- 172.21.15.46:0/839298858 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1224127030 con 0x7f12240a3ff0 2024-09-16T04:46:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1229d58640 1 --2- 172.21.15.46:0/839298858 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12240a3ff0 0x7f12241cc420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1214000c00 tx=0x7f12140046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f121affd640 1 -- 172.21.15.46:0/839298858 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214041020 con 0x7f12240a3ff0 2024-09-16T04:46:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f121affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f121affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f121affd640 10 monclient: dump: 2024-09-16T04:46:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.693 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f121affd640 10 monclient: _finish_auth 0 2024-09-16T04:46:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f121affd640 10 monclient: _check_auth_tickets 2024-09-16T04:46:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f121affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f121affd640 1 -- 172.21.15.46:0/839298858 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1214030970 con 0x7f12240a3ff0 2024-09-16T04:46:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f121affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f121affd640 1 -- 172.21.15.46:0/839298858 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214037040 con 0x7f12240a3ff0 2024-09-16T04:46:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f121affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f121affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f121affd640 10 monclient: dump: 2024-09-16T04:46:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:11.695 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:11.695 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:11.695 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:11.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f122c7bd640 5 monclient: authenticate success, global_id 94166 2024-09-16T04:46:11.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f122c7bd640 10 monclient: _renew_subs 2024-09-16T04:46:11.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f122c7bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:11.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f122c7bd640 1 -- 172.21.15.46:0/839298858 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1224127310 con 0x7f12240a3ff0 2024-09-16T04:46:11.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f1228d56640 4 set_mon_vals no callback set 2024-09-16T04:46:11.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f122c7bd640 10 monclient: _renew_subs 2024-09-16T04:46:11.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f122c7bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:11.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.677+0000 7f122c7bd640 1 -- 172.21.15.46:0/839298858 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f12240a8a70 con 0x7f12240a3ff0 2024-09-16T04:46:11.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.678+0000 7f121affd640 1 -- 172.21.15.46:0/839298858 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f1214044020 con 0x7f12240a3ff0 2024-09-16T04:46:11.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.678+0000 7f122c7bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.678+0000 7f121affd640 1 --2- 172.21.15.46:0/839298858 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1208038360 0x7f120803a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.678+0000 7f122c7bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.678+0000 7f122c7bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.678+0000 7f121affd640 1 -- 172.21.15.46:0/839298858 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(282..282 src has 1..282) ==== 4738+0+0 (secure 0 0 0) 0x7f121406c0e0 con 0x7f12240a3ff0 2024-09-16T04:46:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.678+0000 7f122c7bd640 1 -- 172.21.15.46:0/839298858 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f122409bcf0 con 0x7f12240a3ff0 2024-09-16T04:46:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.679+0000 7f122a559640 1 --2- 172.21.15.46:0/839298858 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1208038360 0x7f120803a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.679+0000 7f122a559640 10 monclient: get_auth_request con 0x7f1208038360 auth_method 0 2024-09-16T04:46:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.679+0000 7f122a559640 1 --2- 172.21.15.46:0/839298858 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1208038360 0x7f120803a820 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f1220006fd0 tx=0x7f1220008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.683+0000 7f121affd640 1 -- 172.21.15.46:0/839298858 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f121404b030 con 0x7f12240a3ff0 2024-09-16T04:46:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.683+0000 7f121affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.683+0000 7f121affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.925+0000 7f122c7bd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:46:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.925+0000 7f122c7bd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.925+0000 7f122c7bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.925+0000 7f122c7bd640 1 -- 172.21.15.46:0/839298858 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1224069150 con 0x7f12240a3ff0 2024-09-16T04:46:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.926+0000 7f121affd640 1 -- 172.21.15.46:0/839298858 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f1214031e80 con 0x7f12240a3ff0 2024-09-16T04:46:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.926+0000 7f121affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.926+0000 7f121affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:11.929 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:46:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.928+0000 7f122c7bd640 1 -- 172.21.15.46:0/839298858 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1208038360 msgr2=0x7f120803a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.928+0000 7f122c7bd640 1 --2- 172.21.15.46:0/839298858 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1208038360 0x7f120803a820 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f1220006fd0 tx=0x7f1220008040 comp rx=0 tx=0).stop 2024-09-16T04:46:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.928+0000 7f122c7bd640 10 monclient: shutdown 2024-09-16T04:46:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.928+0000 7f122c7bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.928+0000 7f122c7bd640 1 -- 172.21.15.46:0/839298858 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12240a3ff0 msgr2=0x7f12241cc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.928+0000 7f122c7bd640 1 --2- 172.21.15.46:0/839298858 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12240a3ff0 0x7f12241cc420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1214000c00 tx=0x7f12140046c0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.928+0000 7f122c7bd640 1 -- 172.21.15.46:0/839298858 shutdown_connections 2024-09-16T04:46:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.928+0000 7f122c7bd640 1 --2- 172.21.15.46:0/839298858 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1208038360 0x7f120803a820 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.928+0000 7f122c7bd640 1 --2- 172.21.15.46:0/839298858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12241369d0 0x7f1224123420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.928+0000 7f122c7bd640 1 --2- 172.21.15.46:0/839298858 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f12240a3ff0 0x7f12241cc420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.928+0000 7f122c7bd640 1 --2- 172.21.15.46:0/839298858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12240a3c10 0x7f12241cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.928+0000 7f122c7bd640 1 -- 172.21.15.46:0/839298858 >> 172.21.15.46:0/839298858 conn(0x7f12240a5d10 msgr2=0x7f122409e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.928+0000 7f122c7bd640 1 -- 172.21.15.46:0/839298858 shutdown_connections 2024-09-16T04:46:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:11.928+0000 7f122c7bd640 1 -- 172.21.15.46:0/839298858 wait complete. 2024-09-16T04:46:14.941 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:46:15.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.051+0000 7f4b24752640 10 monclient: get_monmap_and_config 2024-09-16T04:46:15.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.051+0000 7f4b24752640 10 monclient: build_initial_monmap 2024-09-16T04:46:15.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.051+0000 7f4b24752640 10 monclient: monmap: 2024-09-16T04:46:15.053 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:15.053 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:15.053051+0000 2024-09-16T04:46:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:15.053051+0000 2024-09-16T04:46:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:15.054 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:15.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.052+0000 7f4b24752640 1 Processor -- start 2024-09-16T04:46:15.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.052+0000 7f4b24752640 1 -- start start 2024-09-16T04:46:15.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.052+0000 7f4b24752640 10 monclient: init 2024-09-16T04:46:15.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.052+0000 7f4b24752640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:15.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.052+0000 7f4b24752640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-16T04:46:15.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.052+0000 7f4b24752640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4b1c0a3c10 0x7f4b1c09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.052+0000 7f4b24752640 10 monclient(hunting): picked mon.noname-c con 0x7f4b1c0a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:46:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.052+0000 7f4b24752640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b1c0a3ff0 0x7f4b1c0a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.052+0000 7f4b24752640 10 monclient(hunting): picked mon.noname-a con 0x7f4b1c0a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:46:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.052+0000 7f4b24752640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4b1c1369d0 addr v1:172.21.15.46:6789/0 2024-09-16T04:46:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.052+0000 7f4b24752640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b1c09b940 con 0x7f4b1c1369d0 2024-09-16T04:46:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.052+0000 7f4b24752640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.052+0000 7f4b24752640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4b1c09ab00 con 0x7f4b1c0a3ff0 2024-09-16T04:46:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.052+0000 7f4b24752640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.052+0000 7f4b24752640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f4b1c09b030 con 0x7f4b1c0a3c10 2024-09-16T04:46:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.052+0000 7f4b24752640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.052+0000 7f4b24752640 10 monclient(hunting): authenticate will time out at 1173.468972s 2024-09-16T04:46:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b22cef640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f4b1c1369d0 legacy=0x7f4b1c136e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:46:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b21ced640 1 -- >> v2:172.21.15.46:3300/0 conn(0x7f4b1c0a3ff0 msgr2=0x7f4b1c0a43d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:46:15.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b21ced640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b1c0a3ff0 0x7f4b1c0a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:15.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b224ee640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4b1c0a3c10 0x7f4b1c09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:15.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b224ee640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4b1c0a3c10 0x7f4b1c09bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.46:47966/0 (socket says 172.21.15.46:47966) 2024-09-16T04:46:15.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b224ee640 1 -- 172.21.15.46:0/3345269072 learned_addr learned my addr 172.21.15.46:0/3345269072 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:15.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b224ee640 10 monclient(hunting): get_auth_request con 0x7f4b1c0a3c10 auth_method 0 2024-09-16T04:46:15.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b224ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:15.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b224ee640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:15.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b224ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:15.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b224ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:15.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b224ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:15.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b224ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:15.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b224ee640 10 monclient(hunting): handle_auth_done global_id 6766 payload 274 2024-09-16T04:46:15.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b224ee640 1 -- 172.21.15.46:0/3345269072 >> v2:172.21.15.46:3300/0 conn(0x7f4b1c0a3ff0 msgr2=0x7f4b1c0a43d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:46:15.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b224ee640 1 --2- 172.21.15.46:0/3345269072 >> v2:172.21.15.46:3300/0 conn(0x7f4b1c0a3ff0 0x7f4b1c0a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:15.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b224ee640 1 -- 172.21.15.46:0/3345269072 >> v1:172.21.15.46:6789/0 conn(0x7f4b1c1369d0 legacy=0x7f4b1c136e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:46:15.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b224ee640 10 monclient: _finish_hunting 0 2024-09-16T04:46:15.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b224ee640 1 monclient: found mon.noname-c 2024-09-16T04:46:15.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b224ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:15.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b224ee640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:46:15.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.053+0000 7f4b224ee640 1 -- 172.21.15.46:0/3345269072 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b1c09b310 con 0x7f4b1c0a3c10 2024-09-16T04:46:15.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.054+0000 7f4b224ee640 1 --2- 172.21.15.46:0/3345269072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b1c0a3c10 0x7f4b1c09bcf0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c068240 tx=0x7f4b1802a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=754e8797981778dd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:15.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.054+0000 7f4b214ec640 1 -- 172.21.15.46:0/3345269072 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b18037070 con 0x7f4b1c0a3c10 2024-09-16T04:46:15.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.054+0000 7f4b214ec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:15.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.054+0000 7f4b214ec640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:15.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.054+0000 7f4b214ec640 10 monclient: dump: 2024-09-16T04:46:15.062 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:15.062 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:15.062 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:15.062 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:15.062 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:15.062 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:15.062 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:15.062 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:15.062 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:15.062 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:15.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b214ec640 10 monclient: _finish_auth 0 2024-09-16T04:46:15.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b214ec640 10 monclient: _check_auth_tickets 2024-09-16T04:46:15.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b214ec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:15.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b214ec640 1 -- 172.21.15.46:0/3345269072 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b1802ae40 con 0x7f4b1c0a3c10 2024-09-16T04:46:15.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b24752640 5 monclient: authenticate success, global_id 6766 2024-09-16T04:46:15.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b24752640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:15.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b214ec640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:15.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b24752640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:15.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b214ec640 1 -- 172.21.15.46:0/3345269072 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b18032660 con 0x7f4b1c0a3c10 2024-09-16T04:46:15.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b214ec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:15.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b24752640 4 set_mon_vals no callback set 2024-09-16T04:46:15.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b214ec640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:46:15.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b214ec640 10 monclient: dump: 2024-09-16T04:46:15.063 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:15.064 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:15.064 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:15.064 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:15.064 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:15.064 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:15.064 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:15.064 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:15.064 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:15.064 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:15.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b24752640 10 monclient: shutdown 2024-09-16T04:46:15.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b24752640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:15.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b24752640 1 -- 172.21.15.46:0/3345269072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b1c0a3c10 msgr2=0x7f4b1c09bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:15.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b24752640 1 --2- 172.21.15.46:0/3345269072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b1c0a3c10 0x7f4b1c09bcf0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c068240 tx=0x7f4b1802a130 comp rx=0 tx=0).stop 2024-09-16T04:46:15.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b24752640 1 -- 172.21.15.46:0/3345269072 shutdown_connections 2024-09-16T04:46:15.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b24752640 1 --2- 172.21.15.46:0/3345269072 >> v2:172.21.15.46:3300/0 conn(0x7f4b1c0a3ff0 0x7f4b1c0a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:15.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b24752640 1 --2- 172.21.15.46:0/3345269072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b1c0a3c10 0x7f4b1c09bcf0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:15.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b24752640 1 -- 172.21.15.46:0/3345269072 >> 172.21.15.46:0/3345269072 conn(0x7f4b1c0a6340 msgr2=0x7f4b1c06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:15.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b24752640 1 -- 172.21.15.46:0/3345269072 shutdown_connections 2024-09-16T04:46:15.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.055+0000 7f4b24752640 1 -- 172.21.15.46:0/3345269072 wait complete. 2024-09-16T04:46:15.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.056+0000 7f4b24752640 10 monclient: build_initial_monmap 2024-09-16T04:46:15.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.056+0000 7f4b24752640 10 monclient: monmap: 2024-09-16T04:46:15.065 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:15.065 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:15.065 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:15.065 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:15.065 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:15.066 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:15.066 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:15.066 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:15.066 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:15.066 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:15.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.056+0000 7f4b24752640 1 Processor -- start 2024-09-16T04:46:15.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.056+0000 7f4b24752640 1 -- start start 2024-09-16T04:46:15.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.056+0000 7f4b24752640 10 monclient: init 2024-09-16T04:46:15.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b24752640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:15.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b24752640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:46:15.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b24752640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b1c0a3c10 0x7f4b1c1cbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:15.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b24752640 10 monclient(hunting): picked mon.noname-a con 0x7f4b1c0a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:15.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b24752640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4b1c0a3ff0 0x7f4b1c135b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:15.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b24752640 10 monclient(hunting): picked mon.noname-c con 0x7f4b1c0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:15.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b24752640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b1c1369d0 0x7f4b1c136070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:15.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b24752640 10 monclient(hunting): picked mon.noname-b con 0x7f4b1c1369d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:15.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b24752640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:15.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b24752640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1c0a4910 con 0x7f4b1c0a3c10 2024-09-16T04:46:15.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b24752640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:15.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b24752640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1c0a69c0 con 0x7f4b1c1369d0 2024-09-16T04:46:15.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b24752640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:15.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b24752640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f4b1c0a0400 con 0x7f4b1c0a3ff0 2024-09-16T04:46:15.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b24752640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:15.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b224ee640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b1c0a3c10 msgr2=0x7f4b1c1cbd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:46:15.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b24752640 10 monclient(hunting): authenticate will time out at 1173.473326s 2024-09-16T04:46:15.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b224ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b1c0a3c10 0x7f4b1c1cbd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:15.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b21ced640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4b1c0a3ff0 0x7f4b1c135b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:15.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b21ced640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4b1c0a3ff0 0x7f4b1c135b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39952/0 (socket says 172.21.15.46:39952) 2024-09-16T04:46:15.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b21ced640 1 -- 172.21.15.46:0/1955017452 learned_addr learned my addr 172.21.15.46:0/1955017452 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:15.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b21ced640 10 monclient(hunting): get_auth_request con 0x7f4b1c0a3ff0 auth_method 0 2024-09-16T04:46:15.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b21ced640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:15.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b21ced640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:15.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b21ced640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:15.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b21ced640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:15.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b21ced640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:15.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b21ced640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:15.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b22cef640 1 --2- 172.21.15.46:0/1955017452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b1c1369d0 0x7f4b1c136070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:15.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b22cef640 10 monclient(hunting): get_auth_request con 0x7f4b1c1369d0 auth_method 0 2024-09-16T04:46:15.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b22cef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:15.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b22cef640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:15.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b22cef640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:15.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b21ced640 10 monclient(hunting): handle_auth_done global_id 94169 payload 824 2024-09-16T04:46:15.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b21ced640 1 -- 172.21.15.46:0/1955017452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b1c1369d0 msgr2=0x7f4b1c136070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:15.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b21ced640 1 --2- 172.21.15.46:0/1955017452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b1c1369d0 0x7f4b1c136070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:15.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b21ced640 1 -- 172.21.15.46:0/1955017452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b1c0a3c10 msgr2=0x7f4b1c1cbd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:46:15.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b21ced640 1 --2- 172.21.15.46:0/1955017452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b1c0a3c10 0x7f4b1c1cbd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:15.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b21ced640 10 monclient: _finish_hunting 0 2024-09-16T04:46:15.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b21ced640 1 monclient: found mon.noname-c 2024-09-16T04:46:15.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b21ced640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:15.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b21ced640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:46:15.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b21ced640 1 -- 172.21.15.46:0/1955017452 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b1c127530 con 0x7f4b1c0a3ff0 2024-09-16T04:46:15.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.057+0000 7f4b21ced640 1 --2- 172.21.15.46:0/1955017452 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4b1c0a3ff0 0x7f4b1c135b30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c00b550 tx=0x7f4b0c00ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:15.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.058+0000 7f4b12ffd640 1 -- 172.21.15.46:0/1955017452 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c003cf0 con 0x7f4b1c0a3ff0 2024-09-16T04:46:15.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.058+0000 7f4b12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:15.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.058+0000 7f4b12ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:15.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.058+0000 7f4b12ffd640 10 monclient: dump: 2024-09-16T04:46:15.073 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:15.073 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:15.074 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:15.074 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:15.074 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:15.074 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:15.074 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:15.074 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:15.074 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:15.074 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:15.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.058+0000 7f4b12ffd640 10 monclient: _finish_auth 0 2024-09-16T04:46:15.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.058+0000 7f4b12ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:46:15.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.058+0000 7f4b12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:15.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.058+0000 7f4b12ffd640 1 -- 172.21.15.46:0/1955017452 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b0c015070 con 0x7f4b1c0a3ff0 2024-09-16T04:46:15.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.058+0000 7f4b12ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.058+0000 7f4b12ffd640 1 -- 172.21.15.46:0/1955017452 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c0052b0 con 0x7f4b1c0a3ff0 2024-09-16T04:46:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.058+0000 7f4b12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.058+0000 7f4b12ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.058+0000 7f4b12ffd640 10 monclient: dump: 2024-09-16T04:46:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:15.076 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:15.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.058+0000 7f4b24752640 5 monclient: authenticate success, global_id 94169 2024-09-16T04:46:15.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.058+0000 7f4b24752640 10 monclient: _renew_subs 2024-09-16T04:46:15.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.058+0000 7f4b24752640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:15.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.058+0000 7f4b24752640 1 -- 172.21.15.46:0/1955017452 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b1c1cc290 con 0x7f4b1c0a3ff0 2024-09-16T04:46:15.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.058+0000 7f4b20ceb640 4 set_mon_vals no callback set 2024-09-16T04:46:15.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.058+0000 7f4b24752640 10 monclient: _renew_subs 2024-09-16T04:46:15.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.058+0000 7f4b24752640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:15.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.058+0000 7f4b24752640 1 -- 172.21.15.46:0/1955017452 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b1c0a8650 con 0x7f4b1c0a3ff0 2024-09-16T04:46:15.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.059+0000 7f4b12ffd640 1 -- 172.21.15.46:0/1955017452 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f4b0c005450 con 0x7f4b1c0a3ff0 2024-09-16T04:46:15.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.059+0000 7f4b24752640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:15.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.059+0000 7f4b24752640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:15.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.059+0000 7f4b24752640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:15.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.059+0000 7f4b12ffd640 1 --2- 172.21.15.46:0/1955017452 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4b00038360 0x7f4b0003a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.059+0000 7f4b24752640 1 -- 172.21.15.46:0/1955017452 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b1c09bd70 con 0x7f4b1c0a3ff0 2024-09-16T04:46:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.059+0000 7f4b12ffd640 1 -- 172.21.15.46:0/1955017452 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(282..282 src has 1..282) ==== 4738+0+0 (secure 0 0 0) 0x7f4b0c04b4b0 con 0x7f4b1c0a3ff0 2024-09-16T04:46:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.059+0000 7f4b224ee640 1 --2- 172.21.15.46:0/1955017452 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4b00038360 0x7f4b0003a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.059+0000 7f4b224ee640 10 monclient: get_auth_request con 0x7f4b00038360 auth_method 0 2024-09-16T04:46:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.060+0000 7f4b224ee640 1 --2- 172.21.15.46:0/1955017452 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4b00038360 0x7f4b0003a820 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f4b18034000 tx=0x7f4b1802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.064+0000 7f4b12ffd640 1 -- 172.21.15.46:0/1955017452 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4b0c0193c0 con 0x7f4b1c0a3ff0 2024-09-16T04:46:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.064+0000 7f4b12ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.064+0000 7f4b12ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:15.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.307+0000 7f4b24752640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:46:15.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.308+0000 7f4b24752640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:15.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.308+0000 7f4b24752640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:15.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.308+0000 7f4b24752640 1 -- 172.21.15.46:0/1955017452 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4b1c126d90 con 0x7f4b1c0a3ff0 2024-09-16T04:46:15.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.308+0000 7f4b12ffd640 1 -- 172.21.15.46:0/1955017452 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f4b0c015220 con 0x7f4b1c0a3ff0 2024-09-16T04:46:15.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.308+0000 7f4b12ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:15.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.308+0000 7f4b12ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:15.311 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:46:15.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.310+0000 7f4b24752640 1 -- 172.21.15.46:0/1955017452 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4b00038360 msgr2=0x7f4b0003a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:15.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.310+0000 7f4b24752640 1 --2- 172.21.15.46:0/1955017452 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4b00038360 0x7f4b0003a820 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f4b18034000 tx=0x7f4b1802e000 comp rx=0 tx=0).stop 2024-09-16T04:46:15.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.310+0000 7f4b24752640 10 monclient: shutdown 2024-09-16T04:46:15.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.310+0000 7f4b24752640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:15.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.310+0000 7f4b24752640 1 -- 172.21.15.46:0/1955017452 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4b1c0a3ff0 msgr2=0x7f4b1c135b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:15.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.310+0000 7f4b24752640 1 --2- 172.21.15.46:0/1955017452 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4b1c0a3ff0 0x7f4b1c135b30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c00b550 tx=0x7f4b0c00ba10 comp rx=0 tx=0).stop 2024-09-16T04:46:15.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.310+0000 7f4b24752640 1 -- 172.21.15.46:0/1955017452 shutdown_connections 2024-09-16T04:46:15.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.310+0000 7f4b24752640 1 --2- 172.21.15.46:0/1955017452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b1c1369d0 0x7f4b1c136070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:15.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.310+0000 7f4b24752640 1 --2- 172.21.15.46:0/1955017452 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4b1c0a3ff0 0x7f4b1c135b30 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:15.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.310+0000 7f4b24752640 1 --2- 172.21.15.46:0/1955017452 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4b00038360 0x7f4b0003a820 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:15.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.310+0000 7f4b24752640 1 --2- 172.21.15.46:0/1955017452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b1c0a3c10 0x7f4b1c1cbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:15.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.310+0000 7f4b24752640 1 -- 172.21.15.46:0/1955017452 >> 172.21.15.46:0/1955017452 conn(0x7f4b1c0a6340 msgr2=0x7f4b1c0a7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:15.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.310+0000 7f4b24752640 1 -- 172.21.15.46:0/1955017452 shutdown_connections 2024-09-16T04:46:15.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:15.310+0000 7f4b24752640 1 -- 172.21.15.46:0/1955017452 wait complete. 2024-09-16T04:46:18.324 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:46:18.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.436+0000 7f1406674640 10 monclient: get_monmap_and_config 2024-09-16T04:46:18.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.436+0000 7f1406674640 10 monclient: build_initial_monmap 2024-09-16T04:46:18.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.436+0000 7f1406674640 10 monclient: monmap: 2024-09-16T04:46:18.438 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:18.438 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:18.439 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:18.437659+0000 2024-09-16T04:46:18.439 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:18.437659+0000 2024-09-16T04:46:18.439 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:18.439 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:18.439 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:18.439 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:18.439 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:18.439 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:18.439 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:18.439 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:18.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.436+0000 7f1406674640 1 Processor -- start 2024-09-16T04:46:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f1406674640 1 -- start start 2024-09-16T04:46:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f1406674640 10 monclient: init 2024-09-16T04:46:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f1406674640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f1406674640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-16T04:46:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f1406674640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f140006e160 0x7f14000a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f1406674640 10 monclient(hunting): picked mon.noname-b con 0x7f140006e160 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f1406674640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f14000a2ee0 addr v1:172.21.15.46:6789/0 2024-09-16T04:46:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f1406674640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1400137050 addr v1:172.21.15.114:6789/0 2024-09-16T04:46:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f1406674640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14000a3fd0 con 0x7f14000a2ee0 2024-09-16T04:46:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f1406674640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14000a0bd0 con 0x7f1400137050 2024-09-16T04:46:18.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f1406674640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:18.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f1406674640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f14000a3300 con 0x7f140006e160 2024-09-16T04:46:18.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f1406674640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:18.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f1406674640 10 monclient(hunting): authenticate will time out at 1176.853572s 2024-09-16T04:46:18.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f13ff7fe640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f14000a2ee0 legacy=0x7f1400136940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:46:18.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f13fffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f140006e160 0x7f14000a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:18.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f13fffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f140006e160 0x7f14000a29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39954/0 (socket says 172.21.15.46:39954) 2024-09-16T04:46:18.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f13fffff640 1 -- 172.21.15.46:0/1105443810 learned_addr learned my addr 172.21.15.46:0/1105443810 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:18.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f13fffff640 10 monclient(hunting): get_auth_request con 0x7f140006e160 auth_method 0 2024-09-16T04:46:18.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f13fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:18.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f13fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:18.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f13fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:18.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f13fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:18.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f13fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:18.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.437+0000 7f13fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:18.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.438+0000 7f13fffff640 10 monclient(hunting): handle_auth_done global_id 94172 payload 274 2024-09-16T04:46:18.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.438+0000 7f13fffff640 1 -- 172.21.15.46:0/1105443810 >> v1:172.21.15.114:6789/0 conn(0x7f1400137050 legacy=0x7f140013ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:18.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.438+0000 7f13fffff640 1 -- 172.21.15.46:0/1105443810 >> v1:172.21.15.46:6789/0 conn(0x7f14000a2ee0 legacy=0x7f1400136940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:46:18.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.438+0000 7f13fffff640 10 monclient: _finish_hunting 0 2024-09-16T04:46:18.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.438+0000 7f13fffff640 1 monclient: found mon.noname-b 2024-09-16T04:46:18.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.438+0000 7f13fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:18.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.438+0000 7f13fffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:46:18.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.438+0000 7f13fffff640 1 -- 172.21.15.46:0/1105443810 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14000a0eb0 con 0x7f140006e160 2024-09-16T04:46:18.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.438+0000 7f13fffff640 1 --2- 172.21.15.46:0/1105443810 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f140006e160 0x7f14000a29a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f13f00098a0 tx=0x7f13f002a160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6e0eac3b529958f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:18.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f13feffd640 1 -- 172.21.15.46:0/1105443810 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f0037070 con 0x7f140006e160 2024-09-16T04:46:18.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f13feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:18.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f13feffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:46:18.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f13feffd640 10 monclient: dump: 2024-09-16T04:46:18.452 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:18.452 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:18.452 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:18.452 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:18.453 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:18.453 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:18.453 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:18.453 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:18.453 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:18.453 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:18.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f13feffd640 10 monclient: _finish_auth 0 2024-09-16T04:46:18.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f13feffd640 10 monclient: _check_auth_tickets 2024-09-16T04:46:18.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f13feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:18.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f13feffd640 1 -- 172.21.15.46:0/1105443810 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13f0032440 con 0x7f140006e160 2024-09-16T04:46:18.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f13feffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:18.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f13feffd640 1 -- 172.21.15.46:0/1105443810 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f0032810 con 0x7f140006e160 2024-09-16T04:46:18.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f1406674640 5 monclient: authenticate success, global_id 94172 2024-09-16T04:46:18.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f1406674640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:18.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f1406674640 4 set_mon_vals no callback set 2024-09-16T04:46:18.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f13feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:18.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f13feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:18.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f13feffd640 10 monclient: dump: 2024-09-16T04:46:18.454 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:18.454 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:18.454 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:18.454 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:18.455 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:18.455 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:18.455 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:18.455 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:18.455 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:18.455 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:18.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f1406674640 10 monclient: shutdown 2024-09-16T04:46:18.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f1406674640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:18.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f1406674640 1 -- 172.21.15.46:0/1105443810 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f140006e160 msgr2=0x7f14000a29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:18.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f1406674640 1 --2- 172.21.15.46:0/1105443810 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f140006e160 0x7f14000a29a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f13f00098a0 tx=0x7f13f002a160 comp rx=0 tx=0).stop 2024-09-16T04:46:18.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f1406674640 1 -- 172.21.15.46:0/1105443810 shutdown_connections 2024-09-16T04:46:18.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f1406674640 1 --2- 172.21.15.46:0/1105443810 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f140006e160 0x7f14000a29a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:18.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f1406674640 1 -- 172.21.15.46:0/1105443810 >> 172.21.15.46:0/1105443810 conn(0x7f14000a5c10 msgr2=0x7f14000a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:18.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f1406674640 1 -- 172.21.15.46:0/1105443810 shutdown_connections 2024-09-16T04:46:18.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.439+0000 7f1406674640 1 -- 172.21.15.46:0/1105443810 wait complete. 2024-09-16T04:46:18.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.440+0000 7f1406674640 10 monclient: build_initial_monmap 2024-09-16T04:46:18.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.440+0000 7f1406674640 10 monclient: monmap: 2024-09-16T04:46:18.456 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:18.456 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:18.456 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:18.456 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:18.457 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:18.457 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:18.457 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:18.457 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:18.457 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:18.457 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:18.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.440+0000 7f1406674640 1 Processor -- start 2024-09-16T04:46:18.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.440+0000 7f1406674640 1 -- start start 2024-09-16T04:46:18.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.440+0000 7f1406674640 10 monclient: init 2024-09-16T04:46:18.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1406674640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:18.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1406674640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:46:18.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1406674640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f140006e160 0x7f14001cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:18.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1406674640 10 monclient(hunting): picked mon.noname-a con 0x7f140006e160 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:18.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1406674640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14000a2ee0 0x7f1400135c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:18.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1406674640 10 monclient(hunting): picked mon.noname-b con 0x7f14000a2ee0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:18.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1406674640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1400137050 0x7f1400136160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:18.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1406674640 10 monclient(hunting): picked mon.noname-c con 0x7f1400137050 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:18.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1406674640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:18.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1406674640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f14000a34a0 con 0x7f140006e160 2024-09-16T04:46:18.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1406674640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:18.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1406674640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f14000a5130 con 0x7f14000a2ee0 2024-09-16T04:46:18.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1406674640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:18.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1406674640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f14000a88c0 con 0x7f1400137050 2024-09-16T04:46:18.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1406674640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:18.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f13fffff640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f140006e160 msgr2=0x7f14001cbb60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:46:18.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1406674640 10 monclient(hunting): authenticate will time out at 1176.857473s 2024-09-16T04:46:18.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f13fffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f140006e160 0x7f14001cbb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:18.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1404c11640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1400137050 0x7f1400136160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:18.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1404c11640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1400137050 0x7f1400136160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39968/0 (socket says 172.21.15.46:39968) 2024-09-16T04:46:18.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1404c11640 1 -- 172.21.15.46:0/3532783665 learned_addr learned my addr 172.21.15.46:0/3532783665 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:18.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1404c11640 10 monclient(hunting): get_auth_request con 0x7f1400137050 auth_method 0 2024-09-16T04:46:18.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1404c11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:18.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1404c11640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:18.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1404c11640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:18.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f13ff7fe640 1 --2- 172.21.15.46:0/3532783665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14000a2ee0 0x7f1400135c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:18.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1404c11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:18.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1404c11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:18.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1404c11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:18.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f13ff7fe640 10 monclient(hunting): get_auth_request con 0x7f14000a2ee0 auth_method 0 2024-09-16T04:46:18.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f13ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:18.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f13ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:18.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f13ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:18.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f13ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:18.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f13ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:18.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f13ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:18.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1404c11640 10 monclient(hunting): handle_auth_done global_id 94175 payload 824 2024-09-16T04:46:18.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1404c11640 1 -- 172.21.15.46:0/3532783665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14000a2ee0 msgr2=0x7f1400135c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:18.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1404c11640 1 --2- 172.21.15.46:0/3532783665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14000a2ee0 0x7f1400135c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:18.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1404c11640 1 -- 172.21.15.46:0/3532783665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f140006e160 msgr2=0x7f14001cbb60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:46:18.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1404c11640 1 --2- 172.21.15.46:0/3532783665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f140006e160 0x7f14001cbb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:18.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1404c11640 10 monclient: _finish_hunting 0 2024-09-16T04:46:18.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1404c11640 1 monclient: found mon.noname-c 2024-09-16T04:46:18.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1404c11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:18.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1404c11640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:46:18.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.441+0000 7f1404c11640 1 -- 172.21.15.46:0/3532783665 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14000a0300 con 0x7f1400137050 2024-09-16T04:46:18.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f1404c11640 1 --2- 172.21.15.46:0/3532783665 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1400137050 0x7f1400136160 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f13f800c7a0 tx=0x7f13f800cc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:18.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f13fcff9640 1 -- 172.21.15.46:0/3532783665 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f8003b70 con 0x7f1400137050 2024-09-16T04:46:18.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f13fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:18.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f13fcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:18.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f13fcff9640 10 monclient: dump: 2024-09-16T04:46:18.462 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:18.462 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:18.462 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:18.462 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:18.462 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:18.462 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:18.462 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:18.462 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:18.462 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:18.463 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:18.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f13fcff9640 10 monclient: _finish_auth 0 2024-09-16T04:46:18.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f13fcff9640 10 monclient: _check_auth_tickets 2024-09-16T04:46:18.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f13fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:18.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f13fcff9640 1 -- 172.21.15.46:0/3532783665 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13f8003d30 con 0x7f1400137050 2024-09-16T04:46:18.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f1406674640 5 monclient: authenticate success, global_id 94175 2024-09-16T04:46:18.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f1406674640 10 monclient: _renew_subs 2024-09-16T04:46:18.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f1406674640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:18.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f1406674640 1 -- 172.21.15.46:0/3532783665 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1400123600 con 0x7f1400137050 2024-09-16T04:46:18.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f13fcff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:18.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f13fcff9640 1 -- 172.21.15.46:0/3532783665 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f800aa90 con 0x7f1400137050 2024-09-16T04:46:18.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f13fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:18.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f13fcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:18.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f13fcff9640 10 monclient: dump: 2024-09-16T04:46:18.464 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:18.464 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:18.464 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:18.464 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:18.464 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:18.464 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:18.464 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:18.464 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:18.465 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:18.465 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:18.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f1406674640 10 monclient: _renew_subs 2024-09-16T04:46:18.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f1406674640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:18.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f13fe7fc640 4 set_mon_vals no callback set 2024-09-16T04:46:18.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.442+0000 7f1406674640 1 -- 172.21.15.46:0/3532783665 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f140011f760 con 0x7f1400137050 2024-09-16T04:46:18.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.443+0000 7f13fcff9640 1 -- 172.21.15.46:0/3532783665 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f13f800ac30 con 0x7f1400137050 2024-09-16T04:46:18.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.443+0000 7f1406674640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:18.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.443+0000 7f1406674640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:18.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.443+0000 7f1406674640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:18.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.443+0000 7f1406674640 1 -- 172.21.15.46:0/3532783665 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1400137500 con 0x7f1400137050 2024-09-16T04:46:18.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.443+0000 7f13fcff9640 1 --2- 172.21.15.46:0/3532783665 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f13ec038360 0x7f13ec03a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:18.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.443+0000 7f13fcff9640 1 -- 172.21.15.46:0/3532783665 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(282..282 src has 1..282) ==== 4738+0+0 (secure 0 0 0) 0x7f13f804b6d0 con 0x7f1400137050 2024-09-16T04:46:18.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.443+0000 7f13fffff640 1 --2- 172.21.15.46:0/3532783665 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f13ec038360 0x7f13ec03a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:18.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.443+0000 7f13fffff640 10 monclient: get_auth_request con 0x7f13ec038360 auth_method 0 2024-09-16T04:46:18.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.444+0000 7f13fffff640 1 --2- 172.21.15.46:0/3532783665 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f13ec038360 0x7f13ec03a820 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f13f0034000 tx=0x7f13f002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:18.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.448+0000 7f13fcff9640 1 -- 172.21.15.46:0/3532783665 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f13f8011540 con 0x7f1400137050 2024-09-16T04:46:18.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.448+0000 7f13fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:18.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.448+0000 7f13fcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:18.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.696+0000 7f1406674640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:46:18.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.696+0000 7f1406674640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:18.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.696+0000 7f1406674640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:18.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.696+0000 7f1406674640 1 -- 172.21.15.46:0/3532783665 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1400123280 con 0x7f1400137050 2024-09-16T04:46:18.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.696+0000 7f13fcff9640 1 -- 172.21.15.46:0/3532783665 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f13f8005080 con 0x7f1400137050 2024-09-16T04:46:18.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.696+0000 7f13fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:18.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.696+0000 7f13fcff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:18.699 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:46:18.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.698+0000 7f1406674640 1 -- 172.21.15.46:0/3532783665 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f13ec038360 msgr2=0x7f13ec03a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:18.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.698+0000 7f1406674640 1 --2- 172.21.15.46:0/3532783665 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f13ec038360 0x7f13ec03a820 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f13f0034000 tx=0x7f13f002e000 comp rx=0 tx=0).stop 2024-09-16T04:46:18.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.698+0000 7f1406674640 10 monclient: shutdown 2024-09-16T04:46:18.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.698+0000 7f1406674640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:18.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.698+0000 7f1406674640 1 -- 172.21.15.46:0/3532783665 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1400137050 msgr2=0x7f1400136160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:18.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.698+0000 7f1406674640 1 --2- 172.21.15.46:0/3532783665 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1400137050 0x7f1400136160 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f13f800c7a0 tx=0x7f13f800cc60 comp rx=0 tx=0).stop 2024-09-16T04:46:18.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.698+0000 7f1406674640 1 -- 172.21.15.46:0/3532783665 shutdown_connections 2024-09-16T04:46:18.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.698+0000 7f1406674640 1 --2- 172.21.15.46:0/3532783665 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1400137050 0x7f1400136160 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:18.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.698+0000 7f1406674640 1 --2- 172.21.15.46:0/3532783665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14000a2ee0 0x7f1400135c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:18.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.698+0000 7f1406674640 1 --2- 172.21.15.46:0/3532783665 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f13ec038360 0x7f13ec03a820 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:18.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.698+0000 7f1406674640 1 --2- 172.21.15.46:0/3532783665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f140006e160 0x7f14001cbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:18.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.698+0000 7f1406674640 1 -- 172.21.15.46:0/3532783665 >> 172.21.15.46:0/3532783665 conn(0x7f14000a5c10 msgr2=0x7f140009ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:18.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.698+0000 7f1406674640 1 -- 172.21.15.46:0/3532783665 shutdown_connections 2024-09-16T04:46:18.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:18.698+0000 7f1406674640 1 -- 172.21.15.46:0/3532783665 wait complete. 2024-09-16T04:46:21.712 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:46:21.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.824+0000 7fe4ce6ba640 10 monclient: get_monmap_and_config 2024-09-16T04:46:21.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.824+0000 7fe4ce6ba640 10 monclient: build_initial_monmap 2024-09-16T04:46:21.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.824+0000 7fe4ce6ba640 10 monclient: monmap: 2024-09-16T04:46:21.826 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:21.827 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:21.827 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:21.826137+0000 2024-09-16T04:46:21.827 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:21.826137+0000 2024-09-16T04:46:21.827 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:21.827 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:21.827 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:21.827 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:21.827 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:21.827 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:21.827 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:21.827 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:21.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.825+0000 7fe4ce6ba640 1 Processor -- start 2024-09-16T04:46:21.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.825+0000 7fe4ce6ba640 1 -- start start 2024-09-16T04:46:21.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.825+0000 7fe4ce6ba640 10 monclient: init 2024-09-16T04:46:21.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.825+0000 7fe4ce6ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:21.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.825+0000 7fe4ce6ba640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-16T04:46:21.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.825+0000 7fe4ce6ba640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe4c806e160 addr v1:172.21.15.46:6789/0 2024-09-16T04:46:21.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.826+0000 7fe4ce6ba640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 0x7fe4c813c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:21.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.826+0000 7fe4ce6ba640 10 monclient(hunting): picked mon.noname-b con 0x7fe4c8139c50 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:21.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.826+0000 7fe4ce6ba640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4c813c580 addr v1:172.21.15.114:6789/0 2024-09-16T04:46:21.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.826+0000 7fe4ce6ba640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4c80a3fd0 con 0x7fe4c806e160 2024-09-16T04:46:21.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.826+0000 7fe4ce6ba640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4c80a0bd0 con 0x7fe4c813c580 2024-09-16T04:46:21.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.826+0000 7fe4ce6ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:21.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.826+0000 7fe4ce6ba640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe4c806e540 con 0x7fe4c8139c50 2024-09-16T04:46:21.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.826+0000 7fe4ce6ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:21.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.826+0000 7fe4ce6ba640 10 monclient(hunting): authenticate will time out at 1180.242065s 2024-09-16T04:46:21.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.826+0000 7fe4c7fff640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7fe4c806e160 legacy=0x7fe4c80a29a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:46:21.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.826+0000 7fe4c77fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 0x7fe4c813c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:21.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.826+0000 7fe4c77fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 0x7fe4c813c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:39974/0 (socket says 172.21.15.46:39974) 2024-09-16T04:46:21.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.826+0000 7fe4c77fe640 1 -- 172.21.15.46:0/544058022 learned_addr learned my addr 172.21.15.46:0/544058022 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:21.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.826+0000 7fe4c77fe640 10 monclient(hunting): get_auth_request con 0x7fe4c8139c50 auth_method 0 2024-09-16T04:46:21.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.826+0000 7fe4c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:21.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.826+0000 7fe4c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:21.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.826+0000 7fe4c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:21.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.826+0000 7fe4c77fe640 1 -- 172.21.15.46:0/544058022 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 msgr2=0x7fe4c813c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:46:21.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.826+0000 7fe4c77fe640 1 -- 172.21.15.46:0/544058022 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 msgr2=0x7fe4c813c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:46:21.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.826+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 0x7fe4c813c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:21.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:21.826+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 0x7fe4c813c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:22.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:22.027+0000 7fe4c7fff640 1 -- 172.21.15.46:0/544058022 >> v1:172.21.15.46:6789/0 conn(0x7fe4c806e160 legacy=0x7fe4c80a29a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:46:22.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:22.027+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 0x7fe4c813c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:22.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:22.027+0000 7fe4c77fe640 10 monclient(hunting): get_auth_request con 0x7fe4c8139c50 auth_method 0 2024-09-16T04:46:22.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:22.027+0000 7fe4c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:22.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:22.027+0000 7fe4c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:22.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:22.027+0000 7fe4c77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:46:22.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:22.027+0000 7fe4c77fe640 1 -- 172.21.15.46:0/544058022 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 msgr2=0x7fe4c813c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:46:22.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:22.027+0000 7fe4c77fe640 1 -- 172.21.15.46:0/544058022 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 msgr2=0x7fe4c813c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:46:22.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:22.027+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 0x7fe4c813c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:22.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:22.027+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 0x7fe4c813c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:46:22.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:22.428+0000 7fe4c7fff640 1 -- 172.21.15.46:0/544058022 >> v1:172.21.15.46:6789/0 conn(0x7fe4c806e160 legacy=0x7fe4c80a29a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:46:22.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:22.428+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 0x7fe4c813c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:22.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:22.428+0000 7fe4c77fe640 10 monclient(hunting): get_auth_request con 0x7fe4c8139c50 auth_method 0 2024-09-16T04:46:22.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:22.428+0000 7fe4c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:22.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:22.428+0000 7fe4c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:22.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:22.428+0000 7fe4c77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:46:22.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:22.428+0000 7fe4c77fe640 1 -- 172.21.15.46:0/544058022 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 msgr2=0x7fe4c813c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:46:22.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:22.428+0000 7fe4c77fe640 1 -- 172.21.15.46:0/544058022 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 msgr2=0x7fe4c813c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:46:22.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:22.428+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 0x7fe4c813c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:22.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:22.428+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 0x7fe4c813c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:46:23.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:23.229+0000 7fe4c7fff640 1 -- 172.21.15.46:0/544058022 >> v1:172.21.15.46:6789/0 conn(0x7fe4c806e160 legacy=0x7fe4c80a29a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:46:23.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:23.229+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 0x7fe4c813c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:23.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:23.229+0000 7fe4c77fe640 10 monclient(hunting): get_auth_request con 0x7fe4c8139c50 auth_method 0 2024-09-16T04:46:23.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:23.229+0000 7fe4c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:23.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:23.229+0000 7fe4c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:23.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:23.229+0000 7fe4c77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:46:23.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:23.229+0000 7fe4c77fe640 1 -- 172.21.15.46:0/544058022 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 msgr2=0x7fe4c813c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:46:23.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:23.229+0000 7fe4c77fe640 1 -- 172.21.15.46:0/544058022 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 msgr2=0x7fe4c813c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:46:23.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:23.229+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 0x7fe4c813c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:23.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:23.229+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 0x7fe4c813c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:46:24.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c5ffb640 10 monclient(hunting): tick 2024-09-16T04:46:24.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c5ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:46:24.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c5ffb640 1 monclient(hunting): continuing hunt 2024-09-16T04:46:24.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c5ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:46:24.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c5ffb640 1 -- 172.21.15.46:0/544058022 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 msgr2=0x7fe4c813c040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:46:24.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c5ffb640 1 --2- 172.21.15.46:0/544058022 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8139c50 0x7fe4c813c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:24.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c5ffb640 1 -- 172.21.15.46:0/544058022 >> v1:172.21.15.114:6789/0 conn(0x7fe4c813c580 legacy=0x7fe4c813e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:24.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c5ffb640 1 -- 172.21.15.46:0/544058022 >> v1:172.21.15.46:6789/0 conn(0x7fe4c806e160 legacy=0x7fe4c80a29a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:46:24.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c5ffb640 10 monclient: _add_conns ranks=[3,4] 2024-09-16T04:46:24.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c5ffb640 1 --2- 172.21.15.46:0/544058022 >> v2:172.21.15.46:3300/0 conn(0x7fe4b40036f0 0x7fe4b4005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:24.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c5ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe4b40036f0 addr v2:172.21.15.46:3300/0 2024-09-16T04:46:24.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c5ffb640 1 --2- 172.21.15.46:0/544058022 >> v2:172.21.15.114:3300/0 conn(0x7fe4b4006240 0x7fe4b4008700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:24.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c5ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b4006240 addr v2:172.21.15.114:3300/0 2024-09-16T04:46:24.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c5ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:24.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c5ffb640 1 -- 172.21.15.46:0/544058022 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe4c806e540 con 0x7fe4b40036f0 2024-09-16T04:46:24.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c5ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:24.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c5ffb640 1 -- 172.21.15.46:0/544058022 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fe4b4008dc0 con 0x7fe4b4006240 2024-09-16T04:46:24.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c5ffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:24.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c7fff640 1 -- 172.21.15.46:0/544058022 >> v2:172.21.15.46:3300/0 conn(0x7fe4b40036f0 msgr2=0x7fe4b4005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:46:24.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c7fff640 1 --2- 172.21.15.46:0/544058022 >> v2:172.21.15.46:3300/0 conn(0x7fe4b40036f0 0x7fe4b4005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:24.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> v2:172.21.15.114:3300/0 conn(0x7fe4b4006240 0x7fe4b4008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:24.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c77fe640 10 monclient(hunting): get_auth_request con 0x7fe4b4006240 auth_method 0 2024-09-16T04:46:24.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:24.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:24.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:24.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c77fe640 1 -- 172.21.15.46:0/544058022 >> v2:172.21.15.114:3300/0 conn(0x7fe4b4006240 msgr2=0x7fe4b4008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:46:24.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c77fe640 1 -- 172.21.15.46:0/544058022 >> v2:172.21.15.114:3300/0 conn(0x7fe4b4006240 msgr2=0x7fe4b4008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:46:24.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> v2:172.21.15.114:3300/0 conn(0x7fe4b4006240 0x7fe4b4008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:24.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:24.826+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> v2:172.21.15.114:3300/0 conn(0x7fe4b4006240 0x7fe4b4008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:25.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.027+0000 7fe4c7fff640 1 -- 172.21.15.46:0/544058022 >> v2:172.21.15.46:3300/0 conn(0x7fe4b40036f0 msgr2=0x7fe4b4005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:46:25.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.027+0000 7fe4c7fff640 1 --2- 172.21.15.46:0/544058022 >> v2:172.21.15.46:3300/0 conn(0x7fe4b40036f0 0x7fe4b4005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:46:25.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.027+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> v2:172.21.15.114:3300/0 conn(0x7fe4b4006240 0x7fe4b4008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:25.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.027+0000 7fe4c77fe640 10 monclient(hunting): get_auth_request con 0x7fe4b4006240 auth_method 0 2024-09-16T04:46:25.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.027+0000 7fe4c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:25.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.027+0000 7fe4c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:25.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.027+0000 7fe4c77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:46:25.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.027+0000 7fe4c77fe640 1 -- 172.21.15.46:0/544058022 >> v2:172.21.15.114:3300/0 conn(0x7fe4b4006240 msgr2=0x7fe4b4008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:46:25.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.027+0000 7fe4c77fe640 1 -- 172.21.15.46:0/544058022 >> v2:172.21.15.114:3300/0 conn(0x7fe4b4006240 msgr2=0x7fe4b4008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:46:25.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.027+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> v2:172.21.15.114:3300/0 conn(0x7fe4b4006240 0x7fe4b4008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:25.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.027+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> v2:172.21.15.114:3300/0 conn(0x7fe4b4006240 0x7fe4b4008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:46:25.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.428+0000 7fe4c7fff640 1 -- 172.21.15.46:0/544058022 >> v2:172.21.15.46:3300/0 conn(0x7fe4b40036f0 msgr2=0x7fe4b4005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:46:25.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.428+0000 7fe4c7fff640 1 --2- 172.21.15.46:0/544058022 >> v2:172.21.15.46:3300/0 conn(0x7fe4b40036f0 0x7fe4b4005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:46:25.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.428+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> v2:172.21.15.114:3300/0 conn(0x7fe4b4006240 0x7fe4b4008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:25.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.428+0000 7fe4c77fe640 10 monclient(hunting): get_auth_request con 0x7fe4b4006240 auth_method 0 2024-09-16T04:46:25.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.428+0000 7fe4c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:25.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.428+0000 7fe4c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:25.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.428+0000 7fe4c77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:46:25.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.428+0000 7fe4c77fe640 1 -- 172.21.15.46:0/544058022 >> v2:172.21.15.114:3300/0 conn(0x7fe4b4006240 msgr2=0x7fe4b4008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:46:25.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.428+0000 7fe4c77fe640 1 -- 172.21.15.46:0/544058022 >> v2:172.21.15.114:3300/0 conn(0x7fe4b4006240 msgr2=0x7fe4b4008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:46:25.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.428+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> v2:172.21.15.114:3300/0 conn(0x7fe4b4006240 0x7fe4b4008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:25.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:25.428+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> v2:172.21.15.114:3300/0 conn(0x7fe4b4006240 0x7fe4b4008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:46:26.185 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:26.186 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39853]: starting. 2024-09-16T04:46:26.186 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39853]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-39817 2024-09-16T04:46:26.186 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39853]: created object 0... 2024-09-16T04:46:26.186 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39853]: created object 25... 2024-09-16T04:46:26.186 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39853]: created object 49... 2024-09-16T04:46:26.186 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39853]: finishing. 2024-09-16T04:46:26.186 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39853]: shutting down. 2024-09-16T04:46:26.198 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:26.198 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39855]: starting. 2024-09-16T04:46:26.198 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39855]: listing objects. 2024-09-16T04:46:26.198 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39855]: listed object 0... 2024-09-16T04:46:26.198 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39855]: listed object 25... 2024-09-16T04:46:26.198 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39855]: saw 50 objects 2024-09-16T04:46:26.198 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39855]: shutting down. 2024-09-16T04:46:26.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.229+0000 7fe4c7fff640 1 -- 172.21.15.46:0/544058022 >> v2:172.21.15.46:3300/0 conn(0x7fe4b40036f0 msgr2=0x7fe4b4005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:46:26.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.229+0000 7fe4c7fff640 1 --2- 172.21.15.46:0/544058022 >> v2:172.21.15.46:3300/0 conn(0x7fe4b40036f0 0x7fe4b4005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:46:26.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.229+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> v2:172.21.15.114:3300/0 conn(0x7fe4b4006240 0x7fe4b4008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:26.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.229+0000 7fe4c77fe640 10 monclient(hunting): get_auth_request con 0x7fe4b4006240 auth_method 0 2024-09-16T04:46:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.229+0000 7fe4c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.229+0000 7fe4c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.229+0000 7fe4c77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:46:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.229+0000 7fe4c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.229+0000 7fe4c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.229+0000 7fe4c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.229+0000 7fe4c77fe640 10 monclient(hunting): handle_auth_done global_id 6775 payload 274 2024-09-16T04:46:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.229+0000 7fe4c77fe640 1 -- 172.21.15.46:0/544058022 >> v2:172.21.15.46:3300/0 conn(0x7fe4b40036f0 msgr2=0x7fe4b4005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:46:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.229+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> v2:172.21.15.46:3300/0 conn(0x7fe4b40036f0 0x7fe4b4005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.229+0000 7fe4c77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:46:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.229+0000 7fe4c77fe640 1 monclient: found mon.noname-c 2024-09-16T04:46:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.229+0000 7fe4c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.229+0000 7fe4c77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:46:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.229+0000 7fe4c77fe640 1 -- 172.21.15.46:0/544058022 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c80a0eb0 con 0x7fe4b4006240 2024-09-16T04:46:26.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.230+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/544058022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4b4006240 0x7fe4b4008700 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4006210 tx=0x7fe4bc02be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53499e573f4fe7d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:26.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.230+0000 7fe4c6ffd640 1 -- 172.21.15.46:0/544058022 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc032070 con 0x7fe4b4006240 2024-09-16T04:46:26.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.230+0000 7fe4c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:26.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.230+0000 7fe4c6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.230+0000 7fe4c6ffd640 10 monclient: dump: 2024-09-16T04:46:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:26.235 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.230+0000 7fe4c6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:46:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.230+0000 7fe4c6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:46:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.230+0000 7fe4c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.230+0000 7fe4c6ffd640 1 -- 172.21.15.46:0/544058022 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4bc030070 con 0x7fe4b4006240 2024-09-16T04:46:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.230+0000 7fe4c6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.230+0000 7fe4c6ffd640 1 -- 172.21.15.46:0/544058022 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc00f040 con 0x7fe4b4006240 2024-09-16T04:46:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.230+0000 7fe4ce6ba640 5 monclient: authenticate success, global_id 6775 2024-09-16T04:46:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.230+0000 7fe4ce6ba640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.230+0000 7fe4c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.230+0000 7fe4ce6ba640 4 set_mon_vals no callback set 2024-09-16T04:46:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.230+0000 7fe4c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:46:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.230+0000 7fe4c6ffd640 10 monclient: dump: 2024-09-16T04:46:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:26.237 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.231+0000 7fe4ce6ba640 10 monclient: shutdown 2024-09-16T04:46:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.231+0000 7fe4ce6ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.231+0000 7fe4ce6ba640 1 -- 172.21.15.46:0/544058022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4b4006240 msgr2=0x7fe4b4008700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.231+0000 7fe4ce6ba640 1 --2- 172.21.15.46:0/544058022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4b4006240 0x7fe4b4008700 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4006210 tx=0x7fe4bc02be40 comp rx=0 tx=0).stop 2024-09-16T04:46:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.231+0000 7fe4ccc57640 1 -- 172.21.15.46:0/544058022 reap_dead start 2024-09-16T04:46:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.231+0000 7fe4ce6ba640 1 -- 172.21.15.46:0/544058022 shutdown_connections 2024-09-16T04:46:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.231+0000 7fe4ce6ba640 1 --2- 172.21.15.46:0/544058022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4b4006240 0x7fe4b4008700 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.231+0000 7fe4ce6ba640 1 -- 172.21.15.46:0/544058022 >> 172.21.15.46:0/544058022 conn(0x7fe4c80a5c10 msgr2=0x7fe4c80a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.231+0000 7fe4ce6ba640 1 -- 172.21.15.46:0/544058022 shutdown_connections 2024-09-16T04:46:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.231+0000 7fe4ce6ba640 1 -- 172.21.15.46:0/544058022 wait complete. 2024-09-16T04:46:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.232+0000 7fe4ce6ba640 10 monclient: build_initial_monmap 2024-09-16T04:46:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.232+0000 7fe4ce6ba640 10 monclient: monmap: 2024-09-16T04:46:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:26.239 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.232+0000 7fe4ce6ba640 1 Processor -- start 2024-09-16T04:46:26.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ce6ba640 1 -- start start 2024-09-16T04:46:26.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ce6ba640 10 monclient: init 2024-09-16T04:46:26.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ce6ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:26.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ce6ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:46:26.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ce6ba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4b4006240 0x7fe4c8136c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ce6ba640 10 monclient(hunting): picked mon.noname-a con 0x7fe4b4006240 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:26.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ce6ba640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8137140 0x7fe4c811f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ce6ba640 10 monclient(hunting): picked mon.noname-c con 0x7fe4c8137140 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:26.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ce6ba640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4c811fca0 0x7fe4c8120120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ce6ba640 10 monclient(hunting): picked mon.noname-b con 0x7fe4c811fca0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:26.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ce6ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:26.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ce6ba640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c80a34a0 con 0x7fe4b4006240 2024-09-16T04:46:26.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ce6ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:26.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ce6ba640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c80a4d90 con 0x7fe4c811fca0 2024-09-16T04:46:26.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ce6ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:26.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ce6ba640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe4c80a80b0 con 0x7fe4c8137140 2024-09-16T04:46:26.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ce6ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:26.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4c7fff640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4b4006240 msgr2=0x7fe4c8136c00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:46:26.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ce6ba640 10 monclient(hunting): authenticate will time out at 1184.649604s 2024-09-16T04:46:26.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4c7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4b4006240 0x7fe4c8136c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:26.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4c77fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8137140 0x7fe4c811f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:26.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4c77fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8137140 0x7fe4c811f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:59520/0 (socket says 172.21.15.46:59520) 2024-09-16T04:46:26.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4c77fe640 1 -- 172.21.15.46:0/537938554 learned_addr learned my addr 172.21.15.46:0/537938554 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:26.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4c77fe640 10 monclient(hunting): get_auth_request con 0x7fe4c8137140 auth_method 0 2024-09-16T04:46:26.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:26.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:26.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:26.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ccc57640 1 --2- 172.21.15.46:0/537938554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4c811fca0 0x7fe4c8120120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:26.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:26.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:26.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:26.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ccc57640 10 monclient(hunting): get_auth_request con 0x7fe4c811fca0 auth_method 0 2024-09-16T04:46:26.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ccc57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:26.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ccc57640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:26.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ccc57640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:26.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ccc57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:26.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ccc57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:26.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.233+0000 7fe4ccc57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:26.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c77fe640 10 monclient(hunting): handle_auth_done global_id 94178 payload 824 2024-09-16T04:46:26.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c77fe640 1 -- 172.21.15.46:0/537938554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4c811fca0 msgr2=0x7fe4c8120120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:26.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/537938554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4c811fca0 0x7fe4c8120120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c77fe640 1 -- 172.21.15.46:0/537938554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4b4006240 msgr2=0x7fe4c8136c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:46:26.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/537938554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4b4006240 0x7fe4c8136c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:46:26.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c77fe640 1 monclient: found mon.noname-c 2024-09-16T04:46:26.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:26.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:46:26.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c77fe640 1 -- 172.21.15.46:0/537938554 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c81cc5c0 con 0x7fe4c8137140 2024-09-16T04:46:26.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c77fe640 1 --2- 172.21.15.46:0/537938554 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8137140 0x7fe4c811f760 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc000c00 tx=0x7fe4bc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:26.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c4ff9640 1 -- 172.21.15.46:0/537938554 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc047070 con 0x7fe4c8137140 2024-09-16T04:46:26.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:26.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:26.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c4ff9640 10 monclient: dump: 2024-09-16T04:46:26.247 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:26.247 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:26.247 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.247 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.247 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:26.247 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.248 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:26.248 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:26.248 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:26.248 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c4ff9640 10 monclient: _finish_auth 0 2024-09-16T04:46:26.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:46:26.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:26.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c4ff9640 1 -- 172.21.15.46:0/537938554 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4bc003610 con 0x7fe4c8137140 2024-09-16T04:46:26.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4ce6ba640 5 monclient: authenticate success, global_id 94178 2024-09-16T04:46:26.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4ce6ba640 10 monclient: _renew_subs 2024-09-16T04:46:26.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4ce6ba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:26.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4ce6ba640 1 -- 172.21.15.46:0/537938554 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4c81c1b30 con 0x7fe4c8137140 2024-09-16T04:46:26.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:26.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c4ff9640 1 -- 172.21.15.46:0/537938554 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc00f040 con 0x7fe4c8137140 2024-09-16T04:46:26.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:26.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:26.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c4ff9640 10 monclient: dump: 2024-09-16T04:46:26.249 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:26.249 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:26.249 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.249 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.249 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:26.249 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.249 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:26.249 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:26.249 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:26.250 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4c67fc640 4 set_mon_vals no callback set 2024-09-16T04:46:26.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4ce6ba640 10 monclient: _renew_subs 2024-09-16T04:46:26.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4ce6ba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:26.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.234+0000 7fe4ce6ba640 1 -- 172.21.15.46:0/537938554 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4c8123440 con 0x7fe4c8137140 2024-09-16T04:46:26.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.235+0000 7fe4c4ff9640 1 -- 172.21.15.46:0/537938554 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fe4bc04a070 con 0x7fe4c8137140 2024-09-16T04:46:26.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.235+0000 7fe4ce6ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:26.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.235+0000 7fe4ce6ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:26.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.235+0000 7fe4ce6ba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:26.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.235+0000 7fe4ce6ba640 1 -- 172.21.15.46:0/537938554 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4c80a29a0 con 0x7fe4c8137140 2024-09-16T04:46:26.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.235+0000 7fe4c4ff9640 1 --2- 172.21.15.46:0/537938554 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe4b40390f0 0x7fe4b403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.235+0000 7fe4c4ff9640 1 -- 172.21.15.46:0/537938554 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(283..283 src has 1..283) ==== 4738+0+0 (secure 0 0 0) 0x7fe4bc06cef0 con 0x7fe4c8137140 2024-09-16T04:46:26.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.236+0000 7fe4c7fff640 1 --2- 172.21.15.46:0/537938554 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe4b40390f0 0x7fe4b403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:26.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.236+0000 7fe4c7fff640 10 monclient: get_auth_request con 0x7fe4b40390f0 auth_method 0 2024-09-16T04:46:26.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.236+0000 7fe4c7fff640 1 --2- 172.21.15.46:0/537938554 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe4b40390f0 0x7fe4b403b5b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8003980 tx=0x7fe4b8009060 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:26.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.240+0000 7fe4c4ff9640 1 -- 172.21.15.46:0/537938554 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe4c81c1b30 con 0x7fe4c8137140 2024-09-16T04:46:26.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.240+0000 7fe4c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:26.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.240+0000 7fe4c4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:26.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.483+0000 7fe4ce6ba640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:46:26.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.483+0000 7fe4ce6ba640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:26.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.483+0000 7fe4ce6ba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:26.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.483+0000 7fe4ce6ba640 1 -- 172.21.15.46:0/537938554 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe4c8121a30 con 0x7fe4c8137140 2024-09-16T04:46:26.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.484+0000 7fe4c4ff9640 1 -- 172.21.15.46:0/537938554 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fe4bc0039c0 con 0x7fe4c8137140 2024-09-16T04:46:26.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.484+0000 7fe4c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:26.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.484+0000 7fe4c4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:26.487 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":90,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:46:26.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.486+0000 7fe4ce6ba640 1 -- 172.21.15.46:0/537938554 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe4b40390f0 msgr2=0x7fe4b403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:26.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.486+0000 7fe4ce6ba640 1 --2- 172.21.15.46:0/537938554 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe4b40390f0 0x7fe4b403b5b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8003980 tx=0x7fe4b8009060 comp rx=0 tx=0).stop 2024-09-16T04:46:26.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.486+0000 7fe4ce6ba640 10 monclient: shutdown 2024-09-16T04:46:26.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.486+0000 7fe4ce6ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:26.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.486+0000 7fe4ce6ba640 1 -- 172.21.15.46:0/537938554 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8137140 msgr2=0x7fe4c811f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:26.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.486+0000 7fe4ce6ba640 1 --2- 172.21.15.46:0/537938554 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8137140 0x7fe4c811f760 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc000c00 tx=0x7fe4bc004730 comp rx=0 tx=0).stop 2024-09-16T04:46:26.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.486+0000 7fe4ce6ba640 1 -- 172.21.15.46:0/537938554 shutdown_connections 2024-09-16T04:46:26.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.486+0000 7fe4ce6ba640 1 --2- 172.21.15.46:0/537938554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4c811fca0 0x7fe4c8120120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.486+0000 7fe4ce6ba640 1 --2- 172.21.15.46:0/537938554 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe4c8137140 0x7fe4c811f760 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.486+0000 7fe4ce6ba640 1 --2- 172.21.15.46:0/537938554 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe4b40390f0 0x7fe4b403b5b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.486+0000 7fe4ce6ba640 1 --2- 172.21.15.46:0/537938554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4b4006240 0x7fe4c8136c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.486+0000 7fe4ce6ba640 1 -- 172.21.15.46:0/537938554 >> 172.21.15.46:0/537938554 conn(0x7fe4c80a5c10 msgr2=0x7fe4c8137ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:26.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.486+0000 7fe4ce6ba640 1 -- 172.21.15.46:0/537938554 shutdown_connections 2024-09-16T04:46:26.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.486+0000 7fe4ce6ba640 1 -- 172.21.15.46:0/537938554 wait complete. 2024-09-16T04:46:26.499 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:46:26.499 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T04:46:26.499 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:46:26.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.599+0000 7f44b27c4640 10 monclient: get_monmap_and_config 2024-09-16T04:46:26.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.599+0000 7f44b27c4640 10 monclient: build_initial_monmap 2024-09-16T04:46:26.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.599+0000 7f44b27c4640 10 monclient: monmap: 2024-09-16T04:46:26.602 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:26.602 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:26.602 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:26.601385+0000 2024-09-16T04:46:26.602 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:26.601385+0000 2024-09-16T04:46:26.602 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:26.602 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.603 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:26.603 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:26.603 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:26.603 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:26.603 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:26.603 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.600+0000 7f44b27c4640 1 Processor -- start 2024-09-16T04:46:26.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.600+0000 7f44b27c4640 1 -- start start 2024-09-16T04:46:26.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.600+0000 7f44b27c4640 10 monclient: init 2024-09-16T04:46:26.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.601+0000 7f44b27c4640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:26.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.601+0000 7f44b27c4640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-16T04:46:26.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.601+0000 7f44b27c4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44ac09f810 addr v1:172.21.15.46:6789/0 2024-09-16T04:46:26.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.601+0000 7f44b27c4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f44ac0a0300 0x7f44ac09e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.601+0000 7f44b27c4640 10 monclient(hunting): picked mon.noname-a con 0x7f44ac0a0300 addr v2:172.21.15.46:3300/0 2024-09-16T04:46:26.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.601+0000 7f44abfff640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f44ac09f810 legacy=0x7f44ac09fbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:46:26.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.601+0000 7f44b27c4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44ac138610 addr v1:172.21.15.114:6789/0 2024-09-16T04:46:26.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.601+0000 7f44b27c4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44ac09c310 con 0x7f44ac09f810 2024-09-16T04:46:26.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.601+0000 7f44ab7fe640 1 -- >> v2:172.21.15.46:3300/0 conn(0x7f44ac0a0300 msgr2=0x7f44ac09e2c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:46:26.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.601+0000 7f44b27c4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44ac09c840 con 0x7f44ac138610 2024-09-16T04:46:26.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.601+0000 7f44b27c4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:26.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.601+0000 7f44b27c4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f44ac09d290 con 0x7f44ac0a0300 2024-09-16T04:46:26.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.601+0000 7f44ab7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f44ac0a0300 0x7f44ac09e2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:26.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.601+0000 7f44b27c4640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:26.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.601+0000 7f44b27c4640 10 monclient(hunting): authenticate will time out at 1185.017422s 2024-09-16T04:46:26.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.601+0000 7f44b0d61640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f44ac138610 0x7f44ac13aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.46:57696/0 (socket says 172.21.15.46:57696) 2024-09-16T04:46:26.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.601+0000 7f44b0d61640 1 -- 172.21.15.46:0/3775078029 learned_addr learned my addr 172.21.15.46:0/3775078029 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:46:26.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 1 -- 172.21.15.46:0/3775078029 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2870262927 0 0) 0x7f44ac09c840 con 0x7f44ac138610 2024-09-16T04:46:26.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:46:26.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:26.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:26.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 10 monclient(hunting): my global_id is 6781 2024-09-16T04:46:26.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 1 -- 172.21.15.46:0/3775078029 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4494003520 con 0x7f44ac138610 2024-09-16T04:46:26.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 1 -- 172.21.15.46:0/3775078029 <== mon.1 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1080274245 0 0) 0x7f4494003520 con 0x7f44ac138610 2024-09-16T04:46:26.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 1 -- 172.21.15.46:0/3775078029 >> v2:172.21.15.46:3300/0 conn(0x7f44ac0a0300 msgr2=0x7f44ac09e2c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:46:26.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 1 --2- 172.21.15.46:0/3775078029 >> v2:172.21.15.46:3300/0 conn(0x7f44ac0a0300 0x7f44ac09e2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 1 -- 172.21.15.46:0/3775078029 >> v1:172.21.15.46:6789/0 conn(0x7f44ac09f810 legacy=0x7f44ac09fbf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:46:26.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 10 monclient: _finish_hunting 0 2024-09-16T04:46:26.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 1 monclient: found mon.noname-c-legacy 2024-09-16T04:46:26.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:26.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.114:6789/0 2024-09-16T04:46:26.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 1 -- 172.21.15.46:0/3775078029 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44ac09d850 con 0x7f44ac138610 2024-09-16T04:46:26.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 10 monclient: _finish_auth 0 2024-09-16T04:46:26.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 10 monclient: _check_auth_tickets 2024-09-16T04:46:26.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:26.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 1 -- 172.21.15.46:0/3775078029 <== mon.1 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f44a40019f0 con 0x7f44ac138610 2024-09-16T04:46:26.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:26.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T04:46:26.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 10 monclient: dump: 2024-09-16T04:46:26.619 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:26.619 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:26.619 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.619 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.619 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:26.619 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.619 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:26.619 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:26.619 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:26.619 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T04:46:26.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:26.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 1 -- 172.21.15.46:0/3775078029 >> v1:172.21.15.114:6789/0 conn(0x7f44ac138610 legacy=0x7f44ac13aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:26.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:46:26.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 1 --2- 172.21.15.46:0/3775078029 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44940039b0 0x7f4494003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 10 monclient(hunting): picked mon.c con 0x7f44940039b0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:26.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 1 --2- 172.21.15.46:0/3775078029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4494004800 0x7f4494004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 10 monclient(hunting): picked mon.b con 0x7f4494004800 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:26.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 1 --2- 172.21.15.46:0/3775078029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44940052e0 0x7f449400cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 10 monclient(hunting): picked mon.a con 0x7f44940052e0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:26.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:26.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 1 -- 172.21.15.46:0/3775078029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f449400d2c0 con 0x7f44940052e0 2024-09-16T04:46:26.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:26.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 1 -- 172.21.15.46:0/3775078029 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f449400d470 con 0x7f4494004800 2024-09-16T04:46:26.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:26.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 1 -- 172.21.15.46:0/3775078029 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f449400d670 con 0x7f44940039b0 2024-09-16T04:46:26.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:26.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44aaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:46:26.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44b0d61640 1 -- 172.21.15.46:0/3775078029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44940052e0 msgr2=0x7f449400cad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:46:26.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44b0d61640 1 --2- 172.21.15.46:0/3775078029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44940052e0 0x7f449400cad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:26.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44abfff640 1 --2- 172.21.15.46:0/3775078029 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44940039b0 0x7f4494003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:26.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44ab7fe640 1 --2- 172.21.15.46:0/3775078029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4494004800 0x7f4494004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:26.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44abfff640 10 monclient(hunting): get_auth_request con 0x7f44940039b0 auth_method 0 2024-09-16T04:46:26.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:26.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44abfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:26.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44abfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:46:26.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44ab7fe640 10 monclient(hunting): get_auth_request con 0x7f4494004800 auth_method 0 2024-09-16T04:46:26.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:26.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:26.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44ab7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:46:26.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:26.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:26.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:46:26.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:26.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:26.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44abfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:46:26.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44ab7fe640 10 monclient(hunting): handle_auth_done global_id 6781 payload 293 2024-09-16T04:46:26.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44ab7fe640 1 -- 172.21.15.46:0/3775078029 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44940039b0 msgr2=0x7f4494003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:26.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44ab7fe640 1 --2- 172.21.15.46:0/3775078029 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44940039b0 0x7f4494003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44ab7fe640 1 -- 172.21.15.46:0/3775078029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44940052e0 msgr2=0x7f449400cad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:46:26.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44ab7fe640 1 --2- 172.21.15.46:0/3775078029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44940052e0 0x7f449400cad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44ab7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:46:26.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44ab7fe640 1 monclient: found mon.b 2024-09-16T04:46:26.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:26.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44ab7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.114:3300/0 2024-09-16T04:46:26.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44ab7fe640 1 -- 172.21.15.46:0/3775078029 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4494003520 con 0x7f4494004800 2024-09-16T04:46:26.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44ab7fe640 10 monclient: _finish_auth 0 2024-09-16T04:46:26.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44ab7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:46:26.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44b0d61640 1 -- 172.21.15.46:0/3775078029 reap_dead start 2024-09-16T04:46:26.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:26.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44b27c4640 5 monclient: authenticate success, global_id 6781 2024-09-16T04:46:26.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.602+0000 7f44b27c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:26.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.603+0000 7f44ab7fe640 1 --2- 172.21.15.46:0/3775078029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4494004800 0x7f4494004c80 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f44a0003980 tx=0x7f4494005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=767d1ad844e84dde server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:26.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.603+0000 7f44aaffd640 1 -- 172.21.15.46:0/3775078029 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a0009500 con 0x7f4494004800 2024-09-16T04:46:26.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.603+0000 7f44aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:26.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.603+0000 7f44aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:46:26.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.603+0000 7f44aaffd640 10 monclient: dump: 2024-09-16T04:46:26.627 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:26.627 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:26.627 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.627 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.628 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:26.628 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.628 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:26.628 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:26.628 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:26.628 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.603+0000 7f44b27c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:26.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.603+0000 7f44aaffd640 1 -- 172.21.15.46:0/3775078029 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44a002e070 con 0x7f4494004800 2024-09-16T04:46:26.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.603+0000 7f44aaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:26.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.603+0000 7f44b27c4640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:26.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.603+0000 7f44aaffd640 1 -- 172.21.15.46:0/3775078029 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a0003df0 con 0x7f4494004800 2024-09-16T04:46:26.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.603+0000 7f44aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:26.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.603+0000 7f44b27c4640 4 set_mon_vals no callback set 2024-09-16T04:46:26.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.603+0000 7f44aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:46:26.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.603+0000 7f44aaffd640 10 monclient: dump: 2024-09-16T04:46:26.629 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:26.629 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:26.629 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.630 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.630 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:26.630 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.630 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:26.630 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:26.630 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:26.630 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.604+0000 7f44b27c4640 10 monclient: shutdown 2024-09-16T04:46:26.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.604+0000 7f44b27c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:26.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.604+0000 7f44b27c4640 1 -- 172.21.15.46:0/3775078029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4494004800 msgr2=0x7f4494004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:26.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.604+0000 7f44b27c4640 1 --2- 172.21.15.46:0/3775078029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4494004800 0x7f4494004c80 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f44a0003980 tx=0x7f4494005830 comp rx=0 tx=0).stop 2024-09-16T04:46:26.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.604+0000 7f44b27c4640 1 -- 172.21.15.46:0/3775078029 shutdown_connections 2024-09-16T04:46:26.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.604+0000 7f44b27c4640 1 --2- 172.21.15.46:0/3775078029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4494004800 0x7f4494004c80 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.604+0000 7f44b27c4640 1 -- 172.21.15.46:0/3775078029 >> 172.21.15.46:0/3775078029 conn(0x7f44ac0a5960 msgr2=0x7f44ac0a1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:26.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.604+0000 7f44b27c4640 1 -- 172.21.15.46:0/3775078029 shutdown_connections 2024-09-16T04:46:26.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.604+0000 7f44b27c4640 1 -- 172.21.15.46:0/3775078029 wait complete. 2024-09-16T04:46:26.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.604+0000 7f44b27c4640 10 monclient: build_initial_monmap 2024-09-16T04:46:26.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.604+0000 7f44b27c4640 10 monclient: monmap: 2024-09-16T04:46:26.633 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:26.633 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:26.633 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:26.633 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:26.634 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.605+0000 7f44b27c4640 1 Processor -- start 2024-09-16T04:46:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.605+0000 7f44b27c4640 1 -- start start 2024-09-16T04:46:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.605+0000 7f44b27c4640 10 monclient: init 2024-09-16T04:46:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.605+0000 7f44b27c4640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.605+0000 7f44b27c4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:46:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.605+0000 7f44b27c4640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4494004800 0x7f44ac131840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.605+0000 7f44b27c4640 10 monclient(hunting): picked mon.noname-c con 0x7f4494004800 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.605+0000 7f44b27c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44ac131d80 0x7f44ac1cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.605+0000 7f44b27c4640 10 monclient(hunting): picked mon.noname-a con 0x7f44ac131d80 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:26.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.605+0000 7f44b27c4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44ac1cbf00 0x7f44ac1cc380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.605+0000 7f44b27c4640 10 monclient(hunting): picked mon.noname-b con 0x7f44ac1cbf00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:26.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.605+0000 7f44b27c4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:26.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.605+0000 7f44b27c4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f44ac09e2c0 con 0x7f44ac131d80 2024-09-16T04:46:26.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.605+0000 7f44b27c4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:26.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.605+0000 7f44b27c4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f44ac09f090 con 0x7f44ac1cbf00 2024-09-16T04:46:26.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.605+0000 7f44b27c4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:26.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.605+0000 7f44b27c4640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f44ac09e440 con 0x7f4494004800 2024-09-16T04:46:26.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.605+0000 7f44b27c4640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:26.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.605+0000 7f44b27c4640 10 monclient(hunting): authenticate will time out at 1185.021980s 2024-09-16T04:46:26.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.605+0000 7f44ab7fe640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44ac131d80 msgr2=0x7f44ac1cb9c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:46:26.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44ab7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44ac131d80 0x7f44ac1cb9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:26.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44abfff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4494004800 0x7f44ac131840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:26.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44abfff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4494004800 0x7f44ac131840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:59542/0 (socket says 172.21.15.46:59542) 2024-09-16T04:46:26.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44abfff640 1 -- 172.21.15.46:0/1991587361 learned_addr learned my addr 172.21.15.46:0/1991587361 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:26.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44abfff640 10 monclient(hunting): get_auth_request con 0x7f4494004800 auth_method 0 2024-09-16T04:46:26.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:26.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44abfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:26.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:26.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44b0d61640 1 --2- 172.21.15.46:0/1991587361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44ac1cbf00 0x7f44ac1cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:26.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44b0d61640 10 monclient(hunting): get_auth_request con 0x7f44ac1cbf00 auth_method 0 2024-09-16T04:46:26.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44b0d61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:26.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44b0d61640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:26.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44b0d61640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:26.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:26.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44b0d61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44b0d61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44b0d61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44abfff640 10 monclient(hunting): handle_auth_done global_id 94181 payload 824 2024-09-16T04:46:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44abfff640 1 -- 172.21.15.46:0/1991587361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44ac1cbf00 msgr2=0x7f44ac1cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44abfff640 1 --2- 172.21.15.46:0/1991587361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44ac1cbf00 0x7f44ac1cc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44abfff640 1 -- 172.21.15.46:0/1991587361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44ac131d80 msgr2=0x7f44ac1cb9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:46:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44abfff640 1 --2- 172.21.15.46:0/1991587361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44ac131d80 0x7f44ac1cb9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44abfff640 10 monclient: _finish_hunting 0 2024-09-16T04:46:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44abfff640 1 monclient: found mon.noname-c 2024-09-16T04:46:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44abfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:46:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44abfff640 1 -- 172.21.15.46:0/1991587361 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44ac132200 con 0x7f4494004800 2024-09-16T04:46:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44b0d61640 1 --2- 172.21.15.46:0/1991587361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44ac1cbf00 0x7f44ac1cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44abfff640 1 --2- 172.21.15.46:0/1991587361 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4494004800 0x7f44ac131840 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f449c003fd0 tx=0x7f449c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44a8ff9640 1 -- 172.21.15.46:0/1991587361 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f449c016070 con 0x7f4494004800 2024-09-16T04:46:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44a8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44a8ff9640 10 monclient: dump: 2024-09-16T04:46:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:26.639 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44a8ff9640 10 monclient: _finish_auth 0 2024-09-16T04:46:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44a8ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:46:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44a8ff9640 1 -- 172.21.15.46:0/1991587361 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f449c00e7b0 con 0x7f4494004800 2024-09-16T04:46:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44b27c4640 5 monclient: authenticate success, global_id 94181 2024-09-16T04:46:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44a8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44b27c4640 10 monclient: _renew_subs 2024-09-16T04:46:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44a8ff9640 1 -- 172.21.15.46:0/1991587361 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f449c00eba0 con 0x7f4494004800 2024-09-16T04:46:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44b27c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44b27c4640 1 -- 172.21.15.46:0/1991587361 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44ac121f50 con 0x7f4494004800 2024-09-16T04:46:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44a8ff9640 10 monclient: dump: 2024-09-16T04:46:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:26.641 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44a9ffb640 4 set_mon_vals no callback set 2024-09-16T04:46:26.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44b27c4640 10 monclient: _renew_subs 2024-09-16T04:46:26.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.606+0000 7f44b27c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:26.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.607+0000 7f44b27c4640 1 -- 172.21.15.46:0/1991587361 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f44ac0a4410 con 0x7f4494004800 2024-09-16T04:46:26.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.607+0000 7f44a8ff9640 1 -- 172.21.15.46:0/1991587361 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f449c019070 con 0x7f4494004800 2024-09-16T04:46:26.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.607+0000 7f44b27c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:26.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.607+0000 7f44b27c4640 1 --2- 172.21.15.46:0/1991587361 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44ac121990 0x7f44ac02d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.607+0000 7f44b27c4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:26.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.607+0000 7f44b27c4640 1 -- 172.21.15.46:0/1991587361 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f44ac09c160 con 0x7f44ac121990 2024-09-16T04:46:26.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.607+0000 7f44a8ff9640 1 --2- 172.21.15.46:0/1991587361 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f44980382a0 0x7f449803a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.607+0000 7f44a8ff9640 1 -- 172.21.15.46:0/1991587361 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(283..283 src has 1..283) ==== 4738+0+0 (secure 0 0 0) 0x7f449c04b210 con 0x7f4494004800 2024-09-16T04:46:26.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.607+0000 7f44ab7fe640 1 --2- 172.21.15.46:0/1991587361 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44ac121990 0x7f44ac02d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:26.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.607+0000 7f44ab7fe640 10 monclient: get_auth_request con 0x7f44ac121990 auth_method 0 2024-09-16T04:46:26.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.607+0000 7f44ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:26.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.607+0000 7f44ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:26.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.607+0000 7f44ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:26.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.607+0000 7f44b0d61640 1 --2- 172.21.15.46:0/1991587361 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f44980382a0 0x7f449803a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:26.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.607+0000 7f44b0d61640 10 monclient: get_auth_request con 0x7f44980382a0 auth_method 0 2024-09-16T04:46:26.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.608+0000 7f44ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:26.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.608+0000 7f44ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:26.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.608+0000 7f44ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:26.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.608+0000 7f44ab7fe640 10 monclient(hunting): handle_auth_done global_id 94184 payload 824 2024-09-16T04:46:26.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.608+0000 7f44ab7fe640 1 -- 172.21.15.46:0/1991587361 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f44ac09fbf0 con 0x7f44ac121990 2024-09-16T04:46:26.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.608+0000 7f44b0d61640 1 --2- 172.21.15.46:0/1991587361 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f44980382a0 0x7f449803a760 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f44ac09bf60 tx=0x7f44a4007ef0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:26.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.608+0000 7f44ab7fe640 1 --2- 172.21.15.46:0/1991587361 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44ac121990 0x7f44ac02d7c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f44940057e0 tx=0x7f44a0005a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:26.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.608+0000 7f44a8ff9640 1 -- 172.21.15.46:0/1991587361 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a0005270 con 0x7f44ac121990 2024-09-16T04:46:26.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.608+0000 7f44a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:26.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.608+0000 7f44a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:26.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.608+0000 7f44a8ff9640 10 monclient: dump: 2024-09-16T04:46:26.643 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:26.643 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:26.643 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.643 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.643 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:26.643 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.644 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:26.644 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:26.644 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:26.644 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.608+0000 7f44a8ff9640 1 -- 172.21.15.46:0/1991587361 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f44a002e040 con 0x7f44ac121990 2024-09-16T04:46:26.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.608+0000 7f44a8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:26.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.608+0000 7f44a8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:26.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.608+0000 7f44a8ff9640 1 -- 172.21.15.46:0/1991587361 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44ac121990 msgr2=0x7f44ac02d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:26.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.608+0000 7f44a8ff9640 1 --2- 172.21.15.46:0/1991587361 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44ac121990 0x7f44ac02d7c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f44940057e0 tx=0x7f44a0005a80 comp rx=0 tx=0).stop 2024-09-16T04:46:26.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.608+0000 7f44a8ff9640 1 -- 172.21.15.46:0/1991587361 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44ac121990 msgr2=0x7f44ac02d7c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:26.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.608+0000 7f44a8ff9640 1 --2- 172.21.15.46:0/1991587361 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44ac121990 0x7f44ac02d7c0 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f44940057e0 tx=0x7f44a0005a80 comp rx=0 tx=0).stop 2024-09-16T04:46:26.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44b27c4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:46:26.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44b27c4640 1 --2- 172.21.15.46:0/1991587361 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44ac062df0 0x7f44ac063200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44b27c4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:26.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44b27c4640 1 -- 172.21.15.46:0/1991587361 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f44ac1372c0 con 0x7f44ac062df0 2024-09-16T04:46:26.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44ab7fe640 1 --2- 172.21.15.46:0/1991587361 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44ac062df0 0x7f44ac063200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:26.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44ab7fe640 10 monclient: get_auth_request con 0x7f44ac062df0 auth_method 0 2024-09-16T04:46:26.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:26.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:26.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:26.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:26.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:26.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:26.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44ab7fe640 10 monclient(hunting): handle_auth_done global_id 94187 payload 824 2024-09-16T04:46:26.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44ab7fe640 1 -- 172.21.15.46:0/1991587361 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f44ac06e4c0 con 0x7f44ac062df0 2024-09-16T04:46:26.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44ab7fe640 1 --2- 172.21.15.46:0/1991587361 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44ac062df0 0x7f44ac063200 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f44a0003fd0 tx=0x7f44a0005a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:26.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44a8ff9640 1 -- 172.21.15.46:0/1991587361 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a0046670 con 0x7f44ac062df0 2024-09-16T04:46:26.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:26.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:26.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44a8ff9640 10 monclient: dump: 2024-09-16T04:46:26.649 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:26.649 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:26.649 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.649 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.650 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:26.650 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.650 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:26.650 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:26.650 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:26.650 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44a8ff9640 1 -- 172.21.15.46:0/1991587361 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4533 (secure 0 0 0) 0x7f44a0005fb0 con 0x7f44ac062df0 2024-09-16T04:46:26.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44a8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:46:26.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44a8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:26.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44a8ff9640 1 -- 172.21.15.46:0/1991587361 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44ac062df0 msgr2=0x7f44ac063200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:26.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44a8ff9640 1 --2- 172.21.15.46:0/1991587361 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44ac062df0 0x7f44ac063200 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f44a0003fd0 tx=0x7f44a0005a10 comp rx=0 tx=0).stop 2024-09-16T04:46:26.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44a8ff9640 1 -- 172.21.15.46:0/1991587361 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44ac062df0 msgr2=0x7f44ac063200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:26.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.615+0000 7f44a8ff9640 1 --2- 172.21.15.46:0/1991587361 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44ac062df0 0x7f44ac063200 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f44a0003fd0 tx=0x7f44a0005a10 comp rx=0 tx=0).stop 2024-09-16T04:46:26.652 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:46:26.652 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:46:26.652 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:46:26.653 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:46:26.653 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 90, 2024-09-16T04:46:26.653 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 27870, 2024-09-16T04:46:26.653 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:46:26.653 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:46:26.653 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:46:26.654 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:26.654 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:46:26.654 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:26.654 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:46:26.654 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:46:26.654 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:26.654 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:26.655 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:26.655 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:26.655 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:26.655 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:26.655 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:26.656 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:26.656 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:26.656 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:26.656 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:26.656 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:26.656 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:46:26.657 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:46:26.657 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:26.657 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:26.657 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:26.657 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:26.657 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:26.658 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:26.658 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:26.658 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:26.658 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:26.658 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:26.659 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:26.659 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:26.659 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.659 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:46:26.659 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:46:26.659 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:46:26.660 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:46:26.660 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:46:26.660 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:46:26.660 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:26.660 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:26.660 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:46:26.661 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:46:26.661 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:46:26.661 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:46:26.661 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:46:26.661 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:46:26.662 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:46:26.662 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:26.662 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:46:26.662 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:26.662 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:26.662 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:26.663 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:26.663 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:26.663 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:26.663 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:26.663 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:26.663 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:26.664 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:26.664 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:26.664 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:26.664 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:46:26.664 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.664 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:46:26.665 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.665 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:46:26.665 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:46:26.665 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:26.665 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:26.665 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.666 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:26.666 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:46:26.666 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:26.666 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.666 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.666 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:26.667 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:46:26.667 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:26.667 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:26.667 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:26.667 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.667 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:46:26.667 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:46:26.667 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:26.668 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:26.668 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:26.668 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.668 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.668 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:46:26.668 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:46:26.668 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:26.668 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:26.668 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.668 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:26.668 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:46:26.668 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:26.668 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.669 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.669 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:26.669 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:46:26.669 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:26.669 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:26.669 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:26.669 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.669 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:46:26.669 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:46:26.669 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:26.669 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:26.670 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:26.670 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.670 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.670 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:46:26.670 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:46:26.670 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:26.670 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:26.670 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.670 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:26.670 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:46:26.670 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:26.670 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.670 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.671 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:26.671 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:46:26.671 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:26.671 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:26.671 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:26.671 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.671 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:46:26.671 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:46:26.671 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:26.671 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:26.671 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:26.671 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:26.671 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:26.671 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.672 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:46:26.672 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:46:26.672 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.672 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:26.672 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:26.672 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:26.672 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:26.672 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:26.672 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:46:26.672 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.672 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:26.672 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:26.672 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:46:26.673 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:26.673 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:26.673 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.673 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:46:26.673 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:46:26.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.616+0000 7f44b27c4640 1 -- 172.21.15.46:0/1991587361 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f44980382a0 msgr2=0x7f449803a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:26.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.616+0000 7f44b27c4640 1 --2- 172.21.15.46:0/1991587361 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f44980382a0 0x7f449803a760 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f44ac09bf60 tx=0x7f44a4007ef0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.616+0000 7f44b27c4640 10 monclient: shutdown 2024-09-16T04:46:26.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.616+0000 7f44b27c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:26.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.616+0000 7f44b27c4640 1 -- 172.21.15.46:0/1991587361 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4494004800 msgr2=0x7f44ac131840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:26.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.616+0000 7f44b27c4640 1 --2- 172.21.15.46:0/1991587361 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f4494004800 0x7f44ac131840 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f449c003fd0 tx=0x7f449c009b50 comp rx=0 tx=0).stop 2024-09-16T04:46:26.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.616+0000 7f44b0d61640 1 -- 172.21.15.46:0/1991587361 reap_dead start 2024-09-16T04:46:26.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.616+0000 7f44b0d61640 1 -- 172.21.15.46:0/1991587361 reap_dead start 2024-09-16T04:46:26.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.616+0000 7f44b27c4640 1 -- 172.21.15.46:0/1991587361 shutdown_connections 2024-09-16T04:46:26.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.616+0000 7f44b27c4640 1 -- 172.21.15.46:0/1991587361 >> 172.21.15.46:0/1991587361 conn(0x7f44ac0a5960 msgr2=0x7f44ac072c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:26.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.616+0000 7f44b27c4640 1 -- 172.21.15.46:0/1991587361 shutdown_connections 2024-09-16T04:46:26.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.616+0000 7f44b27c4640 1 -- 172.21.15.46:0/1991587361 wait complete. 2024-09-16T04:46:26.675 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:46:26.675 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:46:26.676 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:46:26.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.776+0000 7f9187817640 10 monclient: get_monmap_and_config 2024-09-16T04:46:26.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.776+0000 7f9187817640 10 monclient: build_initial_monmap 2024-09-16T04:46:26.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.776+0000 7f9187817640 10 monclient: monmap: 2024-09-16T04:46:26.779 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:26.779 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:26.779 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:26.777817+0000 2024-09-16T04:46:26.779 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:26.777817+0000 2024-09-16T04:46:26.779 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:26.780 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.780 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:26.780 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:26.780 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:26.780 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:26.780 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:26.781 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.776+0000 7f9187817640 1 Processor -- start 2024-09-16T04:46:26.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9187817640 1 -- start start 2024-09-16T04:46:26.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9187817640 10 monclient: init 2024-09-16T04:46:26.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9187817640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:26.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9187817640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-16T04:46:26.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9187817640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f91800a3c10 0x7f918009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9187817640 10 monclient(hunting): picked mon.noname-a con 0x7f91800a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:46:26.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9187817640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f91800a3ff0 0x7f91800a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9187817640 10 monclient(hunting): picked mon.noname-c con 0x7f91800a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:46:26.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9187817640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f91801369d0 addr v1:172.21.15.46:6789/0 2024-09-16T04:46:26.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9187817640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f918009b940 con 0x7f91801369d0 2024-09-16T04:46:26.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9187817640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:26.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9187817640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f918009ab00 con 0x7f91800a3c10 2024-09-16T04:46:26.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9187817640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:26.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9187817640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f918009b030 con 0x7f91800a3ff0 2024-09-16T04:46:26.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9187817640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:26.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9187817640 10 monclient(hunting): authenticate will time out at 1185.193635s 2024-09-16T04:46:26.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9185db4640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f91801369d0 legacy=0x7f9180136e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-09-16T04:46:26.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f91855b3640 1 -- >> v2:172.21.15.46:3300/0 conn(0x7f91800a3c10 msgr2=0x7f918009bcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:46:26.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f91855b3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f91800a3c10 0x7f918009bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:26.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9184db2640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f91800a3ff0 0x7f91800a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:26.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9184db2640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f91800a3ff0 0x7f91800a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.46:38498/0 (socket says 172.21.15.46:38498) 2024-09-16T04:46:26.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9184db2640 1 -- 172.21.15.46:0/2371540136 learned_addr learned my addr 172.21.15.46:0/2371540136 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:26.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9184db2640 10 monclient(hunting): get_auth_request con 0x7f91800a3ff0 auth_method 0 2024-09-16T04:46:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9184db2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9184db2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.777+0000 7f9184db2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.778+0000 7f9184db2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.778+0000 7f9184db2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.778+0000 7f9184db2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.778+0000 7f9184db2640 10 monclient(hunting): handle_auth_done global_id 6787 payload 274 2024-09-16T04:46:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.778+0000 7f9184db2640 1 -- 172.21.15.46:0/2371540136 >> v2:172.21.15.46:3300/0 conn(0x7f91800a3c10 msgr2=0x7f918009bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:46:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.778+0000 7f9184db2640 1 --2- 172.21.15.46:0/2371540136 >> v2:172.21.15.46:3300/0 conn(0x7f91800a3c10 0x7f918009bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.778+0000 7f9184db2640 1 -- 172.21.15.46:0/2371540136 >> v1:172.21.15.46:6789/0 conn(0x7f91801369d0 legacy=0x7f9180136e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:46:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.778+0000 7f9184db2640 10 monclient: _finish_hunting 0 2024-09-16T04:46:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.778+0000 7f9184db2640 1 monclient: found mon.noname-c 2024-09-16T04:46:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.778+0000 7f9184db2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.778+0000 7f9184db2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:46:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.778+0000 7f9184db2640 1 -- 172.21.15.46:0/2371540136 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f918009b310 con 0x7f91800a3ff0 2024-09-16T04:46:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.778+0000 7f9184db2640 1 --2- 172.21.15.46:0/2371540136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91800a3ff0 0x7f91800a43d0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f91800a38d0 tx=0x7f917002a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63bd7d344bd97b95 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9177fff640 1 -- 172.21.15.46:0/2371540136 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9170037070 con 0x7f91800a3ff0 2024-09-16T04:46:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9177fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9177fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9177fff640 10 monclient: dump: 2024-09-16T04:46:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:26.800 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9177fff640 10 monclient: _finish_auth 0 2024-09-16T04:46:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9177fff640 10 monclient: _check_auth_tickets 2024-09-16T04:46:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9177fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9177fff640 1 -- 172.21.15.46:0/2371540136 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f917002ae40 con 0x7f91800a3ff0 2024-09-16T04:46:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9187817640 5 monclient: authenticate success, global_id 6787 2024-09-16T04:46:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9187817640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9177fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9187817640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9177fff640 1 -- 172.21.15.46:0/2371540136 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9170032660 con 0x7f91800a3ff0 2024-09-16T04:46:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9177fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9187817640 4 set_mon_vals no callback set 2024-09-16T04:46:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9177fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:46:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9177fff640 10 monclient: dump: 2024-09-16T04:46:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:26.802 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9187817640 10 monclient: shutdown 2024-09-16T04:46:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9187817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9187817640 1 -- 172.21.15.46:0/2371540136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91800a3ff0 msgr2=0x7f91800a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9187817640 1 --2- 172.21.15.46:0/2371540136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91800a3ff0 0x7f91800a43d0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f91800a38d0 tx=0x7f917002a130 comp rx=0 tx=0).stop 2024-09-16T04:46:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9187817640 1 -- 172.21.15.46:0/2371540136 shutdown_connections 2024-09-16T04:46:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9187817640 1 --2- 172.21.15.46:0/2371540136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91800a3ff0 0x7f91800a43d0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9187817640 1 --2- 172.21.15.46:0/2371540136 >> v2:172.21.15.46:3300/0 conn(0x7f91800a3c10 0x7f918009bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9187817640 1 -- 172.21.15.46:0/2371540136 >> 172.21.15.46:0/2371540136 conn(0x7f91800a6340 msgr2=0x7f918006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9187817640 1 -- 172.21.15.46:0/2371540136 shutdown_connections 2024-09-16T04:46:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.779+0000 7f9187817640 1 -- 172.21.15.46:0/2371540136 wait complete. 2024-09-16T04:46:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.780+0000 7f9187817640 10 monclient: build_initial_monmap 2024-09-16T04:46:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.780+0000 7f9187817640 10 monclient: monmap: 2024-09-16T04:46:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:26.804 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:26.804 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.780+0000 7f9187817640 1 Processor -- start 2024-09-16T04:46:26.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.780+0000 7f9187817640 1 -- start start 2024-09-16T04:46:26.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.780+0000 7f9187817640 10 monclient: init 2024-09-16T04:46:26.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.780+0000 7f9187817640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:26.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.780+0000 7f9187817640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:46:26.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9187817640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f91800a3c10 0x7f91801cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9187817640 10 monclient(hunting): picked mon.noname-c con 0x7f91800a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:26.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9187817640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91800a3ff0 0x7f9180123440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9187817640 10 monclient(hunting): picked mon.noname-b con 0x7f91800a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:26.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9187817640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91801369d0 0x7f9180123980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9187817640 10 monclient(hunting): picked mon.noname-a con 0x7f91801369d0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:26.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9187817640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:26.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9187817640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f91800a4910 con 0x7f91801369d0 2024-09-16T04:46:26.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9187817640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:26.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9187817640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f91800a69c0 con 0x7f91800a3ff0 2024-09-16T04:46:26.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9187817640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:26.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9187817640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f9180127db0 con 0x7f91800a3c10 2024-09-16T04:46:26.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9187817640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:26.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9187817640 10 monclient(hunting): authenticate will time out at 1185.197060s 2024-09-16T04:46:26.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9185db4640 1 -- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91801369d0 msgr2=0x7f9180123980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:3300/0 2024-09-16T04:46:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9185db4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91801369d0 0x7f9180123980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91855b3640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f91800a3c10 0x7f91801cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91855b3640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f91800a3c10 0x7f91801cc430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:59580/0 (socket says 172.21.15.46:59580) 2024-09-16T04:46:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91855b3640 1 -- 172.21.15.46:0/4256867525 learned_addr learned my addr 172.21.15.46:0/4256867525 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91855b3640 10 monclient(hunting): get_auth_request con 0x7f91800a3c10 auth_method 0 2024-09-16T04:46:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91855b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91855b3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91855b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91855b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91855b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91855b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9184db2640 1 --2- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91800a3ff0 0x7f9180123440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9184db2640 10 monclient(hunting): get_auth_request con 0x7f91800a3ff0 auth_method 0 2024-09-16T04:46:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9184db2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9184db2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9184db2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9184db2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9184db2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9184db2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91855b3640 10 monclient(hunting): handle_auth_done global_id 94190 payload 824 2024-09-16T04:46:26.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91855b3640 1 -- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91800a3ff0 msgr2=0x7f9180123440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:26.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91855b3640 1 --2- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91800a3ff0 0x7f9180123440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91855b3640 1 -- 172.21.15.46:0/4256867525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91801369d0 msgr2=0x7f9180123980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:46:26.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91855b3640 1 --2- 172.21.15.46:0/4256867525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91801369d0 0x7f9180123980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91855b3640 10 monclient: _finish_hunting 0 2024-09-16T04:46:26.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91855b3640 1 monclient: found mon.noname-c 2024-09-16T04:46:26.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91855b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:26.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91855b3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:46:26.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91855b3640 1 -- 172.21.15.46:0/4256867525 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91801270f0 con 0x7f91800a3c10 2024-09-16T04:46:26.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91855b3640 1 --2- 172.21.15.46:0/4256867525 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f91800a3c10 0x7f91801cc430 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f917c00b5b0 tx=0x7f917c00ba70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:26.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9175ffb640 1 -- 172.21.15.46:0/4256867525 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917c013020 con 0x7f91800a3c10 2024-09-16T04:46:26.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9175ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:26.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9175ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:26.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9175ffb640 10 monclient: dump: 2024-09-16T04:46:26.810 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:26.811 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:26.811 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.811 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.811 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:26.811 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.811 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:26.812 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:26.812 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:26.812 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9175ffb640 10 monclient: _finish_auth 0 2024-09-16T04:46:26.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9175ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:46:26.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9175ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9175ffb640 1 -- 172.21.15.46:0/4256867525 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f917c003dc0 con 0x7f91800a3c10 2024-09-16T04:46:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9175ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9175ffb640 1 -- 172.21.15.46:0/4256867525 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917c0052b0 con 0x7f91800a3c10 2024-09-16T04:46:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9187817640 5 monclient: authenticate success, global_id 94190 2024-09-16T04:46:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9175ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9175ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9175ffb640 10 monclient: dump: 2024-09-16T04:46:26.814 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:26.814 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:26.814 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.814 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.814 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:26.814 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.814 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:26.815 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:26.815 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:26.815 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9187817640 10 monclient: _renew_subs 2024-09-16T04:46:26.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9187817640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:26.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f9187817640 1 -- 172.21.15.46:0/4256867525 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9180125c40 con 0x7f91800a3c10 2024-09-16T04:46:26.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.781+0000 7f91777fe640 4 set_mon_vals no callback set 2024-09-16T04:46:26.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.782+0000 7f9187817640 10 monclient: _renew_subs 2024-09-16T04:46:26.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.782+0000 7f9187817640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:26.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.782+0000 7f9187817640 1 -- 172.21.15.46:0/4256867525 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f91800a85d0 con 0x7f91800a3c10 2024-09-16T04:46:26.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.782+0000 7f9175ffb640 1 -- 172.21.15.46:0/4256867525 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f917c016020 con 0x7f91800a3c10 2024-09-16T04:46:26.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.783+0000 7f9187817640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:26.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.783+0000 7f9187817640 1 --2- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9180126d10 0x7f9180123ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.783+0000 7f9187817640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:26.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.783+0000 7f9187817640 1 -- 172.21.15.46:0/4256867525 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f91800a4a90 con 0x7f9180126d10 2024-09-16T04:46:26.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.783+0000 7f9175ffb640 1 --2- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9164038360 0x7f916403a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.783+0000 7f9175ffb640 1 -- 172.21.15.46:0/4256867525 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(283..283 src has 1..283) ==== 4738+0+0 (secure 0 0 0) 0x7f917c04b320 con 0x7f91800a3c10 2024-09-16T04:46:26.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.783+0000 7f9184db2640 1 --2- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9180126d10 0x7f9180123ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:26.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.783+0000 7f9184db2640 10 monclient: get_auth_request con 0x7f9180126d10 auth_method 0 2024-09-16T04:46:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.783+0000 7f9184db2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.783+0000 7f9184db2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.783+0000 7f9184db2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.783+0000 7f9185db4640 1 --2- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9164038360 0x7f916403a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.783+0000 7f9185db4640 10 monclient: get_auth_request con 0x7f9164038360 auth_method 0 2024-09-16T04:46:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.783+0000 7f9184db2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.783+0000 7f9184db2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.783+0000 7f9184db2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.783+0000 7f9184db2640 10 monclient(hunting): handle_auth_done global_id 6793 payload 824 2024-09-16T04:46:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.783+0000 7f9184db2640 1 -- 172.21.15.46:0/4256867525 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f91800a0400 con 0x7f9180126d10 2024-09-16T04:46:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.783+0000 7f9185db4640 1 --2- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9164038360 0x7f916403a820 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9178003d00 tx=0x7f9178009270 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.783+0000 7f9184db2640 1 --2- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9180126d10 0x7f9180123ec0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f918006ee70 tx=0x7f91700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.784+0000 7f9175ffb640 1 -- 172.21.15.46:0/4256867525 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9170037040 con 0x7f9180126d10 2024-09-16T04:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.784+0000 7f9175ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.784+0000 7f9175ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.784+0000 7f9175ffb640 10 monclient: dump: 2024-09-16T04:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:26.820 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:26.820 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.784+0000 7f9175ffb640 1 -- 172.21.15.46:0/4256867525 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9170032e40 con 0x7f9180126d10 2024-09-16T04:46:26.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.784+0000 7f9175ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:26.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.784+0000 7f9175ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:26.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.784+0000 7f9175ffb640 1 -- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9180126d10 msgr2=0x7f9180123ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:26.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.784+0000 7f9175ffb640 1 --2- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9180126d10 0x7f9180123ec0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f918006ee70 tx=0x7f91700046c0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.784+0000 7f9175ffb640 1 -- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9180126d10 msgr2=0x7f9180123ec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:26.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.784+0000 7f9175ffb640 1 --2- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9180126d10 0x7f9180123ec0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.790+0000 7f9187817640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:46:26.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.790+0000 7f9187817640 1 --2- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9180124400 0x7f918011f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:26.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.790+0000 7f9187817640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:26.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.790+0000 7f9187817640 1 -- 172.21.15.46:0/4256867525 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f918009bd70 con 0x7f9180124400 2024-09-16T04:46:26.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.790+0000 7f9184db2640 1 --2- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9180124400 0x7f918011f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:26.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.790+0000 7f9184db2640 10 monclient: get_auth_request con 0x7f9180124400 auth_method 0 2024-09-16T04:46:26.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.790+0000 7f9184db2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:26.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.790+0000 7f9184db2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:26.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.790+0000 7f9184db2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:26.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.790+0000 7f9184db2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:26.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.790+0000 7f9184db2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:26.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.790+0000 7f9184db2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:26.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.790+0000 7f9184db2640 10 monclient(hunting): handle_auth_done global_id 6796 payload 824 2024-09-16T04:46:26.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.790+0000 7f9184db2640 1 -- 172.21.15.46:0/4256867525 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9180067e20 con 0x7f9180124400 2024-09-16T04:46:26.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9184db2640 1 --2- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9180124400 0x7f918011f860 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f9170000c00 tx=0x7f9170008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:26.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9175ffb640 1 -- 172.21.15.46:0/4256867525 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9170037040 con 0x7f9180124400 2024-09-16T04:46:26.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9175ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:26.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9175ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:46:26.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9175ffb640 10 monclient: dump: 2024-09-16T04:46:26.823 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:26.823 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:26.823 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.823 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:26.823 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:26.823 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:26.823 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:26.823 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:26.823 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:26.823 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:26.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9175ffb640 1 -- 172.21.15.46:0/4256867525 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4870 (secure 0 0 0) 0x7f917003ce20 con 0x7f9180124400 2024-09-16T04:46:26.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9175ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:46:26.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9175ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:26.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9175ffb640 1 -- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9180124400 msgr2=0x7f918011f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:26.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9175ffb640 1 --2- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9180124400 0x7f918011f860 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f9170000c00 tx=0x7f9170008030 comp rx=0 tx=0).stop 2024-09-16T04:46:26.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9175ffb640 1 -- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9180124400 msgr2=0x7f918011f860 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:26.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9175ffb640 1 --2- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9180124400 0x7f918011f860 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:26.824 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:46:26.824 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:46:26.824 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:46:26.824 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:46:26.825 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 90, 2024-09-16T04:46:26.825 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 472694, 2024-09-16T04:46:26.825 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:46:26.825 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:46:26.825 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:46:26.825 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:26.825 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:46:26.825 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:26.825 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:46:26.825 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:46:26.825 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:26.825 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:26.825 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:26.826 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:26.826 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:26.826 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:26.826 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:26.826 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:26.826 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:26.826 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:26.826 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:26.826 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:26.826 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:46:26.826 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:46:26.826 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:26.826 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:26.827 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:26.827 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:26.827 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:26.827 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:26.827 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:26.827 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:26.827 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:26.827 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:26.827 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:26.827 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:26.827 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.827 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:46:26.827 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:46:26.827 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:46:26.828 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:46:26.828 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:46:26.828 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:46:26.828 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:26.828 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:26.828 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:46:26.828 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:46:26.828 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:46:26.828 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:46:26.828 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:46:26.828 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:46:26.828 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:46:26.828 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:26.829 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:46:26.829 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:26.829 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:26.829 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:26.829 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:26.829 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:26.829 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:26.829 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:26.829 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:26.829 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:26.829 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:26.829 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:26.829 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:26.830 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:46:26.830 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.830 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:46:26.830 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.830 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:46:26.830 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:46:26.830 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:26.830 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:26.830 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.830 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:26.830 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:46:26.830 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:26.830 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.831 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.831 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:26.831 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:46:26.831 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:26.831 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:26.831 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:26.831 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.831 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:46:26.831 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:46:26.831 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:26.831 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:26.831 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:26.831 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.832 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.832 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:46:26.832 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:46:26.832 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:26.832 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:26.832 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.832 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:26.832 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:46:26.832 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:26.832 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.832 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.832 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:26.832 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:46:26.832 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:26.833 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:26.833 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:26.833 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.833 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:46:26.833 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:46:26.833 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:26.833 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:26.833 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:26.833 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.833 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.833 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:46:26.833 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:46:26.833 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:26.834 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:26.834 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.834 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:26.834 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:46:26.834 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:26.834 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.834 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.834 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:26.834 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:46:26.834 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:26.834 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:26.834 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:26.834 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.835 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:46:26.835 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:46:26.835 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:26.835 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:26.835 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:26.835 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:26.835 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:26.835 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.835 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:46:26.835 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:46:26.835 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.835 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:26.835 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:26.835 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:26.836 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:26.836 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:26.836 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:46:26.836 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.836 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:26.836 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:26.836 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6 2024-09-16T04:46:26.836 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:26.836 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:26.836 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:46:26.836 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.836 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:26.836 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:26.837 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5 2024-09-16T04:46:26.837 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:26.837 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:26.837 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:46:26.837 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:26.837 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:26.837 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:26.837 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:26.837 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:26.837 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:26.837 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:26.837 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:46:26.837 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:46:26.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9187817640 1 -- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9164038360 msgr2=0x7f916403a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:26.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9187817640 1 --2- 172.21.15.46:0/4256867525 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f9164038360 0x7f916403a820 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9178003d00 tx=0x7f9178009270 comp rx=0 tx=0).stop 2024-09-16T04:46:26.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9187817640 10 monclient: shutdown 2024-09-16T04:46:26.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9187817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:26.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9187817640 1 -- 172.21.15.46:0/4256867525 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f91800a3c10 msgr2=0x7f91801cc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:26.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9187817640 1 --2- 172.21.15.46:0/4256867525 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f91800a3c10 0x7f91801cc430 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f917c00b5b0 tx=0x7f917c00ba70 comp rx=0 tx=0).stop 2024-09-16T04:46:26.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9185db4640 1 -- 172.21.15.46:0/4256867525 reap_dead start 2024-09-16T04:46:26.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9185db4640 1 -- 172.21.15.46:0/4256867525 reap_dead start 2024-09-16T04:46:26.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9187817640 1 -- 172.21.15.46:0/4256867525 shutdown_connections 2024-09-16T04:46:26.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.791+0000 7f9187817640 1 -- 172.21.15.46:0/4256867525 >> 172.21.15.46:0/4256867525 conn(0x7f91800a6340 msgr2=0x7f91800a1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:26.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.792+0000 7f9187817640 1 -- 172.21.15.46:0/4256867525 shutdown_connections 2024-09-16T04:46:26.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:26.792+0000 7f9187817640 1 -- 172.21.15.46:0/4256867525 wait complete. 2024-09-16T04:46:26.843 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:46:26.843 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:46:27.104 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:27.104 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39854]: starting. 2024-09-16T04:46:27.104 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39854]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-39817 2024-09-16T04:46:27.104 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39854]: shutting down. 2024-09-16T04:46:27.105 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:27.105 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:27.105 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:46:27.108 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:46:29.222 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39923]: starting. 2024-09-16T04:46:29.222 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39923]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-39922 2024-09-16T04:46:29.222 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39923]: created object 0... 2024-09-16T04:46:29.222 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39923]: created object 25... 2024-09-16T04:46:29.223 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39923]: created object 49... 2024-09-16T04:46:29.223 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39923]: finishing. 2024-09-16T04:46:29.223 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[39923]: shutting down. 2024-09-16T04:46:30.174 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39924]: starting. 2024-09-16T04:46:30.175 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39924]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-39922 2024-09-16T04:46:30.175 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[39924]: shutting down. 2024-09-16T04:46:32.480 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39960]: starting. 2024-09-16T04:46:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39960]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-39922 2024-09-16T04:46:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39960]: created object 0... 2024-09-16T04:46:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39960]: created object 25... 2024-09-16T04:46:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39960]: created object 49... 2024-09-16T04:46:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39960]: finishing. 2024-09-16T04:46:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[39960]: shutting down. 2024-09-16T04:46:32.494 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:32.494 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39962]: starting. 2024-09-16T04:46:32.494 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39962]: listing objects. 2024-09-16T04:46:32.495 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39962]: listed object 0... 2024-09-16T04:46:32.495 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39962]: listed object 25... 2024-09-16T04:46:32.495 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39962]: saw 50 objects 2024-09-16T04:46:32.495 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[39962]: shutting down. 2024-09-16T04:46:33.182 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:33.182 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39961]: starting. 2024-09-16T04:46:33.182 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39961]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-39922 2024-09-16T04:46:33.182 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[39961]: shutting down. 2024-09-16T04:46:33.183 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:33.183 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:33.183 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:46:33.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:46:35.273 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40031]: starting. 2024-09-16T04:46:35.274 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40031]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-40030 2024-09-16T04:46:35.274 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40031]: created object 0... 2024-09-16T04:46:35.274 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40031]: created object 25... 2024-09-16T04:46:35.274 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40031]: created object 49... 2024-09-16T04:46:35.274 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40031]: finishing. 2024-09-16T04:46:35.274 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40031]: shutting down. 2024-09-16T04:46:36.197 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[40032]: starting. 2024-09-16T04:46:36.197 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[40032]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-40030 2024-09-16T04:46:36.197 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[40032]: shutting down. 2024-09-16T04:46:38.293 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:38.293 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40067]: starting. 2024-09-16T04:46:38.293 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40067]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-40030 2024-09-16T04:46:38.294 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40067]: created object 0... 2024-09-16T04:46:38.294 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40067]: created object 25... 2024-09-16T04:46:38.294 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40067]: created object 49... 2024-09-16T04:46:38.294 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40067]: finishing. 2024-09-16T04:46:38.294 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40067]: shutting down. 2024-09-16T04:46:38.305 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:38.306 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40069]: starting. 2024-09-16T04:46:38.306 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40069]: listing objects. 2024-09-16T04:46:38.306 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40069]: listed object 0... 2024-09-16T04:46:38.306 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40069]: listed object 25... 2024-09-16T04:46:38.306 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40069]: saw 50 objects 2024-09-16T04:46:38.306 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40069]: shutting down. 2024-09-16T04:46:39.162 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:39.163 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[40068]: starting. 2024-09-16T04:46:39.163 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[40068]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-40030 2024-09-16T04:46:39.163 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[40068]: shutting down. 2024-09-16T04:46:39.164 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:39.164 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:39.164 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:46:39.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:46:41.301 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40138]: starting. 2024-09-16T04:46:41.302 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40138]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-40137 2024-09-16T04:46:41.302 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40138]: created object 0... 2024-09-16T04:46:41.302 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40138]: created object 25... 2024-09-16T04:46:41.302 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40138]: created object 49... 2024-09-16T04:46:41.302 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40138]: finishing. 2024-09-16T04:46:41.302 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40138]: shutting down. 2024-09-16T04:46:42.221 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[40139]: starting. 2024-09-16T04:46:42.221 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[40139]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-40137 2024-09-16T04:46:42.222 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[40139]: shutting down. 2024-09-16T04:46:44.307 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:44.307 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40174]: starting. 2024-09-16T04:46:44.307 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40174]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-40137 2024-09-16T04:46:44.307 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40174]: created object 0... 2024-09-16T04:46:44.307 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40174]: created object 25... 2024-09-16T04:46:44.307 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40174]: created object 49... 2024-09-16T04:46:44.307 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40174]: finishing. 2024-09-16T04:46:44.307 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40174]: shutting down. 2024-09-16T04:46:44.320 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:44.320 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40176]: starting. 2024-09-16T04:46:44.321 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40176]: listing objects. 2024-09-16T04:46:44.321 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40176]: listed object 0... 2024-09-16T04:46:44.321 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40176]: listed object 25... 2024-09-16T04:46:44.321 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40176]: saw 50 objects 2024-09-16T04:46:44.321 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40176]: shutting down. 2024-09-16T04:46:45.236 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:45.236 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[40175]: starting. 2024-09-16T04:46:45.236 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[40175]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-40137 2024-09-16T04:46:45.236 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[40175]: shutting down. 2024-09-16T04:46:45.238 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:45.238 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:45.238 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:46:45.240 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:46:46.844 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:46:46.844 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T04:46:46.844 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T04:46:46.845 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T04:46:46.847 INFO:tasks.ceph.mon.a:Started 2024-09-16T04:46:46.848 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:46:46.848 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:46:46.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.973+0000 7f1b3e42e640 10 monclient: get_monmap_and_config 2024-09-16T04:46:46.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.973+0000 7f1b3e42e640 10 monclient: build_initial_monmap 2024-09-16T04:46:46.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.973+0000 7f1b3e42e640 10 monclient: monmap: 2024-09-16T04:46:46.974 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:46.974 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:46.974 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:46.973508+0000 2024-09-16T04:46:46.974 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:46.973508+0000 2024-09-16T04:46:46.974 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:46.974 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:46.974 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:46.975 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:46.975 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:46.975 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:46.975 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:46.975 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:46.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.973+0000 7f1b3e42e640 1 Processor -- start 2024-09-16T04:46:46.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.973+0000 7f1b3e42e640 1 -- start start 2024-09-16T04:46:46.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.973+0000 7f1b3e42e640 10 monclient: init 2024-09-16T04:46:46.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b3e42e640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:46.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b3e42e640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-16T04:46:46.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b3e42e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1b380a3c10 0x7f1b3809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:46.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b3e42e640 10 monclient(hunting): picked mon.noname-c con 0x7f1b380a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:46:46.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b3e42e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1b380a3ff0 0x7f1b380a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:46.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b3e42e640 10 monclient(hunting): picked mon.noname-b con 0x7f1b380a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:46.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b3e42e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b381369d0 addr v1:172.21.15.46:6789/0 2024-09-16T04:46:46.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b3e42e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b3809ab70 con 0x7f1b381369d0 2024-09-16T04:46:46.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b3e42e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:46.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b3e42e640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f1b380a4d60 con 0x7f1b380a3c10 2024-09-16T04:46:46.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b3e42e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:46.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b3e42e640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f1b3809b030 con 0x7f1b380a3ff0 2024-09-16T04:46:46.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b3e42e640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:46.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b3e42e640 10 monclient(hunting): authenticate will time out at 1205.389375s 2024-09-16T04:46:46.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b3c9cb640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1b381369d0 0x7f1b38136e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48896/0 (socket says 172.21.15.46:48896) 2024-09-16T04:46:46.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b3c9cb640 1 -- 172.21.15.46:0/1905374798 learned_addr learned my addr 172.21.15.46:0/1905374798 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:46:46.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b377fe640 1 --2- 172.21.15.46:0/1905374798 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1b380a3ff0 0x7f1b380a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:46.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b377fe640 10 monclient(hunting): get_auth_request con 0x7f1b380a3ff0 auth_method 0 2024-09-16T04:46:46.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:46.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:46.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:46.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b37fff640 1 --2- 172.21.15.46:0/1905374798 >> v2:172.21.15.114:3300/0 conn(0x7f1b380a3c10 0x7f1b3809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:46.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b37fff640 10 monclient(hunting): get_auth_request con 0x7f1b380a3c10 auth_method 0 2024-09-16T04:46:46.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:46.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:46.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:46.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:46.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:46.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:46.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:46.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:46.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b377fe640 10 monclient(hunting): handle_auth_done global_id 94265 payload 274 2024-09-16T04:46:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b377fe640 1 -- 172.21.15.46:0/1905374798 >> v2:172.21.15.114:3300/0 conn(0x7f1b380a3c10 msgr2=0x7f1b3809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b377fe640 1 --2- 172.21.15.46:0/1905374798 >> v2:172.21.15.114:3300/0 conn(0x7f1b380a3c10 0x7f1b3809bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b377fe640 1 -- 172.21.15.46:0/1905374798 >> v1:172.21.15.46:6789/0 conn(0x7f1b381369d0 legacy=0x7f1b38136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b377fe640 10 monclient: _finish_hunting 0 2024-09-16T04:46:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b377fe640 1 monclient: found mon.noname-b 2024-09-16T04:46:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b377fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:46:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.974+0000 7f1b377fe640 1 -- 172.21.15.46:0/1905374798 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3809b310 con 0x7f1b380a3ff0 2024-09-16T04:46:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.975+0000 7f1b377fe640 1 --2- 172.21.15.46:0/1905374798 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1b380a3ff0 0x7f1b380a43d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0097a0 tx=0x7f1b2c02a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=129ed290a7a92405 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.975+0000 7f1b36ffd640 1 -- 172.21.15.46:0/1905374798 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c037070 con 0x7f1b380a3ff0 2024-09-16T04:46:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.975+0000 7f1b36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.975+0000 7f1b36ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:46:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.975+0000 7f1b36ffd640 10 monclient: dump: 2024-09-16T04:46:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:46.987 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.975+0000 7f1b36ffd640 10 monclient: _finish_auth 0 2024-09-16T04:46:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.975+0000 7f1b36ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:46:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.975+0000 7f1b36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.975+0000 7f1b36ffd640 1 -- 172.21.15.46:0/1905374798 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b2c02ae50 con 0x7f1b380a3ff0 2024-09-16T04:46:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.975+0000 7f1b3e42e640 5 monclient: authenticate success, global_id 94265 2024-09-16T04:46:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.975+0000 7f1b3e42e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.975+0000 7f1b36ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.975+0000 7f1b3e42e640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.975+0000 7f1b3e42e640 4 set_mon_vals no callback set 2024-09-16T04:46:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.975+0000 7f1b36ffd640 1 -- 172.21.15.46:0/1905374798 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c032660 con 0x7f1b380a3ff0 2024-09-16T04:46:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.975+0000 7f1b36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.975+0000 7f1b36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.975+0000 7f1b36ffd640 10 monclient: dump: 2024-09-16T04:46:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:46.989 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.976+0000 7f1b3e42e640 10 monclient: shutdown 2024-09-16T04:46:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.976+0000 7f1b3e42e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.976+0000 7f1b3e42e640 1 -- 172.21.15.46:0/1905374798 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1b380a3ff0 msgr2=0x7f1b380a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.976+0000 7f1b3e42e640 1 --2- 172.21.15.46:0/1905374798 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1b380a3ff0 0x7f1b380a43d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0097a0 tx=0x7f1b2c02a100 comp rx=0 tx=0).stop 2024-09-16T04:46:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.976+0000 7f1b3e42e640 1 -- 172.21.15.46:0/1905374798 shutdown_connections 2024-09-16T04:46:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.976+0000 7f1b3e42e640 1 --2- 172.21.15.46:0/1905374798 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1b380a3ff0 0x7f1b380a43d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.976+0000 7f1b3e42e640 1 --2- 172.21.15.46:0/1905374798 >> v2:172.21.15.114:3300/0 conn(0x7f1b380a3c10 0x7f1b3809bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.976+0000 7f1b3e42e640 1 -- 172.21.15.46:0/1905374798 >> 172.21.15.46:0/1905374798 conn(0x7f1b380a5d10 msgr2=0x7f1b380a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.976+0000 7f1b3e42e640 1 -- 172.21.15.46:0/1905374798 shutdown_connections 2024-09-16T04:46:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.976+0000 7f1b3e42e640 1 -- 172.21.15.46:0/1905374798 wait complete. 2024-09-16T04:46:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.977+0000 7f1b3e42e640 10 monclient: build_initial_monmap 2024-09-16T04:46:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.977+0000 7f1b3e42e640 10 monclient: monmap: 2024-09-16T04:46:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:46.991 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.977+0000 7f1b3e42e640 1 Processor -- start 2024-09-16T04:46:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.977+0000 7f1b3e42e640 1 -- start start 2024-09-16T04:46:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.977+0000 7f1b3e42e640 10 monclient: init 2024-09-16T04:46:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3e42e640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3e42e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:46:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3e42e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b380a3c10 0x7f1b381cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3e42e640 10 monclient(hunting): picked mon.noname-a con 0x7f1b380a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3e42e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b380a3ff0 0x7f1b381cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3e42e640 10 monclient(hunting): picked mon.noname-b con 0x7f1b380a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3e42e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1b381369d0 0x7f1b38123280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3e42e640 10 monclient(hunting): picked mon.noname-c con 0x7f1b381369d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3e42e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3e42e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1b380a4910 con 0x7f1b380a3c10 2024-09-16T04:46:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3e42e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3e42e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3806e3d0 con 0x7f1b380a3ff0 2024-09-16T04:46:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3e42e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3e42e640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f1b38127210 con 0x7f1b381369d0 2024-09-16T04:46:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3e42e640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3e42e640 10 monclient(hunting): authenticate will time out at 1205.393343s 2024-09-16T04:46:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b37fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b380a3c10 0x7f1b381cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b37fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b380a3c10 0x7f1b381cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57856/0 (socket says 172.21.15.46:57856) 2024-09-16T04:46:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b37fff640 1 -- 172.21.15.46:0/3231512706 learned_addr learned my addr 172.21.15.46:0/3231512706 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b37fff640 10 monclient(hunting): get_auth_request con 0x7f1b380a3c10 auth_method 0 2024-09-16T04:46:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3c9cb640 1 --2- 172.21.15.46:0/3231512706 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1b381369d0 0x7f1b38123280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3c9cb640 10 monclient(hunting): get_auth_request con 0x7f1b381369d0 auth_method 0 2024-09-16T04:46:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3c9cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3c9cb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3c9cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b37fff640 1 -- 172.21.15.46:0/3231512706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b380a3c10 msgr2=0x7f1b381cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:46:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b37fff640 1 -- 172.21.15.46:0/3231512706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b380a3c10 msgr2=0x7f1b381cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:46:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b37fff640 1 --2- 172.21.15.46:0/3231512706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b380a3c10 0x7f1b381cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b37fff640 1 --2- 172.21.15.46:0/3231512706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b380a3c10 0x7f1b381cbee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3c9cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b377fe640 1 --2- 172.21.15.46:0/3231512706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b380a3ff0 0x7f1b381cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3c9cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:46.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3c9cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:46.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b377fe640 10 monclient(hunting): get_auth_request con 0x7f1b380a3ff0 auth_method 0 2024-09-16T04:46:46.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:46.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:46.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:46.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:46.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:46.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:46.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3c9cb640 10 monclient(hunting): handle_auth_done global_id 94268 payload 824 2024-09-16T04:46:46.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3c9cb640 1 -- 172.21.15.46:0/3231512706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b380a3ff0 msgr2=0x7f1b381cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:46.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3c9cb640 1 --2- 172.21.15.46:0/3231512706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b380a3ff0 0x7f1b381cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:46.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3c9cb640 1 -- 172.21.15.46:0/3231512706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b380a3c10 msgr2=0x7f1b381cbee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:46:46.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3c9cb640 1 --2- 172.21.15.46:0/3231512706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b380a3c10 0x7f1b381cbee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:46.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3c9cb640 10 monclient: _finish_hunting 0 2024-09-16T04:46:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3c9cb640 1 monclient: found mon.noname-c 2024-09-16T04:46:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3c9cb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3c9cb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:46:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3c9cb640 1 -- 172.21.15.46:0/3231512706 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b38125a80 con 0x7f1b381369d0 2024-09-16T04:46:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.978+0000 7f1b3c9cb640 1 --2- 172.21.15.46:0/3231512706 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1b381369d0 0x7f1b38123280 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1b3000caa0 tx=0x7f1b3000cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.979+0000 7f1b34ff9640 1 -- 172.21.15.46:0/3231512706 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b30017070 con 0x7f1b381369d0 2024-09-16T04:46:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.979+0000 7f1b34ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.979+0000 7f1b34ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.979+0000 7f1b34ff9640 10 monclient: dump: 2024-09-16T04:46:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:46.996 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:46.996 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:46.996 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:46.996 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:46.996 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:46.996 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:46.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.979+0000 7f1b34ff9640 10 monclient: _finish_auth 0 2024-09-16T04:46:46.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.979+0000 7f1b34ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:46:46.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.979+0000 7f1b34ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:46.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.979+0000 7f1b34ff9640 1 -- 172.21.15.46:0/3231512706 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b3000ab40 con 0x7f1b381369d0 2024-09-16T04:46:46.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.979+0000 7f1b3e42e640 5 monclient: authenticate success, global_id 94268 2024-09-16T04:46:46.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.979+0000 7f1b3e42e640 10 monclient: _renew_subs 2024-09-16T04:46:46.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.979+0000 7f1b3e42e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:46.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.979+0000 7f1b3e42e640 1 -- 172.21.15.46:0/3231512706 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b380a0400 con 0x7f1b381369d0 2024-09-16T04:46:46.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.979+0000 7f1b34ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:46.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.979+0000 7f1b34ff9640 1 -- 172.21.15.46:0/3231512706 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b30012400 con 0x7f1b381369d0 2024-09-16T04:46:46.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.979+0000 7f1b34ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:46.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.979+0000 7f1b34ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:46.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.979+0000 7f1b34ff9640 10 monclient: dump: 2024-09-16T04:46:46.997 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:46.997 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:46.997 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:46.997 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:46.997 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:46.997 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:46.997 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:46.997 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:46.997 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:46.998 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:46.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.979+0000 7f1b3e42e640 10 monclient: _renew_subs 2024-09-16T04:46:46.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.979+0000 7f1b3e42e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:46.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.979+0000 7f1b3e42e640 1 -- 172.21.15.46:0/3231512706 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b3806daa0 con 0x7f1b381369d0 2024-09-16T04:46:46.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.979+0000 7f1b367fc640 4 set_mon_vals no callback set 2024-09-16T04:46:46.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.980+0000 7f1b34ff9640 1 -- 172.21.15.46:0/3231512706 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f1b3001a070 con 0x7f1b381369d0 2024-09-16T04:46:46.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.980+0000 7f1b3e42e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:46.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.980+0000 7f1b3e42e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:46.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.980+0000 7f1b3e42e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:46.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.980+0000 7f1b3e42e640 1 -- 172.21.15.46:0/3231512706 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b38137580 con 0x7f1b381369d0 2024-09-16T04:46:46.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.980+0000 7f1b34ff9640 1 --2- 172.21.15.46:0/3231512706 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1b240383b0 0x7f1b2403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:46.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.980+0000 7f1b34ff9640 1 -- 172.21.15.46:0/3231512706 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(304..304 src has 1..304) ==== 4738+0+0 (secure 0 0 0) 0x7f1b3004b380 con 0x7f1b381369d0 2024-09-16T04:46:46.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.980+0000 7f1b37fff640 1 --2- 172.21.15.46:0/3231512706 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1b240383b0 0x7f1b2403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:46.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.980+0000 7f1b37fff640 10 monclient: get_auth_request con 0x7f1b240383b0 auth_method 0 2024-09-16T04:46:46.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.980+0000 7f1b37fff640 1 --2- 172.21.15.46:0/3231512706 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1b240383b0 0x7f1b2403a870 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1b280095c0 tx=0x7f1b28008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:46.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.985+0000 7f1b34ff9640 1 -- 172.21.15.46:0/3231512706 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1b30007a00 con 0x7f1b381369d0 2024-09-16T04:46:46.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.985+0000 7f1b34ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:46.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:46.985+0000 7f1b34ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:47.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:47.232+0000 7f1b3e42e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:46:47.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:47.232+0000 7f1b3e42e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:47.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:47.232+0000 7f1b3e42e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:47.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:47.232+0000 7f1b3e42e640 1 -- 172.21.15.46:0/3231512706 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1b38067e40 con 0x7f1b381369d0 2024-09-16T04:46:47.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:47.232+0000 7f1b34ff9640 1 -- 172.21.15.46:0/3231512706 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f1b3001a520 con 0x7f1b381369d0 2024-09-16T04:46:47.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:47.232+0000 7f1b34ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:47.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:47.232+0000 7f1b34ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:47.235 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":90,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:46:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:47.234+0000 7f1b3e42e640 1 -- 172.21.15.46:0/3231512706 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1b240383b0 msgr2=0x7f1b2403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:47.234+0000 7f1b3e42e640 1 --2- 172.21.15.46:0/3231512706 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1b240383b0 0x7f1b2403a870 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1b280095c0 tx=0x7f1b28008040 comp rx=0 tx=0).stop 2024-09-16T04:46:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:47.234+0000 7f1b3e42e640 10 monclient: shutdown 2024-09-16T04:46:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:47.234+0000 7f1b3e42e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:47.234+0000 7f1b3e42e640 1 -- 172.21.15.46:0/3231512706 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1b381369d0 msgr2=0x7f1b38123280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:47.234+0000 7f1b3e42e640 1 --2- 172.21.15.46:0/3231512706 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1b381369d0 0x7f1b38123280 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1b3000caa0 tx=0x7f1b3000cf60 comp rx=0 tx=0).stop 2024-09-16T04:46:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:47.234+0000 7f1b3e42e640 1 -- 172.21.15.46:0/3231512706 shutdown_connections 2024-09-16T04:46:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:47.234+0000 7f1b3e42e640 1 --2- 172.21.15.46:0/3231512706 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f1b381369d0 0x7f1b38123280 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:47.234+0000 7f1b3e42e640 1 --2- 172.21.15.46:0/3231512706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b380a3ff0 0x7f1b381cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:47.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:47.234+0000 7f1b3e42e640 1 --2- 172.21.15.46:0/3231512706 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f1b240383b0 0x7f1b2403a870 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:47.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:47.234+0000 7f1b3e42e640 1 --2- 172.21.15.46:0/3231512706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b380a3c10 0x7f1b381cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:47.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:47.234+0000 7f1b3e42e640 1 -- 172.21.15.46:0/3231512706 >> 172.21.15.46:0/3231512706 conn(0x7f1b380a5d10 msgr2=0x7f1b3809e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:47.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:47.235+0000 7f1b3e42e640 1 -- 172.21.15.46:0/3231512706 shutdown_connections 2024-09-16T04:46:47.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:47.235+0000 7f1b3e42e640 1 -- 172.21.15.46:0/3231512706 wait complete. 2024-09-16T04:46:47.325 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40245]: starting. 2024-09-16T04:46:47.326 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40245]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-40244 2024-09-16T04:46:47.326 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40245]: created object 0... 2024-09-16T04:46:47.326 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40245]: created object 25... 2024-09-16T04:46:47.327 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40245]: created object 49... 2024-09-16T04:46:47.327 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40245]: finishing. 2024-09-16T04:46:47.327 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40245]: shutting down. 2024-09-16T04:46:48.250 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[40246]: starting. 2024-09-16T04:46:48.250 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[40246]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-40244 2024-09-16T04:46:48.251 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[40246]: shutting down. 2024-09-16T04:46:50.252 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:46:50.256 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:50.256 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40281]: starting. 2024-09-16T04:46:50.256 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40281]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-40244 2024-09-16T04:46:50.257 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40281]: created object 0... 2024-09-16T04:46:50.257 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40281]: created object 25... 2024-09-16T04:46:50.257 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40281]: created object 49... 2024-09-16T04:46:50.257 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40281]: finishing. 2024-09-16T04:46:50.257 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40281]: shutting down. 2024-09-16T04:46:50.269 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:50.269 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40283]: starting. 2024-09-16T04:46:50.269 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40283]: listing objects. 2024-09-16T04:46:50.269 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40283]: listed object 0... 2024-09-16T04:46:50.269 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40283]: listed object 25... 2024-09-16T04:46:50.270 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40283]: saw 50 objects 2024-09-16T04:46:50.270 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40283]: shutting down. 2024-09-16T04:46:50.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.356+0000 7f198892c640 10 monclient: get_monmap_and_config 2024-09-16T04:46:50.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.356+0000 7f198892c640 10 monclient: build_initial_monmap 2024-09-16T04:46:50.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.356+0000 7f198892c640 10 monclient: monmap: 2024-09-16T04:46:50.357 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:50.357 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:50.357 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:50.356966+0000 2024-09-16T04:46:50.358 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:50.356966+0000 2024-09-16T04:46:50.358 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:50.358 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:50.358 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:50.358 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:50.358 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:50.358 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:50.358 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:50.358 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:50.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.357+0000 7f198892c640 1 Processor -- start 2024-09-16T04:46:50.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.357+0000 7f198892c640 1 -- start start 2024-09-16T04:46:50.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.357+0000 7f198892c640 10 monclient: init 2024-09-16T04:46:50.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.357+0000 7f198892c640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:50.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.357+0000 7f198892c640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-16T04:46:50.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.357+0000 7f198892c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1984131a10 0x7f198409ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:50.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.357+0000 7f198892c640 10 monclient(hunting): picked mon.noname-a con 0x7f1984131a10 addr v2:172.21.15.46:3300/0 2024-09-16T04:46:50.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.357+0000 7f198892c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f198409e760 0x7f1984131df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:50.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.357+0000 7f198892c640 10 monclient(hunting): picked mon.noname-b con 0x7f198409e760 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:50.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.357+0000 7f198892c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f19841347d0 addr v1:172.21.15.46:6789/0 2024-09-16T04:46:50.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.357+0000 7f198892c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19840a0f40 con 0x7f19841347d0 2024-09-16T04:46:50.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.357+0000 7f198892c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:50.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.357+0000 7f198892c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f19840a1220 con 0x7f1984131a10 2024-09-16T04:46:50.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.357+0000 7f198892c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:50.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.357+0000 7f198892c640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f1984132330 con 0x7f198409e760 2024-09-16T04:46:50.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.357+0000 7f198892c640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:50.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.357+0000 7f198892c640 10 monclient(hunting): authenticate will time out at 1208.772949s 2024-09-16T04:46:50.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f198259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1984131a10 0x7f198409ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:50.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f1981d9b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f198409e760 0x7f1984131df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:50.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f198259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1984131a10 0x7f198409ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57860/0 (socket says 172.21.15.46:57860) 2024-09-16T04:46:50.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f1981d9b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f198409e760 0x7f1984131df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:56808/0 (socket says 172.21.15.46:56808) 2024-09-16T04:46:50.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f198259c640 1 -- 172.21.15.46:0/112919422 learned_addr learned my addr 172.21.15.46:0/112919422 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:50.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f198259c640 10 monclient(hunting): get_auth_request con 0x7f1984131a10 auth_method 0 2024-09-16T04:46:50.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f198259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:50.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f198259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:50.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f198259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:50.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f1981d9b640 10 monclient(hunting): get_auth_request con 0x7f198409e760 auth_method 0 2024-09-16T04:46:50.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f1981d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:50.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f1981d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:50.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f1981d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:50.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f1981d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:50.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f1981d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:50.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f198259c640 1 -- 172.21.15.46:0/112919422 >> v2:172.21.15.46:3300/0 conn(0x7f1984131a10 msgr2=0x7f198409ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:46:50.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f198259c640 1 -- 172.21.15.46:0/112919422 >> v2:172.21.15.46:3300/0 conn(0x7f1984131a10 msgr2=0x7f198409ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:46:50.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f198259c640 1 --2- 172.21.15.46:0/112919422 >> v2:172.21.15.46:3300/0 conn(0x7f1984131a10 0x7f198409ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:50.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f1981d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:50.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f198259c640 1 --2- 172.21.15.46:0/112919422 >> v2:172.21.15.46:3300/0 conn(0x7f1984131a10 0x7f198409ec70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:50.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f1981d9b640 10 monclient(hunting): handle_auth_done global_id 94286 payload 274 2024-09-16T04:46:50.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f1981d9b640 1 -- 172.21.15.46:0/112919422 >> v2:172.21.15.46:3300/0 conn(0x7f1984131a10 msgr2=0x7f198409ec70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:46:50.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f1981d9b640 1 --2- 172.21.15.46:0/112919422 >> v2:172.21.15.46:3300/0 conn(0x7f1984131a10 0x7f198409ec70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:50.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f1981d9b640 1 -- 172.21.15.46:0/112919422 >> v1:172.21.15.46:6789/0 conn(0x7f19841347d0 legacy=0x7f1984134c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:50.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f1981d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:46:50.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f1981d9b640 1 monclient: found mon.noname-b 2024-09-16T04:46:50.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f1981d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:50.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f1981d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:46:50.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.358+0000 7f1981d9b640 1 -- 172.21.15.46:0/112919422 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f198409f810 con 0x7f198409e760 2024-09-16T04:46:50.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f1981d9b640 1 --2- 172.21.15.46:0/112919422 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f198409e760 0x7f1984131df0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f19780097a0 tx=0x7f197802a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff474349c20c2961 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:50.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198159a640 1 -- 172.21.15.46:0/112919422 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1978037070 con 0x7f198409e760 2024-09-16T04:46:50.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:50.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:46:50.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198159a640 10 monclient: dump: 2024-09-16T04:46:50.370 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:50.370 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:50.370 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:50.370 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:50.370 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:50.370 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:50.370 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:50.370 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:50.370 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:50.371 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:50.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198159a640 10 monclient: _finish_auth 0 2024-09-16T04:46:50.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198159a640 10 monclient: _check_auth_tickets 2024-09-16T04:46:50.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:50.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198892c640 5 monclient: authenticate success, global_id 94286 2024-09-16T04:46:50.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198892c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:50.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198159a640 1 -- 172.21.15.46:0/112919422 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f197802adb0 con 0x7f198409e760 2024-09-16T04:46:50.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:50.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198892c640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:50.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198892c640 4 set_mon_vals no callback set 2024-09-16T04:46:50.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198159a640 1 -- 172.21.15.46:0/112919422 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1978032660 con 0x7f198409e760 2024-09-16T04:46:50.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:50.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:50.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198159a640 10 monclient: dump: 2024-09-16T04:46:50.372 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:50.372 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:50.372 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:50.372 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:50.372 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:50.372 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:50.372 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:50.372 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:50.372 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:50.372 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:50.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198892c640 10 monclient: shutdown 2024-09-16T04:46:50.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198892c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:50.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198892c640 1 -- 172.21.15.46:0/112919422 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f198409e760 msgr2=0x7f1984131df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198892c640 1 --2- 172.21.15.46:0/112919422 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f198409e760 0x7f1984131df0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f19780097a0 tx=0x7f197802a060 comp rx=0 tx=0).stop 2024-09-16T04:46:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198892c640 1 -- 172.21.15.46:0/112919422 shutdown_connections 2024-09-16T04:46:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198892c640 1 --2- 172.21.15.46:0/112919422 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f198409e760 0x7f1984131df0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198892c640 1 --2- 172.21.15.46:0/112919422 >> v2:172.21.15.46:3300/0 conn(0x7f1984131a10 0x7f198409ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198892c640 1 -- 172.21.15.46:0/112919422 >> 172.21.15.46:0/112919422 conn(0x7f19840a5960 msgr2=0x7f19840a1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198892c640 1 -- 172.21.15.46:0/112919422 shutdown_connections 2024-09-16T04:46:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.359+0000 7f198892c640 1 -- 172.21.15.46:0/112919422 wait complete. 2024-09-16T04:46:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.360+0000 7f198892c640 10 monclient: build_initial_monmap 2024-09-16T04:46:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.360+0000 7f198892c640 10 monclient: monmap: 2024-09-16T04:46:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:50.374 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:50.374 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:50.374 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:50.374 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:50.374 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:50.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.360+0000 7f198892c640 1 Processor -- start 2024-09-16T04:46:50.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198892c640 1 -- start start 2024-09-16T04:46:50.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198892c640 10 monclient: init 2024-09-16T04:46:50.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198892c640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:50.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198892c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:46:50.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198892c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f198409e760 0x7f19841304e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:50.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198892c640 10 monclient(hunting): picked mon.noname-b con 0x7f198409e760 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:50.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198892c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1984131a10 0x7f1984138250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:50.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198892c640 10 monclient(hunting): picked mon.noname-a con 0x7f1984131a10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:50.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198892c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f19841347d0 0x7f1984138790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:50.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198892c640 10 monclient(hunting): picked mon.noname-c con 0x7f19841347d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:50.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198892c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:50.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198892c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f19841354f0 con 0x7f1984131a10 2024-09-16T04:46:50.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198892c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:50.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198892c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1984135370 con 0x7f198409e760 2024-09-16T04:46:50.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198892c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:50.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198892c640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f198413cb10 con 0x7f19841347d0 2024-09-16T04:46:50.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198892c640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:50.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198892c640 10 monclient(hunting): authenticate will time out at 1208.776502s 2024-09-16T04:46:50.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1981d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1984131a10 0x7f1984138250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:50.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1982d9d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f19841347d0 0x7f1984138790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:50.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1981d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1984131a10 0x7f1984138250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57864/0 (socket says 172.21.15.46:57864) 2024-09-16T04:46:50.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1981d9b640 1 -- 172.21.15.46:0/3573224706 learned_addr learned my addr 172.21.15.46:0/3573224706 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:50.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1981d9b640 10 monclient(hunting): get_auth_request con 0x7f1984131a10 auth_method 0 2024-09-16T04:46:50.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1981d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:50.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1981d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:50.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1981d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:50.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1982d9d640 10 monclient(hunting): get_auth_request con 0x7f19841347d0 auth_method 0 2024-09-16T04:46:50.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1982d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:50.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1982d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:50.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1982d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:50.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1981d9b640 1 -- 172.21.15.46:0/3573224706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1984131a10 msgr2=0x7f1984138250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:46:50.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1981d9b640 1 -- 172.21.15.46:0/3573224706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1984131a10 msgr2=0x7f1984138250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:46:50.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1981d9b640 1 --2- 172.21.15.46:0/3573224706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1984131a10 0x7f1984138250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:46:50.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1982d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:50.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1982d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:50.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1982d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:50.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1981d9b640 1 --2- 172.21.15.46:0/3573224706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1984131a10 0x7f1984138250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:50.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198259c640 1 --2- 172.21.15.46:0/3573224706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f198409e760 0x7f19841304e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:50.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198259c640 10 monclient(hunting): get_auth_request con 0x7f198409e760 auth_method 0 2024-09-16T04:46:50.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:50.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:50.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:50.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1982d9d640 10 monclient(hunting): handle_auth_done global_id 94289 payload 824 2024-09-16T04:46:50.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1982d9d640 1 -- 172.21.15.46:0/3573224706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f198409e760 msgr2=0x7f19841304e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:50.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1982d9d640 1 --2- 172.21.15.46:0/3573224706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f198409e760 0x7f19841304e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:50.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1982d9d640 1 -- 172.21.15.46:0/3573224706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1984131a10 msgr2=0x7f1984138250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:46:50.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1982d9d640 1 --2- 172.21.15.46:0/3573224706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1984131a10 0x7f1984138250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:50.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1982d9d640 10 monclient: _finish_hunting 0 2024-09-16T04:46:50.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1982d9d640 1 monclient: found mon.noname-c 2024-09-16T04:46:50.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1982d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:50.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1982d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:46:50.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f1982d9d640 1 -- 172.21.15.46:0/3573224706 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f198413c1e0 con 0x7f19841347d0 2024-09-16T04:46:50.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.361+0000 7f198259c640 1 --2- 172.21.15.46:0/3573224706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f198409e760 0x7f19841304e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:46:50.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f1982d9d640 1 --2- 172.21.15.46:0/3573224706 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f19841347d0 0x7f1984138790 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f197c00c690 tx=0x7f197c00cb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:50.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f1972ffd640 1 -- 172.21.15.46:0/3573224706 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c003b70 con 0x7f19841347d0 2024-09-16T04:46:50.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f1972ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:50.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f1972ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:50.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f1972ffd640 10 monclient: dump: 2024-09-16T04:46:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:50.380 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f1972ffd640 10 monclient: _finish_auth 0 2024-09-16T04:46:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f1972ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:46:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f1972ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f1972ffd640 1 -- 172.21.15.46:0/3573224706 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f197c003d10 con 0x7f19841347d0 2024-09-16T04:46:50.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f198892c640 5 monclient: authenticate success, global_id 94289 2024-09-16T04:46:50.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f198892c640 10 monclient: _renew_subs 2024-09-16T04:46:50.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f198892c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:50.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f198892c640 1 -- 172.21.15.46:0/3573224706 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f198409c100 con 0x7f19841347d0 2024-09-16T04:46:50.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f1972ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:50.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f1972ffd640 1 -- 172.21.15.46:0/3573224706 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c005310 con 0x7f19841347d0 2024-09-16T04:46:50.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f1972ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:50.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f1972ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:50.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f1972ffd640 10 monclient: dump: 2024-09-16T04:46:50.381 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:50.381 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:50.381 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:50.381 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:50.382 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:50.382 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:50.382 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:50.382 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:50.382 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:50.382 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:50.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f1980d99640 4 set_mon_vals no callback set 2024-09-16T04:46:50.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f198892c640 10 monclient: _renew_subs 2024-09-16T04:46:50.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f198892c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:50.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.362+0000 7f198892c640 1 -- 172.21.15.46:0/3573224706 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f198413bc20 con 0x7f19841347d0 2024-09-16T04:46:50.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.363+0000 7f1972ffd640 1 -- 172.21.15.46:0/3573224706 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f197c0054b0 con 0x7f19841347d0 2024-09-16T04:46:50.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.363+0000 7f1972ffd640 1 --2- 172.21.15.46:0/3573224706 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f19640383b0 0x7f196403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:50.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.363+0000 7f1972ffd640 1 -- 172.21.15.46:0/3573224706 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(308..308 src has 1..308) ==== 4738+0+0 (secure 0 0 0) 0x7f197c04b620 con 0x7f19841347d0 2024-09-16T04:46:50.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.363+0000 7f198892c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:50.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.363+0000 7f198892c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:50.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.363+0000 7f198892c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:50.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.363+0000 7f198892c640 1 -- 172.21.15.46:0/3573224706 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1984131df0 con 0x7f19841347d0 2024-09-16T04:46:50.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.363+0000 7f198259c640 1 --2- 172.21.15.46:0/3573224706 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f19640383b0 0x7f196403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:50.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.363+0000 7f198259c640 10 monclient: get_auth_request con 0x7f19640383b0 auth_method 0 2024-09-16T04:46:50.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.363+0000 7f198259c640 1 --2- 172.21.15.46:0/3573224706 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f19640383b0 0x7f196403a870 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1974009100 tx=0x7f1974008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:50.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.368+0000 7f1972ffd640 1 -- 172.21.15.46:0/3573224706 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f197c0102e0 con 0x7f19841347d0 2024-09-16T04:46:50.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.368+0000 7f1972ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:50.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.368+0000 7f1972ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:50.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.616+0000 7f198892c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:46:50.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.616+0000 7f198892c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:50.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.616+0000 7f198892c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:50.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.616+0000 7f198892c640 1 -- 172.21.15.46:0/3573224706 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f198413a7d0 con 0x7f19841347d0 2024-09-16T04:46:50.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.616+0000 7f1972ffd640 1 -- 172.21.15.46:0/3573224706 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f197c00adc0 con 0x7f19841347d0 2024-09-16T04:46:50.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.616+0000 7f1972ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:50.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.616+0000 7f1972ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:50.618 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":90,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:46:50.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.618+0000 7f198892c640 1 -- 172.21.15.46:0/3573224706 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f19640383b0 msgr2=0x7f196403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:50.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.618+0000 7f198892c640 1 --2- 172.21.15.46:0/3573224706 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f19640383b0 0x7f196403a870 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1974009100 tx=0x7f1974008040 comp rx=0 tx=0).stop 2024-09-16T04:46:50.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.618+0000 7f198892c640 10 monclient: shutdown 2024-09-16T04:46:50.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.618+0000 7f198892c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:50.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.618+0000 7f198892c640 1 -- 172.21.15.46:0/3573224706 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f19841347d0 msgr2=0x7f1984138790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:50.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.618+0000 7f198892c640 1 --2- 172.21.15.46:0/3573224706 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f19841347d0 0x7f1984138790 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f197c00c690 tx=0x7f197c00cb50 comp rx=0 tx=0).stop 2024-09-16T04:46:50.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.618+0000 7f198892c640 1 -- 172.21.15.46:0/3573224706 shutdown_connections 2024-09-16T04:46:50.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.618+0000 7f198892c640 1 --2- 172.21.15.46:0/3573224706 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f19841347d0 0x7f1984138790 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:50.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.618+0000 7f198892c640 1 --2- 172.21.15.46:0/3573224706 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f19640383b0 0x7f196403a870 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:50.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.618+0000 7f198892c640 1 --2- 172.21.15.46:0/3573224706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1984131a10 0x7f1984138250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:50.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.618+0000 7f198892c640 1 --2- 172.21.15.46:0/3573224706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f198409e760 0x7f19841304e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:50.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.618+0000 7f198892c640 1 -- 172.21.15.46:0/3573224706 >> 172.21.15.46:0/3573224706 conn(0x7f19840a5960 msgr2=0x7f1984132bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:50.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.618+0000 7f198892c640 1 -- 172.21.15.46:0/3573224706 shutdown_connections 2024-09-16T04:46:50.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:50.618+0000 7f198892c640 1 -- 172.21.15.46:0/3573224706 wait complete. 2024-09-16T04:46:51.178 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:51.178 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[40282]: starting. 2024-09-16T04:46:51.178 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[40282]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-40244 2024-09-16T04:46:51.178 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[40282]: shutting down. 2024-09-16T04:46:51.179 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:51.179 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:51.179 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:46:51.181 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T04:46:53.322 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40352]: starting. 2024-09-16T04:46:53.322 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40352]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-40351 2024-09-16T04:46:53.322 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40352]: created object 0... 2024-09-16T04:46:53.323 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40352]: created object 25... 2024-09-16T04:46:53.323 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40352]: created object 49... 2024-09-16T04:46:53.323 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40352]: finishing. 2024-09-16T04:46:53.323 INFO:teuthology.orchestra.run.smithi114.stdout:process_1_[40352]: shutting down. 2024-09-16T04:46:53.630 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:46:53.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.736+0000 7ff44e21e640 10 monclient: get_monmap_and_config 2024-09-16T04:46:53.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.736+0000 7ff44e21e640 10 monclient: build_initial_monmap 2024-09-16T04:46:53.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.736+0000 7ff44e21e640 10 monclient: monmap: 2024-09-16T04:46:53.737 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:53.737 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:53.737 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:53.736867+0000 2024-09-16T04:46:53.738 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:53.736867+0000 2024-09-16T04:46:53.738 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:53.738 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:53.738 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:53.738 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:53.738 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:53.738 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:53.738 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:53.738 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:53.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.737+0000 7ff44e21e640 1 Processor -- start 2024-09-16T04:46:53.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.737+0000 7ff44e21e640 1 -- start start 2024-09-16T04:46:53.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.737+0000 7ff44e21e640 10 monclient: init 2024-09-16T04:46:53.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.737+0000 7ff44e21e640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:53.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.737+0000 7ff44e21e640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-16T04:46:53.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.737+0000 7ff44e21e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4480a3c10 0x7ff44809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:53.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.737+0000 7ff44e21e640 10 monclient(hunting): picked mon.noname-b con 0x7ff4480a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:53.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.737+0000 7ff44e21e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff4480a3ff0 0x7ff4480a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:53.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.737+0000 7ff44e21e640 10 monclient(hunting): picked mon.noname-a con 0x7ff4480a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:46:53.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.737+0000 7ff44e21e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff4481369d0 0x7ff448136e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:53.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.737+0000 7ff44e21e640 10 monclient(hunting): picked mon.noname-c con 0x7ff4481369d0 addr v2:172.21.15.114:3300/0 2024-09-16T04:46:53.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.737+0000 7ff44e21e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:53.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.737+0000 7ff44e21e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff44809b940 con 0x7ff4480a3ff0 2024-09-16T04:46:53.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.737+0000 7ff44e21e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:53.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.737+0000 7ff44e21e640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ff44809ab00 con 0x7ff4481369d0 2024-09-16T04:46:53.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.737+0000 7ff44e21e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:53.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.737+0000 7ff44e21e640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7ff44809b030 con 0x7ff4480a3c10 2024-09-16T04:46:53.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.737+0000 7ff44e21e640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:53.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.737+0000 7ff44e21e640 10 monclient(hunting): authenticate will time out at 1212.152899s 2024-09-16T04:46:53.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff4477fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4480a3c10 0x7ff44809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:53.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff43ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff4480a3ff0 0x7ff4480a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:53.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff4477fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4480a3c10 0x7ff44809bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:56836/0 (socket says 172.21.15.46:56836) 2024-09-16T04:46:53.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff4477fe640 1 -- 172.21.15.46:0/2848759723 learned_addr learned my addr 172.21.15.46:0/2848759723 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:53.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff43ffff640 10 monclient(hunting): get_auth_request con 0x7ff4480a3ff0 auth_method 0 2024-09-16T04:46:53.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff43ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:53.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff43ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:53.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff43ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:53.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff4477fe640 10 monclient(hunting): get_auth_request con 0x7ff4480a3c10 auth_method 0 2024-09-16T04:46:53.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff4477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:53.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff4477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:53.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff4477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:53.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff43ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:53.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff43ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:53.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff43ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:53.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff4477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:53.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff4477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:53.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff4477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:53.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff447fff640 1 --2- 172.21.15.46:0/2848759723 >> v2:172.21.15.114:3300/0 conn(0x7ff4481369d0 0x7ff448136e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:53.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff447fff640 10 monclient(hunting): get_auth_request con 0x7ff4481369d0 auth_method 0 2024-09-16T04:46:53.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff447fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:53.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff447fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:53.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff447fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:53.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff43ffff640 10 monclient(hunting): handle_auth_done global_id 104097 payload 274 2024-09-16T04:46:53.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff43ffff640 1 -- 172.21.15.46:0/2848759723 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4480a3c10 msgr2=0x7ff44809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:53.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff43ffff640 1 --2- 172.21.15.46:0/2848759723 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4480a3c10 0x7ff44809bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:53.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff43ffff640 1 -- 172.21.15.46:0/2848759723 >> v2:172.21.15.114:3300/0 conn(0x7ff4481369d0 msgr2=0x7ff448136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:53.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff43ffff640 1 --2- 172.21.15.46:0/2848759723 >> v2:172.21.15.114:3300/0 conn(0x7ff4481369d0 0x7ff448136e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:53.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff43ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:46:53.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff43ffff640 1 monclient: found mon.noname-a 2024-09-16T04:46:53.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff43ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:53.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff43ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:46:53.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff43ffff640 1 -- 172.21.15.46:0/2848759723 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff44809b310 con 0x7ff4480a3ff0 2024-09-16T04:46:53.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.738+0000 7ff4477fe640 1 --2- 172.21.15.46:0/2848759723 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4480a3c10 0x7ff44809bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:53.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff43ffff640 1 --2- 172.21.15.46:0/2848759723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480a3ff0 0x7ff4480a43d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff448068160 tx=0x7ff430029fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c0fb6ace70615ea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:53.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff446ffd640 1 -- 172.21.15.46:0/2848759723 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff430037070 con 0x7ff4480a3ff0 2024-09-16T04:46:53.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff446ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:53.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff446ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:46:53.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff446ffd640 10 monclient: dump: 2024-09-16T04:46:53.751 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:53.751 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:53.751 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:53.751 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:53.751 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:53.751 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:53.751 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:53.751 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:53.751 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:53.751 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:53.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff446ffd640 10 monclient: _finish_auth 0 2024-09-16T04:46:53.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff446ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:46:53.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff446ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:53.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff446ffd640 1 -- 172.21.15.46:0/2848759723 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff43002ad30 con 0x7ff4480a3ff0 2024-09-16T04:46:53.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff446ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:53.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff446ffd640 1 -- 172.21.15.46:0/2848759723 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff430032660 con 0x7ff4480a3ff0 2024-09-16T04:46:53.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff44e21e640 5 monclient: authenticate success, global_id 104097 2024-09-16T04:46:53.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff44e21e640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:53.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff44e21e640 4 set_mon_vals no callback set 2024-09-16T04:46:53.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff446ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:53.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff446ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:53.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff446ffd640 10 monclient: dump: 2024-09-16T04:46:53.752 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:53.752 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:53.752 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:53.753 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:53.753 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:53.753 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:53.753 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:53.753 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:53.753 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:53.753 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:53.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff44e21e640 10 monclient: shutdown 2024-09-16T04:46:53.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff44e21e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:53.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff44e21e640 1 -- 172.21.15.46:0/2848759723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480a3ff0 msgr2=0x7ff4480a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:53.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff44e21e640 1 --2- 172.21.15.46:0/2848759723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480a3ff0 0x7ff4480a43d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff448068160 tx=0x7ff430029fe0 comp rx=0 tx=0).stop 2024-09-16T04:46:53.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff44e21e640 1 -- 172.21.15.46:0/2848759723 shutdown_connections 2024-09-16T04:46:53.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff44e21e640 1 --2- 172.21.15.46:0/2848759723 >> v2:172.21.15.114:3300/0 conn(0x7ff4481369d0 0x7ff448136e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:53.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff44e21e640 1 --2- 172.21.15.46:0/2848759723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480a3ff0 0x7ff4480a43d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:53.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff44e21e640 1 --2- 172.21.15.46:0/2848759723 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4480a3c10 0x7ff44809bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:53.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff44e21e640 1 -- 172.21.15.46:0/2848759723 >> 172.21.15.46:0/2848759723 conn(0x7ff4480a6340 msgr2=0x7ff44806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:53.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff44e21e640 1 -- 172.21.15.46:0/2848759723 shutdown_connections 2024-09-16T04:46:53.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.739+0000 7ff44e21e640 1 -- 172.21.15.46:0/2848759723 wait complete. 2024-09-16T04:46:53.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.740+0000 7ff44e21e640 10 monclient: build_initial_monmap 2024-09-16T04:46:53.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.740+0000 7ff44e21e640 10 monclient: monmap: 2024-09-16T04:46:53.754 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:53.754 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:53.754 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:53.754 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:53.754 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:53.754 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:53.754 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:53.755 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:53.755 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:53.755 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:53.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.740+0000 7ff44e21e640 1 Processor -- start 2024-09-16T04:46:53.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.740+0000 7ff44e21e640 1 -- start start 2024-09-16T04:46:53.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.740+0000 7ff44e21e640 10 monclient: init 2024-09-16T04:46:53.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff44e21e640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:53.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff44e21e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:46:53.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff44e21e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480a3c10 0x7ff4481cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:53.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff44e21e640 10 monclient(hunting): picked mon.noname-a con 0x7ff4480a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:53.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff44e21e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4480a3ff0 0x7ff4481cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:53.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff44e21e640 10 monclient(hunting): picked mon.noname-c con 0x7ff4480a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:53.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff44e21e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4481369d0 0x7ff448123400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:53.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff44e21e640 10 monclient(hunting): picked mon.noname-b con 0x7ff4481369d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:53.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff44e21e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:53.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff44e21e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff4480a4910 con 0x7ff4480a3c10 2024-09-16T04:46:53.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff44e21e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:53.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff44e21e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff4480a69c0 con 0x7ff4481369d0 2024-09-16T04:46:53.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff44e21e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:53.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff44e21e640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7ff448127280 con 0x7ff4480a3ff0 2024-09-16T04:46:53.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff44e21e640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:53.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff44e21e640 10 monclient(hunting): authenticate will time out at 1212.156370s 2024-09-16T04:46:53.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff4477fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480a3c10 0x7ff4481cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:53.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff43ffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4480a3ff0 0x7ff4481cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:53.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff4477fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480a3c10 0x7ff4481cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57908/0 (socket says 172.21.15.46:57908) 2024-09-16T04:46:53.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff4477fe640 1 -- 172.21.15.46:0/1664673700 learned_addr learned my addr 172.21.15.46:0/1664673700 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:53.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff4477fe640 10 monclient(hunting): get_auth_request con 0x7ff4480a3c10 auth_method 0 2024-09-16T04:46:53.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff4477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:53.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff43ffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4480a3ff0 0x7ff4481cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:56838/0 (socket says 172.21.15.46:56838) 2024-09-16T04:46:53.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff4477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:53.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff4477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:53.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff43ffff640 10 monclient(hunting): get_auth_request con 0x7ff4480a3ff0 auth_method 0 2024-09-16T04:46:53.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff43ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:53.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff43ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:53.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff43ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:53.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff4477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:53.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff4477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:53.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff4477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:53.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff43ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:53.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff43ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff43ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff447fff640 1 --2- 172.21.15.46:0/1664673700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4481369d0 0x7ff448123400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff447fff640 10 monclient(hunting): get_auth_request con 0x7ff4481369d0 auth_method 0 2024-09-16T04:46:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff447fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff447fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff447fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff4477fe640 10 monclient(hunting): handle_auth_done global_id 104100 payload 824 2024-09-16T04:46:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff4477fe640 1 -- 172.21.15.46:0/1664673700 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4480a3ff0 msgr2=0x7ff4481cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff4477fe640 1 --2- 172.21.15.46:0/1664673700 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4480a3ff0 0x7ff4481cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff4477fe640 1 -- 172.21.15.46:0/1664673700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4481369d0 msgr2=0x7ff448123400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff4477fe640 1 --2- 172.21.15.46:0/1664673700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4481369d0 0x7ff448123400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff4477fe640 10 monclient: _finish_hunting 0 2024-09-16T04:46:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff4477fe640 1 monclient: found mon.noname-a 2024-09-16T04:46:53.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff4477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:53.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff4477fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:46:53.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff4477fe640 1 -- 172.21.15.46:0/1664673700 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4481253c0 con 0x7ff4480a3c10 2024-09-16T04:46:53.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff43ffff640 1 --2- 172.21.15.46:0/1664673700 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4480a3ff0 0x7ff4481cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:53.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.741+0000 7ff4477fe640 1 --2- 172.21.15.46:0/1664673700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480a3c10 0x7ff4481cbee0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff43800e9d0 tx=0x7ff43800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:53.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff444ff9640 1 -- 172.21.15.46:0/1664673700 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff438017070 con 0x7ff4480a3c10 2024-09-16T04:46:53.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff444ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:53.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff444ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:46:53.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff444ff9640 10 monclient: dump: 2024-09-16T04:46:53.759 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:53.759 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:53.759 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:53.759 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:53.760 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff444ff9640 10 monclient: _finish_auth 0 2024-09-16T04:46:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff444ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:46:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff444ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff44e21e640 5 monclient: authenticate success, global_id 104100 2024-09-16T04:46:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff444ff9640 1 -- 172.21.15.46:0/1664673700 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff43800ca60 con 0x7ff4480a3c10 2024-09-16T04:46:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff44e21e640 10 monclient: _renew_subs 2024-09-16T04:46:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff44e21e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff44e21e640 1 -- 172.21.15.46:0/1664673700 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff448126fa0 con 0x7ff4480a3c10 2024-09-16T04:46:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff444ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff444ff9640 1 -- 172.21.15.46:0/1664673700 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43800ce10 con 0x7ff4480a3c10 2024-09-16T04:46:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff444ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff444ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff444ff9640 10 monclient: dump: 2024-09-16T04:46:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:53.762 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff44e21e640 10 monclient: _renew_subs 2024-09-16T04:46:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff44e21e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff44e21e640 1 -- 172.21.15.46:0/1664673700 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4480a8940 con 0x7ff4480a3c10 2024-09-16T04:46:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff4467fc640 4 set_mon_vals no callback set 2024-09-16T04:46:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.742+0000 7ff444ff9640 1 -- 172.21.15.46:0/1664673700 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7ff43801a070 con 0x7ff4480a3c10 2024-09-16T04:46:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.743+0000 7ff44e21e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.743+0000 7ff44e21e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.743+0000 7ff44e21e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.743+0000 7ff44e21e640 1 -- 172.21.15.46:0/1664673700 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4480a43d0 con 0x7ff4480a3c10 2024-09-16T04:46:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.743+0000 7ff444ff9640 1 --2- 172.21.15.46:0/1664673700 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff4280383b0 0x7ff42803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.743+0000 7ff444ff9640 1 -- 172.21.15.46:0/1664673700 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(311..311 src has 1..311) ==== 4738+0+0 (secure 0 0 0) 0x7ff43804b700 con 0x7ff4480a3c10 2024-09-16T04:46:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.743+0000 7ff43ffff640 1 --2- 172.21.15.46:0/1664673700 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff4280383b0 0x7ff42803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.743+0000 7ff43ffff640 10 monclient: get_auth_request con 0x7ff4280383b0 auth_method 0 2024-09-16T04:46:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.744+0000 7ff43ffff640 1 --2- 172.21.15.46:0/1664673700 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff4280383b0 0x7ff42803a870 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff430034000 tx=0x7ff43002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.748+0000 7ff444ff9640 1 -- 172.21.15.46:0/1664673700 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff438011d50 con 0x7ff4480a3c10 2024-09-16T04:46:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.748+0000 7ff444ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.748+0000 7ff444ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:53.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.996+0000 7ff44e21e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:46:53.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.996+0000 7ff44e21e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:53.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.996+0000 7ff44e21e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:53.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.996+0000 7ff44e21e640 1 -- 172.21.15.46:0/1664673700 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff448067e20 con 0x7ff4480a3c10 2024-09-16T04:46:53.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.996+0000 7ff444ff9640 1 -- 172.21.15.46:0/1664673700 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7ff43801b3b0 con 0x7ff4480a3c10 2024-09-16T04:46:53.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.996+0000 7ff444ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:53.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.996+0000 7ff444ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:53.998 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:46:54.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.998+0000 7ff44e21e640 1 -- 172.21.15.46:0/1664673700 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff4280383b0 msgr2=0x7ff42803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.998+0000 7ff44e21e640 1 --2- 172.21.15.46:0/1664673700 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff4280383b0 0x7ff42803a870 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff430034000 tx=0x7ff43002e000 comp rx=0 tx=0).stop 2024-09-16T04:46:54.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.998+0000 7ff44e21e640 10 monclient: shutdown 2024-09-16T04:46:54.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.998+0000 7ff44e21e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:54.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.998+0000 7ff44e21e640 1 -- 172.21.15.46:0/1664673700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480a3c10 msgr2=0x7ff4481cbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.998+0000 7ff44e21e640 1 --2- 172.21.15.46:0/1664673700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480a3c10 0x7ff4481cbee0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff43800e9d0 tx=0x7ff43800ee90 comp rx=0 tx=0).stop 2024-09-16T04:46:54.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.998+0000 7ff44e21e640 1 -- 172.21.15.46:0/1664673700 shutdown_connections 2024-09-16T04:46:54.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.998+0000 7ff44e21e640 1 --2- 172.21.15.46:0/1664673700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4481369d0 0x7ff448123400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.998+0000 7ff44e21e640 1 --2- 172.21.15.46:0/1664673700 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff4480a3ff0 0x7ff4481cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.998+0000 7ff44e21e640 1 --2- 172.21.15.46:0/1664673700 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff4280383b0 0x7ff42803a870 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.998+0000 7ff44e21e640 1 --2- 172.21.15.46:0/1664673700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480a3c10 0x7ff4481cbee0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.998+0000 7ff44e21e640 1 -- 172.21.15.46:0/1664673700 >> 172.21.15.46:0/1664673700 conn(0x7ff4480a6340 msgr2=0x7ff4480a11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:54.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.999+0000 7ff44e21e640 1 -- 172.21.15.46:0/1664673700 shutdown_connections 2024-09-16T04:46:54.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:53.999+0000 7ff44e21e640 1 -- 172.21.15.46:0/1664673700 wait complete. 2024-09-16T04:46:54.010 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:46:54.010 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:46:54.010 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:46:54.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.112+0000 7fc496023640 10 monclient: get_monmap_and_config 2024-09-16T04:46:54.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.112+0000 7fc496023640 10 monclient: build_initial_monmap 2024-09-16T04:46:54.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.112+0000 7fc496023640 10 monclient: monmap: 2024-09-16T04:46:54.113 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:54.113 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.113 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:54.112622+0000 2024-09-16T04:46:54.114 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:54.112622+0000 2024-09-16T04:46:54.114 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:54.114 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.114 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:54.114 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:54.114 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:54.114 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:54.114 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:54.114 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.112+0000 7fc496023640 1 Processor -- start 2024-09-16T04:46:54.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.112+0000 7fc496023640 1 -- start start 2024-09-16T04:46:54.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.112+0000 7fc496023640 10 monclient: init 2024-09-16T04:46:54.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc496023640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:54.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc496023640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-16T04:46:54.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc496023640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc4900a3c10 0x7fc49009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc496023640 10 monclient(hunting): picked mon.noname-b con 0x7fc4900a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:54.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc496023640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc4900a3ff0 0x7fc4900a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc496023640 10 monclient(hunting): picked mon.noname-a con 0x7fc4900a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:46:54.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc496023640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc4901369d0 addr v1:172.21.15.46:6789/0 2024-09-16T04:46:54.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc496023640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc49009b940 con 0x7fc4901369d0 2024-09-16T04:46:54.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc496023640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc496023640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc49009ab00 con 0x7fc4900a3ff0 2024-09-16T04:46:54.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc496023640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc496023640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fc49009b030 con 0x7fc4900a3c10 2024-09-16T04:46:54.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc496023640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:54.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc496023640 10 monclient(hunting): authenticate will time out at 1212.528438s 2024-09-16T04:46:54.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc4900a3ff0 0x7fc4900a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48f7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc4900a3c10 0x7fc49009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48f7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc4900a3c10 0x7fc49009bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:56842/0 (socket says 172.21.15.46:56842) 2024-09-16T04:46:54.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc4900a3ff0 0x7fc4900a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57920/0 (socket says 172.21.15.46:57920) 2024-09-16T04:46:54.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc4901369d0 0x7fc490136e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48914/0 (socket says 172.21.15.46:48914) 2024-09-16T04:46:54.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48f7fe640 1 -- 172.21.15.46:0/217578199 learned_addr learned my addr 172.21.15.46:0/217578199 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:54.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48f7fe640 10 monclient(hunting): get_auth_request con 0x7fc4900a3c10 auth_method 0 2024-09-16T04:46:54.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48effd640 10 monclient(hunting): get_auth_request con 0x7fc4900a3ff0 auth_method 0 2024-09-16T04:46:54.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48f7fe640 10 monclient(hunting): handle_auth_done global_id 94307 payload 274 2024-09-16T04:46:54.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48f7fe640 1 -- 172.21.15.46:0/217578199 >> v2:172.21.15.46:3300/0 conn(0x7fc4900a3ff0 msgr2=0x7fc4900a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:54.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48f7fe640 1 --2- 172.21.15.46:0/217578199 >> v2:172.21.15.46:3300/0 conn(0x7fc4900a3ff0 0x7fc4900a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48f7fe640 1 -- 172.21.15.46:0/217578199 >> v1:172.21.15.46:6789/0 conn(0x7fc4901369d0 legacy=0x7fc490136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:46:54.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48f7fe640 1 monclient: found mon.noname-b 2024-09-16T04:46:54.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:54.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:46:54.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48f7fe640 1 -- 172.21.15.46:0/217578199 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc49009b310 con 0x7fc4900a3c10 2024-09-16T04:46:54.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.113+0000 7fc48effd640 1 --2- 172.21.15.46:0/217578199 >> v2:172.21.15.46:3300/0 conn(0x7fc4900a3ff0 0x7fc4900a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:54.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.114+0000 7fc48f7fe640 1 --2- 172.21.15.46:0/217578199 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc4900a3c10 0x7fc49009bcf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc480009e50 tx=0x7fc48002a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf7bb61ba7f5c9b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:54.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.114+0000 7fc48e7fc640 1 -- 172.21.15.46:0/217578199 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc480036070 con 0x7fc4900a3c10 2024-09-16T04:46:54.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.114+0000 7fc48e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.114+0000 7fc48e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:46:54.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.114+0000 7fc48e7fc640 10 monclient: dump: 2024-09-16T04:46:54.129 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.129 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.129 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.129 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.129 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.129 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.129 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.129 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.129 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.129 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.114+0000 7fc48e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:46:54.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.114+0000 7fc48e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:46:54.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.114+0000 7fc48e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:54.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.114+0000 7fc48e7fc640 1 -- 172.21.15.46:0/217578199 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc480031600 con 0x7fc4900a3c10 2024-09-16T04:46:54.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.114+0000 7fc496023640 5 monclient: authenticate success, global_id 94307 2024-09-16T04:46:54.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.114+0000 7fc496023640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:54.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.114+0000 7fc48e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:54.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.114+0000 7fc48e7fc640 1 -- 172.21.15.46:0/217578199 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4800319d0 con 0x7fc4900a3c10 2024-09-16T04:46:54.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.114+0000 7fc48e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.114+0000 7fc48e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:54.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.114+0000 7fc48e7fc640 10 monclient: dump: 2024-09-16T04:46:54.130 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.130 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.130 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.131 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.131 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.131 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.131 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.131 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.131 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.131 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.114+0000 7fc496023640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:54.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.114+0000 7fc496023640 4 set_mon_vals no callback set 2024-09-16T04:46:54.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.115+0000 7fc496023640 10 monclient: shutdown 2024-09-16T04:46:54.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.115+0000 7fc496023640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:54.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.115+0000 7fc496023640 1 -- 172.21.15.46:0/217578199 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc4900a3c10 msgr2=0x7fc49009bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.115+0000 7fc496023640 1 --2- 172.21.15.46:0/217578199 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc4900a3c10 0x7fc49009bcf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc480009e50 tx=0x7fc48002a5f0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.115+0000 7fc496023640 1 -- 172.21.15.46:0/217578199 shutdown_connections 2024-09-16T04:46:54.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.115+0000 7fc496023640 1 --2- 172.21.15.46:0/217578199 >> v2:172.21.15.46:3300/0 conn(0x7fc4900a3ff0 0x7fc4900a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.115+0000 7fc496023640 1 --2- 172.21.15.46:0/217578199 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc4900a3c10 0x7fc49009bcf0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.115+0000 7fc496023640 1 -- 172.21.15.46:0/217578199 >> 172.21.15.46:0/217578199 conn(0x7fc4900a6340 msgr2=0x7fc49006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:54.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.115+0000 7fc496023640 1 -- 172.21.15.46:0/217578199 shutdown_connections 2024-09-16T04:46:54.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.115+0000 7fc496023640 1 -- 172.21.15.46:0/217578199 wait complete. 2024-09-16T04:46:54.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.115+0000 7fc496023640 10 monclient: build_initial_monmap 2024-09-16T04:46:54.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.115+0000 7fc496023640 10 monclient: monmap: 2024-09-16T04:46:54.132 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:54.132 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:54.132 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:54.132 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:54.132 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:54.132 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.133 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:54.133 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:54.133 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:54.133 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.116+0000 7fc496023640 1 Processor -- start 2024-09-16T04:46:54.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.116+0000 7fc496023640 1 -- start start 2024-09-16T04:46:54.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.116+0000 7fc496023640 10 monclient: init 2024-09-16T04:46:54.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.116+0000 7fc496023640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:54.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.116+0000 7fc496023640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:46:54.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.116+0000 7fc496023640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4900a3c10 0x7fc4901cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.116+0000 7fc496023640 10 monclient(hunting): picked mon.noname-b con 0x7fc4900a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:54.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.116+0000 7fc496023640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4900a3ff0 0x7fc4901cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.116+0000 7fc496023640 10 monclient(hunting): picked mon.noname-a con 0x7fc4900a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:54.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.116+0000 7fc496023640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc4901369d0 0x7fc490123230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc496023640 10 monclient(hunting): picked mon.noname-c con 0x7fc4901369d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:54.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc496023640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc496023640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc4900a4910 con 0x7fc4900a3ff0 2024-09-16T04:46:54.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc496023640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc496023640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc4900a69c0 con 0x7fc4900a3c10 2024-09-16T04:46:54.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc496023640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc496023640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fc490126610 con 0x7fc4901369d0 2024-09-16T04:46:54.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc496023640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:54.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc496023640 10 monclient(hunting): authenticate will time out at 1212.532032s 2024-09-16T04:46:54.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48ffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc4901369d0 0x7fc490123230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4900a3ff0 0x7fc4901cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4900a3ff0 0x7fc4901cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57924/0 (socket says 172.21.15.46:57924) 2024-09-16T04:46:54.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48ffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc4901369d0 0x7fc490123230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:56856/0 (socket says 172.21.15.46:56856) 2024-09-16T04:46:54.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48effd640 1 -- 172.21.15.46:0/2867193183 learned_addr learned my addr 172.21.15.46:0/2867193183 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:54.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48effd640 10 monclient(hunting): get_auth_request con 0x7fc4900a3ff0 auth_method 0 2024-09-16T04:46:54.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48ffff640 10 monclient(hunting): get_auth_request con 0x7fc4901369d0 auth_method 0 2024-09-16T04:46:54.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48f7fe640 1 --2- 172.21.15.46:0/2867193183 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4900a3c10 0x7fc4901cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48f7fe640 10 monclient(hunting): get_auth_request con 0x7fc4900a3c10 auth_method 0 2024-09-16T04:46:54.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48ffff640 10 monclient(hunting): handle_auth_done global_id 94310 payload 824 2024-09-16T04:46:54.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48ffff640 1 -- 172.21.15.46:0/2867193183 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4900a3c10 msgr2=0x7fc4901cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48ffff640 1 --2- 172.21.15.46:0/2867193183 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4900a3c10 0x7fc4901cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48ffff640 1 -- 172.21.15.46:0/2867193183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4900a3ff0 msgr2=0x7fc4901cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48ffff640 1 --2- 172.21.15.46:0/2867193183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4900a3ff0 0x7fc4901cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:46:54.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48ffff640 1 monclient: found mon.noname-c 2024-09-16T04:46:54.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:54.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:46:54.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48ffff640 1 -- 172.21.15.46:0/2867193183 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc490126ee0 con 0x7fc4901369d0 2024-09-16T04:46:54.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48effd640 1 --2- 172.21.15.46:0/2867193183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4900a3ff0 0x7fc4901cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:54.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48ffff640 1 --2- 172.21.15.46:0/2867193183 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc4901369d0 0x7fc490123230 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc48800c3f0 tx=0x7fc48800c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:54.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc477fff640 1 -- 172.21.15.46:0/2867193183 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4880074e0 con 0x7fc4901369d0 2024-09-16T04:46:54.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc477fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc477fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:54.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc477fff640 10 monclient: dump: 2024-09-16T04:46:54.138 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.138 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.138 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.138 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.138 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.138 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.138 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.138 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.138 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.138 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc477fff640 10 monclient: _finish_auth 0 2024-09-16T04:46:54.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc477fff640 10 monclient: _check_auth_tickets 2024-09-16T04:46:54.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc477fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:54.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc496023640 5 monclient: authenticate success, global_id 94310 2024-09-16T04:46:54.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc477fff640 1 -- 172.21.15.46:0/2867193183 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc488007680 con 0x7fc4901369d0 2024-09-16T04:46:54.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc496023640 10 monclient: _renew_subs 2024-09-16T04:46:54.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc496023640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:54.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc496023640 1 -- 172.21.15.46:0/2867193183 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc490125a30 con 0x7fc4901369d0 2024-09-16T04:46:54.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc477fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:54.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc477fff640 1 -- 172.21.15.46:0/2867193183 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4880049e0 con 0x7fc4901369d0 2024-09-16T04:46:54.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc477fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc477fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:54.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc477fff640 10 monclient: dump: 2024-09-16T04:46:54.140 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.140 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.140 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.140 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.140 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.140 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.140 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.140 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.140 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.140 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc496023640 10 monclient: _renew_subs 2024-09-16T04:46:54.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc48dffb640 4 set_mon_vals no callback set 2024-09-16T04:46:54.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc496023640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:54.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.117+0000 7fc496023640 1 -- 172.21.15.46:0/2867193183 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4900a8900 con 0x7fc4901369d0 2024-09-16T04:46:54.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.118+0000 7fc477fff640 1 -- 172.21.15.46:0/2867193183 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fc48801a020 con 0x7fc4901369d0 2024-09-16T04:46:54.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.118+0000 7fc496023640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:54.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.118+0000 7fc496023640 1 --2- 172.21.15.46:0/2867193183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc490126af0 0x7fc490123770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.118+0000 7fc496023640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.118+0000 7fc496023640 1 -- 172.21.15.46:0/2867193183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc4900a4a90 con 0x7fc490126af0 2024-09-16T04:46:54.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.118+0000 7fc477fff640 1 --2- 172.21.15.46:0/2867193183 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc47c0383b0 0x7fc47c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.118+0000 7fc48f7fe640 1 --2- 172.21.15.46:0/2867193183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc490126af0 0x7fc490123770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.118+0000 7fc477fff640 1 -- 172.21.15.46:0/2867193183 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(311..311 src has 1..311) ==== 4738+0+0 (secure 0 0 0) 0x7fc48804ad20 con 0x7fc4901369d0 2024-09-16T04:46:54.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.118+0000 7fc48f7fe640 10 monclient: get_auth_request con 0x7fc490126af0 auth_method 0 2024-09-16T04:46:54.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.118+0000 7fc48f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.118+0000 7fc48f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.118+0000 7fc48f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.119+0000 7fc48f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.119+0000 7fc48f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.119+0000 7fc48f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.119+0000 7fc48f7fe640 10 monclient(hunting): handle_auth_done global_id 104112 payload 824 2024-09-16T04:46:54.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.119+0000 7fc48f7fe640 1 -- 172.21.15.46:0/2867193183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc4900a2ba0 con 0x7fc490126af0 2024-09-16T04:46:54.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.119+0000 7fc48effd640 1 --2- 172.21.15.46:0/2867193183 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc47c0383b0 0x7fc47c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.119+0000 7fc48f7fe640 1 --2- 172.21.15.46:0/2867193183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc490126af0 0x7fc490123770 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc490068160 tx=0x7fc480004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:54.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.119+0000 7fc48effd640 10 monclient: get_auth_request con 0x7fc47c0383b0 auth_method 0 2024-09-16T04:46:54.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.119+0000 7fc477fff640 1 -- 172.21.15.46:0/2867193183 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc480036040 con 0x7fc490126af0 2024-09-16T04:46:54.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.119+0000 7fc477fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.119+0000 7fc477fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:54.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.119+0000 7fc477fff640 10 monclient: dump: 2024-09-16T04:46:54.143 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.143 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.143 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.143 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.143 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.143 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.143 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.143 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.143 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.143 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.119+0000 7fc477fff640 1 -- 172.21.15.46:0/2867193183 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc48003c640 con 0x7fc490126af0 2024-09-16T04:46:54.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.119+0000 7fc477fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:54.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.119+0000 7fc48effd640 1 --2- 172.21.15.46:0/2867193183 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc47c0383b0 0x7fc47c03a870 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc484009560 tx=0x7fc484009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:54.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.119+0000 7fc477fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:54.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.119+0000 7fc477fff640 1 -- 172.21.15.46:0/2867193183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc490126af0 msgr2=0x7fc490123770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.119+0000 7fc477fff640 1 --2- 172.21.15.46:0/2867193183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc490126af0 0x7fc490123770 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc490068160 tx=0x7fc480004a40 comp rx=0 tx=0).stop 2024-09-16T04:46:54.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.119+0000 7fc477fff640 1 -- 172.21.15.46:0/2867193183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc490126af0 msgr2=0x7fc490123770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:54.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.119+0000 7fc477fff640 1 --2- 172.21.15.46:0/2867193183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc490126af0 0x7fc490123770 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.125+0000 7fc496023640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:46:54.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.125+0000 7fc496023640 1 --2- 172.21.15.46:0/2867193183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc490123cb0 0x7fc4901240c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.125+0000 7fc496023640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.125+0000 7fc496023640 1 -- 172.21.15.46:0/2867193183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc490124600 con 0x7fc490123cb0 2024-09-16T04:46:54.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc48f7fe640 1 --2- 172.21.15.46:0/2867193183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc490123cb0 0x7fc4901240c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc48f7fe640 10 monclient: get_auth_request con 0x7fc490123cb0 auth_method 0 2024-09-16T04:46:54.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc48f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc48f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc48f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc48f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc48f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc48f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc48f7fe640 10 monclient(hunting): handle_auth_done global_id 104115 payload 824 2024-09-16T04:46:54.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc48f7fe640 1 -- 172.21.15.46:0/2867193183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc49009bcf0 con 0x7fc490123cb0 2024-09-16T04:46:54.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc48f7fe640 1 --2- 172.21.15.46:0/2867193183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc490123cb0 0x7fc4901240c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc48002a630 tx=0x7fc48002f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:54.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc477fff640 1 -- 172.21.15.46:0/2867193183 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc480050600 con 0x7fc490123cb0 2024-09-16T04:46:54.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc477fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc477fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:54.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc477fff640 10 monclient: dump: 2024-09-16T04:46:54.147 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.147 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.148 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.148 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.148 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.148 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.148 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.148 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.148 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.148 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc477fff640 1 -- 172.21.15.46:0/2867193183 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7fc480050600 con 0x7fc490123cb0 2024-09-16T04:46:54.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc477fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:46:54.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc477fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:54.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc477fff640 1 -- 172.21.15.46:0/2867193183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc490123cb0 msgr2=0x7fc4901240c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc477fff640 1 --2- 172.21.15.46:0/2867193183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc490123cb0 0x7fc4901240c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc48002a630 tx=0x7fc48002f000 comp rx=0 tx=0).stop 2024-09-16T04:46:54.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc477fff640 1 -- 172.21.15.46:0/2867193183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc490123cb0 msgr2=0x7fc4901240c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:54.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.126+0000 7fc477fff640 1 --2- 172.21.15.46:0/2867193183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc490123cb0 0x7fc4901240c0 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc48002a630 tx=0x7fc48002f000 comp rx=0 tx=0).stop 2024-09-16T04:46:54.149 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:46:54.149 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:46:54.149 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:46:54.149 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:46:54.149 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 94, 2024-09-16T04:46:54.149 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 7183, 2024-09-16T04:46:54.149 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:46:54.149 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:46:54.150 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:46:54.150 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:46:54.150 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:54.150 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:46:54.150 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:54.150 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:46:54.150 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:46:54.150 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:54.150 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:54.150 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:54.150 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:54.150 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:54.150 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:54.150 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:54.150 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:54.151 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:54.151 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:54.151 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:54.151 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:54.151 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:46:54.151 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:46:54.151 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:54.151 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:54.151 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:54.151 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:54.151 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:54.151 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:54.151 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:54.152 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:54.152 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:54.152 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:54.152 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:54.152 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:54.152 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.152 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:46:54.152 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:46:54.152 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:46:54.152 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:46:54.152 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:46:54.152 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:46:54.152 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:54.152 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:54.153 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:46:54.153 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:46:54.153 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:46:54.153 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:46:54.153 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:46:54.153 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:46:54.153 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:46:54.153 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:54.153 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:46:54.153 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:54.153 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:54.153 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:54.153 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:54.153 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:54.154 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:54.154 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:54.154 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:54.154 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:54.154 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:54.154 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:54.154 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:54.154 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:46:54.154 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.154 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:46:54.154 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.154 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:46:54.155 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:46:54.155 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:54.155 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:54.155 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.155 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:54.155 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:46:54.155 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:54.155 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.155 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.155 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:54.156 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:46:54.156 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:54.156 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:54.156 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:54.156 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.156 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:46:54.156 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:46:54.156 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:54.156 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:54.156 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:54.156 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.156 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.156 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:46:54.157 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:46:54.157 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:54.157 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:54.157 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.157 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:54.157 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:46:54.157 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:54.157 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.157 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.157 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:54.157 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:46:54.157 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:54.158 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:54.158 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:54.158 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.158 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:46:54.158 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:46:54.158 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:54.158 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:54.158 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:54.158 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.158 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.158 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:46:54.158 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:46:54.158 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:54.159 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:54.159 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.159 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:54.159 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:46:54.159 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:54.159 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.159 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.159 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:54.159 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:46:54.159 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:54.160 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:54.160 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:54.160 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.160 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:46:54.160 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:46:54.160 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:54.160 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:54.160 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:54.160 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:54.160 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:54.160 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.160 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:46:54.160 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:46:54.161 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.161 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:54.161 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:54.161 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:54.161 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:54.161 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:54.161 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:46:54.161 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.161 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:54.161 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:54.161 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:54.161 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:54.161 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:54.162 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.162 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:46:54.162 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:46:54.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.127+0000 7fc496023640 1 -- 172.21.15.46:0/2867193183 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc47c0383b0 msgr2=0x7fc47c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.127+0000 7fc496023640 1 --2- 172.21.15.46:0/2867193183 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fc47c0383b0 0x7fc47c03a870 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc484009560 tx=0x7fc484009190 comp rx=0 tx=0).stop 2024-09-16T04:46:54.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.127+0000 7fc496023640 10 monclient: shutdown 2024-09-16T04:46:54.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.127+0000 7fc496023640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:54.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.127+0000 7fc496023640 1 -- 172.21.15.46:0/2867193183 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc4901369d0 msgr2=0x7fc490123230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.127+0000 7fc496023640 1 --2- 172.21.15.46:0/2867193183 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fc4901369d0 0x7fc490123230 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc48800c3f0 tx=0x7fc48800c8b0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.127+0000 7fc48ffff640 1 -- 172.21.15.46:0/2867193183 reap_dead start 2024-09-16T04:46:54.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.127+0000 7fc48ffff640 1 -- 172.21.15.46:0/2867193183 reap_dead start 2024-09-16T04:46:54.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.127+0000 7fc496023640 1 -- 172.21.15.46:0/2867193183 shutdown_connections 2024-09-16T04:46:54.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.127+0000 7fc496023640 1 -- 172.21.15.46:0/2867193183 >> 172.21.15.46:0/2867193183 conn(0x7fc4900a6340 msgr2=0x7fc49009f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:54.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.127+0000 7fc496023640 1 -- 172.21.15.46:0/2867193183 shutdown_connections 2024-09-16T04:46:54.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.127+0000 7fc496023640 1 -- 172.21.15.46:0/2867193183 wait complete. 2024-09-16T04:46:54.164 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:46:54.164 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:46:54.165 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:46:54.245 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[40353]: starting. 2024-09-16T04:46:54.245 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[40353]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-40351 2024-09-16T04:46:54.245 INFO:teuthology.orchestra.run.smithi114.stdout:process_2_[40353]: shutting down. 2024-09-16T04:46:54.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.269+0000 7f024c6ba640 10 monclient: get_monmap_and_config 2024-09-16T04:46:54.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.269+0000 7f024c6ba640 10 monclient: build_initial_monmap 2024-09-16T04:46:54.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.269+0000 7f024c6ba640 10 monclient: monmap: 2024-09-16T04:46:54.270 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:54.270 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.270 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:54.269780+0000 2024-09-16T04:46:54.272 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:54.269780+0000 2024-09-16T04:46:54.272 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:54.272 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.272 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:54.272 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:54.272 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:54.272 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:54.272 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:54.272 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.269+0000 7f024c6ba640 1 Processor -- start 2024-09-16T04:46:54.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.269+0000 7f024c6ba640 1 -- start start 2024-09-16T04:46:54.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.269+0000 7f024c6ba640 10 monclient: init 2024-09-16T04:46:54.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024c6ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:54.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024c6ba640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-16T04:46:54.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024c6ba640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f02440a3c10 0x7f024409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024c6ba640 10 monclient(hunting): picked mon.noname-b con 0x7f02440a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:54.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024c6ba640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02440a3ff0 0x7f02440a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024c6ba640 10 monclient(hunting): picked mon.noname-a con 0x7f02440a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:46:54.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024c6ba640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f02441369d0 addr v1:172.21.15.46:6789/0 2024-09-16T04:46:54.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024c6ba640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f024409ab70 con 0x7f02441369d0 2024-09-16T04:46:54.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024c6ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024c6ba640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f02440a4d60 con 0x7f02440a3ff0 2024-09-16T04:46:54.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024c6ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024c6ba640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f024409b030 con 0x7f02440a3c10 2024-09-16T04:46:54.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024c6ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:54.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024a456640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f02440a3c10 0x7f024409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024c6ba640 10 monclient(hunting): authenticate will time out at 1212.685717s 2024-09-16T04:46:54.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024a456640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f02440a3c10 0x7f024409bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:56860/0 (socket says 172.21.15.46:56860) 2024-09-16T04:46:54.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024a456640 1 -- 172.21.15.46:0/148650591 learned_addr learned my addr 172.21.15.46:0/148650591 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:54.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024a456640 10 monclient(hunting): get_auth_request con 0x7f02440a3c10 auth_method 0 2024-09-16T04:46:54.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024a456640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024a456640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024a456640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024a456640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024a456640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f024a456640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f0249c55640 1 --2- 172.21.15.46:0/148650591 >> v2:172.21.15.46:3300/0 conn(0x7f02440a3ff0 0x7f02440a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f0249c55640 10 monclient(hunting): get_auth_request con 0x7f02440a3ff0 auth_method 0 2024-09-16T04:46:54.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f0249c55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f0249c55640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.270+0000 7f0249c55640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.271+0000 7f024a456640 10 monclient(hunting): handle_auth_done global_id 94331 payload 274 2024-09-16T04:46:54.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.271+0000 7f024a456640 1 -- 172.21.15.46:0/148650591 >> v2:172.21.15.46:3300/0 conn(0x7f02440a3ff0 msgr2=0x7f02440a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:54.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.271+0000 7f024a456640 1 --2- 172.21.15.46:0/148650591 >> v2:172.21.15.46:3300/0 conn(0x7f02440a3ff0 0x7f02440a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.271+0000 7f024a456640 1 -- 172.21.15.46:0/148650591 >> v1:172.21.15.46:6789/0 conn(0x7f02441369d0 legacy=0x7f0244136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:54.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.271+0000 7f024a456640 10 monclient: _finish_hunting 0 2024-09-16T04:46:54.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.271+0000 7f024a456640 1 monclient: found mon.noname-b 2024-09-16T04:46:54.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.271+0000 7f024a456640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:54.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.271+0000 7f024a456640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:46:54.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.271+0000 7f024a456640 1 -- 172.21.15.46:0/148650591 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f024409b310 con 0x7f02440a3c10 2024-09-16T04:46:54.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.271+0000 7f024a456640 1 --2- 172.21.15.46:0/148650591 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f02440a3c10 0x7f024409bcf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0240009870 tx=0x7f024002a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e2a5892039102db server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:54.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f0249454640 1 -- 172.21.15.46:0/148650591 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0240037070 con 0x7f02440a3c10 2024-09-16T04:46:54.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f0249454640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f0249454640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:46:54.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f0249454640 10 monclient: dump: 2024-09-16T04:46:54.284 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.284 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.284 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.284 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.284 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.284 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.284 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.284 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.285 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.285 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f0249454640 10 monclient: _finish_auth 0 2024-09-16T04:46:54.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f0249454640 10 monclient: _check_auth_tickets 2024-09-16T04:46:54.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f0249454640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:54.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f0249454640 1 -- 172.21.15.46:0/148650591 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0240032440 con 0x7f02440a3c10 2024-09-16T04:46:54.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f024c6ba640 5 monclient: authenticate success, global_id 94331 2024-09-16T04:46:54.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f024c6ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:54.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f0249454640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:54.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f024c6ba640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:54.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f0249454640 1 -- 172.21.15.46:0/148650591 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0240032810 con 0x7f02440a3c10 2024-09-16T04:46:54.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f0249454640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f024c6ba640 4 set_mon_vals no callback set 2024-09-16T04:46:54.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f0249454640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:54.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f0249454640 10 monclient: dump: 2024-09-16T04:46:54.286 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.286 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.286 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.286 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.286 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.286 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.286 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.286 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.286 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.286 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f024c6ba640 10 monclient: shutdown 2024-09-16T04:46:54.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f024c6ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:54.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f024c6ba640 1 -- 172.21.15.46:0/148650591 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f02440a3c10 msgr2=0x7f024409bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f024c6ba640 1 --2- 172.21.15.46:0/148650591 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f02440a3c10 0x7f024409bcf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0240009870 tx=0x7f024002a1c0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f024c6ba640 1 -- 172.21.15.46:0/148650591 shutdown_connections 2024-09-16T04:46:54.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f024c6ba640 1 --2- 172.21.15.46:0/148650591 >> v2:172.21.15.46:3300/0 conn(0x7f02440a3ff0 0x7f02440a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f024c6ba640 1 --2- 172.21.15.46:0/148650591 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f02440a3c10 0x7f024409bcf0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f024c6ba640 1 -- 172.21.15.46:0/148650591 >> 172.21.15.46:0/148650591 conn(0x7f02440a5d10 msgr2=0x7f02440a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:54.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f024c6ba640 1 -- 172.21.15.46:0/148650591 shutdown_connections 2024-09-16T04:46:54.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.272+0000 7f024c6ba640 1 -- 172.21.15.46:0/148650591 wait complete. 2024-09-16T04:46:54.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.273+0000 7f024c6ba640 10 monclient: build_initial_monmap 2024-09-16T04:46:54.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.273+0000 7f024c6ba640 10 monclient: monmap: 2024-09-16T04:46:54.287 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:54.287 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:54.287 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:54.287 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:54.287 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:54.287 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.288 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:54.288 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:54.288 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:54.288 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.273+0000 7f024c6ba640 1 Processor -- start 2024-09-16T04:46:54.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.273+0000 7f024c6ba640 1 -- start start 2024-09-16T04:46:54.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.273+0000 7f024c6ba640 10 monclient: init 2024-09-16T04:46:54.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024c6ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:54.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024c6ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:46:54.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024c6ba640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f02440a3c10 0x7f02441cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024c6ba640 10 monclient(hunting): picked mon.noname-c con 0x7f02440a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:54.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024c6ba640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02440a3ff0 0x7f02441cc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024c6ba640 10 monclient(hunting): picked mon.noname-b con 0x7f02440a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:54.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024c6ba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02441369d0 0x7f02441231e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024c6ba640 10 monclient(hunting): picked mon.noname-a con 0x7f02441369d0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:54.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024c6ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024c6ba640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f02440a4910 con 0x7f02441369d0 2024-09-16T04:46:54.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024c6ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024c6ba640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f024406e3d0 con 0x7f02440a3ff0 2024-09-16T04:46:54.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024c6ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024c6ba640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f0244127b00 con 0x7f02440a3c10 2024-09-16T04:46:54.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024c6ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:54.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024c6ba640 10 monclient(hunting): authenticate will time out at 1212.689135s 2024-09-16T04:46:54.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024a456640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f02440a3c10 0x7f02441cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024a456640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f02440a3c10 0x7f02441cbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:56876/0 (socket says 172.21.15.46:56876) 2024-09-16T04:46:54.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024a456640 1 -- 172.21.15.46:0/1095544885 learned_addr learned my addr 172.21.15.46:0/1095544885 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:54.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024a456640 10 monclient(hunting): get_auth_request con 0x7f02440a3c10 auth_method 0 2024-09-16T04:46:54.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024a456640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024a456640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024a456640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024ac57640 1 --2- 172.21.15.46:0/1095544885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02441369d0 0x7f02441231e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024ac57640 10 monclient(hunting): get_auth_request con 0x7f02441369d0 auth_method 0 2024-09-16T04:46:54.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024ac57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024ac57640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024ac57640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024a456640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024a456640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024a456640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024ac57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024ac57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024ac57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f0249c55640 1 --2- 172.21.15.46:0/1095544885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02440a3ff0 0x7f02441cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f0249c55640 10 monclient(hunting): get_auth_request con 0x7f02440a3ff0 auth_method 0 2024-09-16T04:46:54.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f0249c55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f0249c55640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f0249c55640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024a456640 10 monclient(hunting): handle_auth_done global_id 94334 payload 824 2024-09-16T04:46:54.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024a456640 1 -- 172.21.15.46:0/1095544885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02440a3ff0 msgr2=0x7f02441cc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024a456640 1 --2- 172.21.15.46:0/1095544885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02440a3ff0 0x7f02441cc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024a456640 1 -- 172.21.15.46:0/1095544885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02441369d0 msgr2=0x7f02441231e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024a456640 1 --2- 172.21.15.46:0/1095544885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02441369d0 0x7f02441231e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024a456640 10 monclient: _finish_hunting 0 2024-09-16T04:46:54.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024a456640 1 monclient: found mon.noname-c 2024-09-16T04:46:54.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024a456640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:54.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024a456640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:46:54.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024a456640 1 -- 172.21.15.46:0/1095544885 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0244125990 con 0x7f02440a3c10 2024-09-16T04:46:54.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024a456640 1 --2- 172.21.15.46:0/1095544885 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f02440a3c10 0x7f02441cbf50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0240000c00 tx=0x7f02400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:54.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f023affd640 1 -- 172.21.15.46:0/1095544885 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0240041020 con 0x7f02440a3c10 2024-09-16T04:46:54.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f023affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f023affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:54.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f023affd640 10 monclient: dump: 2024-09-16T04:46:54.292 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.292 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.292 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.292 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.292 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.292 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.292 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.292 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.292 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.293 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f023affd640 10 monclient: _finish_auth 0 2024-09-16T04:46:54.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f023affd640 10 monclient: _check_auth_tickets 2024-09-16T04:46:54.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f023affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:54.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024c6ba640 5 monclient: authenticate success, global_id 94334 2024-09-16T04:46:54.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024c6ba640 10 monclient: _renew_subs 2024-09-16T04:46:54.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024c6ba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:54.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.274+0000 7f024c6ba640 1 -- 172.21.15.46:0/1095544885 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0244127120 con 0x7f02440a3c10 2024-09-16T04:46:54.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.275+0000 7f023affd640 1 -- 172.21.15.46:0/1095544885 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02400306c0 con 0x7f02440a3c10 2024-09-16T04:46:54.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.275+0000 7f023affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:54.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.275+0000 7f023affd640 1 -- 172.21.15.46:0/1095544885 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0240037040 con 0x7f02440a3c10 2024-09-16T04:46:54.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.275+0000 7f023affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.275+0000 7f023affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:54.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.275+0000 7f023affd640 10 monclient: dump: 2024-09-16T04:46:54.294 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.294 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.294 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.294 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.294 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.294 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.294 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.294 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.294 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.294 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.275+0000 7f024c6ba640 10 monclient: _renew_subs 2024-09-16T04:46:54.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.275+0000 7f0248c53640 4 set_mon_vals no callback set 2024-09-16T04:46:54.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.275+0000 7f024c6ba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:54.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.275+0000 7f024c6ba640 1 -- 172.21.15.46:0/1095544885 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f02440a0400 con 0x7f02440a3c10 2024-09-16T04:46:54.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.275+0000 7f023affd640 1 -- 172.21.15.46:0/1095544885 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f0240044020 con 0x7f02440a3c10 2024-09-16T04:46:54.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.276+0000 7f023affd640 1 --2- 172.21.15.46:0/1095544885 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f02280383b0 0x7f022803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.276+0000 7f023affd640 1 -- 172.21.15.46:0/1095544885 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(312..312 src has 1..312) ==== 4346+0+0 (secure 0 0 0) 0x7f024006bdd0 con 0x7f02440a3c10 2024-09-16T04:46:54.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.276+0000 7f0249c55640 1 --2- 172.21.15.46:0/1095544885 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f02280383b0 0x7f022803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.276+0000 7f0249c55640 10 monclient: get_auth_request con 0x7f02280383b0 auth_method 0 2024-09-16T04:46:54.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.276+0000 7f024c6ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:54.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.276+0000 7f024c6ba640 1 --2- 172.21.15.46:0/1095544885 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0244126ac0 0x7f0244123720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.276+0000 7f024c6ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.276+0000 7f024c6ba640 1 -- 172.21.15.46:0/1095544885 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f024409bcf0 con 0x7f0244126ac0 2024-09-16T04:46:54.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.276+0000 7f024ac57640 1 --2- 172.21.15.46:0/1095544885 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0244126ac0 0x7f0244123720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.276+0000 7f024ac57640 10 monclient: get_auth_request con 0x7f0244126ac0 auth_method 0 2024-09-16T04:46:54.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.276+0000 7f024ac57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.276+0000 7f024ac57640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.276+0000 7f024ac57640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.277+0000 7f024ac57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.277+0000 7f024ac57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.277+0000 7f0249c55640 1 --2- 172.21.15.46:0/1095544885 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f02280383b0 0x7f022803a870 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0244136770 tx=0x7f0234008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:54.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.277+0000 7f024ac57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.277+0000 7f024ac57640 10 monclient(hunting): handle_auth_done global_id 94337 payload 824 2024-09-16T04:46:54.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.277+0000 7f024ac57640 1 -- 172.21.15.46:0/1095544885 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0244069150 con 0x7f0244126ac0 2024-09-16T04:46:54.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.277+0000 7f024ac57640 1 --2- 172.21.15.46:0/1095544885 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0244126ac0 0x7f0244123720 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f023c00f8b0 tx=0x7f023c00fd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:54.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.277+0000 7f023affd640 1 -- 172.21.15.46:0/1095544885 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023c003b80 con 0x7f0244126ac0 2024-09-16T04:46:54.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.277+0000 7f023affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.277+0000 7f023affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:54.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.277+0000 7f023affd640 10 monclient: dump: 2024-09-16T04:46:54.296 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.296 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.296 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.297 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.297 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.297 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.297 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.297 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.297 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.297 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.277+0000 7f023affd640 1 -- 172.21.15.46:0/1095544885 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f023c018070 con 0x7f0244126ac0 2024-09-16T04:46:54.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.277+0000 7f023affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:54.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.277+0000 7f023affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:54.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.277+0000 7f023affd640 1 -- 172.21.15.46:0/1095544885 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0244126ac0 msgr2=0x7f0244123720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.277+0000 7f023affd640 1 --2- 172.21.15.46:0/1095544885 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0244126ac0 0x7f0244123720 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f023c00f8b0 tx=0x7f023c00fd70 comp rx=0 tx=0).stop 2024-09-16T04:46:54.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.277+0000 7f023affd640 1 -- 172.21.15.46:0/1095544885 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0244126ac0 msgr2=0x7f0244123720 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:54.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.277+0000 7f023affd640 1 --2- 172.21.15.46:0/1095544885 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0244126ac0 0x7f0244123720 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f023c00f8b0 tx=0x7f023c00fd70 comp rx=0 tx=0).stop 2024-09-16T04:46:54.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.293+0000 7f024c6ba640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:46:54.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.293+0000 7f024c6ba640 1 --2- 172.21.15.46:0/1095544885 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0244123c60 0x7f0244124070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.293+0000 7f024c6ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.293+0000 7f024c6ba640 1 -- 172.21.15.46:0/1095544885 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f02441245b0 con 0x7f0244123c60 2024-09-16T04:46:54.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.293+0000 7f024ac57640 1 --2- 172.21.15.46:0/1095544885 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0244123c60 0x7f0244124070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.293+0000 7f024ac57640 10 monclient: get_auth_request con 0x7f0244123c60 auth_method 0 2024-09-16T04:46:54.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.293+0000 7f024ac57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.293+0000 7f024ac57640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.293+0000 7f024ac57640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.293+0000 7f024ac57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.293+0000 7f024ac57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.293+0000 7f024ac57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.294+0000 7f024ac57640 10 monclient(hunting): handle_auth_done global_id 94340 payload 824 2024-09-16T04:46:54.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.294+0000 7f024ac57640 1 -- 172.21.15.46:0/1095544885 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0244067e40 con 0x7f0244123c60 2024-09-16T04:46:54.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.294+0000 7f024ac57640 1 --2- 172.21.15.46:0/1095544885 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0244123c60 0x7f0244124070 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f023c017000 tx=0x7f023c0014a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:54.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.294+0000 7f023affd640 1 -- 172.21.15.46:0/1095544885 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023c006b50 con 0x7f0244123c60 2024-09-16T04:46:54.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.294+0000 7f023affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.294+0000 7f023affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:54.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.294+0000 7f023affd640 10 monclient: dump: 2024-09-16T04:46:54.301 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.301 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.301 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.301 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.301 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.301 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.301 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.301 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.301 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.301 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.294+0000 7f023affd640 1 -- 172.21.15.46:0/1095544885 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f023c006b50 con 0x7f0244123c60 2024-09-16T04:46:54.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.294+0000 7f023affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:46:54.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.294+0000 7f023affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:54.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.294+0000 7f023affd640 1 -- 172.21.15.46:0/1095544885 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0244123c60 msgr2=0x7f0244124070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.294+0000 7f023affd640 1 --2- 172.21.15.46:0/1095544885 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0244123c60 0x7f0244124070 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f023c017000 tx=0x7f023c0014a0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.294+0000 7f023affd640 1 -- 172.21.15.46:0/1095544885 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0244123c60 msgr2=0x7f0244124070 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:54.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.294+0000 7f023affd640 1 --2- 172.21.15.46:0/1095544885 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0244123c60 0x7f0244124070 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f023c017000 tx=0x7f023c0014a0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.302 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:46:54.302 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:46:54.302 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:46:54.302 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:46:54.303 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 94, 2024-09-16T04:46:54.303 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 55549, 2024-09-16T04:46:54.303 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:46:54.303 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:46:54.303 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:46:54.303 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:46:54.303 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:54.303 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 1, 2024-09-16T04:46:54.303 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:54.303 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:46:54.303 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:46:54.303 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:54.303 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:54.303 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:54.304 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:54.304 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:54.304 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:54.304 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:54.304 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:54.304 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:54.304 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:54.304 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:54.304 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:54.304 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:46:54.304 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:46:54.304 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:54.304 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:54.304 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:54.305 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:54.305 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:54.305 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:54.305 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:54.305 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:54.305 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:54.305 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:54.305 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:54.305 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:54.305 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.305 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:46:54.305 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:46:54.306 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:46:54.306 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:46:54.306 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:46:54.306 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:46:54.306 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:54.306 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:54.306 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:46:54.306 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:46:54.306 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:46:54.306 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:46:54.306 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:46:54.306 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:46:54.307 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:46:54.307 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:54.307 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:46:54.307 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:54.307 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:54.307 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:54.307 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:54.307 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:54.307 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:54.307 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:54.307 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:54.307 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:54.307 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:54.307 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:54.308 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:54.308 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:46:54.308 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.308 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:46:54.308 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.308 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:46:54.308 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:46:54.308 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:54.308 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:54.308 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.308 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:54.308 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:46:54.308 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:54.308 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.309 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.309 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:54.309 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:46:54.309 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:54.309 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:54.309 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:54.309 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.309 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:46:54.309 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:46:54.309 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:54.309 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:54.309 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:54.309 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.309 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.310 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:46:54.310 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:46:54.310 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:54.310 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:54.310 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.310 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:54.310 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:46:54.310 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:54.310 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.310 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.310 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:54.310 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:46:54.310 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:54.310 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:54.311 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:54.311 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.311 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:46:54.311 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:46:54.311 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:54.311 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:54.311 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:54.311 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.311 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.311 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:46:54.311 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:46:54.311 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:54.312 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:54.312 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.312 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:54.312 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:46:54.312 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:54.312 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.312 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.312 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:54.313 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:46:54.313 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:54.313 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:54.313 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:54.313 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.313 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:46:54.313 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:46:54.313 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:54.313 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:54.313 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:54.313 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:54.313 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:54.313 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.314 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:46:54.314 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:46:54.314 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.314 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:54.314 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:54.314 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:54.314 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:54.314 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:54.314 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:46:54.314 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.314 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:54.314 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:54.314 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:54.315 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:54.315 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:54.315 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:46:54.315 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.315 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:54.315 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:54.315 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:46:54.315 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:54.315 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:54.315 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.315 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:46:54.315 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:46:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.295+0000 7f024c6ba640 1 -- 172.21.15.46:0/1095544885 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f02280383b0 msgr2=0x7f022803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.295+0000 7f024c6ba640 1 --2- 172.21.15.46:0/1095544885 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f02280383b0 0x7f022803a870 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0244136770 tx=0x7f0234008040 comp rx=0 tx=0).stop 2024-09-16T04:46:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.295+0000 7f024c6ba640 10 monclient: shutdown 2024-09-16T04:46:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.295+0000 7f024c6ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.295+0000 7f024c6ba640 1 -- 172.21.15.46:0/1095544885 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f02440a3c10 msgr2=0x7f02441cbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.295+0000 7f024ac57640 1 -- 172.21.15.46:0/1095544885 reap_dead start 2024-09-16T04:46:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.295+0000 7f024c6ba640 1 --2- 172.21.15.46:0/1095544885 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f02440a3c10 0x7f02441cbf50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0240000c00 tx=0x7f02400046c0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.295+0000 7f024c6ba640 1 -- 172.21.15.46:0/1095544885 shutdown_connections 2024-09-16T04:46:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.295+0000 7f024c6ba640 1 --2- 172.21.15.46:0/1095544885 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f02440a3c10 0x7f02441cbf50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.295+0000 7f024c6ba640 1 -- 172.21.15.46:0/1095544885 >> 172.21.15.46:0/1095544885 conn(0x7f02440a5d10 msgr2=0x7f02440a2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.295+0000 7f024c6ba640 1 -- 172.21.15.46:0/1095544885 shutdown_connections 2024-09-16T04:46:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.295+0000 7f024c6ba640 1 -- 172.21.15.46:0/1095544885 wait complete. 2024-09-16T04:46:54.318 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:46:54.318 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:46:54.318 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:46:54.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.419+0000 7f63afc23640 10 monclient: get_monmap_and_config 2024-09-16T04:46:54.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.419+0000 7f63afc23640 10 monclient: build_initial_monmap 2024-09-16T04:46:54.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.419+0000 7f63afc23640 10 monclient: monmap: 2024-09-16T04:46:54.420 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:54.420 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.420 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:54.419616+0000 2024-09-16T04:46:54.420 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:54.419616+0000 2024-09-16T04:46:54.421 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:54.421 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.421 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:54.421 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:54.421 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:54.421 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:54.421 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:54.421 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.419+0000 7f63afc23640 1 Processor -- start 2024-09-16T04:46:54.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.419+0000 7f63afc23640 1 -- start start 2024-09-16T04:46:54.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.419+0000 7f63afc23640 10 monclient: init 2024-09-16T04:46:54.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63afc23640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:54.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63afc23640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T04:46:54.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63afc23640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63a80a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:46:54.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63afc23640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f63a80a3ff0 0x7f63a80a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63afc23640 10 monclient(hunting): picked mon.noname-b con 0x7f63a80a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:54.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63afc23640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f63a813be90 0x7f63a813e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63afc23640 10 monclient(hunting): picked mon.noname-c con 0x7f63a813be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:46:54.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63afc23640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63a809ab70 con 0x7f63a80a3c10 2024-09-16T04:46:54.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63afc23640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63afc23640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f63a80a4d60 con 0x7f63a813be90 2024-09-16T04:46:54.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63afc23640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63afc23640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f63a809b030 con 0x7f63a80a3ff0 2024-09-16T04:46:54.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63afc23640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:54.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63afc23640 10 monclient(hunting): authenticate will time out at 1212.835448s 2024-09-16T04:46:54.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ad1be640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f63a80a3ff0 0x7f63a80a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ad9bf640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f63a80a3c10 0x7f63a809bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45072/0 (socket says 172.21.15.46:45072) 2024-09-16T04:46:54.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ad9bf640 1 -- 172.21.15.46:0/3198603724 learned_addr learned my addr 172.21.15.46:0/3198603724 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:46:54.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ad1be640 10 monclient(hunting): get_auth_request con 0x7f63a80a3ff0 auth_method 0 2024-09-16T04:46:54.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ad1be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ad1be640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ad1be640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ad1be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ad1be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ad1be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ae1c0640 1 --2- 172.21.15.46:0/3198603724 >> v2:172.21.15.114:3300/0 conn(0x7f63a813be90 0x7f63a813e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ae1c0640 10 monclient(hunting): get_auth_request con 0x7f63a813be90 auth_method 0 2024-09-16T04:46:54.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ae1c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ae1c0640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ae1c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ae1c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ae1c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ae1c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ad1be640 10 monclient(hunting): handle_auth_done global_id 94343 payload 274 2024-09-16T04:46:54.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ad1be640 1 -- 172.21.15.46:0/3198603724 >> v2:172.21.15.114:3300/0 conn(0x7f63a813be90 msgr2=0x7f63a813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:54.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ad1be640 1 --2- 172.21.15.46:0/3198603724 >> v2:172.21.15.114:3300/0 conn(0x7f63a813be90 0x7f63a813e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ac9bd640 1 -- 172.21.15.46:0/3198603724 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2526642099 0 0) 0x7f63a809ab70 con 0x7f63a80a3c10 2024-09-16T04:46:54.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ad1be640 1 -- 172.21.15.46:0/3198603724 >> v1:172.21.15.46:6789/0 conn(0x7f63a80a3c10 legacy=0x7f63a809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ad1be640 10 monclient: _finish_hunting 0 2024-09-16T04:46:54.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ad1be640 1 monclient: found mon.noname-b 2024-09-16T04:46:54.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ad1be640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:54.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ad1be640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:46:54.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ad1be640 1 -- 172.21.15.46:0/3198603724 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63a809b310 con 0x7f63a80a3ff0 2024-09-16T04:46:54.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.420+0000 7f63ac9bd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:46:54.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.421+0000 7f63ad1be640 1 --2- 172.21.15.46:0/3198603724 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f63a80a3ff0 0x7f63a80a43d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f63980097a0 tx=0x7f639802a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38ef9de31311979 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:54.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.421+0000 7f63ac9bd640 1 -- 172.21.15.46:0/3198603724 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6398037070 con 0x7f63a80a3ff0 2024-09-16T04:46:54.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.421+0000 7f63ac9bd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.421+0000 7f63ac9bd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:46:54.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.421+0000 7f63ac9bd640 10 monclient: dump: 2024-09-16T04:46:54.434 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.434 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.434 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.434 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.434 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.434 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.434 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.434 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.434 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.434 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.421+0000 7f63ac9bd640 10 monclient: _finish_auth 0 2024-09-16T04:46:54.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.421+0000 7f63ac9bd640 10 monclient: _check_auth_tickets 2024-09-16T04:46:54.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.421+0000 7f63ac9bd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:54.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.421+0000 7f63afc23640 5 monclient: authenticate success, global_id 94343 2024-09-16T04:46:54.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.421+0000 7f63afc23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:54.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.421+0000 7f63ac9bd640 1 -- 172.21.15.46:0/3198603724 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f639802adb0 con 0x7f63a80a3ff0 2024-09-16T04:46:54.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.421+0000 7f63ac9bd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:54.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.421+0000 7f63afc23640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:54.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.421+0000 7f63afc23640 4 set_mon_vals no callback set 2024-09-16T04:46:54.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.421+0000 7f63ac9bd640 1 -- 172.21.15.46:0/3198603724 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6398032660 con 0x7f63a80a3ff0 2024-09-16T04:46:54.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.421+0000 7f63ac9bd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.421+0000 7f63ac9bd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:54.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.421+0000 7f63ac9bd640 10 monclient: dump: 2024-09-16T04:46:54.435 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.435 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.435 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.435 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.436 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.436 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.436 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.436 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.436 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.436 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.422+0000 7f63afc23640 10 monclient: shutdown 2024-09-16T04:46:54.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.422+0000 7f63afc23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:54.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.422+0000 7f63afc23640 1 -- 172.21.15.46:0/3198603724 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f63a80a3ff0 msgr2=0x7f63a80a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.422+0000 7f63afc23640 1 --2- 172.21.15.46:0/3198603724 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f63a80a3ff0 0x7f63a80a43d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f63980097a0 tx=0x7f639802a060 comp rx=0 tx=0).stop 2024-09-16T04:46:54.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.422+0000 7f63afc23640 1 -- 172.21.15.46:0/3198603724 shutdown_connections 2024-09-16T04:46:54.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.422+0000 7f63afc23640 1 --2- 172.21.15.46:0/3198603724 >> v2:172.21.15.114:3300/0 conn(0x7f63a813be90 0x7f63a813e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.422+0000 7f63afc23640 1 --2- 172.21.15.46:0/3198603724 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f63a80a3ff0 0x7f63a80a43d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.422+0000 7f63afc23640 1 -- 172.21.15.46:0/3198603724 >> 172.21.15.46:0/3198603724 conn(0x7f63a80a5d10 msgr2=0x7f63a80a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:54.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.422+0000 7f63afc23640 1 -- 172.21.15.46:0/3198603724 shutdown_connections 2024-09-16T04:46:54.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.422+0000 7f63afc23640 1 -- 172.21.15.46:0/3198603724 wait complete. 2024-09-16T04:46:54.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63afc23640 10 monclient: build_initial_monmap 2024-09-16T04:46:54.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63afc23640 10 monclient: monmap: 2024-09-16T04:46:54.437 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:54.437 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:54.437 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:54.437 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:54.437 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:54.437 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.438 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:54.438 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:54.438 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:54.438 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63afc23640 1 Processor -- start 2024-09-16T04:46:54.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63afc23640 1 -- start start 2024-09-16T04:46:54.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63afc23640 10 monclient: init 2024-09-16T04:46:54.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63afc23640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:54.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63afc23640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:46:54.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63afc23640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a80a3c10 0x7f63a81cb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63afc23640 10 monclient(hunting): picked mon.noname-b con 0x7f63a80a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:54.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63afc23640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a80a3ff0 0x7f63a8136ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63afc23640 10 monclient(hunting): picked mon.noname-a con 0x7f63a80a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:54.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63afc23640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f63a813be90 0x7f63a80a1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63afc23640 10 monclient(hunting): picked mon.noname-c con 0x7f63a813be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:54.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63afc23640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63afc23640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f63a806e550 con 0x7f63a80a3ff0 2024-09-16T04:46:54.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63afc23640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63afc23640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f63a806e3d0 con 0x7f63a80a3c10 2024-09-16T04:46:54.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63afc23640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63afc23640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f63a813e7c0 con 0x7f63a813be90 2024-09-16T04:46:54.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63afc23640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:54.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63afc23640 10 monclient(hunting): authenticate will time out at 1212.838879s 2024-09-16T04:46:54.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63ad1be640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a80a3ff0 0x7f63a8136ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63ae1c0640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f63a813be90 0x7f63a80a1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63ad1be640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a80a3ff0 0x7f63a8136ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35128/0 (socket says 172.21.15.46:35128) 2024-09-16T04:46:54.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ad1be640 1 -- 172.21.15.46:0/4061320898 learned_addr learned my addr 172.21.15.46:0/4061320898 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:54.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ad1be640 10 monclient(hunting): get_auth_request con 0x7f63a80a3ff0 auth_method 0 2024-09-16T04:46:54.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.423+0000 7f63ae1c0640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f63a813be90 0x7f63a80a1530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:35772/0 (socket says 172.21.15.46:35772) 2024-09-16T04:46:54.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ad1be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ad1be640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ad1be640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ae1c0640 10 monclient(hunting): get_auth_request con 0x7f63a813be90 auth_method 0 2024-09-16T04:46:54.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ae1c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ae1c0640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ae1c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ad9bf640 1 --2- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a80a3c10 0x7f63a81cb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ad9bf640 10 monclient(hunting): get_auth_request con 0x7f63a80a3c10 auth_method 0 2024-09-16T04:46:54.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ad9bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ad9bf640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ad9bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ad1be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ad1be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ad1be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ae1c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ae1c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ae1c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ad9bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ad9bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ad9bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ae1c0640 10 monclient(hunting): handle_auth_done global_id 94346 payload 824 2024-09-16T04:46:54.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ae1c0640 1 -- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a80a3c10 msgr2=0x7f63a81cb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ae1c0640 1 --2- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a80a3c10 0x7f63a81cb860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ae1c0640 1 -- 172.21.15.46:0/4061320898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a80a3ff0 msgr2=0x7f63a8136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ae1c0640 1 --2- 172.21.15.46:0/4061320898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a80a3ff0 0x7f63a8136ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ae1c0640 10 monclient: _finish_hunting 0 2024-09-16T04:46:54.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ae1c0640 1 monclient: found mon.noname-c 2024-09-16T04:46:54.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ae1c0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:54.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ae1c0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:46:54.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ae1c0640 1 -- 172.21.15.46:0/4061320898 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63a8121390 con 0x7f63a813be90 2024-09-16T04:46:54.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ad1be640 1 --2- 172.21.15.46:0/4061320898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a80a3ff0 0x7f63a8136ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:54.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63ae1c0640 1 --2- 172.21.15.46:0/4061320898 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f63a813be90 0x7f63a80a1530 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f63a000efc0 tx=0x7f63a000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:54.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f639e7fc640 1 -- 172.21.15.46:0/4061320898 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a0018020 con 0x7f63a813be90 2024-09-16T04:46:54.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f639e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f639e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:54.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f639e7fc640 10 monclient: dump: 2024-09-16T04:46:54.443 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.443 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.443 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.443 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.443 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.443 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.443 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.443 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.443 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.443 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f639e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:46:54.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f639e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:46:54.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f639e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:54.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63afc23640 5 monclient: authenticate success, global_id 94346 2024-09-16T04:46:54.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f639e7fc640 1 -- 172.21.15.46:0/4061320898 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63a0014480 con 0x7f63a813be90 2024-09-16T04:46:54.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f639e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:54.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f639e7fc640 1 -- 172.21.15.46:0/4061320898 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a0009b10 con 0x7f63a813be90 2024-09-16T04:46:54.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63afc23640 10 monclient: _renew_subs 2024-09-16T04:46:54.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63afc23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:54.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63afc23640 1 -- 172.21.15.46:0/4061320898 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63a81210b0 con 0x7f63a813be90 2024-09-16T04:46:54.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f639e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f639e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:54.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f639f7fe640 4 set_mon_vals no callback set 2024-09-16T04:46:54.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f639e7fc640 10 monclient: dump: 2024-09-16T04:46:54.444 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.444 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.444 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.445 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.445 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.445 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.445 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.445 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.445 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.445 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63afc23640 10 monclient: _renew_subs 2024-09-16T04:46:54.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63afc23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:54.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.424+0000 7f63afc23640 1 -- 172.21.15.46:0/4061320898 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f63a81cc6f0 con 0x7f63a813be90 2024-09-16T04:46:54.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.425+0000 7f63afc23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:54.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.425+0000 7f63afc23640 1 --2- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a8137220 0x7f63a80a1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.425+0000 7f63afc23640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.425+0000 7f63afc23640 1 -- 172.21.15.46:0/4061320898 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f63a813c6a0 con 0x7f63a8137220 2024-09-16T04:46:54.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.425+0000 7f639e7fc640 1 -- 172.21.15.46:0/4061320898 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f63a0005d10 con 0x7f63a813be90 2024-09-16T04:46:54.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.425+0000 7f63ad9bf640 1 --2- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a8137220 0x7f63a80a1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.425+0000 7f639e7fc640 1 --2- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f638c0383b0 0x7f638c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.425+0000 7f63ad9bf640 10 monclient: get_auth_request con 0x7f63a8137220 auth_method 0 2024-09-16T04:46:54.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.425+0000 7f63ad9bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.425+0000 7f63ad9bf640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.425+0000 7f63ad9bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.425+0000 7f639e7fc640 1 -- 172.21.15.46:0/4061320898 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(312..312 src has 1..312) ==== 4346+0+0 (secure 0 0 0) 0x7f63a004c2a0 con 0x7f63a813be90 2024-09-16T04:46:54.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.425+0000 7f63ad9bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.425+0000 7f63ad9bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.425+0000 7f63ad9bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.425+0000 7f63ad1be640 1 --2- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f638c0383b0 0x7f638c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.426+0000 7f63ad1be640 10 monclient: get_auth_request con 0x7f638c0383b0 auth_method 0 2024-09-16T04:46:54.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.426+0000 7f63ad9bf640 10 monclient(hunting): handle_auth_done global_id 6994 payload 824 2024-09-16T04:46:54.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.426+0000 7f63ad9bf640 1 -- 172.21.15.46:0/4061320898 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f63a80a43d0 con 0x7f63a8137220 2024-09-16T04:46:54.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.426+0000 7f63ad9bf640 1 --2- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a8137220 0x7f63a80a1a70 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f63a4000f60 tx=0x7f63a40018d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:54.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.426+0000 7f639e7fc640 1 -- 172.21.15.46:0/4061320898 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a4011580 con 0x7f63a8137220 2024-09-16T04:46:54.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.426+0000 7f639e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.426+0000 7f639e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:46:54.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.426+0000 7f639e7fc640 10 monclient: dump: 2024-09-16T04:46:54.447 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.447 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.447 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.447 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.447 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.447 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.448 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.448 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.448 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.448 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.426+0000 7f63ad1be640 1 --2- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f638c0383b0 0x7f638c03a870 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f639802a0a0 tx=0x7f639802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:54.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.426+0000 7f639e7fc640 1 -- 172.21.15.46:0/4061320898 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f63a4018070 con 0x7f63a8137220 2024-09-16T04:46:54.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.426+0000 7f639e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:54.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.426+0000 7f639e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:54.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.426+0000 7f639e7fc640 1 -- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a8137220 msgr2=0x7f63a80a1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.426+0000 7f639e7fc640 1 --2- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a8137220 0x7f63a80a1a70 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f63a4000f60 tx=0x7f63a40018d0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.426+0000 7f639e7fc640 1 -- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a8137220 msgr2=0x7f63a80a1a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:54.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.426+0000 7f639e7fc640 1 --2- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a8137220 0x7f63a80a1a70 secure :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f63a4000f60 tx=0x7f63a40018d0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.432+0000 7f63afc23640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:46:54.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.432+0000 7f63afc23640 1 --2- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a811fa60 0x7f63a80a1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.432+0000 7f63afc23640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.432+0000 7f63afc23640 1 -- 172.21.15.46:0/4061320898 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f63a8069150 con 0x7f63a811fa60 2024-09-16T04:46:54.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.433+0000 7f63ad9bf640 1 --2- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a811fa60 0x7f63a80a1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.433+0000 7f63ad9bf640 10 monclient: get_auth_request con 0x7f63a811fa60 auth_method 0 2024-09-16T04:46:54.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.433+0000 7f63ad9bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.433+0000 7f63ad9bf640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.433+0000 7f63ad9bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.433+0000 7f63ad9bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.433+0000 7f63ad9bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.433+0000 7f63ad9bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.433+0000 7f63ad9bf640 10 monclient(hunting): handle_auth_done global_id 6997 payload 824 2024-09-16T04:46:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.433+0000 7f63ad9bf640 1 -- 172.21.15.46:0/4061320898 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f63a8067e40 con 0x7f63a811fa60 2024-09-16T04:46:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.433+0000 7f63ad9bf640 1 --2- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a811fa60 0x7f63a80a1fb0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f63a4000f60 tx=0x7f63a4001080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.433+0000 7f639e7fc640 1 -- 172.21.15.46:0/4061320898 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a4011c30 con 0x7f63a811fa60 2024-09-16T04:46:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.433+0000 7f639e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.433+0000 7f639e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:46:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.433+0000 7f639e7fc640 10 monclient: dump: 2024-09-16T04:46:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.452 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.434+0000 7f639e7fc640 1 -- 172.21.15.46:0/4061320898 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f63a4011c30 con 0x7f63a811fa60 2024-09-16T04:46:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.434+0000 7f639e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:46:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.434+0000 7f639e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.434+0000 7f639e7fc640 1 -- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a811fa60 msgr2=0x7f63a80a1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.434+0000 7f639e7fc640 1 --2- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a811fa60 0x7f63a80a1fb0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f63a4000f60 tx=0x7f63a4001080 comp rx=0 tx=0).stop 2024-09-16T04:46:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.434+0000 7f639e7fc640 1 -- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a811fa60 msgr2=0x7f63a80a1fb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.434+0000 7f639e7fc640 1 --2- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a811fa60 0x7f63a80a1fb0 secure :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f63a4000f60 tx=0x7f63a4001080 comp rx=0 tx=0).stop 2024-09-16T04:46:54.453 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:46:54.453 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:46:54.453 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:46:54.453 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:46:54.453 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 94, 2024-09-16T04:46:54.453 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 500335, 2024-09-16T04:46:54.453 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:46:54.453 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:46:54.453 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:46:54.454 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:46:54.454 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:54.454 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 1, 2024-09-16T04:46:54.454 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:54.454 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:46:54.454 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:46:54.454 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:54.454 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:54.454 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:54.454 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:54.454 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:54.454 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:54.454 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:54.455 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:54.455 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:54.455 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:54.455 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:54.455 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:54.455 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:46:54.455 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:46:54.455 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:54.455 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:54.455 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:54.455 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:54.455 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:54.455 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:54.455 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:54.456 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:54.456 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:54.456 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:54.456 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:54.456 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:54.456 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.456 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:46:54.456 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:46:54.456 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:46:54.456 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:46:54.456 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:46:54.456 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:46:54.456 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:54.457 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:54.457 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:46:54.457 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:46:54.457 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:46:54.457 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:46:54.457 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:46:54.457 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:46:54.457 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:46:54.457 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:54.457 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:46:54.457 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:54.457 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:54.457 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:54.457 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:54.458 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:54.458 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:54.458 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:54.458 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:54.458 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:54.458 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:54.458 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:54.458 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:54.458 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:46:54.458 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.458 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:46:54.458 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.458 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:46:54.459 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:46:54.459 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:54.459 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:54.459 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.459 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:54.459 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:46:54.459 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:54.459 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.459 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.459 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:54.459 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:46:54.459 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:54.459 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:54.459 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:54.460 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.460 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:46:54.460 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:46:54.460 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:54.460 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:54.460 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:54.460 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.460 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.460 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:46:54.460 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:46:54.460 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:54.460 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:54.460 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.460 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:54.461 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:46:54.461 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:54.461 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.461 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.461 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:54.461 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:46:54.461 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:54.461 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:54.461 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:54.461 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.461 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:46:54.461 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:46:54.461 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:54.461 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:54.462 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:54.462 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.462 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.462 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:46:54.462 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:46:54.462 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:54.462 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:54.462 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.462 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:54.462 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:46:54.462 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:54.462 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.462 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.462 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:54.463 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:46:54.463 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:54.463 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:54.463 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:54.463 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.463 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:46:54.463 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:46:54.463 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:54.463 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:54.463 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:54.463 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:54.463 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:54.463 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.463 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:46:54.464 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:46:54.464 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.464 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:54.464 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:54.464 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:54.464 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:54.464 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:54.464 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:46:54.464 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.464 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:54.464 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:54.464 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5 2024-09-16T04:46:54.464 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:54.464 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:54.465 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:46:54.465 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.465 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:54.465 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:54.465 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6 2024-09-16T04:46:54.465 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:54.465 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:54.465 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:46:54.465 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:54.465 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:54.465 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:54.465 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:54.465 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:54.465 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:54.466 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:54.466 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:46:54.466 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:46:54.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.434+0000 7f63afc23640 1 -- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f638c0383b0 msgr2=0x7f638c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.434+0000 7f63afc23640 1 --2- 172.21.15.46:0/4061320898 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f638c0383b0 0x7f638c03a870 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f639802a0a0 tx=0x7f639802e000 comp rx=0 tx=0).stop 2024-09-16T04:46:54.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.434+0000 7f63afc23640 10 monclient: shutdown 2024-09-16T04:46:54.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.434+0000 7f63afc23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:54.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.434+0000 7f63afc23640 1 -- 172.21.15.46:0/4061320898 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f63a813be90 msgr2=0x7f63a80a1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.434+0000 7f63afc23640 1 --2- 172.21.15.46:0/4061320898 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f63a813be90 0x7f63a80a1530 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f63a000efc0 tx=0x7f63a000c490 comp rx=0 tx=0).stop 2024-09-16T04:46:54.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.434+0000 7f63ae1c0640 1 -- 172.21.15.46:0/4061320898 reap_dead start 2024-09-16T04:46:54.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.434+0000 7f63ae1c0640 1 -- 172.21.15.46:0/4061320898 reap_dead start 2024-09-16T04:46:54.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.434+0000 7f63afc23640 1 -- 172.21.15.46:0/4061320898 shutdown_connections 2024-09-16T04:46:54.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.434+0000 7f63afc23640 1 -- 172.21.15.46:0/4061320898 >> 172.21.15.46:0/4061320898 conn(0x7f63a80a5d10 msgr2=0x7f63a809f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:54.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.434+0000 7f63afc23640 1 -- 172.21.15.46:0/4061320898 shutdown_connections 2024-09-16T04:46:54.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.434+0000 7f63afc23640 1 -- 172.21.15.46:0/4061320898 wait complete. 2024-09-16T04:46:54.468 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:46:54.468 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:46:54.468 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:46:54.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.568+0000 7ff2f2247640 10 monclient: get_monmap_and_config 2024-09-16T04:46:54.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.568+0000 7ff2f2247640 10 monclient: build_initial_monmap 2024-09-16T04:46:54.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.568+0000 7ff2f2247640 10 monclient: monmap: 2024-09-16T04:46:54.569 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:54.569 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.569 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:54.568522+0000 2024-09-16T04:46:54.569 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:54.568522+0000 2024-09-16T04:46:54.569 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:54.569 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.569 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:54.569 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:54.570 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:54.570 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:54.570 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:54.570 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.568+0000 7ff2f2247640 1 Processor -- start 2024-09-16T04:46:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.568+0000 7ff2f2247640 1 -- start start 2024-09-16T04:46:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.568+0000 7ff2f2247640 10 monclient: init 2024-09-16T04:46:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2f2247640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2f2247640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-16T04:46:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2f2247640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff2ec0a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:46:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2f2247640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff2ec0a3ff0 0x7ff2ec0a4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2f2247640 10 monclient(hunting): picked mon.noname-c con 0x7ff2ec0a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:46:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2f2247640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff2ec13be90 addr v1:172.21.15.46:6789/0 2024-09-16T04:46:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2f2247640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2ec09ab70 con 0x7ff2ec13be90 2024-09-16T04:46:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2f2247640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2ec0a4d60 con 0x7ff2ec0a3c10 2024-09-16T04:46:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2f2247640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2f2247640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ff2ec09b030 con 0x7ff2ec0a3ff0 2024-09-16T04:46:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2f2247640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2f2247640 10 monclient(hunting): authenticate will time out at 1212.984325s 2024-09-16T04:46:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2ebfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff2ec13be90 0x7ff2ec13e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45080/0 (socket says 172.21.15.46:45080) 2024-09-16T04:46:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2ebfff640 1 -- 172.21.15.46:0/2264704235 learned_addr learned my addr 172.21.15.46:0/2264704235 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:46:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2eaffd640 1 --2- 172.21.15.46:0/2264704235 >> v2:172.21.15.114:3300/0 conn(0x7ff2ec0a3ff0 0x7ff2ec0a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2eaffd640 10 monclient(hunting): get_auth_request con 0x7ff2ec0a3ff0 auth_method 0 2024-09-16T04:46:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2ea7fc640 1 -- 172.21.15.46:0/2264704235 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2326219097 0 0) 0x7ff2ec09ab70 con 0x7ff2ec13be90 2024-09-16T04:46:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:46:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2ea7fc640 10 monclient(hunting): my global_id is 104151 2024-09-16T04:46:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2ea7fc640 1 -- 172.21.15.46:0/2264704235 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2d4003520 con 0x7ff2ec13be90 2024-09-16T04:46:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2eaffd640 10 monclient(hunting): handle_auth_done global_id 7000 payload 274 2024-09-16T04:46:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2eaffd640 1 -- 172.21.15.46:0/2264704235 >> v1:172.21.15.114:6789/0 conn(0x7ff2ec0a3c10 legacy=0x7ff2ec09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2eaffd640 1 -- 172.21.15.46:0/2264704235 >> v1:172.21.15.46:6789/0 conn(0x7ff2ec13be90 legacy=0x7ff2ec13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2eaffd640 10 monclient: _finish_hunting 0 2024-09-16T04:46:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2eaffd640 1 monclient: found mon.noname-c 2024-09-16T04:46:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2eaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:46:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.569+0000 7ff2eaffd640 1 -- 172.21.15.46:0/2264704235 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2ec09b310 con 0x7ff2ec0a3ff0 2024-09-16T04:46:54.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.570+0000 7ff2eaffd640 1 --2- 172.21.15.46:0/2264704235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2ec0a3ff0 0x7ff2ec0a4400 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7ff2e00097a0 tx=0x7ff2e002a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58f0cd7ceec3bef6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:54.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.570+0000 7ff2ea7fc640 1 -- 172.21.15.46:0/2264704235 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e0037070 con 0x7ff2ec0a3ff0 2024-09-16T04:46:54.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.570+0000 7ff2ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.570+0000 7ff2ea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:54.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.570+0000 7ff2ea7fc640 10 monclient: dump: 2024-09-16T04:46:54.580 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.580 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.580 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.580 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.580 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.580 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.580 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.580 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.580 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.581 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.570+0000 7ff2ea7fc640 10 monclient: _finish_auth 0 2024-09-16T04:46:54.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.570+0000 7ff2ea7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:46:54.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.570+0000 7ff2ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:54.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.570+0000 7ff2f2247640 5 monclient: authenticate success, global_id 7000 2024-09-16T04:46:54.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.570+0000 7ff2f2247640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:54.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.570+0000 7ff2ea7fc640 1 -- 172.21.15.46:0/2264704235 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2e0032440 con 0x7ff2ec0a3ff0 2024-09-16T04:46:54.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.570+0000 7ff2ea7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:54.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.570+0000 7ff2ea7fc640 1 -- 172.21.15.46:0/2264704235 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e002ae60 con 0x7ff2ec0a3ff0 2024-09-16T04:46:54.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.570+0000 7ff2f2247640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:54.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.570+0000 7ff2f2247640 4 set_mon_vals no callback set 2024-09-16T04:46:54.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.570+0000 7ff2ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.570+0000 7ff2ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:46:54.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.570+0000 7ff2ea7fc640 10 monclient: dump: 2024-09-16T04:46:54.582 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.582 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.582 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.582 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.582 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.582 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.582 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.582 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.582 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.582 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.571+0000 7ff2f2247640 10 monclient: shutdown 2024-09-16T04:46:54.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.571+0000 7ff2f2247640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:54.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.571+0000 7ff2f2247640 1 -- 172.21.15.46:0/2264704235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2ec0a3ff0 msgr2=0x7ff2ec0a4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.571+0000 7ff2f2247640 1 --2- 172.21.15.46:0/2264704235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2ec0a3ff0 0x7ff2ec0a4400 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7ff2e00097a0 tx=0x7ff2e002a110 comp rx=0 tx=0).stop 2024-09-16T04:46:54.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.571+0000 7ff2f2247640 1 -- 172.21.15.46:0/2264704235 shutdown_connections 2024-09-16T04:46:54.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.571+0000 7ff2f2247640 1 --2- 172.21.15.46:0/2264704235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2ec0a3ff0 0x7ff2ec0a4400 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.571+0000 7ff2f2247640 1 -- 172.21.15.46:0/2264704235 >> 172.21.15.46:0/2264704235 conn(0x7ff2ec0a5d10 msgr2=0x7ff2ec0a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:54.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.571+0000 7ff2f2247640 1 -- 172.21.15.46:0/2264704235 shutdown_connections 2024-09-16T04:46:54.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.571+0000 7ff2f2247640 1 -- 172.21.15.46:0/2264704235 wait complete. 2024-09-16T04:46:54.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.572+0000 7ff2f2247640 10 monclient: build_initial_monmap 2024-09-16T04:46:54.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.572+0000 7ff2f2247640 10 monclient: monmap: 2024-09-16T04:46:54.583 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:54.583 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:54.583 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:54.583 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:54.583 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:54.583 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.583 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:54.584 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:54.584 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:54.584 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.572+0000 7ff2f2247640 1 Processor -- start 2024-09-16T04:46:54.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.572+0000 7ff2f2247640 1 -- start start 2024-09-16T04:46:54.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.572+0000 7ff2f2247640 10 monclient: init 2024-09-16T04:46:54.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.572+0000 7ff2f2247640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:54.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:46:54.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff2ec0a3c10 0x7ff2ec1cba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 10 monclient(hunting): picked mon.noname-c con 0x7ff2ec0a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:54.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2ec0a3ff0 0x7ff2ec136dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 10 monclient(hunting): picked mon.noname-b con 0x7ff2ec0a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:54.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2ec13be90 0x7ff2ec138e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 10 monclient(hunting): picked mon.noname-a con 0x7ff2ec13be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:54.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ec06e550 con 0x7ff2ec13be90 2024-09-16T04:46:54.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ec06e3d0 con 0x7ff2ec0a3ff0 2024-09-16T04:46:54.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:54.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7ff2ec13f9a0 con 0x7ff2ec0a3c10 2024-09-16T04:46:54.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:54.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 10 monclient(hunting): authenticate will time out at 1212.988074s 2024-09-16T04:46:54.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2ebfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2ec13be90 0x7ff2ec138e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eb7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff2ec0a3c10 0x7ff2ec1cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2ebfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2ec13be90 0x7ff2ec138e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35144/0 (socket says 172.21.15.46:35144) 2024-09-16T04:46:54.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2ebfff640 1 -- 172.21.15.46:0/1352913227 learned_addr learned my addr 172.21.15.46:0/1352913227 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:54.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2ebfff640 10 monclient(hunting): get_auth_request con 0x7ff2ec13be90 auth_method 0 2024-09-16T04:46:54.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eb7fe640 10 monclient(hunting): get_auth_request con 0x7ff2ec0a3c10 auth_method 0 2024-09-16T04:46:54.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eaffd640 1 --2- 172.21.15.46:0/1352913227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2ec0a3ff0 0x7ff2ec136dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eaffd640 10 monclient(hunting): get_auth_request con 0x7ff2ec0a3ff0 auth_method 0 2024-09-16T04:46:54.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:54.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:54.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:54.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:54.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:54.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:54.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eb7fe640 10 monclient(hunting): handle_auth_done global_id 94349 payload 824 2024-09-16T04:46:54.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eb7fe640 1 -- 172.21.15.46:0/1352913227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2ec0a3ff0 msgr2=0x7ff2ec136dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eb7fe640 1 --2- 172.21.15.46:0/1352913227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2ec0a3ff0 0x7ff2ec136dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eb7fe640 1 -- 172.21.15.46:0/1352913227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2ec13be90 msgr2=0x7ff2ec138e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eb7fe640 1 --2- 172.21.15.46:0/1352913227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2ec13be90 0x7ff2ec138e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eb7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:46:54.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eb7fe640 1 monclient: found mon.noname-c 2024-09-16T04:46:54.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:54.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:46:54.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eb7fe640 1 -- 172.21.15.46:0/1352913227 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2ec120740 con 0x7ff2ec0a3c10 2024-09-16T04:46:54.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2ebfff640 1 --2- 172.21.15.46:0/1352913227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2ec13be90 0x7ff2ec138e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:54.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2eb7fe640 1 --2- 172.21.15.46:0/1352913227 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff2ec0a3c10 0x7ff2ec1cba10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc00c370 tx=0x7ff2dc00c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:54.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2d3fff640 1 -- 172.21.15.46:0/1352913227 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc0052d0 con 0x7ff2ec0a3c10 2024-09-16T04:46:54.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2d3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:54.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2d3fff640 10 monclient: dump: 2024-09-16T04:46:54.588 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.588 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.588 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.588 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.588 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.589 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.589 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.589 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.589 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.589 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2d3fff640 10 monclient: _finish_auth 0 2024-09-16T04:46:54.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2d3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:46:54.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:54.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2d3fff640 1 -- 172.21.15.46:0/1352913227 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2dc005490 con 0x7ff2ec0a3c10 2024-09-16T04:46:54.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 5 monclient: authenticate success, global_id 94349 2024-09-16T04:46:54.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2d3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:54.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2d3fff640 1 -- 172.21.15.46:0/1352913227 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc007260 con 0x7ff2ec0a3c10 2024-09-16T04:46:54.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 10 monclient: _renew_subs 2024-09-16T04:46:54.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 1 -- 172.21.15.46:0/1352913227 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2ec09d330 con 0x7ff2ec0a3c10 2024-09-16T04:46:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2d3fff640 10 monclient: dump: 2024-09-16T04:46:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:54.590 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2e9ffb640 4 set_mon_vals no callback set 2024-09-16T04:46:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 10 monclient: _renew_subs 2024-09-16T04:46:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.573+0000 7ff2f2247640 1 -- 172.21.15.46:0/1352913227 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2ec120180 con 0x7ff2ec0a3c10 2024-09-16T04:46:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.574+0000 7ff2f2247640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.574+0000 7ff2f2247640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.575+0000 7ff2f2247640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.575+0000 7ff2d3fff640 1 -- 172.21.15.46:0/1352913227 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7ff2dc007400 con 0x7ff2ec0a3c10 2024-09-16T04:46:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.575+0000 7ff2f2247640 1 -- 172.21.15.46:0/1352913227 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2ec09c6d0 con 0x7ff2ec0a3c10 2024-09-16T04:46:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.575+0000 7ff2d3fff640 1 --2- 172.21.15.46:0/1352913227 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff2d80383b0 0x7ff2d803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.575+0000 7ff2d3fff640 1 -- 172.21.15.46:0/1352913227 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(312..312 src has 1..312) ==== 4346+0+0 (secure 0 0 0) 0x7ff2dc04aa60 con 0x7ff2ec0a3c10 2024-09-16T04:46:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.575+0000 7ff2eaffd640 1 --2- 172.21.15.46:0/1352913227 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff2d80383b0 0x7ff2d803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.575+0000 7ff2eaffd640 10 monclient: get_auth_request con 0x7ff2d80383b0 auth_method 0 2024-09-16T04:46:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.575+0000 7ff2eaffd640 1 --2- 172.21.15.46:0/1352913227 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff2d80383b0 0x7ff2d803a870 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0034000 tx=0x7ff2e002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:54.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.580+0000 7ff2d3fff640 1 -- 172.21.15.46:0/1352913227 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff2ec09c6d0 con 0x7ff2ec0a3c10 2024-09-16T04:46:54.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.580+0000 7ff2d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:54.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.580+0000 7ff2d3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:54.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.822+0000 7ff2f2247640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:46:54.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.822+0000 7ff2f2247640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:46:54.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.822+0000 7ff2f2247640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:54.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.822+0000 7ff2f2247640 1 -- 172.21.15.46:0/1352913227 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7ff2ec0a4400 con 0x7ff2ec0a3c10 2024-09-16T04:46:54.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.824+0000 7ff2d3fff640 1 -- 172.21.15.46:0/1352913227 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7ff2dc0129f0 con 0x7ff2ec0a3c10 2024-09-16T04:46:54.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.824+0000 7ff2d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:46:54.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.824+0000 7ff2d3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:54.827 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.824+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.827 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.824+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=11,kv=10,logm=60} crc {auth=1638558206,config=3541143443,health=4265645880,kv=510788098,logm=67133590}) 2024-09-16T04:46:54.827 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.824+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=18,config=2,health=11,kv=12,logm=57} crc {auth=2990534795,config=3541143443,health=3274674665,kv=4017288519,logm=3999910629}) 2024-09-16T04:46:54.827 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.824+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.827 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.824+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=11,kv=10,logm=60} crc {auth=1638558206,config=3541143443,health=4265645880,kv=510788098,logm=67133590}) 2024-09-16T04:46:54.828 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.824+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=15,config=2,health=12,kv=12,logm=59} crc {auth=3714375405,config=3541143443,health=2280860798,kv=4017288519,logm=4066897150}) 2024-09-16T04:46:54.828 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.825+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.828 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.825+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2987004694}) 2024-09-16T04:46:54.828 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.825+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=765067796}) 2024-09-16T04:46:54.828 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.825+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.828 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.825+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2987004694}) 2024-09-16T04:46:54.828 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.825+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=882426488}) 2024-09-16T04:46:54.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.826+0000 7ff2f2247640 1 -- 172.21.15.46:0/1352913227 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff2d80383b0 msgr2=0x7ff2d803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.826+0000 7ff2f2247640 1 --2- 172.21.15.46:0/1352913227 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff2d80383b0 0x7ff2d803a870 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0034000 tx=0x7ff2e002e000 comp rx=0 tx=0).stop 2024-09-16T04:46:54.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.826+0000 7ff2f2247640 10 monclient: shutdown 2024-09-16T04:46:54.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.826+0000 7ff2f2247640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:54.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.826+0000 7ff2f2247640 1 -- 172.21.15.46:0/1352913227 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff2ec0a3c10 msgr2=0x7ff2ec1cba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:54.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.826+0000 7ff2f2247640 1 --2- 172.21.15.46:0/1352913227 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff2ec0a3c10 0x7ff2ec1cba10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc00c370 tx=0x7ff2dc00c830 comp rx=0 tx=0).stop 2024-09-16T04:46:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.826+0000 7ff2f2247640 1 -- 172.21.15.46:0/1352913227 shutdown_connections 2024-09-16T04:46:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.826+0000 7ff2f2247640 1 --2- 172.21.15.46:0/1352913227 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ff2d80383b0 0x7ff2d803a870 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.826+0000 7ff2f2247640 1 --2- 172.21.15.46:0/1352913227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2ec13be90 0x7ff2ec138e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.826+0000 7ff2f2247640 1 --2- 172.21.15.46:0/1352913227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2ec0a3ff0 0x7ff2ec136dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.826+0000 7ff2f2247640 1 --2- 172.21.15.46:0/1352913227 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ff2ec0a3c10 0x7ff2ec1cba10 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.826+0000 7ff2f2247640 1 -- 172.21.15.46:0/1352913227 >> 172.21.15.46:0/1352913227 conn(0x7ff2ec0a5d10 msgr2=0x7ff2ec09f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.826+0000 7ff2f2247640 1 -- 172.21.15.46:0/1352913227 shutdown_connections 2024-09-16T04:46:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:54.826+0000 7ff2f2247640 1 -- 172.21.15.46:0/1352913227 wait complete. 2024-09-16T04:46:54.833 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.826+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.833 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.826+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3635585067}) 2024-09-16T04:46:54.833 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.826+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=355780666}) 2024-09-16T04:46:54.833 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.826+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.833 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.826+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3635585067}) 2024-09-16T04:46:54.833 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.826+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3663965816}) 2024-09-16T04:46:54.833 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.827+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.833 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.827+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3512407162}) 2024-09-16T04:46:54.833 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.827+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1938897997}) 2024-09-16T04:46:54.834 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.827+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.834 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.827+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3512407162}) 2024-09-16T04:46:54.834 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.827+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3773302717}) 2024-09-16T04:46:54.834 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.828+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.834 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.828+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3470347156}) 2024-09-16T04:46:54.834 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.828+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1921827788}) 2024-09-16T04:46:54.834 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.828+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.834 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.828+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3470347156}) 2024-09-16T04:46:54.834 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.828+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3128927720}) 2024-09-16T04:46:54.834 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.829+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.834 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.829+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3659258208}) 2024-09-16T04:46:54.834 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.829+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3115501748}) 2024-09-16T04:46:54.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.829+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.829+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3659258208}) 2024-09-16T04:46:54.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.829+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4045107421}) 2024-09-16T04:46:54.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.830+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.830+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1445125996}) 2024-09-16T04:46:54.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.830+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=711250488}) 2024-09-16T04:46:54.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.830+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.830+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1445125996}) 2024-09-16T04:46:54.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.830+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=800721417}) 2024-09-16T04:46:54.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.831+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.831+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3781808755}) 2024-09-16T04:46:54.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.831+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2451414618}) 2024-09-16T04:46:54.835 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.831+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.836 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.831+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3781808755}) 2024-09-16T04:46:54.836 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.831+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1411236014}) 2024-09-16T04:46:54.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.832+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.832+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986136820}) 2024-09-16T04:46:54.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.832+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2636493514}) 2024-09-16T04:46:54.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.832+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.832+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986136820}) 2024-09-16T04:46:54.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.832+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3600365001}) 2024-09-16T04:46:54.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.833+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.833+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2970644286}) 2024-09-16T04:46:54.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.833+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3656710352}) 2024-09-16T04:46:54.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.833+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.838 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.833+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2970644286}) 2024-09-16T04:46:54.839 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.833+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3874323003}) 2024-09-16T04:46:54.839 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.834+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.839 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.834+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3154657797}) 2024-09-16T04:46:54.839 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.834+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3533401777}) 2024-09-16T04:46:54.839 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.834+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.839 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.834+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3154657797}) 2024-09-16T04:46:54.839 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.835+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=818957684}) 2024-09-16T04:46:54.839 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.835+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.839 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.835+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3209418590}) 2024-09-16T04:46:54.840 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.835+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2741221391}) 2024-09-16T04:46:54.840 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.835+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.840 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.836+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3209418590}) 2024-09-16T04:46:54.840 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.836+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4040014864}) 2024-09-16T04:46:54.840 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.837+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.840 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.837+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2514789585}) 2024-09-16T04:46:54.840 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.837+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=838816679}) 2024-09-16T04:46:54.840 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.837+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.840 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.837+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2514789585}) 2024-09-16T04:46:54.840 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.837+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=224015913}) 2024-09-16T04:46:54.842 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:46:54.845 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.838+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.845 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.838+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=882979454}) 2024-09-16T04:46:54.845 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.838+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2628515332}) 2024-09-16T04:46:54.845 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.838+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.845 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.838+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=882979454}) 2024-09-16T04:46:54.845 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.838+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2246402494}) 2024-09-16T04:46:54.845 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.839+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.845 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.839+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774042520}) 2024-09-16T04:46:54.845 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.839+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2030615604}) 2024-09-16T04:46:54.846 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.839+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.846 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.839+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774042520}) 2024-09-16T04:46:54.846 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.839+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1093778509}) 2024-09-16T04:46:54.846 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.840+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.846 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.840+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3083793541}) 2024-09-16T04:46:54.846 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.840+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2477519549}) 2024-09-16T04:46:54.846 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.840+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.846 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.840+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3083793541}) 2024-09-16T04:46:54.846 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.840+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3787689825}) 2024-09-16T04:46:54.846 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.841+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.847 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.841+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2070215097}) 2024-09-16T04:46:54.847 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.841+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4242255877}) 2024-09-16T04:46:54.847 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.841+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.847 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.841+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2070215097}) 2024-09-16T04:46:54.847 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.841+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3372212309}) 2024-09-16T04:46:54.847 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.842+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.847 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.842+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=711009158}) 2024-09-16T04:46:54.847 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.842+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4128048888}) 2024-09-16T04:46:54.847 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.842+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.847 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.842+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=711009158}) 2024-09-16T04:46:54.847 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.842+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=136160963}) 2024-09-16T04:46:54.847 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.843+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.847 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.843+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2006593383}) 2024-09-16T04:46:54.848 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.843+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4001361277}) 2024-09-16T04:46:54.848 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.843+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.848 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.843+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2006593383}) 2024-09-16T04:46:54.848 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.843+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=450871774}) 2024-09-16T04:46:54.848 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.844+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.848 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.844+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2490152956}) 2024-09-16T04:46:54.848 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.844+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2504250525}) 2024-09-16T04:46:54.848 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.844+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.848 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.844+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2490152956}) 2024-09-16T04:46:54.848 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.844+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3044501943}) 2024-09-16T04:46:54.850 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.845+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.850 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.845+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2966286705}) 2024-09-16T04:46:54.850 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.845+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3466073849}) 2024-09-16T04:46:54.851 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.845+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.851 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.845+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2966286705}) 2024-09-16T04:46:54.851 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.845+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1771734877}) 2024-09-16T04:46:54.851 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.846+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.851 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.846+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4150891223}) 2024-09-16T04:46:54.851 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.846+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3817788071}) 2024-09-16T04:46:54.851 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.846+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.851 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.846+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4150891223}) 2024-09-16T04:46:54.851 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.846+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1039996011}) 2024-09-16T04:46:54.851 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.847+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.851 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.847+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2182931978}) 2024-09-16T04:46:54.851 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.847+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=216709109}) 2024-09-16T04:46:54.851 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.847+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.852 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.847+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2182931978}) 2024-09-16T04:46:54.852 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.847+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4143971991}) 2024-09-16T04:46:54.852 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.848+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.852 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.848+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1679245024}) 2024-09-16T04:46:54.852 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.848+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2749406936}) 2024-09-16T04:46:54.852 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.848+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.852 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.848+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1679245024}) 2024-09-16T04:46:54.852 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.848+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=145522720}) 2024-09-16T04:46:54.852 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.848+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.852 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.848+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3771565517}) 2024-09-16T04:46:54.852 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.848+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978599705}) 2024-09-16T04:46:54.852 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.848+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.852 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.848+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3771565517}) 2024-09-16T04:46:54.852 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.848+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=719422856}) 2024-09-16T04:46:54.854 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.850+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.854 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.850+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=5,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=45} crc {logm=2102557572,mdsmap=3956149183,mgr=1081907435,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=284593343,mon_config_key=1385660123,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=4027623034,osd_snap=723727393,osdmap=693127241}) 2024-09-16T04:46:54.854 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.850+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=5,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=46} crc {logm=2102557572,mdsmap=3956149183,mgr=1081907435,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=284593343,mon_config_key=3267531759,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=4027623034,osd_snap=723727393,osdmap=919128941}) 2024-09-16T04:46:54.854 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.850+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.854 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.850+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=5,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=45} crc {logm=2102557572,mdsmap=3956149183,mgr=1081907435,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=284593343,mon_config_key=1385660123,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=4027623034,osd_snap=723727393,osdmap=693127241}) 2024-09-16T04:46:54.854 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.850+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=4,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=9,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=51} crc {logm=4285473996,mdsmap=3956149183,mgr=1652149447,mgr_metadata=4119166318,mgrstat=454564010,mon_config_key=2274580102,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=4027623034,osd_snap=723727393,osdmap=3285176095}) 2024-09-16T04:46:54.854 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.851+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.854 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.851+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3013482554}) 2024-09-16T04:46:54.854 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.851+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1365852781}) 2024-09-16T04:46:54.855 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.851+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.855 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.851+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3013482554}) 2024-09-16T04:46:54.855 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.851+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2665662722}) 2024-09-16T04:46:54.855 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.852+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.855 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.852+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2118926411}) 2024-09-16T04:46:54.855 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.852+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=776357473}) 2024-09-16T04:46:54.855 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.852+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.855 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.852+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2118926411}) 2024-09-16T04:46:54.855 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.852+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3632668602}) 2024-09-16T04:46:54.856 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.854+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.856 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.854+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1008228967}) 2024-09-16T04:46:54.856 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.854+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1531659336}) 2024-09-16T04:46:54.856 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.854+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.856 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.854+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1008228967}) 2024-09-16T04:46:54.856 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.854+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1031427094}) 2024-09-16T04:46:54.857 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.856+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.857 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.856+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3210748643}) 2024-09-16T04:46:54.858 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.856+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1061289486}) 2024-09-16T04:46:54.858 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.856+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.858 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.856+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3210748643}) 2024-09-16T04:46:54.858 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.856+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1956989965}) 2024-09-16T04:46:54.864 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.864+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.864 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.864+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=20} crc {osdmap=1145723592}) 2024-09-16T04:46:54.865 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.864+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=18} crc {osdmap=3747254618}) 2024-09-16T04:46:54.865 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.864+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:46:54.865 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.864+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=20} crc {osdmap=1145723592}) 2024-09-16T04:46:54.865 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:46:54.864+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=18} crc {osdmap=414479200}) 2024-09-16T04:46:55.842 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:46:55.843 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:46:55.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.955+0000 7f42ef027640 10 monclient: get_monmap_and_config 2024-09-16T04:46:55.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.955+0000 7f42ef027640 10 monclient: build_initial_monmap 2024-09-16T04:46:55.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.955+0000 7f42ef027640 10 monclient: monmap: 2024-09-16T04:46:55.956 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:55.956 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:55.956 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:55.955641+0000 2024-09-16T04:46:55.956 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:55.955641+0000 2024-09-16T04:46:55.956 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:55.957 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:55.957 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:55.957 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:55.957 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:55.957 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:55.957 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:55.957 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:55.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.955+0000 7f42ef027640 1 Processor -- start 2024-09-16T04:46:55.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42ef027640 1 -- start start 2024-09-16T04:46:55.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42ef027640 10 monclient: init 2024-09-16T04:46:55.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42ef027640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:55.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42ef027640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-16T04:46:55.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42ef027640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f42e80a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:46:55.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42ef027640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f42e80a3ff0 0x7f42e80a4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:55.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42ef027640 10 monclient(hunting): picked mon.noname-a con 0x7f42e80a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:46:55.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42ef027640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f42e813be90 addr v1:172.21.15.114:6789/0 2024-09-16T04:46:55.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42ef027640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42e809ab70 con 0x7f42e80a3c10 2024-09-16T04:46:55.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42ef027640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42e80a4d60 con 0x7f42e813be90 2024-09-16T04:46:55.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42ef027640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:55.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42ef027640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f42e809b030 con 0x7f42e80a3ff0 2024-09-16T04:46:55.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42ef027640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:55.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42ef027640 10 monclient(hunting): authenticate will time out at 1214.371598s 2024-09-16T04:46:55.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42e7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f42e80a3ff0 0x7f42e80a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:55.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42ecdc3640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f42e80a3c10 0x7f42e809bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45084/0 (socket says 172.21.15.46:45084) 2024-09-16T04:46:55.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42ecdc3640 1 -- 172.21.15.46:0/1576471477 learned_addr learned my addr 172.21.15.46:0/1576471477 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:46:55.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42e7fff640 10 monclient(hunting): get_auth_request con 0x7f42e80a3ff0 auth_method 0 2024-09-16T04:46:55.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:55.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:55.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:55.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:55.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:55.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:55.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.956+0000 7f42e7fff640 10 monclient(hunting): handle_auth_done global_id 104157 payload 274 2024-09-16T04:46:55.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.957+0000 7f42e7fff640 1 -- 172.21.15.46:0/1576471477 >> v1:172.21.15.114:6789/0 conn(0x7f42e813be90 legacy=0x7f42e813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:55.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.957+0000 7f42e7fff640 1 -- 172.21.15.46:0/1576471477 >> v1:172.21.15.46:6789/0 conn(0x7f42e80a3c10 legacy=0x7f42e809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:55.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.957+0000 7f42e7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:46:55.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.957+0000 7f42e7fff640 1 monclient: found mon.noname-a 2024-09-16T04:46:55.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.957+0000 7f42e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:55.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.957+0000 7f42e7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:46:55.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.957+0000 7f42e7fff640 1 -- 172.21.15.46:0/1576471477 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e809b310 con 0x7f42e80a3ff0 2024-09-16T04:46:55.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.957+0000 7f42e7fff640 1 --2- 172.21.15.46:0/1576471477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42e80a3ff0 0x7f42e80a4400 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f42d8009d90 tx=0x7f42d802a580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a378eea825c7f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:55.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.957+0000 7f42e77fe640 1 -- 172.21.15.46:0/1576471477 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d8007740 con 0x7f42e80a3ff0 2024-09-16T04:46:55.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.957+0000 7f42e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:55.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.957+0000 7f42e77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:46:55.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.957+0000 7f42e77fe640 10 monclient: dump: 2024-09-16T04:46:55.970 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:55.970 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:55.970 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:55.970 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:55.970 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:55.970 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:55.970 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:55.970 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:55.970 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:55.971 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:55.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.957+0000 7f42e77fe640 10 monclient: _finish_auth 0 2024-09-16T04:46:55.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.957+0000 7f42e77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:46:55.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.957+0000 7f42e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:55.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.958+0000 7f42e77fe640 1 -- 172.21.15.46:0/1576471477 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42d80078e0 con 0x7f42e80a3ff0 2024-09-16T04:46:55.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.958+0000 7f42ef027640 5 monclient: authenticate success, global_id 104157 2024-09-16T04:46:55.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.958+0000 7f42ef027640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:55.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.958+0000 7f42e77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:55.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.958+0000 7f42e77fe640 1 -- 172.21.15.46:0/1576471477 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d8007cb0 con 0x7f42e80a3ff0 2024-09-16T04:46:55.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.958+0000 7f42e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:55.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.958+0000 7f42e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:55.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.958+0000 7f42e77fe640 10 monclient: dump: 2024-09-16T04:46:55.971 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:55.972 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:55.972 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:55.972 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:55.972 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:55.972 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:55.972 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:55.972 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:55.972 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:55.972 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:55.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.958+0000 7f42ef027640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:55.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.958+0000 7f42ef027640 4 set_mon_vals no callback set 2024-09-16T04:46:55.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.958+0000 7f42ef027640 10 monclient: shutdown 2024-09-16T04:46:55.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.958+0000 7f42ef027640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:55.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.958+0000 7f42ef027640 1 -- 172.21.15.46:0/1576471477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42e80a3ff0 msgr2=0x7f42e80a4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:55.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.958+0000 7f42ef027640 1 --2- 172.21.15.46:0/1576471477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42e80a3ff0 0x7f42e80a4400 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f42d8009d90 tx=0x7f42d802a580 comp rx=0 tx=0).stop 2024-09-16T04:46:55.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.958+0000 7f42ef027640 1 -- 172.21.15.46:0/1576471477 shutdown_connections 2024-09-16T04:46:55.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.958+0000 7f42ef027640 1 --2- 172.21.15.46:0/1576471477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42e80a3ff0 0x7f42e80a4400 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:55.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.958+0000 7f42ef027640 1 -- 172.21.15.46:0/1576471477 >> 172.21.15.46:0/1576471477 conn(0x7f42e80a5d10 msgr2=0x7f42e80a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:55.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.958+0000 7f42ef027640 1 -- 172.21.15.46:0/1576471477 shutdown_connections 2024-09-16T04:46:55.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.958+0000 7f42ef027640 1 -- 172.21.15.46:0/1576471477 wait complete. 2024-09-16T04:46:55.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.959+0000 7f42ef027640 10 monclient: build_initial_monmap 2024-09-16T04:46:55.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.959+0000 7f42ef027640 10 monclient: monmap: 2024-09-16T04:46:55.973 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:55.973 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:55.973 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:55.973 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:55.974 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:55.974 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:55.974 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:55.974 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:55.974 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:55.974 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:55.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.959+0000 7f42ef027640 1 Processor -- start 2024-09-16T04:46:55.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.959+0000 7f42ef027640 1 -- start start 2024-09-16T04:46:55.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.959+0000 7f42ef027640 10 monclient: init 2024-09-16T04:46:55.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ef027640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:55.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ef027640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:46:55.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ef027640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42e80a3c10 0x7f42e8136ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:55.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ef027640 10 monclient(hunting): picked mon.noname-a con 0x7f42e80a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:55.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ef027640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f42e80a3ff0 0x7f42e80a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:55.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ef027640 10 monclient(hunting): picked mon.noname-c con 0x7f42e80a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:55.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ef027640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42e813be90 0x7f42e80a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:55.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ef027640 10 monclient(hunting): picked mon.noname-b con 0x7f42e813be90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:55.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ef027640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:55.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ef027640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f42e806e550 con 0x7f42e80a3c10 2024-09-16T04:46:55.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ef027640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:55.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ef027640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f42e806e3d0 con 0x7f42e813be90 2024-09-16T04:46:55.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ef027640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:55.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ef027640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f42e813f9a0 con 0x7f42e80a3ff0 2024-09-16T04:46:55.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ef027640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:55.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ef027640 10 monclient(hunting): authenticate will time out at 1214.375505s 2024-09-16T04:46:55.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ecdc3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42e80a3c10 0x7f42e8136ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:55.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42e7fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f42e80a3ff0 0x7f42e80a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:55.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ecdc3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42e80a3c10 0x7f42e8136ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35158/0 (socket says 172.21.15.46:35158) 2024-09-16T04:46:55.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ecdc3640 1 -- 172.21.15.46:0/3814431282 learned_addr learned my addr 172.21.15.46:0/3814431282 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:55.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ecdc3640 10 monclient(hunting): get_auth_request con 0x7f42e80a3c10 auth_method 0 2024-09-16T04:46:55.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ecdc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:55.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ecdc3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:55.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ecdc3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:55.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42e7fff640 10 monclient(hunting): get_auth_request con 0x7f42e80a3ff0 auth_method 0 2024-09-16T04:46:55.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:55.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:55.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:55.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ecdc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:55.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ecdc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:55.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ecdc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:55.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:55.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:55.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:55.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ed5c4640 1 --2- 172.21.15.46:0/3814431282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42e813be90 0x7f42e80a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:55.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ecdc3640 10 monclient(hunting): handle_auth_done global_id 104163 payload 824 2024-09-16T04:46:55.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ecdc3640 1 -- 172.21.15.46:0/3814431282 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f42e80a3ff0 msgr2=0x7f42e80a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:55.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ecdc3640 1 --2- 172.21.15.46:0/3814431282 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f42e80a3ff0 0x7f42e80a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:55.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ecdc3640 1 -- 172.21.15.46:0/3814431282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42e813be90 msgr2=0x7f42e80a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:55.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ecdc3640 1 --2- 172.21.15.46:0/3814431282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42e813be90 0x7f42e80a0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:55.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ecdc3640 10 monclient: _finish_hunting 0 2024-09-16T04:46:55.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ecdc3640 1 monclient: found mon.noname-a 2024-09-16T04:46:55.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ecdc3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:55.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ecdc3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:46:55.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.960+0000 7f42ecdc3640 1 -- 172.21.15.46:0/3814431282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e809d5c0 con 0x7f42e80a3c10 2024-09-16T04:46:55.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42ecdc3640 1 --2- 172.21.15.46:0/3814431282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42e80a3c10 0x7f42e8136ca0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f42d000c400 tx=0x7f42d000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:55.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42e57fa640 1 -- 172.21.15.46:0/3814431282 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d0017070 con 0x7f42e80a3c10 2024-09-16T04:46:55.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:55.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42e57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:46:55.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42e57fa640 10 monclient: dump: 2024-09-16T04:46:55.978 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:55.978 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:55.978 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:55.978 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:55.978 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:55.978 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:55.978 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:55.979 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:55.979 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:55.979 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:55.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42e57fa640 10 monclient: _finish_auth 0 2024-09-16T04:46:55.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42e57fa640 10 monclient: _check_auth_tickets 2024-09-16T04:46:55.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:55.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42ef027640 5 monclient: authenticate success, global_id 104163 2024-09-16T04:46:55.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42e57fa640 1 -- 172.21.15.46:0/3814431282 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42d0005ce0 con 0x7f42e80a3c10 2024-09-16T04:46:55.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42e57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:55.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42ef027640 10 monclient: _renew_subs 2024-09-16T04:46:55.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42e57fa640 1 -- 172.21.15.46:0/3814431282 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d00072f0 con 0x7f42e80a3c10 2024-09-16T04:46:55.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42ef027640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:55.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42ef027640 1 -- 172.21.15.46:0/3814431282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42e813f690 con 0x7f42e80a3c10 2024-09-16T04:46:55.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:55.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42e57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:55.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42e57fa640 10 monclient: dump: 2024-09-16T04:46:55.980 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:55.980 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:55.980 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:55.980 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:55.980 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:55.980 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:55.980 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:55.980 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:55.980 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:55.980 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:55.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42e6ffd640 4 set_mon_vals no callback set 2024-09-16T04:46:55.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42ef027640 10 monclient: _renew_subs 2024-09-16T04:46:55.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42ef027640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:55.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.961+0000 7f42ef027640 1 -- 172.21.15.46:0/3814431282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42e8120480 con 0x7f42e80a3c10 2024-09-16T04:46:55.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.962+0000 7f42e57fa640 1 -- 172.21.15.46:0/3814431282 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f42d001a070 con 0x7f42e80a3c10 2024-09-16T04:46:55.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.962+0000 7f42ef027640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:55.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.962+0000 7f42ef027640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:55.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.962+0000 7f42ef027640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:55.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.962+0000 7f42e57fa640 1 --2- 172.21.15.46:0/3814431282 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f42c80383b0 0x7f42c803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:55.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.962+0000 7f42ef027640 1 -- 172.21.15.46:0/3814431282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42e80a4400 con 0x7f42e80a3c10 2024-09-16T04:46:55.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.962+0000 7f42e57fa640 1 -- 172.21.15.46:0/3814431282 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4738+0+0 (secure 0 0 0) 0x7f42d004ab80 con 0x7f42e80a3c10 2024-09-16T04:46:55.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.962+0000 7f42e7fff640 1 --2- 172.21.15.46:0/3814431282 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f42c80383b0 0x7f42c803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:55.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.962+0000 7f42e7fff640 10 monclient: get_auth_request con 0x7f42c80383b0 auth_method 0 2024-09-16T04:46:55.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.963+0000 7f42e7fff640 1 --2- 172.21.15.46:0/3814431282 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f42c80383b0 0x7f42c803a870 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f42d8034000 tx=0x7f42d802f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:55.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.967+0000 7f42e57fa640 1 -- 172.21.15.46:0/3814431282 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f42d0019e10 con 0x7f42e80a3c10 2024-09-16T04:46:55.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.967+0000 7f42e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:55.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:55.967+0000 7f42e57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:56.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.214+0000 7f42ef027640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:46:56.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.214+0000 7f42ef027640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:56.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.214+0000 7f42ef027640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:56.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.214+0000 7f42ef027640 1 -- 172.21.15.46:0/3814431282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f42e80a2c70 con 0x7f42e80a3c10 2024-09-16T04:46:56.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.214+0000 7f42e57fa640 1 -- 172.21.15.46:0/3814431282 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f42d0010920 con 0x7f42e80a3c10 2024-09-16T04:46:56.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.214+0000 7f42e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:56.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.214+0000 7f42e57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:56.216 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:46:56.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.216+0000 7f42ef027640 1 -- 172.21.15.46:0/3814431282 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f42c80383b0 msgr2=0x7f42c803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.216+0000 7f42ef027640 1 --2- 172.21.15.46:0/3814431282 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f42c80383b0 0x7f42c803a870 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f42d8034000 tx=0x7f42d802f000 comp rx=0 tx=0).stop 2024-09-16T04:46:56.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.216+0000 7f42ef027640 10 monclient: shutdown 2024-09-16T04:46:56.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.216+0000 7f42ef027640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:56.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.216+0000 7f42ef027640 1 -- 172.21.15.46:0/3814431282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42e80a3c10 msgr2=0x7f42e8136ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.216+0000 7f42ef027640 1 --2- 172.21.15.46:0/3814431282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42e80a3c10 0x7f42e8136ca0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f42d000c400 tx=0x7f42d000c8c0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.216+0000 7f42ef027640 1 -- 172.21.15.46:0/3814431282 shutdown_connections 2024-09-16T04:46:56.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.216+0000 7f42ef027640 1 --2- 172.21.15.46:0/3814431282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42e813be90 0x7f42e80a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.216+0000 7f42ef027640 1 --2- 172.21.15.46:0/3814431282 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f42e80a3ff0 0x7f42e80a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.216+0000 7f42ef027640 1 --2- 172.21.15.46:0/3814431282 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f42c80383b0 0x7f42c803a870 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.216+0000 7f42ef027640 1 --2- 172.21.15.46:0/3814431282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42e80a3c10 0x7f42e8136ca0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.216+0000 7f42ef027640 1 -- 172.21.15.46:0/3814431282 >> 172.21.15.46:0/3814431282 conn(0x7f42e80a5d10 msgr2=0x7f42e809f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:56.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.216+0000 7f42ef027640 1 -- 172.21.15.46:0/3814431282 shutdown_connections 2024-09-16T04:46:56.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.216+0000 7f42ef027640 1 -- 172.21.15.46:0/3814431282 wait complete. 2024-09-16T04:46:56.227 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:46:56.228 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:46:56.228 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:46:56.228 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:46:56.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.342+0000 7fdf81b18640 10 monclient: get_monmap_and_config 2024-09-16T04:46:56.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.342+0000 7fdf81b18640 10 monclient: build_initial_monmap 2024-09-16T04:46:56.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.342+0000 7fdf81b18640 10 monclient: monmap: 2024-09-16T04:46:56.343 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:56.343 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.343 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:56.342657+0000 2024-09-16T04:46:56.343 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:56.342657+0000 2024-09-16T04:46:56.343 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:56.344 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.344 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:56.344 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:56.344 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:56.344 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:56.344 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:56.344 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.345 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:56.345 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40388]: starting. 2024-09-16T04:46:56.345 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40388]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-40351 2024-09-16T04:46:56.345 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40388]: created object 0... 2024-09-16T04:46:56.345 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40388]: created object 25... 2024-09-16T04:46:56.345 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40388]: created object 49... 2024-09-16T04:46:56.345 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40388]: finishing. 2024-09-16T04:46:56.345 INFO:teuthology.orchestra.run.smithi114.stdout:process_3_[40388]: shutting down. 2024-09-16T04:46:56.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.342+0000 7fdf81b18640 1 Processor -- start 2024-09-16T04:46:56.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf81b18640 1 -- start start 2024-09-16T04:46:56.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf81b18640 10 monclient: init 2024-09-16T04:46:56.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf81b18640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:56.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf81b18640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-16T04:46:56.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf81b18640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdf7c0a3c10 0x7fdf7c09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf81b18640 10 monclient(hunting): picked mon.noname-a con 0x7fdf7c0a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:46:56.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf81b18640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdf7c0a3ff0 0x7fdf7c0a4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf81b18640 10 monclient(hunting): picked mon.noname-c con 0x7fdf7c0a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:46:56.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf81b18640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdf7c1369d0 0x7fdf7c136e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf81b18640 10 monclient(hunting): picked mon.noname-b con 0x7fdf7c1369d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:56.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf81b18640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf81b18640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fdf7c09ab70 con 0x7fdf7c0a3c10 2024-09-16T04:46:56.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf81b18640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf81b18640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fdf7c0a4d60 con 0x7fdf7c0a3ff0 2024-09-16T04:46:56.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf81b18640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf81b18640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fdf7c09b030 con 0x7fdf7c1369d0 2024-09-16T04:46:56.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf81b18640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:56.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf81b18640 10 monclient(hunting): authenticate will time out at 1214.758538s 2024-09-16T04:46:56.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdf7c0a3c10 0x7fdf7c09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7bfff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdf7c1369d0 0x7fdf7c136e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdf7c0a3c10 0x7fdf7c09bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35166/0 (socket says 172.21.15.46:35166) 2024-09-16T04:46:56.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7b7fe640 1 -- 172.21.15.46:0/2379267972 learned_addr learned my addr 172.21.15.46:0/2379267972 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:56.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7b7fe640 10 monclient(hunting): get_auth_request con 0x7fdf7c0a3c10 auth_method 0 2024-09-16T04:46:56.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7bfff640 10 monclient(hunting): get_auth_request con 0x7fdf7c1369d0 auth_method 0 2024-09-16T04:46:56.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7affd640 1 --2- 172.21.15.46:0/2379267972 >> v2:172.21.15.114:3300/0 conn(0x7fdf7c0a3ff0 0x7fdf7c0a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7affd640 10 monclient(hunting): get_auth_request con 0x7fdf7c0a3ff0 auth_method 0 2024-09-16T04:46:56.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7b7fe640 10 monclient(hunting): handle_auth_done global_id 104166 payload 274 2024-09-16T04:46:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7b7fe640 1 -- 172.21.15.46:0/2379267972 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdf7c1369d0 msgr2=0x7fdf7c136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7b7fe640 1 --2- 172.21.15.46:0/2379267972 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdf7c1369d0 0x7fdf7c136e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7b7fe640 1 -- 172.21.15.46:0/2379267972 >> v2:172.21.15.114:3300/0 conn(0x7fdf7c0a3ff0 msgr2=0x7fdf7c0a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7b7fe640 1 --2- 172.21.15.46:0/2379267972 >> v2:172.21.15.114:3300/0 conn(0x7fdf7c0a3ff0 0x7fdf7c0a4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:46:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7b7fe640 1 monclient: found mon.noname-a 2024-09-16T04:46:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:46:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7b7fe640 1 -- 172.21.15.46:0/2379267972 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf7c09b310 con 0x7fdf7c0a3c10 2024-09-16T04:46:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.343+0000 7fdf7bfff640 1 --2- 172.21.15.46:0/2379267972 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdf7c1369d0 0x7fdf7c136e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.344+0000 7fdf7b7fe640 1 --2- 172.21.15.46:0/2379267972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c0a3c10 0x7fdf7c09bcf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c06edc0 tx=0x7fdf6c02a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6576aa548d782a3f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.344+0000 7fdf7a7fc640 1 -- 172.21.15.46:0/2379267972 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c037070 con 0x7fdf7c0a3c10 2024-09-16T04:46:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.344+0000 7fdf7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.344+0000 7fdf7a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:46:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.344+0000 7fdf7a7fc640 10 monclient: dump: 2024-09-16T04:46:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.361 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.344+0000 7fdf7a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:46:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.344+0000 7fdf7a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:46:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.344+0000 7fdf7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.344+0000 7fdf81b18640 5 monclient: authenticate success, global_id 104166 2024-09-16T04:46:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.344+0000 7fdf81b18640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.344+0000 7fdf7a7fc640 1 -- 172.21.15.46:0/2379267972 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf6c02ae00 con 0x7fdf7c0a3c10 2024-09-16T04:46:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.344+0000 7fdf7a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.344+0000 7fdf81b18640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.344+0000 7fdf81b18640 4 set_mon_vals no callback set 2024-09-16T04:46:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.344+0000 7fdf7a7fc640 1 -- 172.21.15.46:0/2379267972 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c032660 con 0x7fdf7c0a3c10 2024-09-16T04:46:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.344+0000 7fdf7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.344+0000 7fdf7a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.344+0000 7fdf7a7fc640 10 monclient: dump: 2024-09-16T04:46:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.363 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.363 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.363 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.363 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.345+0000 7fdf81b18640 10 monclient: shutdown 2024-09-16T04:46:56.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.345+0000 7fdf81b18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:56.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.345+0000 7fdf81b18640 1 -- 172.21.15.46:0/2379267972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c0a3c10 msgr2=0x7fdf7c09bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.345+0000 7fdf81b18640 1 --2- 172.21.15.46:0/2379267972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c0a3c10 0x7fdf7c09bcf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c06edc0 tx=0x7fdf6c02a0b0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.345+0000 7fdf81b18640 1 -- 172.21.15.46:0/2379267972 shutdown_connections 2024-09-16T04:46:56.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.345+0000 7fdf81b18640 1 --2- 172.21.15.46:0/2379267972 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdf7c1369d0 0x7fdf7c136e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.345+0000 7fdf81b18640 1 --2- 172.21.15.46:0/2379267972 >> v2:172.21.15.114:3300/0 conn(0x7fdf7c0a3ff0 0x7fdf7c0a4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.345+0000 7fdf81b18640 1 --2- 172.21.15.46:0/2379267972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c0a3c10 0x7fdf7c09bcf0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.345+0000 7fdf81b18640 1 -- 172.21.15.46:0/2379267972 >> 172.21.15.46:0/2379267972 conn(0x7fdf7c0a5d10 msgr2=0x7fdf7c0a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:56.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.345+0000 7fdf81b18640 1 -- 172.21.15.46:0/2379267972 shutdown_connections 2024-09-16T04:46:56.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.345+0000 7fdf81b18640 1 -- 172.21.15.46:0/2379267972 wait complete. 2024-09-16T04:46:56.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.346+0000 7fdf81b18640 10 monclient: build_initial_monmap 2024-09-16T04:46:56.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.346+0000 7fdf81b18640 10 monclient: monmap: 2024-09-16T04:46:56.364 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:56.364 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:56.364 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:56.364 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:56.364 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:56.364 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.365 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:56.365 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:56.365 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:56.365 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.346+0000 7fdf81b18640 1 Processor -- start 2024-09-16T04:46:56.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.346+0000 7fdf81b18640 1 -- start start 2024-09-16T04:46:56.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.346+0000 7fdf81b18640 10 monclient: init 2024-09-16T04:46:56.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.346+0000 7fdf81b18640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:56.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.346+0000 7fdf81b18640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:46:56.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.346+0000 7fdf81b18640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c0a3c10 0x7fdf7c1cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.346+0000 7fdf81b18640 10 monclient(hunting): picked mon.noname-a con 0x7fdf7c0a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:56.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.346+0000 7fdf81b18640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdf7c0a3ff0 0x7fdf7c1cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf81b18640 10 monclient(hunting): picked mon.noname-c con 0x7fdf7c0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:56.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf81b18640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf7c1369d0 0x7fdf7c123090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf81b18640 10 monclient(hunting): picked mon.noname-b con 0x7fdf7c1369d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:56.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf81b18640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf81b18640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c06e3d0 con 0x7fdf7c0a3c10 2024-09-16T04:46:56.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf81b18640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf81b18640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c0a4940 con 0x7fdf7c1369d0 2024-09-16T04:46:56.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf81b18640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf81b18640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fdf7c06e550 con 0x7fdf7c0a3ff0 2024-09-16T04:46:56.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf81b18640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:56.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf81b18640 10 monclient(hunting): authenticate will time out at 1214.762050s 2024-09-16T04:46:56.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7affd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdf7c0a3ff0 0x7fdf7c1cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c0a3c10 0x7fdf7c1cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c0a3c10 0x7fdf7c1cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35170/0 (socket says 172.21.15.46:35170) 2024-09-16T04:46:56.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7affd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdf7c0a3ff0 0x7fdf7c1cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:35798/0 (socket says 172.21.15.46:35798) 2024-09-16T04:46:56.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7b7fe640 1 -- 172.21.15.46:0/32921082 learned_addr learned my addr 172.21.15.46:0/32921082 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:56.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7b7fe640 10 monclient(hunting): get_auth_request con 0x7fdf7c0a3c10 auth_method 0 2024-09-16T04:46:56.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7affd640 10 monclient(hunting): get_auth_request con 0x7fdf7c0a3ff0 auth_method 0 2024-09-16T04:46:56.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7bfff640 1 --2- 172.21.15.46:0/32921082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf7c1369d0 0x7fdf7c123090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7bfff640 10 monclient(hunting): get_auth_request con 0x7fdf7c1369d0 auth_method 0 2024-09-16T04:46:56.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7affd640 10 monclient(hunting): handle_auth_done global_id 94358 payload 824 2024-09-16T04:46:56.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7affd640 1 -- 172.21.15.46:0/32921082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf7c1369d0 msgr2=0x7fdf7c123090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7affd640 1 --2- 172.21.15.46:0/32921082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf7c1369d0 0x7fdf7c123090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7affd640 1 -- 172.21.15.46:0/32921082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c0a3c10 msgr2=0x7fdf7c1cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7affd640 1 --2- 172.21.15.46:0/32921082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c0a3c10 0x7fdf7c1cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7affd640 10 monclient: _finish_hunting 0 2024-09-16T04:46:56.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7affd640 1 monclient: found mon.noname-c 2024-09-16T04:46:56.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:56.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:46:56.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7affd640 1 -- 172.21.15.46:0/32921082 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf7c125130 con 0x7fdf7c0a3ff0 2024-09-16T04:46:56.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7b7fe640 1 --2- 172.21.15.46:0/32921082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c0a3c10 0x7fdf7c1cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:56.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf7affd640 1 --2- 172.21.15.46:0/32921082 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdf7c0a3ff0 0x7fdf7c1cc420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000e880 tx=0x7fdf7000ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:56.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf63fff640 1 -- 172.21.15.46:0/32921082 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70017070 con 0x7fdf7c0a3ff0 2024-09-16T04:46:56.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf63fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:56.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf63fff640 10 monclient: dump: 2024-09-16T04:46:56.369 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.370 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.370 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.370 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.370 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.370 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.370 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.370 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.370 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.370 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf63fff640 10 monclient: _finish_auth 0 2024-09-16T04:46:56.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf63fff640 10 monclient: _check_auth_tickets 2024-09-16T04:46:56.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:56.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf63fff640 1 -- 172.21.15.46:0/32921082 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf7000c940 con 0x7fdf7c0a3ff0 2024-09-16T04:46:56.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf81b18640 5 monclient: authenticate success, global_id 94358 2024-09-16T04:46:56.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf63fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:56.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf63fff640 1 -- 172.21.15.46:0/32921082 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7000cd30 con 0x7fdf7c0a3ff0 2024-09-16T04:46:56.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf81b18640 10 monclient: _renew_subs 2024-09-16T04:46:56.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf81b18640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:56.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf81b18640 1 -- 172.21.15.46:0/32921082 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf7c1257f0 con 0x7fdf7c0a3ff0 2024-09-16T04:46:56.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf63fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:56.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf63fff640 10 monclient: dump: 2024-09-16T04:46:56.371 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.372 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.372 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.372 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.372 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.372 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.372 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.372 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.373 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.373 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf79ffb640 4 set_mon_vals no callback set 2024-09-16T04:46:56.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf81b18640 10 monclient: _renew_subs 2024-09-16T04:46:56.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf81b18640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:56.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.347+0000 7fdf81b18640 1 -- 172.21.15.46:0/32921082 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf7c1266e0 con 0x7fdf7c0a3ff0 2024-09-16T04:46:56.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.348+0000 7fdf81b18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:56.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.348+0000 7fdf81b18640 1 --2- 172.21.15.46:0/32921082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c126920 0x7fdf7c1235d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.348+0000 7fdf81b18640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.348+0000 7fdf81b18640 1 -- 172.21.15.46:0/32921082 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c069150 con 0x7fdf7c126920 2024-09-16T04:46:56.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.348+0000 7fdf7b7fe640 1 --2- 172.21.15.46:0/32921082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c126920 0x7fdf7c1235d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.348+0000 7fdf63fff640 1 -- 172.21.15.46:0/32921082 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fdf7001a070 con 0x7fdf7c0a3ff0 2024-09-16T04:46:56.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.348+0000 7fdf7b7fe640 10 monclient: get_auth_request con 0x7fdf7c126920 auth_method 0 2024-09-16T04:46:56.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.348+0000 7fdf7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.348+0000 7fdf7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.348+0000 7fdf7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.348+0000 7fdf7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.348+0000 7fdf7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.348+0000 7fdf7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.349+0000 7fdf7b7fe640 10 monclient(hunting): handle_auth_done global_id 104172 payload 824 2024-09-16T04:46:56.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.349+0000 7fdf63fff640 1 --2- 172.21.15.46:0/32921082 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdf680383b0 0x7fdf6803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.349+0000 7fdf7b7fe640 1 -- 172.21.15.46:0/32921082 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdf7c126d00 con 0x7fdf7c126920 2024-09-16T04:46:56.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.349+0000 7fdf63fff640 1 -- 172.21.15.46:0/32921082 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(314..314 src has 1..314) ==== 4738+0+0 (secure 0 0 0) 0x7fdf7004b710 con 0x7fdf7c0a3ff0 2024-09-16T04:46:56.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.349+0000 7fdf7b7fe640 1 --2- 172.21.15.46:0/32921082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c126920 0x7fdf7c1235d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0090b0 tx=0x7fdf6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:56.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.349+0000 7fdf7bfff640 1 --2- 172.21.15.46:0/32921082 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdf680383b0 0x7fdf6803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.349+0000 7fdf7bfff640 10 monclient: get_auth_request con 0x7fdf680383b0 auth_method 0 2024-09-16T04:46:56.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.349+0000 7fdf63fff640 1 -- 172.21.15.46:0/32921082 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c043070 con 0x7fdf7c126920 2024-09-16T04:46:56.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.349+0000 7fdf63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.349+0000 7fdf63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:56.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.349+0000 7fdf63fff640 10 monclient: dump: 2024-09-16T04:46:56.375 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.375 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.375 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.375 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.375 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.375 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.375 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.375 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.375 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.375 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.349+0000 7fdf63fff640 1 -- 172.21.15.46:0/32921082 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdf6c03c850 con 0x7fdf7c126920 2024-09-16T04:46:56.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.349+0000 7fdf63fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:56.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.349+0000 7fdf63fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:56.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.349+0000 7fdf63fff640 1 -- 172.21.15.46:0/32921082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c126920 msgr2=0x7fdf7c1235d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.349+0000 7fdf63fff640 1 --2- 172.21.15.46:0/32921082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c126920 0x7fdf7c1235d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0090b0 tx=0x7fdf6c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.349+0000 7fdf63fff640 1 -- 172.21.15.46:0/32921082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c126920 msgr2=0x7fdf7c1235d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:56.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.349+0000 7fdf63fff640 1 --2- 172.21.15.46:0/32921082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c126920 0x7fdf7c1235d0 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0090b0 tx=0x7fdf6c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.349+0000 7fdf7bfff640 1 --2- 172.21.15.46:0/32921082 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdf680383b0 0x7fdf6803a870 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fdf74011420 tx=0x7fdf7400b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.355+0000 7fdf81b18640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:46:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.355+0000 7fdf81b18640 1 --2- 172.21.15.46:0/32921082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c123b10 0x7fdf7c123f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.355+0000 7fdf81b18640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.355+0000 7fdf81b18640 1 -- 172.21.15.46:0/32921082 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c124460 con 0x7fdf7c123b10 2024-09-16T04:46:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf7b7fe640 1 --2- 172.21.15.46:0/32921082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c123b10 0x7fdf7c123f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf7b7fe640 10 monclient: get_auth_request con 0x7fdf7c123b10 auth_method 0 2024-09-16T04:46:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf7b7fe640 10 monclient(hunting): handle_auth_done global_id 104175 payload 824 2024-09-16T04:46:56.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf7b7fe640 1 -- 172.21.15.46:0/32921082 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdf7c1cc960 con 0x7fdf7c123b10 2024-09-16T04:46:56.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf7b7fe640 1 --2- 172.21.15.46:0/32921082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c123b10 0x7fdf7c123f20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c069320 tx=0x7fdf6c009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:56.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf63fff640 1 -- 172.21.15.46:0/32921082 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c037040 con 0x7fdf7c123b10 2024-09-16T04:46:56.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:56.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf63fff640 10 monclient: dump: 2024-09-16T04:46:56.379 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.379 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.379 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.379 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.379 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.379 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.379 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.379 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.379 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.379 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf63fff640 1 -- 172.21.15.46:0/32921082 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7fdf6c003960 con 0x7fdf7c123b10 2024-09-16T04:46:56.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf63fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:46:56.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf63fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:56.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf63fff640 1 -- 172.21.15.46:0/32921082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c123b10 msgr2=0x7fdf7c123f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf63fff640 1 --2- 172.21.15.46:0/32921082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c123b10 0x7fdf7c123f20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c069320 tx=0x7fdf6c009240 comp rx=0 tx=0).stop 2024-09-16T04:46:56.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf63fff640 1 -- 172.21.15.46:0/32921082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c123b10 msgr2=0x7fdf7c123f20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:56.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.356+0000 7fdf63fff640 1 --2- 172.21.15.46:0/32921082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf7c123b10 0x7fdf7c123f20 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c069320 tx=0x7fdf6c009240 comp rx=0 tx=0).stop 2024-09-16T04:46:56.380 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:46:56.380 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:46:56.381 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:46:56.381 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:46:56.381 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 94, 2024-09-16T04:46:56.381 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 9413, 2024-09-16T04:46:56.381 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:46:56.381 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:46:56.381 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:46:56.381 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:46:56.381 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:56.381 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 3, 2024-09-16T04:46:56.381 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:56.381 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:46:56.381 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:46:56.382 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:56.382 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:56.382 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:56.382 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:56.382 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:56.382 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:56.382 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:56.382 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:56.382 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:56.382 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:56.382 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:56.382 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:56.382 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:46:56.382 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:46:56.382 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:56.383 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:56.383 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:56.383 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:56.383 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:56.383 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:56.383 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:56.383 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:56.383 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:56.383 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:56.383 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:56.383 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:56.383 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.383 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:46:56.383 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:46:56.384 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:46:56.384 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:46:56.384 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:46:56.384 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:46:56.384 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:56.384 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:56.384 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:46:56.384 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:46:56.384 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:46:56.384 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:46:56.384 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:46:56.384 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:46:56.384 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:46:56.384 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:56.385 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:46:56.385 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:56.385 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:56.385 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:56.385 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:56.385 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:56.385 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:56.385 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:56.385 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:56.385 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:56.385 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:56.385 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:56.386 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:56.386 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:46:56.386 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.386 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:46:56.386 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.386 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:46:56.386 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:46:56.386 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:56.386 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:56.386 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.386 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:56.386 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:46:56.386 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:56.387 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.387 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.387 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:56.387 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:46:56.387 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:56.387 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:56.387 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:56.387 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.387 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:46:56.387 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:46:56.387 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:56.387 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:56.387 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:56.387 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.388 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.388 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:46:56.388 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:46:56.388 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:56.388 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:56.388 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.388 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:56.388 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:46:56.388 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:56.388 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.388 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.388 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:56.388 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:46:56.389 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:56.389 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:56.389 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:56.389 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.389 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:46:56.389 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:46:56.389 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:56.389 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:56.389 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:56.389 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.389 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.389 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:46:56.389 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:46:56.389 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:56.390 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:56.390 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.390 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:56.390 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:46:56.390 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:56.390 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.390 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.390 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:56.390 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:46:56.390 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:56.390 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:56.390 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:56.390 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.391 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:46:56.391 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:46:56.391 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:56.391 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:56.391 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:56.391 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:56.391 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:56.391 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.391 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:46:56.391 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:46:56.391 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.391 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:56.392 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:56.392 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:56.392 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:56.392 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:56.392 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:46:56.392 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.392 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:56.392 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:56.392 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:46:56.392 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:56.392 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:56.392 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.392 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:46:56.393 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:46:56.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.357+0000 7fdf81b18640 1 -- 172.21.15.46:0/32921082 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdf680383b0 msgr2=0x7fdf6803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.357+0000 7fdf81b18640 1 --2- 172.21.15.46:0/32921082 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdf680383b0 0x7fdf6803a870 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fdf74011420 tx=0x7fdf7400b040 comp rx=0 tx=0).stop 2024-09-16T04:46:56.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.357+0000 7fdf81b18640 10 monclient: shutdown 2024-09-16T04:46:56.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.357+0000 7fdf81b18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:56.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.357+0000 7fdf81b18640 1 -- 172.21.15.46:0/32921082 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdf7c0a3ff0 msgr2=0x7fdf7c1cc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.357+0000 7fdf81b18640 1 --2- 172.21.15.46:0/32921082 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdf7c0a3ff0 0x7fdf7c1cc420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000e880 tx=0x7fdf7000ed40 comp rx=0 tx=0).stop 2024-09-16T04:46:56.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.357+0000 7fdf7bfff640 1 -- 172.21.15.46:0/32921082 reap_dead start 2024-09-16T04:46:56.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.357+0000 7fdf81b18640 1 -- 172.21.15.46:0/32921082 shutdown_connections 2024-09-16T04:46:56.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.357+0000 7fdf81b18640 1 --2- 172.21.15.46:0/32921082 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdf7c0a3ff0 0x7fdf7c1cc420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.357+0000 7fdf81b18640 1 -- 172.21.15.46:0/32921082 >> 172.21.15.46:0/32921082 conn(0x7fdf7c0a5d10 msgr2=0x7fdf7c09e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:56.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.357+0000 7fdf81b18640 1 -- 172.21.15.46:0/32921082 shutdown_connections 2024-09-16T04:46:56.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.357+0000 7fdf81b18640 1 -- 172.21.15.46:0/32921082 wait complete. 2024-09-16T04:46:56.395 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:46:56.395 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:46:56.395 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:46:56.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.496+0000 7f13d7b52640 10 monclient: get_monmap_and_config 2024-09-16T04:46:56.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.496+0000 7f13d7b52640 10 monclient: build_initial_monmap 2024-09-16T04:46:56.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.496+0000 7f13d7b52640 10 monclient: monmap: 2024-09-16T04:46:56.497 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:56.498 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.498 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:56.496668+0000 2024-09-16T04:46:56.498 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:56.496668+0000 2024-09-16T04:46:56.498 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:56.498 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.498 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:56.498 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:56.498 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:56.498 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:56.498 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:56.499 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.496+0000 7f13d7b52640 1 Processor -- start 2024-09-16T04:46:56.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d7b52640 1 -- start start 2024-09-16T04:46:56.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d7b52640 10 monclient: init 2024-09-16T04:46:56.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d7b52640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:56.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d7b52640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-16T04:46:56.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d7b52640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d006e160 0x7f13d00a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d7b52640 10 monclient(hunting): picked mon.noname-b con 0x7f13d006e160 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:56.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d7b52640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f13d00a2ee0 0x7f13d0136940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d7b52640 10 monclient(hunting): picked mon.noname-c con 0x7f13d00a2ee0 addr v2:172.21.15.114:3300/0 2024-09-16T04:46:56.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d7b52640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f13d0136e80 addr v1:172.21.15.114:6789/0 2024-09-16T04:46:56.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d7b52640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d00a3fd0 con 0x7f13d0136e80 2024-09-16T04:46:56.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d7b52640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d7b52640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f13d00a3300 con 0x7f13d00a2ee0 2024-09-16T04:46:56.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d7b52640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d7b52640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f13d006e540 con 0x7f13d006e160 2024-09-16T04:46:56.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d7b52640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:56.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d7b52640 10 monclient(hunting): authenticate will time out at 1214.912635s 2024-09-16T04:46:56.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d58ee640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d006e160 0x7f13d00a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d58ee640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d006e160 0x7f13d00a29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:35812/0 (socket says 172.21.15.46:35812) 2024-09-16T04:46:56.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d58ee640 1 -- 172.21.15.46:0/2410287424 learned_addr learned my addr 172.21.15.46:0/2410287424 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:56.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d58ee640 10 monclient(hunting): get_auth_request con 0x7f13d006e160 auth_method 0 2024-09-16T04:46:56.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d58ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d58ee640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d58ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d58ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d58ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d58ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d50ed640 1 --2- 172.21.15.46:0/2410287424 >> v2:172.21.15.114:3300/0 conn(0x7f13d00a2ee0 0x7f13d0136940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d58ee640 10 monclient(hunting): handle_auth_done global_id 94361 payload 274 2024-09-16T04:46:56.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d58ee640 1 -- 172.21.15.46:0/2410287424 >> v2:172.21.15.114:3300/0 conn(0x7f13d00a2ee0 msgr2=0x7f13d0136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:56.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.497+0000 7f13d58ee640 1 --2- 172.21.15.46:0/2410287424 >> v2:172.21.15.114:3300/0 conn(0x7f13d00a2ee0 0x7f13d0136940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d58ee640 1 -- 172.21.15.46:0/2410287424 >> v1:172.21.15.114:6789/0 conn(0x7f13d0136e80 legacy=0x7f13d013dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:56.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d58ee640 10 monclient: _finish_hunting 0 2024-09-16T04:46:56.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d58ee640 1 monclient: found mon.noname-b 2024-09-16T04:46:56.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d58ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:56.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d58ee640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:46:56.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d58ee640 1 -- 172.21.15.46:0/2410287424 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d00a0bd0 con 0x7f13d006e160 2024-09-16T04:46:56.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d50ed640 10 monclient: get_auth_request con 0x7f13d00a2ee0 auth_method 0 2024-09-16T04:46:56.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d50ed640 1 --2- 172.21.15.46:0/2410287424 >> v2:172.21.15.114:3300/0 conn(0x7f13d00a2ee0 0x7f13d0136940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:46:56.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d58ee640 1 --2- 172.21.15.46:0/2410287424 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d006e160 0x7f13d00a29a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f13cc009870 tx=0x7f13cc02a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94b25484bccee184 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:56.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d48ec640 1 -- 172.21.15.46:0/2410287424 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc037070 con 0x7f13d006e160 2024-09-16T04:46:56.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d48ec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d48ec640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:46:56.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d48ec640 10 monclient: dump: 2024-09-16T04:46:56.514 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.514 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.514 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.515 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.515 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.515 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.515 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.515 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.515 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.515 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d48ec640 10 monclient: _finish_auth 0 2024-09-16T04:46:56.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d48ec640 10 monclient: _check_auth_tickets 2024-09-16T04:46:56.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d48ec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:56.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d7b52640 5 monclient: authenticate success, global_id 94361 2024-09-16T04:46:56.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d48ec640 1 -- 172.21.15.46:0/2410287424 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13cc02ae10 con 0x7f13d006e160 2024-09-16T04:46:56.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d48ec640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:56.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d7b52640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:56.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d48ec640 1 -- 172.21.15.46:0/2410287424 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc032660 con 0x7f13d006e160 2024-09-16T04:46:56.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d7b52640 4 set_mon_vals no callback set 2024-09-16T04:46:56.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d48ec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d48ec640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:56.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.498+0000 7f13d48ec640 10 monclient: dump: 2024-09-16T04:46:56.516 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.516 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.516 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.516 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.516 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.517 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.517 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.517 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.517 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.517 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.499+0000 7f13d7b52640 10 monclient: shutdown 2024-09-16T04:46:56.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.499+0000 7f13d7b52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:56.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.499+0000 7f13d7b52640 1 -- 172.21.15.46:0/2410287424 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d006e160 msgr2=0x7f13d00a29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.499+0000 7f13d7b52640 1 --2- 172.21.15.46:0/2410287424 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d006e160 0x7f13d00a29a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f13cc009870 tx=0x7f13cc02a100 comp rx=0 tx=0).stop 2024-09-16T04:46:56.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.499+0000 7f13d7b52640 1 -- 172.21.15.46:0/2410287424 shutdown_connections 2024-09-16T04:46:56.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.499+0000 7f13d7b52640 1 --2- 172.21.15.46:0/2410287424 >> v2:172.21.15.114:3300/0 conn(0x7f13d00a2ee0 0x7f13d0136940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.499+0000 7f13d7b52640 1 --2- 172.21.15.46:0/2410287424 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d006e160 0x7f13d00a29a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.499+0000 7f13d7b52640 1 -- 172.21.15.46:0/2410287424 >> 172.21.15.46:0/2410287424 conn(0x7f13d00a5c10 msgr2=0x7f13d00a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:56.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.499+0000 7f13d7b52640 1 -- 172.21.15.46:0/2410287424 shutdown_connections 2024-09-16T04:46:56.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.499+0000 7f13d7b52640 1 -- 172.21.15.46:0/2410287424 wait complete. 2024-09-16T04:46:56.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.500+0000 7f13d7b52640 10 monclient: build_initial_monmap 2024-09-16T04:46:56.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.500+0000 7f13d7b52640 10 monclient: monmap: 2024-09-16T04:46:56.518 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:56.518 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:56.518 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:56.518 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:56.518 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:56.519 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.519 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:56.519 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:56.519 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:56.519 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.500+0000 7f13d7b52640 1 Processor -- start 2024-09-16T04:46:56.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.500+0000 7f13d7b52640 1 -- start start 2024-09-16T04:46:56.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.500+0000 7f13d7b52640 10 monclient: init 2024-09-16T04:46:56.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.500+0000 7f13d7b52640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:56.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:46:56.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13d006e160 0x7f13d01cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 10 monclient(hunting): picked mon.noname-b con 0x7f13d006e160 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:56.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d00a2ee0 0x7f13d0135de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 10 monclient(hunting): picked mon.noname-c con 0x7f13d00a2ee0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:56.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13d0136e80 0x7f13d0136320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 10 monclient(hunting): picked mon.noname-a con 0x7f13d0136e80 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:56.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f13d00a34a0 con 0x7f13d0136e80 2024-09-16T04:46:56.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f13d00a5130 con 0x7f13d006e160 2024-09-16T04:46:56.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f13d01237a0 con 0x7f13d00a2ee0 2024-09-16T04:46:56.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:56.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 10 monclient(hunting): authenticate will time out at 1214.916100s 2024-09-16T04:46:56.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d50ed640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d00a2ee0 0x7f13d0135de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d50ed640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d00a2ee0 0x7f13d0135de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:35814/0 (socket says 172.21.15.46:35814) 2024-09-16T04:46:56.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d50ed640 1 -- 172.21.15.46:0/2277459516 learned_addr learned my addr 172.21.15.46:0/2277459516 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:56.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d50ed640 10 monclient(hunting): get_auth_request con 0x7f13d00a2ee0 auth_method 0 2024-09-16T04:46:56.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d50ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d50ed640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d50ed640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d60ef640 1 --2- 172.21.15.46:0/2277459516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13d0136e80 0x7f13d0136320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d50ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d50ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d50ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d58ee640 1 --2- 172.21.15.46:0/2277459516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13d006e160 0x7f13d01cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d60ef640 10 monclient(hunting): get_auth_request con 0x7f13d0136e80 auth_method 0 2024-09-16T04:46:56.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d60ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d60ef640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d60ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d58ee640 10 monclient(hunting): get_auth_request con 0x7f13d006e160 auth_method 0 2024-09-16T04:46:56.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d58ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d58ee640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d58ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d60ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d60ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d60ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d58ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d58ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d58ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d50ed640 10 monclient(hunting): handle_auth_done global_id 94364 payload 824 2024-09-16T04:46:56.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d50ed640 1 -- 172.21.15.46:0/2277459516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13d006e160 msgr2=0x7f13d01cb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d50ed640 1 --2- 172.21.15.46:0/2277459516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13d006e160 0x7f13d01cb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d50ed640 1 -- 172.21.15.46:0/2277459516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13d0136e80 msgr2=0x7f13d0136320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d50ed640 1 --2- 172.21.15.46:0/2277459516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13d0136e80 0x7f13d0136320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d50ed640 10 monclient: _finish_hunting 0 2024-09-16T04:46:56.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d50ed640 1 monclient: found mon.noname-c 2024-09-16T04:46:56.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d50ed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:56.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d50ed640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:46:56.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d50ed640 1 -- 172.21.15.46:0/2277459516 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d00a0300 con 0x7f13d00a2ee0 2024-09-16T04:46:56.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d50ed640 1 --2- 172.21.15.46:0/2277459516 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d00a2ee0 0x7f13d0135de0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f13c000c7f0 tx=0x7f13c000ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:56.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13c67fc640 1 -- 172.21.15.46:0/2277459516 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c0004230 con 0x7f13d00a2ee0 2024-09-16T04:46:56.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13c67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:56.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13c67fc640 10 monclient: dump: 2024-09-16T04:46:56.525 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.525 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.525 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.525 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.525 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.525 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.525 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.525 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.525 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.525 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13c67fc640 10 monclient: _finish_auth 0 2024-09-16T04:46:56.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13c67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:46:56.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:56.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13c67fc640 1 -- 172.21.15.46:0/2277459516 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13c00043d0 con 0x7f13d00a2ee0 2024-09-16T04:46:56.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 5 monclient: authenticate success, global_id 94364 2024-09-16T04:46:56.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 10 monclient: _renew_subs 2024-09-16T04:46:56.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:56.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 1 -- 172.21.15.46:0/2277459516 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d01231e0 con 0x7f13d00a2ee0 2024-09-16T04:46:56.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13c67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:56.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13c67fc640 1 -- 172.21.15.46:0/2277459516 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c0007a70 con 0x7f13d00a2ee0 2024-09-16T04:46:56.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:56.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13c67fc640 10 monclient: dump: 2024-09-16T04:46:56.527 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.527 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.527 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.527 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.527 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.527 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.527 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.528 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.528 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.528 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13c7fff640 4 set_mon_vals no callback set 2024-09-16T04:46:56.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 10 monclient: _renew_subs 2024-09-16T04:46:56.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:56.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.501+0000 7f13d7b52640 1 -- 172.21.15.46:0/2277459516 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d0121d90 con 0x7f13d00a2ee0 2024-09-16T04:46:56.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.502+0000 7f13d7b52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:56.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.502+0000 7f13d7b52640 1 --2- 172.21.15.46:0/2277459516 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d011f9a0 0x7f13d011fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.502+0000 7f13d7b52640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.502+0000 7f13d7b52640 1 -- 172.21.15.46:0/2277459516 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f13d00a3620 con 0x7f13d011f9a0 2024-09-16T04:46:56.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.502+0000 7f13c67fc640 1 -- 172.21.15.46:0/2277459516 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f13c0007c10 con 0x7f13d00a2ee0 2024-09-16T04:46:56.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.502+0000 7f13d58ee640 1 --2- 172.21.15.46:0/2277459516 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d011f9a0 0x7f13d011fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.502+0000 7f13d58ee640 10 monclient: get_auth_request con 0x7f13d011f9a0 auth_method 0 2024-09-16T04:46:56.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.502+0000 7f13d58ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.502+0000 7f13d58ee640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.502+0000 7f13d58ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.502+0000 7f13d58ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.502+0000 7f13d58ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.502+0000 7f13d58ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.502+0000 7f13c67fc640 1 --2- 172.21.15.46:0/2277459516 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f13b40383b0 0x7f13b403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.502+0000 7f13c67fc640 1 -- 172.21.15.46:0/2277459516 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(314..314 src has 1..314) ==== 4738+0+0 (secure 0 0 0) 0x7f13c004b800 con 0x7f13d00a2ee0 2024-09-16T04:46:56.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.502+0000 7f13d58ee640 10 monclient(hunting): handle_auth_done global_id 94367 payload 824 2024-09-16T04:46:56.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.503+0000 7f13d58ee640 1 -- 172.21.15.46:0/2277459516 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f13d00a29a0 con 0x7f13d011f9a0 2024-09-16T04:46:56.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.503+0000 7f13d60ef640 1 --2- 172.21.15.46:0/2277459516 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f13b40383b0 0x7f13b403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.503+0000 7f13d60ef640 10 monclient: get_auth_request con 0x7f13b40383b0 auth_method 0 2024-09-16T04:46:56.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.503+0000 7f13d58ee640 1 --2- 172.21.15.46:0/2277459516 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d011f9a0 0x7f13d011fd80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f13cc034000 tx=0x7f13cc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:56.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.503+0000 7f13c67fc640 1 -- 172.21.15.46:0/2277459516 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc037040 con 0x7f13d011f9a0 2024-09-16T04:46:56.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.503+0000 7f13c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.503+0000 7f13c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:56.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.503+0000 7f13c67fc640 10 monclient: dump: 2024-09-16T04:46:56.530 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.530 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.530 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.531 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.531 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.531 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.531 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.531 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.531 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.531 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.503+0000 7f13d60ef640 1 --2- 172.21.15.46:0/2277459516 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f13b40383b0 0x7f13b403a870 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f13d0122e30 tx=0x7f13c8003d10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:56.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.503+0000 7f13c67fc640 1 -- 172.21.15.46:0/2277459516 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f13cc032e40 con 0x7f13d011f9a0 2024-09-16T04:46:56.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.503+0000 7f13c67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:56.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.503+0000 7f13c67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:56.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.503+0000 7f13c67fc640 1 -- 172.21.15.46:0/2277459516 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d011f9a0 msgr2=0x7f13d011fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.503+0000 7f13c67fc640 1 --2- 172.21.15.46:0/2277459516 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d011f9a0 0x7f13d011fd80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f13cc034000 tx=0x7f13cc0046c0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.503+0000 7f13c67fc640 1 -- 172.21.15.46:0/2277459516 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d011f9a0 msgr2=0x7f13d011fd80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:56.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.503+0000 7f13c67fc640 1 --2- 172.21.15.46:0/2277459516 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d011f9a0 0x7f13d011fd80 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f13cc034000 tx=0x7f13cc0046c0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.509+0000 7f13d7b52640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:46:56.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.509+0000 7f13d7b52640 1 --2- 172.21.15.46:0/2277459516 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d01202c0 0x7f13d009b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.509+0000 7f13d7b52640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.509+0000 7f13d7b52640 1 -- 172.21.15.46:0/2277459516 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f13d01cbef0 con 0x7f13d01202c0 2024-09-16T04:46:56.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13d58ee640 1 --2- 172.21.15.46:0/2277459516 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d01202c0 0x7f13d009b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13d58ee640 10 monclient: get_auth_request con 0x7f13d01202c0 auth_method 0 2024-09-16T04:46:56.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13d58ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13d58ee640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13d58ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13d58ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13d58ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13d58ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13d58ee640 10 monclient(hunting): handle_auth_done global_id 94370 payload 824 2024-09-16T04:46:56.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13d58ee640 1 -- 172.21.15.46:0/2277459516 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f13d0067e00 con 0x7f13d01202c0 2024-09-16T04:46:56.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13d58ee640 1 --2- 172.21.15.46:0/2277459516 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d01202c0 0x7f13d009b570 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f13cc000c00 tx=0x7f13cc008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:56.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13c67fc640 1 -- 172.21.15.46:0/2277459516 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc037040 con 0x7f13d01202c0 2024-09-16T04:46:56.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:56.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13c67fc640 10 monclient: dump: 2024-09-16T04:46:56.537 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.537 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.537 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.538 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13c67fc640 1 -- 172.21.15.46:0/2277459516 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f13cc04c820 con 0x7f13d01202c0 2024-09-16T04:46:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13c67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:46:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13c67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13c67fc640 1 -- 172.21.15.46:0/2277459516 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d01202c0 msgr2=0x7f13d009b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13c67fc640 1 --2- 172.21.15.46:0/2277459516 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d01202c0 0x7f13d009b570 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f13cc000c00 tx=0x7f13cc008030 comp rx=0 tx=0).stop 2024-09-16T04:46:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13c67fc640 1 -- 172.21.15.46:0/2277459516 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d01202c0 msgr2=0x7f13d009b570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:56.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.510+0000 7f13c67fc640 1 --2- 172.21.15.46:0/2277459516 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d01202c0 0x7f13d009b570 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.539 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:46:56.539 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:46:56.539 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:46:56.539 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:46:56.539 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 94, 2024-09-16T04:46:56.539 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 57765, 2024-09-16T04:46:56.539 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:46:56.539 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:46:56.539 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:46:56.539 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:46:56.539 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:56.540 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 3, 2024-09-16T04:46:56.540 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:56.540 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:46:56.540 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:46:56.540 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:56.540 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:56.540 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:56.540 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:56.540 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:56.540 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:56.540 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:56.540 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:56.540 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:56.540 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:56.541 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:56.541 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:56.541 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:46:56.541 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:46:56.541 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:56.541 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:56.541 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:56.541 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:56.541 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:56.541 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:56.541 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:56.541 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:56.541 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:56.542 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:56.542 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:56.542 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:56.542 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.542 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:46:56.542 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:46:56.542 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:46:56.542 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:46:56.542 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:46:56.542 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:46:56.542 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:56.542 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:56.542 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:46:56.543 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:46:56.543 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:46:56.543 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:46:56.543 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:46:56.543 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:46:56.543 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:46:56.543 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:56.543 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:46:56.543 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:56.543 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:56.543 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:56.543 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:56.543 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:56.543 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:56.544 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:56.544 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:56.544 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:56.544 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:56.544 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:56.544 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:56.544 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:46:56.544 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.544 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:46:56.544 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.544 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:46:56.544 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:46:56.544 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:56.545 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:56.545 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.545 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:56.545 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:46:56.545 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:56.545 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.545 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.545 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:56.545 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:46:56.545 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:56.545 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:56.545 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:56.545 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.546 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:46:56.546 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:46:56.546 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:56.546 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:56.546 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:56.546 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.546 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.546 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:46:56.546 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:46:56.546 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:56.546 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:56.546 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.546 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:56.546 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:46:56.547 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:56.547 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.547 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.547 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:56.547 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:46:56.547 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:56.547 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:56.547 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:56.547 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.547 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:46:56.547 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:46:56.547 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:56.547 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:56.547 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:56.548 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.548 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.548 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:46:56.548 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:46:56.548 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:56.548 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:56.548 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.548 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:56.548 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:46:56.548 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:56.548 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.548 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.548 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:56.549 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:46:56.549 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:56.549 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:56.549 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:56.549 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.549 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:46:56.549 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:46:56.549 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:56.549 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:56.549 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:56.549 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:56.549 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:56.549 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.549 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:46:56.550 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:46:56.550 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.550 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:56.550 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:56.550 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:56.550 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:56.550 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:56.550 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:46:56.550 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.550 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:56.550 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:56.550 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:56.550 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:56.550 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:56.551 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:46:56.551 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.551 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:56.551 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:56.551 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 2 2024-09-16T04:46:56.551 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:56.551 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:56.551 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.551 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:46:56.551 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:46:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.511+0000 7f13d7b52640 1 -- 172.21.15.46:0/2277459516 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f13b40383b0 msgr2=0x7f13b403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.511+0000 7f13d7b52640 1 --2- 172.21.15.46:0/2277459516 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f13b40383b0 0x7f13b403a870 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f13d0122e30 tx=0x7f13c8003d10 comp rx=0 tx=0).stop 2024-09-16T04:46:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.511+0000 7f13d7b52640 10 monclient: shutdown 2024-09-16T04:46:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.511+0000 7f13d7b52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.511+0000 7f13d7b52640 1 -- 172.21.15.46:0/2277459516 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d00a2ee0 msgr2=0x7f13d0135de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.511+0000 7f13d7b52640 1 --2- 172.21.15.46:0/2277459516 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f13d00a2ee0 0x7f13d0135de0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f13c000c7f0 tx=0x7f13c000ccb0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.511+0000 7f13d60ef640 1 -- 172.21.15.46:0/2277459516 reap_dead start 2024-09-16T04:46:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.511+0000 7f13d60ef640 1 -- 172.21.15.46:0/2277459516 reap_dead start 2024-09-16T04:46:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.511+0000 7f13d7b52640 1 -- 172.21.15.46:0/2277459516 shutdown_connections 2024-09-16T04:46:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.511+0000 7f13d7b52640 1 -- 172.21.15.46:0/2277459516 >> 172.21.15.46:0/2277459516 conn(0x7f13d00a5c10 msgr2=0x7f13d009efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.511+0000 7f13d7b52640 1 -- 172.21.15.46:0/2277459516 shutdown_connections 2024-09-16T04:46:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.511+0000 7f13d7b52640 1 -- 172.21.15.46:0/2277459516 wait complete. 2024-09-16T04:46:56.554 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:46:56.554 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:46:56.554 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:46:56.556 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:46:56.557 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40390]: starting. 2024-09-16T04:46:56.557 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40390]: listing objects. 2024-09-16T04:46:56.557 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40390]: listed object 0... 2024-09-16T04:46:56.557 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40390]: listed object 25... 2024-09-16T04:46:56.557 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40390]: saw 50 objects 2024-09-16T04:46:56.557 INFO:teuthology.orchestra.run.smithi114.stdout:process_5_[40390]: shutting down. 2024-09-16T04:46:56.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.655+0000 7fcc88696640 10 monclient: get_monmap_and_config 2024-09-16T04:46:56.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.655+0000 7fcc88696640 10 monclient: build_initial_monmap 2024-09-16T04:46:56.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.655+0000 7fcc88696640 10 monclient: monmap: 2024-09-16T04:46:56.656 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:56.656 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.656 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:56.655537+0000 2024-09-16T04:46:56.656 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:56.655537+0000 2024-09-16T04:46:56.656 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:56.656 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.657 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:56.657 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:56.657 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:56.657 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:56.657 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:56.657 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.655+0000 7fcc88696640 1 Processor -- start 2024-09-16T04:46:56.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.655+0000 7fcc88696640 1 -- start start 2024-09-16T04:46:56.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.655+0000 7fcc88696640 10 monclient: init 2024-09-16T04:46:56.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc88696640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:56.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc88696640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-16T04:46:56.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc88696640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcc800a3c10 0x7fcc8009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc88696640 10 monclient(hunting): picked mon.noname-b con 0x7fcc800a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:56.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc88696640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcc800a3ff0 0x7fcc800a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc88696640 10 monclient(hunting): picked mon.noname-a con 0x7fcc800a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:46:56.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc88696640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc801369d0 addr v1:172.21.15.114:6789/0 2024-09-16T04:46:56.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc88696640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc8009b940 con 0x7fcc801369d0 2024-09-16T04:46:56.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc88696640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc88696640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fcc8009ab00 con 0x7fcc800a3ff0 2024-09-16T04:46:56.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc88696640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc88696640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fcc8009b030 con 0x7fcc800a3c10 2024-09-16T04:46:56.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc88696640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:56.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc88696640 10 monclient(hunting): authenticate will time out at 1215.071341s 2024-09-16T04:46:56.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc85c31640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcc800a3ff0 0x7fcc800a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc86432640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcc800a3c10 0x7fcc8009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc85c31640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcc800a3ff0 0x7fcc800a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35196/0 (socket says 172.21.15.46:35196) 2024-09-16T04:46:56.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc85c31640 1 -- 172.21.15.46:0/3123148529 learned_addr learned my addr 172.21.15.46:0/3123148529 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:56.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc85c31640 10 monclient(hunting): get_auth_request con 0x7fcc800a3ff0 auth_method 0 2024-09-16T04:46:56.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc85c31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc85c31640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc85c31640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc86432640 10 monclient(hunting): get_auth_request con 0x7fcc800a3c10 auth_method 0 2024-09-16T04:46:56.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc86432640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc86432640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc86432640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc85c31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc85c31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc85c31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc86432640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc86432640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc86432640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc85c31640 10 monclient(hunting): handle_auth_done global_id 104181 payload 274 2024-09-16T04:46:56.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc85c31640 1 -- 172.21.15.46:0/3123148529 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcc800a3c10 msgr2=0x7fcc8009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:46:56.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc85c31640 1 --2- 172.21.15.46:0/3123148529 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcc800a3c10 0x7fcc8009bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc85c31640 1 -- 172.21.15.46:0/3123148529 >> v1:172.21.15.114:6789/0 conn(0x7fcc801369d0 legacy=0x7fcc80136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc85c31640 10 monclient: _finish_hunting 0 2024-09-16T04:46:56.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc85c31640 1 monclient: found mon.noname-a 2024-09-16T04:46:56.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc85c31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:56.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc85c31640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:46:56.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.656+0000 7fcc85c31640 1 -- 172.21.15.46:0/3123148529 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc8009b310 con 0x7fcc800a3ff0 2024-09-16T04:46:56.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.657+0000 7fcc85c31640 1 --2- 172.21.15.46:0/3123148529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc800a3ff0 0x7fcc800a43d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcc74009870 tx=0x7fcc7402a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=988f7fc007dc9588 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:56.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.657+0000 7fcc85430640 1 -- 172.21.15.46:0/3123148529 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc74037070 con 0x7fcc800a3ff0 2024-09-16T04:46:56.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.657+0000 7fcc85430640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.657+0000 7fcc85430640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:46:56.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.657+0000 7fcc85430640 10 monclient: dump: 2024-09-16T04:46:56.669 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.669 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.669 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.669 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.670 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.670 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.670 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.670 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.670 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.670 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.657+0000 7fcc85430640 10 monclient: _finish_auth 0 2024-09-16T04:46:56.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.657+0000 7fcc85430640 10 monclient: _check_auth_tickets 2024-09-16T04:46:56.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.657+0000 7fcc85430640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:56.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.657+0000 7fcc88696640 5 monclient: authenticate success, global_id 104181 2024-09-16T04:46:56.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.657+0000 7fcc88696640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:56.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.657+0000 7fcc85430640 1 -- 172.21.15.46:0/3123148529 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc74032440 con 0x7fcc800a3ff0 2024-09-16T04:46:56.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.657+0000 7fcc85430640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:56.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.657+0000 7fcc88696640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:56.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.657+0000 7fcc88696640 4 set_mon_vals no callback set 2024-09-16T04:46:56.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.657+0000 7fcc85430640 1 -- 172.21.15.46:0/3123148529 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc74032810 con 0x7fcc800a3ff0 2024-09-16T04:46:56.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.657+0000 7fcc85430640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.657+0000 7fcc85430640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:56.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.657+0000 7fcc85430640 10 monclient: dump: 2024-09-16T04:46:56.671 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.671 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.671 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.671 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.671 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.671 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.671 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.671 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.671 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.672 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.658+0000 7fcc88696640 10 monclient: shutdown 2024-09-16T04:46:56.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.658+0000 7fcc88696640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:56.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.658+0000 7fcc88696640 1 -- 172.21.15.46:0/3123148529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc800a3ff0 msgr2=0x7fcc800a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.658+0000 7fcc88696640 1 --2- 172.21.15.46:0/3123148529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc800a3ff0 0x7fcc800a43d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcc74009870 tx=0x7fcc7402a130 comp rx=0 tx=0).stop 2024-09-16T04:46:56.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.658+0000 7fcc88696640 1 -- 172.21.15.46:0/3123148529 shutdown_connections 2024-09-16T04:46:56.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.658+0000 7fcc88696640 1 --2- 172.21.15.46:0/3123148529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc800a3ff0 0x7fcc800a43d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.658+0000 7fcc88696640 1 --2- 172.21.15.46:0/3123148529 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcc800a3c10 0x7fcc8009bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.658+0000 7fcc88696640 1 -- 172.21.15.46:0/3123148529 >> 172.21.15.46:0/3123148529 conn(0x7fcc800a6340 msgr2=0x7fcc8006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:56.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.658+0000 7fcc88696640 1 -- 172.21.15.46:0/3123148529 shutdown_connections 2024-09-16T04:46:56.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.658+0000 7fcc88696640 1 -- 172.21.15.46:0/3123148529 wait complete. 2024-09-16T04:46:56.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.658+0000 7fcc88696640 10 monclient: build_initial_monmap 2024-09-16T04:46:56.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.658+0000 7fcc88696640 10 monclient: monmap: 2024-09-16T04:46:56.672 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:56.673 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:56.673 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:56.673 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:56.673 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:56.673 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.673 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:56.673 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:56.673 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:56.673 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc88696640 1 Processor -- start 2024-09-16T04:46:56.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc88696640 1 -- start start 2024-09-16T04:46:56.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc88696640 10 monclient: init 2024-09-16T04:46:56.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc88696640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:56.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc88696640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:46:56.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc88696640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc800a3c10 0x7fcc801cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc88696640 10 monclient(hunting): picked mon.noname-b con 0x7fcc800a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:56.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc88696640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcc800a3ff0 0x7fcc801cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc88696640 10 monclient(hunting): picked mon.noname-c con 0x7fcc800a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:56.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc88696640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc801369d0 0x7fcc80123230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc88696640 10 monclient(hunting): picked mon.noname-a con 0x7fcc801369d0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:56.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc88696640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc88696640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcc800a4910 con 0x7fcc801369d0 2024-09-16T04:46:56.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc88696640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc88696640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcc800a69c0 con 0x7fcc800a3c10 2024-09-16T04:46:56.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc88696640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc88696640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fcc80126610 con 0x7fcc800a3ff0 2024-09-16T04:46:56.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc88696640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:56.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc88696640 10 monclient(hunting): authenticate will time out at 1215.074680s 2024-09-16T04:46:56.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc85c31640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcc800a3ff0 0x7fcc801cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc86c33640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc801369d0 0x7fcc80123230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc85c31640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcc800a3ff0 0x7fcc801cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:35860/0 (socket says 172.21.15.46:35860) 2024-09-16T04:46:56.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc85c31640 1 -- 172.21.15.46:0/1786065391 learned_addr learned my addr 172.21.15.46:0/1786065391 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:56.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc85c31640 10 monclient(hunting): get_auth_request con 0x7fcc800a3ff0 auth_method 0 2024-09-16T04:46:56.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc85c31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc85c31640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc85c31640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc86432640 1 -- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc800a3c10 msgr2=0x7fcc801cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:46:56.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc86c33640 10 monclient(hunting): get_auth_request con 0x7fcc801369d0 auth_method 0 2024-09-16T04:46:56.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc86c33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc86432640 1 -- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc800a3c10 msgr2=0x7fcc801cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:46:56.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc86c33640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc86c33640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc86432640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc800a3c10 0x7fcc801cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-09-16T04:46:56.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc86432640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc800a3c10 0x7fcc801cbee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:56.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc85c31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc85c31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc85c31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc86c33640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc86c33640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.659+0000 7fcc86c33640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc86c33640 10 monclient(hunting): handle_auth_done global_id 104184 payload 824 2024-09-16T04:46:56.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc86c33640 1 -- 172.21.15.46:0/1786065391 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcc800a3ff0 msgr2=0x7fcc801cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc86c33640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcc800a3ff0 0x7fcc801cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc86c33640 1 -- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc800a3c10 msgr2=0x7fcc801cbee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:46:56.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc86c33640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc800a3c10 0x7fcc801cbee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc86c33640 10 monclient: _finish_hunting 0 2024-09-16T04:46:56.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc86c33640 1 monclient: found mon.noname-a 2024-09-16T04:46:56.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc86c33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:56.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc86c33640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:46:56.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc86c33640 1 -- 172.21.15.46:0/1786065391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc80126ee0 con 0x7fcc801369d0 2024-09-16T04:46:56.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc85c31640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcc800a3ff0 0x7fcc801cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:56.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc86c33640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc801369d0 0x7fcc80123230 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000c700 tx=0x7fcc7000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:56.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc7effd640 1 -- 172.21.15.46:0/1786065391 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc70003b70 con 0x7fcc801369d0 2024-09-16T04:46:56.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc7effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:46:56.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc7effd640 10 monclient: dump: 2024-09-16T04:46:56.677 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.677 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.677 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.677 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.678 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.678 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.678 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.678 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.678 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.678 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc7effd640 10 monclient: _finish_auth 0 2024-09-16T04:46:56.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc7effd640 10 monclient: _check_auth_tickets 2024-09-16T04:46:56.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:56.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc7effd640 1 -- 172.21.15.46:0/1786065391 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc70003d10 con 0x7fcc801369d0 2024-09-16T04:46:56.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc88696640 5 monclient: authenticate success, global_id 104184 2024-09-16T04:46:56.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc88696640 10 monclient: _renew_subs 2024-09-16T04:46:56.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc88696640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:56.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc88696640 1 -- 172.21.15.46:0/1786065391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc80125a30 con 0x7fcc801369d0 2024-09-16T04:46:56.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc7effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:56.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc7effd640 1 -- 172.21.15.46:0/1786065391 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc70005310 con 0x7fcc801369d0 2024-09-16T04:46:56.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:46:56.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc7effd640 10 monclient: dump: 2024-09-16T04:46:56.679 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.679 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.679 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.679 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.679 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.679 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.679 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.679 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.679 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.680 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc88696640 10 monclient: _renew_subs 2024-09-16T04:46:56.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc88696640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:46:56.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc88696640 1 -- 172.21.15.46:0/1786065391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc800a8900 con 0x7fcc801369d0 2024-09-16T04:46:56.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.660+0000 7fcc84c2f640 4 set_mon_vals no callback set 2024-09-16T04:46:56.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.661+0000 7fcc88696640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:56.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.661+0000 7fcc88696640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc80126af0 0x7fcc80123770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.661+0000 7fcc88696640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.661+0000 7fcc88696640 1 -- 172.21.15.46:0/1786065391 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcc800a4a90 con 0x7fcc80126af0 2024-09-16T04:46:56.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.661+0000 7fcc7effd640 1 -- 172.21.15.46:0/1786065391 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fcc700054b0 con 0x7fcc801369d0 2024-09-16T04:46:56.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.661+0000 7fcc86432640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc80126af0 0x7fcc80123770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.661+0000 7fcc7effd640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcc6c0383b0 0x7fcc6c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.661+0000 7fcc86432640 10 monclient: get_auth_request con 0x7fcc80126af0 auth_method 0 2024-09-16T04:46:56.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.661+0000 7fcc86432640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.661+0000 7fcc86432640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.661+0000 7fcc86432640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.661+0000 7fcc7effd640 1 -- 172.21.15.46:0/1786065391 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(314..314 src has 1..314) ==== 4738+0+0 (secure 0 0 0) 0x7fcc7004b460 con 0x7fcc801369d0 2024-09-16T04:46:56.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.661+0000 7fcc86432640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.661+0000 7fcc86432640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.661+0000 7fcc86432640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.661+0000 7fcc85c31640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcc6c0383b0 0x7fcc6c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.661+0000 7fcc85c31640 10 monclient: get_auth_request con 0x7fcc6c0383b0 auth_method 0 2024-09-16T04:46:56.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.661+0000 7fcc86432640 10 monclient(hunting): handle_auth_done global_id 7021 payload 824 2024-09-16T04:46:56.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.661+0000 7fcc86432640 1 -- 172.21.15.46:0/1786065391 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcc8009bcf0 con 0x7fcc80126af0 2024-09-16T04:46:56.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.662+0000 7fcc86432640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc80126af0 0x7fcc80123770 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800e810 tx=0x7fcc7800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:56.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.662+0000 7fcc7effd640 1 -- 172.21.15.46:0/1786065391 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc78017070 con 0x7fcc80126af0 2024-09-16T04:46:56.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.662+0000 7fcc7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.662+0000 7fcc85c31640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcc6c0383b0 0x7fcc6c03a870 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fcc740090b0 tx=0x7fcc7402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:56.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.662+0000 7fcc7effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:46:56.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.662+0000 7fcc7effd640 10 monclient: dump: 2024-09-16T04:46:56.682 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.682 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.682 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.682 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.682 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.682 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.682 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.682 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.682 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.682 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.662+0000 7fcc7effd640 1 -- 172.21.15.46:0/1786065391 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcc7800c930 con 0x7fcc80126af0 2024-09-16T04:46:56.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.662+0000 7fcc7effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:56.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.662+0000 7fcc7effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:56.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.662+0000 7fcc7effd640 1 -- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc80126af0 msgr2=0x7fcc80123770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.662+0000 7fcc7effd640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc80126af0 0x7fcc80123770 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800e810 tx=0x7fcc7800ecd0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.662+0000 7fcc7effd640 1 -- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc80126af0 msgr2=0x7fcc80123770 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:56.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.662+0000 7fcc7effd640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc80126af0 0x7fcc80123770 secure :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800e810 tx=0x7fcc7800ecd0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.668+0000 7fcc88696640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:46:56.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.668+0000 7fcc88696640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc80123cb0 0x7fcc801240c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.668+0000 7fcc88696640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.668+0000 7fcc88696640 1 -- 172.21.15.46:0/1786065391 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcc80124600 con 0x7fcc80123cb0 2024-09-16T04:46:56.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.668+0000 7fcc86432640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc80123cb0 0x7fcc801240c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.668+0000 7fcc86432640 10 monclient: get_auth_request con 0x7fcc80123cb0 auth_method 0 2024-09-16T04:46:56.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.668+0000 7fcc86432640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.668+0000 7fcc86432640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.668+0000 7fcc86432640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.668+0000 7fcc86432640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.668+0000 7fcc86432640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.668+0000 7fcc86432640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.669+0000 7fcc86432640 10 monclient(hunting): handle_auth_done global_id 7024 payload 824 2024-09-16T04:46:56.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.669+0000 7fcc86432640 1 -- 172.21.15.46:0/1786065391 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcc80067e20 con 0x7fcc80123cb0 2024-09-16T04:46:56.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.669+0000 7fcc86432640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc80123cb0 0x7fcc801240c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800e810 tx=0x7fcc780046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:56.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.669+0000 7fcc7effd640 1 -- 172.21.15.46:0/1786065391 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc78017030 con 0x7fcc80123cb0 2024-09-16T04:46:56.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.669+0000 7fcc7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.669+0000 7fcc7effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:46:56.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.669+0000 7fcc7effd640 10 monclient: dump: 2024-09-16T04:46:56.686 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.686 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.686 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.686 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.686 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.686 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.686 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.686 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.687 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.687 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.669+0000 7fcc7effd640 1 -- 172.21.15.46:0/1786065391 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7fcc78005c60 con 0x7fcc80123cb0 2024-09-16T04:46:56.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.669+0000 7fcc7effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:46:56.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.669+0000 7fcc7effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:56.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.669+0000 7fcc7effd640 1 -- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc80123cb0 msgr2=0x7fcc801240c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.669+0000 7fcc7effd640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc80123cb0 0x7fcc801240c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800e810 tx=0x7fcc780046c0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.669+0000 7fcc7effd640 1 -- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc80123cb0 msgr2=0x7fcc801240c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:46:56.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.669+0000 7fcc7effd640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc80123cb0 0x7fcc801240c0 secure :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800e810 tx=0x7fcc780046c0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.687 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:46:56.688 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:46:56.688 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:46:56.688 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:46:56.688 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 94, 2024-09-16T04:46:56.688 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 502571, 2024-09-16T04:46:56.688 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:46:56.688 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:46:56.688 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:46:56.689 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:46:56.689 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:56.689 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 3, 2024-09-16T04:46:56.689 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:56.689 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:46:56.689 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:46:56.689 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:56.689 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:56.689 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:56.689 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:56.689 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:56.689 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:56.689 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:56.690 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:56.690 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:56.690 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:56.690 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:56.690 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:56.690 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:46:56.690 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:46:56.690 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:56.690 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:56.690 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:56.690 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:56.690 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:56.690 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:56.690 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:56.690 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:56.691 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:56.691 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:56.691 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:56.691 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:56.691 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.691 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:46:56.691 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:46:56.691 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:46:56.691 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:46:56.691 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:46:56.691 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:46:56.691 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:56.691 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:46:56.691 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:46:56.691 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:46:56.692 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:46:56.692 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:46:56.692 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:46:56.692 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:46:56.692 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:46:56.692 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:46:56.692 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:46:56.692 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:46:56.692 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:46:56.692 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:46:56.692 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:46:56.692 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:46:56.692 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:46:56.692 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:46:56.692 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:46:56.693 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:46:56.693 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:46:56.693 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:46:56.693 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:56.693 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:46:56.693 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.693 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:46:56.693 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.693 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:46:56.693 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:46:56.693 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:56.693 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:56.693 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.694 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:56.694 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:46:56.694 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:56.694 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.694 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.694 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:56.694 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:46:56.694 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:56.694 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:56.694 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:56.694 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.694 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:46:56.694 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:46:56.694 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:56.695 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:56.695 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:56.695 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.695 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.695 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:46:56.695 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:46:56.695 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:56.695 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:56.695 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.695 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:56.695 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:46:56.695 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:56.695 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.695 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.695 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:56.696 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:46:56.696 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:56.696 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:56.696 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:56.696 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.696 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:46:56.696 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:46:56.696 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:56.696 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:56.696 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:56.696 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.696 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.696 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:46:56.696 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:46:56.697 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:46:56.697 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:46:56.697 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.697 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:46:56.697 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:46:56.697 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:56.697 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.697 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.697 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:46:56.697 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:46:56.697 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:46:56.698 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:56.698 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:56.698 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.698 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:46:56.698 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:46:56.698 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:46:56.698 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:46:56.698 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:46:56.698 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:56.698 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:56.698 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.698 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:46:56.698 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:46:56.698 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.699 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:56.699 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:56.699 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:56.699 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:56.699 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:56.699 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:46:56.699 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.699 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:56.699 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:56.699 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5 2024-09-16T04:46:56.699 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:56.699 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:56.699 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:46:56.699 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.700 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:56.700 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:56.700 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5 2024-09-16T04:46:56.700 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:56.700 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:46:56.700 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:46:56.700 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:46:56.700 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:46:56.700 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:46:56.700 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:46:56.700 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:46:56.700 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:46:56.700 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:46:56.700 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:46:56.701 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:46:56.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.670+0000 7fcc88696640 1 -- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcc6c0383b0 msgr2=0x7fcc6c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.670+0000 7fcc88696640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcc6c0383b0 0x7fcc6c03a870 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fcc740090b0 tx=0x7fcc7402e000 comp rx=0 tx=0).stop 2024-09-16T04:46:56.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.670+0000 7fcc88696640 10 monclient: shutdown 2024-09-16T04:46:56.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.670+0000 7fcc88696640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:56.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.670+0000 7fcc88696640 1 -- 172.21.15.46:0/1786065391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc801369d0 msgr2=0x7fcc80123230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.670+0000 7fcc88696640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc801369d0 0x7fcc80123230 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000c700 tx=0x7fcc7000cbc0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.670+0000 7fcc86c33640 1 -- 172.21.15.46:0/1786065391 reap_dead start 2024-09-16T04:46:56.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.670+0000 7fcc88696640 1 -- 172.21.15.46:0/1786065391 shutdown_connections 2024-09-16T04:46:56.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.670+0000 7fcc88696640 1 --2- 172.21.15.46:0/1786065391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc801369d0 0x7fcc80123230 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.670+0000 7fcc88696640 1 -- 172.21.15.46:0/1786065391 >> 172.21.15.46:0/1786065391 conn(0x7fcc800a6340 msgr2=0x7fcc8009f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:56.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.670+0000 7fcc88696640 1 -- 172.21.15.46:0/1786065391 shutdown_connections 2024-09-16T04:46:56.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.670+0000 7fcc88696640 1 -- 172.21.15.46:0/1786065391 wait complete. 2024-09-16T04:46:56.703 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:46:56.703 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T04:46:56.703 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:46:56.703 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T04:46:56.703 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T04:46:56.704 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T04:46:56.704 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:46:56.719 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T04:46:56.719 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:46:56.719 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:46:56.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.818+0000 7f4889b49640 10 monclient: get_monmap_and_config 2024-09-16T04:46:56.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.818+0000 7f4889b49640 10 monclient: build_initial_monmap 2024-09-16T04:46:56.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.818+0000 7f4889b49640 10 monclient: monmap: 2024-09-16T04:46:56.819 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:56.819 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.819 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:46:56.818771+0000 2024-09-16T04:46:56.819 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:46:56.818771+0000 2024-09-16T04:46:56.819 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:56.820 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.820 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:46:56.820 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:46:56.820 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:46:56.820 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:46:56.820 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:46:56.820 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4889b49640 1 Processor -- start 2024-09-16T04:46:56.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4889b49640 1 -- start start 2024-09-16T04:46:56.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4889b49640 10 monclient: init 2024-09-16T04:46:56.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4889b49640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:56.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4889b49640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-16T04:46:56.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4889b49640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f48840a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:46:56.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4889b49640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f48840a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:46:56.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4889b49640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f488413cea0 0x7f488413f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4889b49640 10 monclient(hunting): picked mon.noname-b con 0x7f488413cea0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:56.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4889b49640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f488409b940 con 0x7f48840a3ff0 2024-09-16T04:46:56.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4889b49640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f488409ab00 con 0x7f48840a3c10 2024-09-16T04:46:56.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4889b49640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4889b49640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f488409b030 con 0x7f488413cea0 2024-09-16T04:46:56.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4889b49640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:56.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4889b49640 10 monclient(hunting): authenticate will time out at 1215.234708s 2024-09-16T04:46:56.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f48837fe640 1 -- >> v1:172.21.15.114:6789/0 conn(0x7f48840a3c10 legacy=0x7f488409bcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T04:46:56.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4882ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f48840a3ff0 0x7f48840a43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45098/0 (socket says 172.21.15.46:45098) 2024-09-16T04:46:56.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4883fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f488413cea0 0x7f488413f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4882ffd640 1 -- 172.21.15.46:0/2073930338 learned_addr learned my addr 172.21.15.46:0/2073930338 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:46:56.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4883fff640 10 monclient(hunting): get_auth_request con 0x7f488413cea0 auth_method 0 2024-09-16T04:46:56.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4883fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4883fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4883fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4883fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4883fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.819+0000 7f4883fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f4883fff640 10 monclient(hunting): handle_auth_done global_id 94379 payload 274 2024-09-16T04:46:56.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f4883fff640 1 -- 172.21.15.46:0/2073930338 >> v1:172.21.15.114:6789/0 conn(0x7f48840a3c10 legacy=0x7f488409bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:46:56.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f4883fff640 1 -- 172.21.15.46:0/2073930338 >> v1:172.21.15.46:6789/0 conn(0x7f48840a3ff0 legacy=0x7f48840a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f4883fff640 10 monclient: _finish_hunting 0 2024-09-16T04:46:56.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f4883fff640 1 monclient: found mon.noname-b 2024-09-16T04:46:56.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f4883fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:56.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f4883fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:46:56.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f4883fff640 1 -- 172.21.15.46:0/2073930338 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488409b310 con 0x7f488413cea0 2024-09-16T04:46:56.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f48827fc640 1 -- 172.21.15.46:0/2073930338 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1479606217 0 0) 0x7f488409b940 con 0x7f48840a3ff0 2024-09-16T04:46:56.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f48827fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:46:56.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f4883fff640 1 --2- 172.21.15.46:0/2073930338 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f488413cea0 0x7f488413f290 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f487c0097d0 tx=0x7f487c02a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83463f3e64ad541e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:56.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f48827fc640 1 -- 172.21.15.46:0/2073930338 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487c037070 con 0x7f488413cea0 2024-09-16T04:46:56.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f48827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f48827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:46:56.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f48827fc640 10 monclient: dump: 2024-09-16T04:46:56.834 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.834 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.834 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.834 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.834 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.834 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.834 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.834 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.834 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.834 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f48827fc640 10 monclient: _finish_auth 0 2024-09-16T04:46:56.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f48827fc640 10 monclient: _check_auth_tickets 2024-09-16T04:46:56.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f48827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:56.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f4889b49640 5 monclient: authenticate success, global_id 94379 2024-09-16T04:46:56.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f4889b49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:46:56.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f48827fc640 1 -- 172.21.15.46:0/2073930338 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f487c02ae40 con 0x7f488413cea0 2024-09-16T04:46:56.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f48827fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:56.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f4889b49640 10 monclient: get_monmap_and_config success 2024-09-16T04:46:56.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f4889b49640 4 set_mon_vals no callback set 2024-09-16T04:46:56.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f48827fc640 1 -- 172.21.15.46:0/2073930338 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487c032660 con 0x7f488413cea0 2024-09-16T04:46:56.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f48827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f48827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:56.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.820+0000 7f48827fc640 10 monclient: dump: 2024-09-16T04:46:56.835 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.835 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.835 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.836 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.836 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.836 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.836 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.836 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.836 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.836 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.821+0000 7f4889b49640 10 monclient: shutdown 2024-09-16T04:46:56.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.821+0000 7f4889b49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:56.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.821+0000 7f4889b49640 1 -- 172.21.15.46:0/2073930338 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f488413cea0 msgr2=0x7f488413f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.821+0000 7f4889b49640 1 --2- 172.21.15.46:0/2073930338 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f488413cea0 0x7f488413f290 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f487c0097d0 tx=0x7f487c02a130 comp rx=0 tx=0).stop 2024-09-16T04:46:56.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.821+0000 7f4889b49640 1 -- 172.21.15.46:0/2073930338 shutdown_connections 2024-09-16T04:46:56.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.821+0000 7f4889b49640 1 --2- 172.21.15.46:0/2073930338 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f488413cea0 0x7f488413f290 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.821+0000 7f4889b49640 1 -- 172.21.15.46:0/2073930338 >> 172.21.15.46:0/2073930338 conn(0x7f48840a6340 msgr2=0x7f488406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:56.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.821+0000 7f4889b49640 1 -- 172.21.15.46:0/2073930338 shutdown_connections 2024-09-16T04:46:56.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.821+0000 7f4889b49640 1 -- 172.21.15.46:0/2073930338 wait complete. 2024-09-16T04:46:56.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.822+0000 7f4889b49640 10 monclient: build_initial_monmap 2024-09-16T04:46:56.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.822+0000 7f4889b49640 10 monclient: monmap: 2024-09-16T04:46:56.837 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:46:56.837 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:46:56.837 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:46:56.837 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:46:56.837 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:46:56.837 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.837 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:46:56.837 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:46:56.837 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:46:56.838 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.822+0000 7f4889b49640 1 Processor -- start 2024-09-16T04:46:56.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.822+0000 7f4889b49640 1 -- start start 2024-09-16T04:46:56.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.822+0000 7f4889b49640 10 monclient: init 2024-09-16T04:46:56.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.822+0000 7f4889b49640 10 monclient: _reopen_session rank -1 2024-09-16T04:46:56.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.822+0000 7f4889b49640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:46:56.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.822+0000 7f4889b49640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f48840a3c10 0x7f48841cbb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4889b49640 10 monclient(hunting): picked mon.noname-c con 0x7f48840a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:46:56.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4889b49640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48840a3ff0 0x7f4884136e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4889b49640 10 monclient(hunting): picked mon.noname-a con 0x7f48840a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:46:56.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4889b49640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f488413cea0 0x7f4884138e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4889b49640 10 monclient(hunting): picked mon.noname-b con 0x7f488413cea0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:46:56.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4889b49640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4889b49640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f488413f950 con 0x7f48840a3ff0 2024-09-16T04:46:56.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4889b49640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4889b49640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f488413f7d0 con 0x7f488413cea0 2024-09-16T04:46:56.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4889b49640 10 monclient(hunting): start opening mon connection 2024-09-16T04:46:56.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4889b49640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f488413fad0 con 0x7f48840a3c10 2024-09-16T04:46:56.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4889b49640 10 monclient(hunting): _renew_subs 2024-09-16T04:46:56.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4889b49640 10 monclient(hunting): authenticate will time out at 1215.238060s 2024-09-16T04:46:56.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f48837fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f48840a3c10 0x7f48841cbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4882ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48840a3ff0 0x7f4884136e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4882ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48840a3ff0 0x7f4884136e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35226/0 (socket says 172.21.15.46:35226) 2024-09-16T04:46:56.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4882ffd640 1 -- 172.21.15.46:0/3695630882 learned_addr learned my addr 172.21.15.46:0/3695630882 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:46:56.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4882ffd640 10 monclient(hunting): get_auth_request con 0x7f48840a3ff0 auth_method 0 2024-09-16T04:46:56.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f48837fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f48840a3c10 0x7f48841cbb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:35888/0 (socket says 172.21.15.46:35888) 2024-09-16T04:46:56.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4882ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4882ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f48837fe640 10 monclient(hunting): get_auth_request con 0x7f48840a3c10 auth_method 0 2024-09-16T04:46:56.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f48837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:46:56.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f48837fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:46:56.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f48837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:46:56.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4883fff640 1 -- 172.21.15.46:0/3695630882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f488413cea0 msgr2=0x7f4884138e70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T04:46:56.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4883fff640 1 --2- 172.21.15.46:0/3695630882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f488413cea0 0x7f4884138e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:46:56.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4882ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4882ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4882ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f48837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:46:56.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f48837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:46:56.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f48837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:46:56.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f48837fe640 10 monclient(hunting): handle_auth_done global_id 94382 payload 824 2024-09-16T04:46:56.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f48837fe640 1 -- 172.21.15.46:0/3695630882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f488413cea0 msgr2=0x7f4884138e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:46:56.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f48837fe640 1 --2- 172.21.15.46:0/3695630882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f488413cea0 0x7f4884138e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f48837fe640 1 -- 172.21.15.46:0/3695630882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48840a3ff0 msgr2=0x7f4884136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:56.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f48837fe640 1 --2- 172.21.15.46:0/3695630882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48840a3ff0 0x7f4884136e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:56.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f48837fe640 10 monclient: _finish_hunting 0 2024-09-16T04:46:56.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f48837fe640 1 monclient: found mon.noname-c 2024-09-16T04:46:56.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f48837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:46:56.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f48837fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:46:56.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f48837fe640 1 -- 172.21.15.46:0/3695630882 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48841204d0 con 0x7f48840a3c10 2024-09-16T04:46:56.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4882ffd640 1 --2- 172.21.15.46:0/3695630882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48840a3ff0 0x7f4884136e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:46:56.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f48837fe640 1 --2- 172.21.15.46:0/3695630882 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f48840a3c10 0x7f48841cbb40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f487400be00 tx=0x7f4874009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:56.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f486bfff640 1 -- 172.21.15.46:0/3695630882 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4874015470 con 0x7f48840a3c10 2024-09-16T04:46:56.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f486bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f486bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:46:56.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f486bfff640 10 monclient: dump: 2024-09-16T04:46:56.842 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.842 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.843 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f486bfff640 10 monclient: _finish_auth 0 2024-09-16T04:46:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f486bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:46:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f486bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:46:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f486bfff640 1 -- 172.21.15.46:0/3695630882 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4874010070 con 0x7f48840a3c10 2024-09-16T04:46:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f486bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:46:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f486bfff640 1 -- 172.21.15.46:0/3695630882 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4874015890 con 0x7f48840a3c10 2024-09-16T04:46:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4889b49640 5 monclient: authenticate success, global_id 94382 2024-09-16T04:46:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4889b49640 10 monclient: _renew_subs 2024-09-16T04:46:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4889b49640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4889b49640 1 -- 172.21.15.46:0/3695630882 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48841207b0 con 0x7f48840a3c10 2024-09-16T04:46:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f486bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:46:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f486bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:46:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4881ffb640 4 set_mon_vals no callback set 2024-09-16T04:46:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f486bfff640 10 monclient: dump: 2024-09-16T04:46:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:46:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:46:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:46:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:46:56.845 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:46:56.845 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:46:56.845 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:46:56.845 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:46:56.845 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:46:56.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4889b49640 10 monclient: _renew_subs 2024-09-16T04:46:56.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4889b49640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:56.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.823+0000 7f4889b49640 1 -- 172.21.15.46:0/3695630882 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f488413ef50 con 0x7f48840a3c10 2024-09-16T04:46:56.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.824+0000 7f486bfff640 1 -- 172.21.15.46:0/3695630882 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f4874015a30 con 0x7f48840a3c10 2024-09-16T04:46:56.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.824+0000 7f486bfff640 1 --2- 172.21.15.46:0/3695630882 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4870038360 0x7f487003a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:46:56.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.824+0000 7f486bfff640 1 -- 172.21.15.46:0/3695630882 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(314..314 src has 1..314) ==== 4738+0+0 (secure 0 0 0) 0x7f487404bb40 con 0x7f48840a3c10 2024-09-16T04:46:56.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.824+0000 7f4889b49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:56.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.824+0000 7f4889b49640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:56.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.824+0000 7f4889b49640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:56.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.824+0000 7f4889b49640 1 -- 172.21.15.46:0/3695630882 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48840a8b00 con 0x7f48840a3c10 2024-09-16T04:46:56.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.825+0000 7f4882ffd640 1 --2- 172.21.15.46:0/3695630882 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4870038360 0x7f487003a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:46:56.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.825+0000 7f4882ffd640 10 monclient: get_auth_request con 0x7f4870038360 auth_method 0 2024-09-16T04:46:56.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.825+0000 7f4882ffd640 1 --2- 172.21.15.46:0/3695630882 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4870038360 0x7f487003a820 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f48841209c0 tx=0x7f4878007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:46:56.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.829+0000 7f486bfff640 1 -- 172.21.15.46:0/3695630882 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f48840a8b00 con 0x7f48840a3c10 2024-09-16T04:46:56.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.829+0000 7f486bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:46:56.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:56.829+0000 7f486bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:46:57.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:57.071+0000 7f4889b49640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:46:57.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:57.071+0000 7f4889b49640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:57.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:57.071+0000 7f4889b49640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:46:57.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:57.071+0000 7f4889b49640 1 -- 172.21.15.46:0/3695630882 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f488409bcf0 con 0x7f48840a3c10 2024-09-16T04:46:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:57.071+0000 7f486bfff640 1 -- 172.21.15.46:0/3695630882 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f4874005070 con 0x7f48840a3c10 2024-09-16T04:46:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:57.071+0000 7f486bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:46:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:57.071+0000 7f486bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:46:57.073 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:46:57.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:57.073+0000 7f4889b49640 1 -- 172.21.15.46:0/3695630882 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4870038360 msgr2=0x7f487003a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:57.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:57.073+0000 7f4889b49640 1 --2- 172.21.15.46:0/3695630882 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4870038360 0x7f487003a820 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f48841209c0 tx=0x7f4878007e10 comp rx=0 tx=0).stop 2024-09-16T04:46:57.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:57.073+0000 7f4889b49640 10 monclient: shutdown 2024-09-16T04:46:57.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:57.073+0000 7f4889b49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:46:57.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:57.073+0000 7f4889b49640 1 -- 172.21.15.46:0/3695630882 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f48840a3c10 msgr2=0x7f48841cbb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:46:57.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:57.073+0000 7f4889b49640 1 --2- 172.21.15.46:0/3695630882 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f48840a3c10 0x7f48841cbb40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f487400be00 tx=0x7f4874009f70 comp rx=0 tx=0).stop 2024-09-16T04:46:57.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:57.073+0000 7f4889b49640 1 -- 172.21.15.46:0/3695630882 shutdown_connections 2024-09-16T04:46:57.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:57.073+0000 7f4889b49640 1 --2- 172.21.15.46:0/3695630882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f488413cea0 0x7f4884138e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:57.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:57.073+0000 7f4889b49640 1 --2- 172.21.15.46:0/3695630882 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f4870038360 0x7f487003a820 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:57.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:57.073+0000 7f4889b49640 1 --2- 172.21.15.46:0/3695630882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48840a3ff0 0x7f4884136e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:57.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:57.073+0000 7f4889b49640 1 --2- 172.21.15.46:0/3695630882 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f48840a3c10 0x7f48841cbb40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:46:57.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:57.073+0000 7f4889b49640 1 -- 172.21.15.46:0/3695630882 >> 172.21.15.46:0/3695630882 conn(0x7f48840a6340 msgr2=0x7f48840a2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:46:57.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:57.073+0000 7f4889b49640 1 -- 172.21.15.46:0/3695630882 shutdown_connections 2024-09-16T04:46:57.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:46:57.073+0000 7f4889b49640 1 -- 172.21.15.46:0/3695630882 wait complete. 2024-09-16T04:47:00.086 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:47:00.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.197+0000 7efd527af640 10 monclient: get_monmap_and_config 2024-09-16T04:47:00.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.197+0000 7efd527af640 10 monclient: build_initial_monmap 2024-09-16T04:47:00.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.197+0000 7efd527af640 10 monclient: monmap: 2024-09-16T04:47:00.199 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:00.199 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:00.199 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:00.198213+0000 2024-09-16T04:47:00.199 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:00.198213+0000 2024-09-16T04:47:00.199 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:00.199 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:00.199 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:00.199 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:00.199 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:00.199 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:00.199 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:00.199 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:00.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.198+0000 7efd527af640 1 Processor -- start 2024-09-16T04:47:00.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.198+0000 7efd527af640 1 -- start start 2024-09-16T04:47:00.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.198+0000 7efd527af640 10 monclient: init 2024-09-16T04:47:00.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd527af640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:00.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd527af640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-16T04:47:00.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd527af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efd4c0a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:47:00.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd527af640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd4c0a3ff0 0x7efd4c0a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:00.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd527af640 10 monclient(hunting): picked mon.noname-b con 0x7efd4c0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:00.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd527af640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efd4c13be90 0x7efd4c13e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:00.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd527af640 10 monclient(hunting): picked mon.noname-c con 0x7efd4c13be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:47:00.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd527af640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd4c09b940 con 0x7efd4c0a3c10 2024-09-16T04:47:00.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd527af640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:00.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd527af640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7efd4c09ab00 con 0x7efd4c13be90 2024-09-16T04:47:00.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd527af640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:00.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd527af640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7efd4c09b030 con 0x7efd4c0a3ff0 2024-09-16T04:47:00.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd527af640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:00.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd527af640 10 monclient(hunting): authenticate will time out at 1218.614249s 2024-09-16T04:47:00.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd4bfff640 1 -- >> v1:172.21.15.114:6789/0 conn(0x7efd4c0a3c10 legacy=0x7efd4c09bcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T04:47:00.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd50d4c640 1 -- >> v2:172.21.15.114:3300/0 conn(0x7efd4c13be90 msgr2=0x7efd4c13e280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T04:47:00.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd50d4c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efd4c13be90 0x7efd4c13e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:47:00.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd4b7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd4c0a3ff0 0x7efd4c0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:00.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd4b7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd4c0a3ff0 0x7efd4c0a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:35896/0 (socket says 172.21.15.46:35896) 2024-09-16T04:47:00.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd4b7fe640 1 -- 172.21.15.46:0/127240298 learned_addr learned my addr 172.21.15.46:0/127240298 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:00.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd4b7fe640 10 monclient(hunting): get_auth_request con 0x7efd4c0a3ff0 auth_method 0 2024-09-16T04:47:00.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:00.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:00.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd4b7fe640 10 monclient(hunting): handle_auth_done global_id 94385 payload 274 2024-09-16T04:47:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd4b7fe640 1 -- 172.21.15.46:0/127240298 >> v2:172.21.15.114:3300/0 conn(0x7efd4c13be90 msgr2=0x7efd4c13e280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:47:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd4b7fe640 1 --2- 172.21.15.46:0/127240298 >> v2:172.21.15.114:3300/0 conn(0x7efd4c13be90 0x7efd4c13e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd4b7fe640 1 -- 172.21.15.46:0/127240298 >> v1:172.21.15.114:6789/0 conn(0x7efd4c0a3c10 legacy=0x7efd4c09bcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:47:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd4b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:47:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd4b7fe640 1 monclient: found mon.noname-b 2024-09-16T04:47:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd4b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:47:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.199+0000 7efd4b7fe640 1 -- 172.21.15.46:0/127240298 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd4c09b310 con 0x7efd4c0a3ff0 2024-09-16T04:47:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.200+0000 7efd4b7fe640 1 --2- 172.21.15.46:0/127240298 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd4c0a3ff0 0x7efd4c0a43d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efd40009870 tx=0x7efd4002a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28c8e7f718c71f88 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:00.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.200+0000 7efd4affd640 1 -- 172.21.15.46:0/127240298 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd40037070 con 0x7efd4c0a3ff0 2024-09-16T04:47:00.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.200+0000 7efd4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:00.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.200+0000 7efd4affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:47:00.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.200+0000 7efd4affd640 10 monclient: dump: 2024-09-16T04:47:00.210 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:00.210 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:00.210 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:00.210 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:00.210 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:00.210 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:00.210 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:00.210 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:00.210 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:00.210 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:00.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.200+0000 7efd4affd640 10 monclient: _finish_auth 0 2024-09-16T04:47:00.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.200+0000 7efd4affd640 10 monclient: _check_auth_tickets 2024-09-16T04:47:00.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.200+0000 7efd4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:00.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.200+0000 7efd4affd640 1 -- 172.21.15.46:0/127240298 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd40032440 con 0x7efd4c0a3ff0 2024-09-16T04:47:00.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.200+0000 7efd527af640 5 monclient: authenticate success, global_id 94385 2024-09-16T04:47:00.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.200+0000 7efd527af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:00.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.200+0000 7efd4affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:00.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.200+0000 7efd527af640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:00.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.200+0000 7efd4affd640 1 -- 172.21.15.46:0/127240298 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd40032810 con 0x7efd4c0a3ff0 2024-09-16T04:47:00.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.200+0000 7efd4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:00.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.200+0000 7efd527af640 4 set_mon_vals no callback set 2024-09-16T04:47:00.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.200+0000 7efd4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:00.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.200+0000 7efd4affd640 10 monclient: dump: 2024-09-16T04:47:00.211 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:00.211 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:00.212 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:00.212 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:00.212 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:00.212 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:00.212 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:00.212 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:00.212 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:00.212 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:00.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.201+0000 7efd527af640 10 monclient: shutdown 2024-09-16T04:47:00.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.201+0000 7efd527af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:00.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.201+0000 7efd527af640 1 -- 172.21.15.46:0/127240298 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd4c0a3ff0 msgr2=0x7efd4c0a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:00.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.201+0000 7efd527af640 1 --2- 172.21.15.46:0/127240298 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd4c0a3ff0 0x7efd4c0a43d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efd40009870 tx=0x7efd4002a1a0 comp rx=0 tx=0).stop 2024-09-16T04:47:00.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.201+0000 7efd527af640 1 -- 172.21.15.46:0/127240298 shutdown_connections 2024-09-16T04:47:00.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.201+0000 7efd527af640 1 --2- 172.21.15.46:0/127240298 >> v2:172.21.15.114:3300/0 conn(0x7efd4c13be90 0x7efd4c13e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:00.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.201+0000 7efd527af640 1 --2- 172.21.15.46:0/127240298 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd4c0a3ff0 0x7efd4c0a43d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:00.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.201+0000 7efd527af640 1 -- 172.21.15.46:0/127240298 >> 172.21.15.46:0/127240298 conn(0x7efd4c0a6340 msgr2=0x7efd4c06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:00.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.201+0000 7efd527af640 1 -- 172.21.15.46:0/127240298 shutdown_connections 2024-09-16T04:47:00.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.201+0000 7efd527af640 1 -- 172.21.15.46:0/127240298 wait complete. 2024-09-16T04:47:00.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.201+0000 7efd527af640 10 monclient: build_initial_monmap 2024-09-16T04:47:00.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.201+0000 7efd527af640 10 monclient: monmap: 2024-09-16T04:47:00.213 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:00.213 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:00.213 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:00.213 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:00.213 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:00.213 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:00.213 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:00.214 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:00.214 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:00.214 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:00.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.202+0000 7efd527af640 1 Processor -- start 2024-09-16T04:47:00.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.202+0000 7efd527af640 1 -- start start 2024-09-16T04:47:00.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.202+0000 7efd527af640 10 monclient: init 2024-09-16T04:47:00.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.202+0000 7efd527af640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:00.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.202+0000 7efd527af640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:47:00.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.202+0000 7efd527af640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd4c0a3c10 0x7efd4c133d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:00.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.202+0000 7efd527af640 10 monclient(hunting): picked mon.noname-a con 0x7efd4c0a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:00.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.202+0000 7efd527af640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd4c0a3ff0 0x7efd4c1342c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:00.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.202+0000 7efd527af640 10 monclient(hunting): picked mon.noname-b con 0x7efd4c0a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:00.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.202+0000 7efd527af640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd4c13be90 0x7efd4c1cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.202+0000 7efd527af640 10 monclient(hunting): picked mon.noname-c con 0x7efd4c13be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.202+0000 7efd527af640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.202+0000 7efd527af640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efd4c0a6b40 con 0x7efd4c0a3c10 2024-09-16T04:47:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.202+0000 7efd527af640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.202+0000 7efd527af640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efd4c0a69c0 con 0x7efd4c0a3ff0 2024-09-16T04:47:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.202+0000 7efd527af640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.202+0000 7efd527af640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7efd4c13e7c0 con 0x7efd4c13be90 2024-09-16T04:47:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.202+0000 7efd527af640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.202+0000 7efd527af640 10 monclient(hunting): authenticate will time out at 1218.617887s 2024-09-16T04:47:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd50d4c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd4c13be90 0x7efd4c1cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd4c0a3c10 0x7efd4c133d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd50d4c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd4c13be90 0x7efd4c1cbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:35904/0 (socket says 172.21.15.46:35904) 2024-09-16T04:47:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4b7fe640 1 -- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd4c0a3ff0 msgr2=0x7efd4c1342c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T04:47:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd50d4c640 1 -- 172.21.15.46:0/3363032718 learned_addr learned my addr 172.21.15.46:0/3363032718 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd50d4c640 10 monclient(hunting): get_auth_request con 0x7efd4c13be90 auth_method 0 2024-09-16T04:47:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd50d4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd50d4c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd50d4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4b7fe640 1 --2- 172.21.15.46:0/3363032718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd4c0a3ff0 0x7efd4c1342c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:47:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4bfff640 10 monclient(hunting): get_auth_request con 0x7efd4c0a3c10 auth_method 0 2024-09-16T04:47:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd50d4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd50d4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd50d4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4bfff640 10 monclient(hunting): handle_auth_done global_id 104193 payload 824 2024-09-16T04:47:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4bfff640 1 -- 172.21.15.46:0/3363032718 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd4c13be90 msgr2=0x7efd4c1cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4bfff640 1 --2- 172.21.15.46:0/3363032718 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd4c13be90 0x7efd4c1cbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4bfff640 1 -- 172.21.15.46:0/3363032718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd4c0a3ff0 msgr2=0x7efd4c1342c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:47:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4bfff640 1 --2- 172.21.15.46:0/3363032718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd4c0a3ff0 0x7efd4c1342c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:47:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4bfff640 1 monclient: found mon.noname-a 2024-09-16T04:47:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4bfff640 1 -- 172.21.15.46:0/3363032718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd4c09ef00 con 0x7efd4c0a3c10 2024-09-16T04:47:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd50d4c640 1 --2- 172.21.15.46:0/3363032718 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd4c13be90 0x7efd4c1cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4bfff640 1 --2- 172.21.15.46:0/3363032718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd4c0a3c10 0x7efd4c133d80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efd3c003930 tx=0x7efd3c00b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd48ff9640 1 -- 172.21.15.46:0/3363032718 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3c003dc0 con 0x7efd4c0a3c10 2024-09-16T04:47:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd48ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd48ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd48ff9640 10 monclient: dump: 2024-09-16T04:47:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:00.218 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd48ff9640 10 monclient: _finish_auth 0 2024-09-16T04:47:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd48ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:47:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd48ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd48ff9640 1 -- 172.21.15.46:0/3363032718 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd3c015070 con 0x7efd4c0a3c10 2024-09-16T04:47:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd527af640 5 monclient: authenticate success, global_id 104193 2024-09-16T04:47:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd527af640 10 monclient: _renew_subs 2024-09-16T04:47:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd527af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd527af640 1 -- 172.21.15.46:0/3363032718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd4c09da50 con 0x7efd4c0a3c10 2024-09-16T04:47:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd48ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd48ff9640 1 -- 172.21.15.46:0/3363032718 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3c005290 con 0x7efd4c0a3c10 2024-09-16T04:47:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd48ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd48ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd48ff9640 10 monclient: dump: 2024-09-16T04:47:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:00.220 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:00.220 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:00.220 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:00.220 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:00.220 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:00.220 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:00.220 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:00.220 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:00.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd527af640 10 monclient: _renew_subs 2024-09-16T04:47:00.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd527af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:00.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd527af640 1 -- 172.21.15.46:0/3363032718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd4c134800 con 0x7efd4c0a3c10 2024-09-16T04:47:00.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.203+0000 7efd4a7fc640 4 set_mon_vals no callback set 2024-09-16T04:47:00.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.204+0000 7efd48ff9640 1 -- 172.21.15.46:0/3363032718 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7efd3c00f070 con 0x7efd4c0a3c10 2024-09-16T04:47:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.204+0000 7efd527af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.204+0000 7efd527af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.204+0000 7efd527af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.204+0000 7efd527af640 1 -- 172.21.15.46:0/3363032718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd4c0a4450 con 0x7efd4c0a3c10 2024-09-16T04:47:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.204+0000 7efd48ff9640 1 --2- 172.21.15.46:0/3363032718 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7efd38038390 0x7efd3803a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.204+0000 7efd48ff9640 1 -- 172.21.15.46:0/3363032718 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(314..314 src has 1..314) ==== 4738+0+0 (secure 0 0 0) 0x7efd3c04afa0 con 0x7efd4c0a3c10 2024-09-16T04:47:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.205+0000 7efd4b7fe640 1 --2- 172.21.15.46:0/3363032718 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7efd38038390 0x7efd3803a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.205+0000 7efd4b7fe640 10 monclient: get_auth_request con 0x7efd38038390 auth_method 0 2024-09-16T04:47:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.205+0000 7efd4b7fe640 1 --2- 172.21.15.46:0/3363032718 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7efd38038390 0x7efd3803a850 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7efd40034000 tx=0x7efd4002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.210+0000 7efd48ff9640 1 -- 172.21.15.46:0/3363032718 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7efd3c00d040 con 0x7efd4c0a3c10 2024-09-16T04:47:00.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.210+0000 7efd48ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:00.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.210+0000 7efd48ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:00.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.451+0000 7efd527af640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:47:00.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.451+0000 7efd527af640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:47:00.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.451+0000 7efd527af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:00.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.451+0000 7efd527af640 1 -- 172.21.15.46:0/3363032718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7efd4c09f110 con 0x7efd4c0a3c10 2024-09-16T04:47:00.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.452+0000 7efd48ff9640 1 -- 172.21.15.46:0/3363032718 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7efd3c0056c0 con 0x7efd4c0a3c10 2024-09-16T04:47:00.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.452+0000 7efd48ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:47:00.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.452+0000 7efd48ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:47:00.454 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:47:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.454+0000 7efd527af640 1 -- 172.21.15.46:0/3363032718 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7efd38038390 msgr2=0x7efd3803a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.454+0000 7efd527af640 1 --2- 172.21.15.46:0/3363032718 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7efd38038390 0x7efd3803a850 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7efd40034000 tx=0x7efd4002e000 comp rx=0 tx=0).stop 2024-09-16T04:47:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.454+0000 7efd527af640 10 monclient: shutdown 2024-09-16T04:47:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.454+0000 7efd527af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.454+0000 7efd527af640 1 -- 172.21.15.46:0/3363032718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd4c0a3c10 msgr2=0x7efd4c133d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.454+0000 7efd527af640 1 --2- 172.21.15.46:0/3363032718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd4c0a3c10 0x7efd4c133d80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efd3c003930 tx=0x7efd3c00b9b0 comp rx=0 tx=0).stop 2024-09-16T04:47:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.454+0000 7efd527af640 1 -- 172.21.15.46:0/3363032718 shutdown_connections 2024-09-16T04:47:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.454+0000 7efd527af640 1 --2- 172.21.15.46:0/3363032718 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7efd4c13be90 0x7efd4c1cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.454+0000 7efd527af640 1 --2- 172.21.15.46:0/3363032718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd4c0a3ff0 0x7efd4c1342c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.454+0000 7efd527af640 1 --2- 172.21.15.46:0/3363032718 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7efd38038390 0x7efd3803a850 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.454+0000 7efd527af640 1 --2- 172.21.15.46:0/3363032718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd4c0a3c10 0x7efd4c133d80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.454+0000 7efd527af640 1 -- 172.21.15.46:0/3363032718 >> 172.21.15.46:0/3363032718 conn(0x7efd4c0a6340 msgr2=0x7efd4c13b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.454+0000 7efd527af640 1 -- 172.21.15.46:0/3363032718 shutdown_connections 2024-09-16T04:47:00.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:00.454+0000 7efd527af640 1 -- 172.21.15.46:0/3363032718 wait complete. 2024-09-16T04:47:03.467 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:47:03.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.579+0000 7fd7ca6f1640 10 monclient: get_monmap_and_config 2024-09-16T04:47:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.579+0000 7fd7ca6f1640 10 monclient: build_initial_monmap 2024-09-16T04:47:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.579+0000 7fd7ca6f1640 10 monclient: monmap: 2024-09-16T04:47:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:03.580179+0000 2024-09-16T04:47:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:03.580179+0000 2024-09-16T04:47:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:03.582 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:03.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.580+0000 7fd7ca6f1640 1 Processor -- start 2024-09-16T04:47:03.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.580+0000 7fd7ca6f1640 1 -- start start 2024-09-16T04:47:03.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.580+0000 7fd7ca6f1640 10 monclient: init 2024-09-16T04:47:03.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.580+0000 7fd7ca6f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:03.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7ca6f1640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T04:47:03.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7ca6f1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd7c406e160 addr v1:172.21.15.46:6789/0 2024-09-16T04:47:03.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7ca6f1640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c4139c50 0x7fd7c413c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:03.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7ca6f1640 10 monclient(hunting): picked mon.noname-b con 0x7fd7c4139c50 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:03.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7ca6f1640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd7c413c580 0x7fd7c413e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:03.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7ca6f1640 10 monclient(hunting): picked mon.noname-c con 0x7fd7c413c580 addr v2:172.21.15.114:3300/0 2024-09-16T04:47:03.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7ca6f1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7c40a3fd0 con 0x7fd7c406e160 2024-09-16T04:47:03.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7ca6f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:03.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7ca6f1640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fd7c406e540 con 0x7fd7c413c580 2024-09-16T04:47:03.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7ca6f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:03.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7ca6f1640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd7c40a0bd0 con 0x7fd7c4139c50 2024-09-16T04:47:03.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7ca6f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:03.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7ca6f1640 10 monclient(hunting): authenticate will time out at 1221.996112s 2024-09-16T04:47:03.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7c37fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c4139c50 0x7fd7c413c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:03.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7c3fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd7c406e160 0x7fd7c40a29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45104/0 (socket says 172.21.15.46:45104) 2024-09-16T04:47:03.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7c3fff640 1 -- 172.21.15.46:0/1758314069 learned_addr learned my addr 172.21.15.46:0/1758314069 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:47:03.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7c37fe640 10 monclient(hunting): get_auth_request con 0x7fd7c4139c50 auth_method 0 2024-09-16T04:47:03.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:03.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:03.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:03.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7c8c8e640 1 -- 172.21.15.46:0/1758314069 >> v2:172.21.15.114:3300/0 conn(0x7fd7c413c580 msgr2=0x7fd7c413e970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T04:47:03.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7c8c8e640 1 --2- 172.21.15.46:0/1758314069 >> v2:172.21.15.114:3300/0 conn(0x7fd7c413c580 0x7fd7c413e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:47:03.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:03.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7c37fe640 10 monclient(hunting): handle_auth_done global_id 94391 payload 274 2024-09-16T04:47:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7c37fe640 1 -- 172.21.15.46:0/1758314069 >> v2:172.21.15.114:3300/0 conn(0x7fd7c413c580 msgr2=0x7fd7c413e970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:47:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7c37fe640 1 --2- 172.21.15.46:0/1758314069 >> v2:172.21.15.114:3300/0 conn(0x7fd7c413c580 0x7fd7c413e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7c37fe640 1 -- 172.21.15.46:0/1758314069 >> v1:172.21.15.46:6789/0 conn(0x7fd7c406e160 legacy=0x7fd7c40a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7c37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:47:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7c37fe640 1 monclient: found mon.noname-b 2024-09-16T04:47:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7c37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:47:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.581+0000 7fd7c37fe640 1 -- 172.21.15.46:0/1758314069 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c40a0eb0 con 0x7fd7c4139c50 2024-09-16T04:47:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7c37fe640 1 --2- 172.21.15.46:0/1758314069 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c4139c50 0x7fd7c413c040 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd7b80097a0 tx=0x7fd7b802a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8744b42b42382b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7c2ffd640 1 -- 172.21.15.46:0/1758314069 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8037070 con 0x7fd7c4139c50 2024-09-16T04:47:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7c2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:47:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7c2ffd640 10 monclient: dump: 2024-09-16T04:47:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:03.593 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7c2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:47:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7c2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:47:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7c2ffd640 1 -- 172.21.15.46:0/1758314069 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7b802add0 con 0x7fd7c4139c50 2024-09-16T04:47:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7ca6f1640 5 monclient: authenticate success, global_id 94391 2024-09-16T04:47:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7ca6f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7c2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7ca6f1640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7c2ffd640 1 -- 172.21.15.46:0/1758314069 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8032660 con 0x7fd7c4139c50 2024-09-16T04:47:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7ca6f1640 4 set_mon_vals no callback set 2024-09-16T04:47:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7c2ffd640 10 monclient: dump: 2024-09-16T04:47:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:03.595 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:03.595 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:03.595 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:03.595 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:03.595 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:03.595 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:03.595 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:03.595 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:03.595 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:03.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7ca6f1640 10 monclient: shutdown 2024-09-16T04:47:03.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7ca6f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:03.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7ca6f1640 1 -- 172.21.15.46:0/1758314069 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c4139c50 msgr2=0x7fd7c413c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:03.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.582+0000 7fd7ca6f1640 1 --2- 172.21.15.46:0/1758314069 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c4139c50 0x7fd7c413c040 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd7b80097a0 tx=0x7fd7b802a080 comp rx=0 tx=0).stop 2024-09-16T04:47:03.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.583+0000 7fd7ca6f1640 1 -- 172.21.15.46:0/1758314069 shutdown_connections 2024-09-16T04:47:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.583+0000 7fd7ca6f1640 1 --2- 172.21.15.46:0/1758314069 >> v2:172.21.15.114:3300/0 conn(0x7fd7c413c580 0x7fd7c413e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.583+0000 7fd7ca6f1640 1 --2- 172.21.15.46:0/1758314069 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c4139c50 0x7fd7c413c040 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.583+0000 7fd7ca6f1640 1 -- 172.21.15.46:0/1758314069 >> 172.21.15.46:0/1758314069 conn(0x7fd7c40a5c10 msgr2=0x7fd7c40a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.583+0000 7fd7ca6f1640 1 -- 172.21.15.46:0/1758314069 shutdown_connections 2024-09-16T04:47:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.583+0000 7fd7ca6f1640 1 -- 172.21.15.46:0/1758314069 wait complete. 2024-09-16T04:47:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.583+0000 7fd7ca6f1640 10 monclient: build_initial_monmap 2024-09-16T04:47:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.583+0000 7fd7ca6f1640 10 monclient: monmap: 2024-09-16T04:47:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:03.597 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:03.597 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:03.597 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:03.597 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:03.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.583+0000 7fd7ca6f1640 1 Processor -- start 2024-09-16T04:47:03.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7ca6f1640 1 -- start start 2024-09-16T04:47:03.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7ca6f1640 10 monclient: init 2024-09-16T04:47:03.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7ca6f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:03.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7ca6f1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:47:03.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7ca6f1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c406e160 0x7fd7c409d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:03.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7ca6f1640 10 monclient(hunting): picked mon.noname-b con 0x7fd7c406e160 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:03.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7ca6f1640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c4139c50 0x7fd7c409d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:03.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7ca6f1640 10 monclient(hunting): picked mon.noname-c con 0x7fd7c4139c50 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:03.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7ca6f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c413c580 0x7fd7c409ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:03.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7ca6f1640 10 monclient(hunting): picked mon.noname-a con 0x7fd7c413c580 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:03.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7ca6f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:03.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7ca6f1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c40a34a0 con 0x7fd7c413c580 2024-09-16T04:47:03.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7ca6f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:03.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7ca6f1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c40a4d90 con 0x7fd7c406e160 2024-09-16T04:47:03.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7ca6f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:03.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7ca6f1640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd7c40a8930 con 0x7fd7c4139c50 2024-09-16T04:47:03.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7ca6f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:03.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7ca6f1640 10 monclient(hunting): authenticate will time out at 1221.999670s 2024-09-16T04:47:03.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7c3fff640 1 -- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c406e160 msgr2=0x7fd7c409d330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T04:47:03.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7c37fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c4139c50 0x7fd7c409d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:03.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7c3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c406e160 0x7fd7c409d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:47:03.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7c8c8e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c413c580 0x7fd7c409ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:03.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7c37fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c4139c50 0x7fd7c409d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:35918/0 (socket says 172.21.15.46:35918) 2024-09-16T04:47:03.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7c37fe640 1 -- 172.21.15.46:0/2395720404 learned_addr learned my addr 172.21.15.46:0/2395720404 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:03.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7c37fe640 10 monclient(hunting): get_auth_request con 0x7fd7c4139c50 auth_method 0 2024-09-16T04:47:03.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7c8c8e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c413c580 0x7fd7c409ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35252/0 (socket says 172.21.15.46:35252) 2024-09-16T04:47:03.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:03.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:03.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:03.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7c8c8e640 10 monclient(hunting): get_auth_request con 0x7fd7c413c580 auth_method 0 2024-09-16T04:47:03.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7c8c8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:03.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7c8c8e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:03.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7c8c8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:03.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:03.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:03.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:03.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7c8c8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:03.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7c8c8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:03.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.584+0000 7fd7c8c8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:03.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c37fe640 10 monclient(hunting): handle_auth_done global_id 94394 payload 824 2024-09-16T04:47:03.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c37fe640 1 -- 172.21.15.46:0/2395720404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c406e160 msgr2=0x7fd7c409d330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:47:03.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c37fe640 1 --2- 172.21.15.46:0/2395720404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c406e160 0x7fd7c409d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:03.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c37fe640 1 -- 172.21.15.46:0/2395720404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c413c580 msgr2=0x7fd7c409ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:03.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c37fe640 1 --2- 172.21.15.46:0/2395720404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c413c580 0x7fd7c409ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:03.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:47:03.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c37fe640 1 monclient: found mon.noname-c 2024-09-16T04:47:03.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:03.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:47:03.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c37fe640 1 -- 172.21.15.46:0/2395720404 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c4133990 con 0x7fd7c4139c50 2024-09-16T04:47:03.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c8c8e640 1 --2- 172.21.15.46:0/2395720404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c413c580 0x7fd7c409ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:03.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c37fe640 1 --2- 172.21.15.46:0/2395720404 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c4139c50 0x7fd7c409d870 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd7b80097a0 tx=0x7fd7b80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:03.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c0ff9640 1 -- 172.21.15.46:0/2395720404 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8041070 con 0x7fd7c4139c50 2024-09-16T04:47:03.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:03.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:47:03.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c0ff9640 10 monclient: dump: 2024-09-16T04:47:03.601 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:03.601 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:03.601 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:03.601 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:03.601 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:03.602 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:03.602 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:03.602 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:03.602 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:03.602 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:03.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c0ff9640 10 monclient: _finish_auth 0 2024-09-16T04:47:03.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c0ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:47:03.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:03.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c0ff9640 1 -- 172.21.15.46:0/2395720404 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7b8032e10 con 0x7fd7c4139c50 2024-09-16T04:47:03.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:03.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c0ff9640 1 -- 172.21.15.46:0/2395720404 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8037040 con 0x7fd7c4139c50 2024-09-16T04:47:03.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7ca6f1640 5 monclient: authenticate success, global_id 94394 2024-09-16T04:47:03.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7ca6f1640 10 monclient: _renew_subs 2024-09-16T04:47:03.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7ca6f1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:03.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7ca6f1640 1 -- 172.21.15.46:0/2395720404 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7c4120960 con 0x7fd7c4139c50 2024-09-16T04:47:03.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:03.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:03.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c0ff9640 10 monclient: dump: 2024-09-16T04:47:03.603 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:03.603 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:03.603 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:03.603 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:03.603 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:03.604 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:03.604 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:03.604 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:03.604 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:03.604 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:03.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7c27fc640 4 set_mon_vals no callback set 2024-09-16T04:47:03.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7ca6f1640 10 monclient: _renew_subs 2024-09-16T04:47:03.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7ca6f1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:03.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.585+0000 7fd7ca6f1640 1 -- 172.21.15.46:0/2395720404 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7c41203a0 con 0x7fd7c4139c50 2024-09-16T04:47:03.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.586+0000 7fd7c0ff9640 1 -- 172.21.15.46:0/2395720404 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fd7b8044070 con 0x7fd7c4139c50 2024-09-16T04:47:03.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.586+0000 7fd7ca6f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:03.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.586+0000 7fd7c0ff9640 1 --2- 172.21.15.46:0/2395720404 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd7b0038360 0x7fd7b003a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:03.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.586+0000 7fd7ca6f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:03.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.586+0000 7fd7ca6f1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:03.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.586+0000 7fd7c0ff9640 1 -- 172.21.15.46:0/2395720404 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(314..314 src has 1..314) ==== 4738+0+0 (secure 0 0 0) 0x7fd7b806be90 con 0x7fd7c4139c50 2024-09-16T04:47:03.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.586+0000 7fd7ca6f1640 1 -- 172.21.15.46:0/2395720404 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7c404ffb0 con 0x7fd7c4139c50 2024-09-16T04:47:03.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.586+0000 7fd7c3fff640 1 --2- 172.21.15.46:0/2395720404 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd7b0038360 0x7fd7b003a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:03.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.586+0000 7fd7c3fff640 10 monclient: get_auth_request con 0x7fd7b0038360 auth_method 0 2024-09-16T04:47:03.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.587+0000 7fd7c3fff640 1 --2- 172.21.15.46:0/2395720404 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd7b0038360 0x7fd7b003a820 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4004810 tx=0x7fd7b4007b00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:03.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.591+0000 7fd7c0ff9640 1 -- 172.21.15.46:0/2395720404 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd7b809ad50 con 0x7fd7c4139c50 2024-09-16T04:47:03.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.591+0000 7fd7c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:03.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.591+0000 7fd7c0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:03.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.836+0000 7fd7ca6f1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:47:03.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.836+0000 7fd7ca6f1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:47:03.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.836+0000 7fd7ca6f1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:03.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.836+0000 7fd7ca6f1640 1 -- 172.21.15.46:0/2395720404 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd7c40a3140 con 0x7fd7c4139c50 2024-09-16T04:47:03.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.836+0000 7fd7c0ff9640 1 -- 172.21.15.46:0/2395720404 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fd7c404ffb0 con 0x7fd7c4139c50 2024-09-16T04:47:03.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.836+0000 7fd7c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:47:03.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.836+0000 7fd7c0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:47:03.838 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:47:03.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.838+0000 7fd7ca6f1640 1 -- 172.21.15.46:0/2395720404 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd7b0038360 msgr2=0x7fd7b003a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:03.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.838+0000 7fd7ca6f1640 1 --2- 172.21.15.46:0/2395720404 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd7b0038360 0x7fd7b003a820 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4004810 tx=0x7fd7b4007b00 comp rx=0 tx=0).stop 2024-09-16T04:47:03.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.838+0000 7fd7ca6f1640 10 monclient: shutdown 2024-09-16T04:47:03.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.838+0000 7fd7ca6f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:03.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.838+0000 7fd7ca6f1640 1 -- 172.21.15.46:0/2395720404 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c4139c50 msgr2=0x7fd7c409d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:03.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.838+0000 7fd7ca6f1640 1 --2- 172.21.15.46:0/2395720404 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c4139c50 0x7fd7c409d870 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd7b80097a0 tx=0x7fd7b80046c0 comp rx=0 tx=0).stop 2024-09-16T04:47:03.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.838+0000 7fd7ca6f1640 1 -- 172.21.15.46:0/2395720404 shutdown_connections 2024-09-16T04:47:03.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.838+0000 7fd7ca6f1640 1 --2- 172.21.15.46:0/2395720404 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd7b0038360 0x7fd7b003a820 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:03.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.838+0000 7fd7ca6f1640 1 --2- 172.21.15.46:0/2395720404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c413c580 0x7fd7c409ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:03.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.838+0000 7fd7ca6f1640 1 --2- 172.21.15.46:0/2395720404 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd7c4139c50 0x7fd7c409d870 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.838+0000 7fd7ca6f1640 1 --2- 172.21.15.46:0/2395720404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c406e160 0x7fd7c409d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.839+0000 7fd7ca6f1640 1 -- 172.21.15.46:0/2395720404 >> 172.21.15.46:0/2395720404 conn(0x7fd7c40a5c10 msgr2=0x7fd7c40a6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.839+0000 7fd7ca6f1640 1 -- 172.21.15.46:0/2395720404 shutdown_connections 2024-09-16T04:47:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:03.839+0000 7fd7ca6f1640 1 -- 172.21.15.46:0/2395720404 wait complete. 2024-09-16T04:47:06.851 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:47:06.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.963+0000 7fcaf429b640 10 monclient: get_monmap_and_config 2024-09-16T04:47:06.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.963+0000 7fcaf429b640 10 monclient: build_initial_monmap 2024-09-16T04:47:06.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.963+0000 7fcaf429b640 10 monclient: monmap: 2024-09-16T04:47:06.964 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:06.964 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:06.964 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:06.963757+0000 2024-09-16T04:47:06.964 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:06.963757+0000 2024-09-16T04:47:06.964 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:06.964 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:06.965 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:06.965 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:06.965 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:06.965 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:06.965 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:06.965 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:06.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.963+0000 7fcaf429b640 1 Processor -- start 2024-09-16T04:47:06.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.963+0000 7fcaf429b640 1 -- start start 2024-09-16T04:47:06.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.963+0000 7fcaf429b640 10 monclient: init 2024-09-16T04:47:06.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf429b640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:06.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf429b640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-16T04:47:06.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf429b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 0x7fcaec09bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:06.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf429b640 10 monclient(hunting): picked mon.noname-b con 0x7fcaec0a3ba0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:06.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf429b640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcaec09c1c0 0x7fcaec0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf429b640 10 monclient(hunting): picked mon.noname-c con 0x7fcaec09c1c0 addr v2:172.21.15.114:3300/0 2024-09-16T04:47:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf429b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcaec0a44c0 addr v1:172.21.15.46:6789/0 2024-09-16T04:47:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf429b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaec09afc0 con 0x7fcaec0a44c0 2024-09-16T04:47:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf429b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf429b640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fcaec09b860 con 0x7fcaec09c1c0 2024-09-16T04:47:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf429b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf429b640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fcaec09b2a0 con 0x7fcaec0a3ba0 2024-09-16T04:47:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf429b640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf429b640 10 monclient(hunting): authenticate will time out at 1225.379685s 2024-09-16T04:47:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf2037640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 0x7fcaec09bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf2838640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcaec0a44c0 0x7fcaec135950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47616/0 (socket says 172.21.15.46:47616) 2024-09-16T04:47:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf2838640 1 -- 172.21.15.46:0/699441235 learned_addr learned my addr 172.21.15.46:0/699441235 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:47:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf1836640 1 -- 172.21.15.46:0/699441235 >> v2:172.21.15.114:3300/0 conn(0x7fcaec09c1c0 msgr2=0x7fcaec0a3f80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T04:47:06.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf2037640 10 monclient(hunting): get_auth_request con 0x7fcaec0a3ba0 auth_method 0 2024-09-16T04:47:06.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf2037640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:06.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf2037640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:06.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf2037640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:06.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf1836640 1 --2- 172.21.15.46:0/699441235 >> v2:172.21.15.114:3300/0 conn(0x7fcaec09c1c0 0x7fcaec0a3f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:47:06.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf2037640 1 -- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 msgr2=0x7fcaec09bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:47:06.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf2037640 1 -- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 msgr2=0x7fcaec09bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:47:06.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.964+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 0x7fcaec09bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:47:06.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:06.965+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 0x7fcaec09bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:47:07.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.165+0000 7fcaf1836640 1 -- 172.21.15.46:0/699441235 >> v2:172.21.15.114:3300/0 conn(0x7fcaec09c1c0 msgr2=0x7fcaec0a3f80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T04:47:07.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.165+0000 7fcaf1836640 1 --2- 172.21.15.46:0/699441235 >> v2:172.21.15.114:3300/0 conn(0x7fcaec09c1c0 0x7fcaec0a3f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:47:07.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.166+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 0x7fcaec09bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:07.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.166+0000 7fcaf2037640 10 monclient(hunting): get_auth_request con 0x7fcaec0a3ba0 auth_method 0 2024-09-16T04:47:07.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.166+0000 7fcaf2037640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:07.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.166+0000 7fcaf2037640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:07.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.166+0000 7fcaf2037640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:47:07.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.166+0000 7fcaf2037640 1 -- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 msgr2=0x7fcaec09bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:47:07.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.166+0000 7fcaf2037640 1 -- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 msgr2=0x7fcaec09bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:47:07.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.166+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 0x7fcaec09bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:47:07.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.166+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 0x7fcaec09bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:47:07.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.566+0000 7fcaf1836640 1 -- 172.21.15.46:0/699441235 >> v2:172.21.15.114:3300/0 conn(0x7fcaec09c1c0 msgr2=0x7fcaec0a3f80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T04:47:07.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.566+0000 7fcaf1836640 1 --2- 172.21.15.46:0/699441235 >> v2:172.21.15.114:3300/0 conn(0x7fcaec09c1c0 0x7fcaec0a3f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:47:07.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.567+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 0x7fcaec09bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:07.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.567+0000 7fcaf2037640 10 monclient(hunting): get_auth_request con 0x7fcaec0a3ba0 auth_method 0 2024-09-16T04:47:07.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.567+0000 7fcaf2037640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:07.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.567+0000 7fcaf2037640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:07.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.567+0000 7fcaf2037640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:47:07.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.567+0000 7fcaf2037640 1 -- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 msgr2=0x7fcaec09bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:47:07.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.567+0000 7fcaf2037640 1 -- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 msgr2=0x7fcaec09bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:47:07.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.567+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 0x7fcaec09bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:47:07.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:07.567+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 0x7fcaec09bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:47:08.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:08.367+0000 7fcaf1836640 1 -- 172.21.15.46:0/699441235 >> v2:172.21.15.114:3300/0 conn(0x7fcaec09c1c0 msgr2=0x7fcaec0a3f80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T04:47:08.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:08.367+0000 7fcaf1836640 1 --2- 172.21.15.46:0/699441235 >> v2:172.21.15.114:3300/0 conn(0x7fcaec09c1c0 0x7fcaec0a3f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:47:08.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:08.368+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 0x7fcaec09bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:08.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:08.368+0000 7fcaf2037640 10 monclient(hunting): get_auth_request con 0x7fcaec0a3ba0 auth_method 0 2024-09-16T04:47:08.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:08.368+0000 7fcaf2037640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:08.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:08.368+0000 7fcaf2037640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:08.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:08.368+0000 7fcaf2037640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:47:08.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:08.368+0000 7fcaf2037640 1 -- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 msgr2=0x7fcaec09bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:47:08.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:08.368+0000 7fcaf2037640 1 -- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 msgr2=0x7fcaec09bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:47:08.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:08.368+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 0x7fcaec09bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:47:08.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:08.368+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 0x7fcaec09bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:47:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcae3fff640 10 monclient(hunting): tick 2024-09-16T04:47:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcae3fff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:47:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcae3fff640 1 monclient(hunting): continuing hunt 2024-09-16T04:47:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcae3fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:47:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcae3fff640 1 -- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 msgr2=0x7fcaec09bc80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:47:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcae3fff640 1 --2- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec0a3ba0 0x7fcaec09bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcae3fff640 1 -- 172.21.15.46:0/699441235 >> v2:172.21.15.114:3300/0 conn(0x7fcaec09c1c0 msgr2=0x7fcaec0a3f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:47:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcae3fff640 1 --2- 172.21.15.46:0/699441235 >> v2:172.21.15.114:3300/0 conn(0x7fcaec09c1c0 0x7fcaec0a3f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcae3fff640 1 -- 172.21.15.46:0/699441235 >> v1:172.21.15.46:6789/0 conn(0x7fcaec0a44c0 legacy=0x7fcaec135950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcae3fff640 10 monclient: _add_conns ranks=[3,1] 2024-09-16T04:47:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcae3fff640 1 --2- 172.21.15.46:0/699441235 >> v2:172.21.15.46:3300/0 conn(0x7fcad00036e0 0x7fcad0005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcae3fff640 10 monclient(hunting): picked mon.noname-a con 0x7fcad00036e0 addr v2:172.21.15.46:3300/0 2024-09-16T04:47:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcae3fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcad0006190 addr v1:172.21.15.114:6789/0 2024-09-16T04:47:09.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcae3fff640 1 -- 172.21.15.46:0/699441235 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaec09b580 con 0x7fcad0006190 2024-09-16T04:47:09.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcae3fff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:09.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcae3fff640 1 -- 172.21.15.46:0/699441235 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fcaec09b860 con 0x7fcad00036e0 2024-09-16T04:47:09.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcae3fff640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:09.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcaf1836640 1 -- 172.21.15.46:0/699441235 >> v1:172.21.15.114:6789/0 conn(0x7fcad0006190 legacy=0x7fcad0008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T04:47:09.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> v2:172.21.15.46:3300/0 conn(0x7fcad00036e0 0x7fcad0005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:09.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcaf2037640 10 monclient(hunting): get_auth_request con 0x7fcad00036e0 auth_method 0 2024-09-16T04:47:09.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcaf2037640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:09.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcaf2037640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:09.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcaf2037640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:09.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcaf2037640 1 -- 172.21.15.46:0/699441235 >> v2:172.21.15.46:3300/0 conn(0x7fcad00036e0 msgr2=0x7fcad0005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:47:09.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcaf2037640 1 -- 172.21.15.46:0/699441235 >> v2:172.21.15.46:3300/0 conn(0x7fcad00036e0 msgr2=0x7fcad0005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:47:09.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> v2:172.21.15.46:3300/0 conn(0x7fcad00036e0 0x7fcad0005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:47:09.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:09.965+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> v2:172.21.15.46:3300/0 conn(0x7fcad00036e0 0x7fcad0005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:47:10.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:10.166+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> v2:172.21.15.46:3300/0 conn(0x7fcad00036e0 0x7fcad0005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:10.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:10.166+0000 7fcaf1836640 1 -- 172.21.15.46:0/699441235 >> v1:172.21.15.114:6789/0 conn(0x7fcad0006190 legacy=0x7fcad0008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T04:47:10.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:10.166+0000 7fcaf2037640 10 monclient(hunting): get_auth_request con 0x7fcad00036e0 auth_method 0 2024-09-16T04:47:10.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:10.166+0000 7fcaf2037640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:10.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:10.166+0000 7fcaf2037640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:10.166+0000 7fcaf2037640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:47:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:10.166+0000 7fcaf2037640 1 -- 172.21.15.46:0/699441235 >> v2:172.21.15.46:3300/0 conn(0x7fcad00036e0 msgr2=0x7fcad0005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:47:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:10.166+0000 7fcaf2037640 1 -- 172.21.15.46:0/699441235 >> v2:172.21.15.46:3300/0 conn(0x7fcad00036e0 msgr2=0x7fcad0005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:47:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:10.166+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> v2:172.21.15.46:3300/0 conn(0x7fcad00036e0 0x7fcad0005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:47:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:10.166+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> v2:172.21.15.46:3300/0 conn(0x7fcad00036e0 0x7fcad0005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:47:10.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:10.567+0000 7fcaf1836640 1 -- 172.21.15.46:0/699441235 >> v1:172.21.15.114:6789/0 conn(0x7fcad0006190 legacy=0x7fcad0008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T04:47:10.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:10.567+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> v2:172.21.15.46:3300/0 conn(0x7fcad00036e0 0x7fcad0005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:10.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:10.567+0000 7fcaf2037640 10 monclient(hunting): get_auth_request con 0x7fcad00036e0 auth_method 0 2024-09-16T04:47:10.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:10.567+0000 7fcaf2037640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:10.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:10.567+0000 7fcaf2037640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:10.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:10.567+0000 7fcaf2037640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:47:10.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:10.567+0000 7fcaf2037640 1 -- 172.21.15.46:0/699441235 >> v2:172.21.15.46:3300/0 conn(0x7fcad00036e0 msgr2=0x7fcad0005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:47:10.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:10.567+0000 7fcaf2037640 1 -- 172.21.15.46:0/699441235 >> v2:172.21.15.46:3300/0 conn(0x7fcad00036e0 msgr2=0x7fcad0005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:47:10.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:10.567+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> v2:172.21.15.46:3300/0 conn(0x7fcad00036e0 0x7fcad0005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:47:10.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:10.567+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> v2:172.21.15.46:3300/0 conn(0x7fcad00036e0 0x7fcad0005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:47:11.275 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:47:11.276 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[40389]: starting. 2024-09-16T04:47:11.276 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[40389]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-40351 2024-09-16T04:47:11.276 INFO:teuthology.orchestra.run.smithi114.stdout:process_4_[40389]: shutting down. 2024-09-16T04:47:11.277 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:47:11.277 INFO:teuthology.orchestra.run.smithi114.stdout:******************************* 2024-09-16T04:47:11.277 INFO:teuthology.orchestra.run.smithi114.stdout:******* SUCCESS ********** 2024-09-16T04:47:11.280 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-16T04:47:11.294 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-16T04:47:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.368+0000 7fcaf1836640 1 -- 172.21.15.46:0/699441235 >> v1:172.21.15.114:6789/0 conn(0x7fcad0006190 legacy=0x7fcad0008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T04:47:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.368+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> v2:172.21.15.46:3300/0 conn(0x7fcad00036e0 0x7fcad0005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.368+0000 7fcaf2037640 10 monclient(hunting): get_auth_request con 0x7fcad00036e0 auth_method 0 2024-09-16T04:47:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.368+0000 7fcaf2037640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.368+0000 7fcaf2037640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.368+0000 7fcaf2037640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:47:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.368+0000 7fcaf2037640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.368+0000 7fcaf2037640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.368+0000 7fcaf2037640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.368+0000 7fcaf2037640 10 monclient(hunting): handle_auth_done global_id 104199 payload 274 2024-09-16T04:47:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.368+0000 7fcaf2037640 1 -- 172.21.15.46:0/699441235 >> v1:172.21.15.114:6789/0 conn(0x7fcad0006190 legacy=0x7fcad0008650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:47:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.368+0000 7fcaf2037640 10 monclient: _finish_hunting 0 2024-09-16T04:47:11.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.368+0000 7fcaf2037640 1 monclient: found mon.noname-a 2024-09-16T04:47:11.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.368+0000 7fcaf2037640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:11.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.368+0000 7fcaf2037640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:11.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.368+0000 7fcaf2037640 1 -- 172.21.15.46:0/699441235 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcad000afa0 con 0x7fcad00036e0 2024-09-16T04:47:11.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.369+0000 7fcaf2037640 1 --2- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcad00036e0 0x7fcad0005b40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcae8006fd0 tx=0x7fcae802bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e47a573726021765 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.369+0000 7fcaf1035640 1 -- 172.21.15.46:0/699441235 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae8032070 con 0x7fcad00036e0 2024-09-16T04:47:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.369+0000 7fcaf1035640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.369+0000 7fcaf1035640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.369+0000 7fcaf1035640 10 monclient: dump: 2024-09-16T04:47:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:11.378 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.369+0000 7fcaf1035640 10 monclient: _finish_auth 0 2024-09-16T04:47:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.369+0000 7fcaf1035640 10 monclient: _check_auth_tickets 2024-09-16T04:47:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.369+0000 7fcaf1035640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.369+0000 7fcaf1035640 1 -- 172.21.15.46:0/699441235 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcae8030070 con 0x7fcad00036e0 2024-09-16T04:47:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.369+0000 7fcaf429b640 5 monclient: authenticate success, global_id 104199 2024-09-16T04:47:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.369+0000 7fcaf429b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.369+0000 7fcaf1035640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.369+0000 7fcaf429b640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.369+0000 7fcaf429b640 4 set_mon_vals no callback set 2024-09-16T04:47:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.369+0000 7fcaf1035640 1 -- 172.21.15.46:0/699441235 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae8004270 con 0x7fcad00036e0 2024-09-16T04:47:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.369+0000 7fcaf1035640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.369+0000 7fcaf1035640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.369+0000 7fcaf1035640 10 monclient: dump: 2024-09-16T04:47:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:11.380 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:11.380 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:11.380 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:11.380 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:11.380 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:11.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.370+0000 7fcaf429b640 10 monclient: shutdown 2024-09-16T04:47:11.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.370+0000 7fcaf429b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:11.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.370+0000 7fcaf429b640 1 -- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcad00036e0 msgr2=0x7fcad0005b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:11.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.370+0000 7fcaf429b640 1 --2- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcad00036e0 0x7fcad0005b40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcae8006fd0 tx=0x7fcae802bf90 comp rx=0 tx=0).stop 2024-09-16T04:47:11.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.370+0000 7fcaf2838640 1 -- 172.21.15.46:0/699441235 reap_dead start 2024-09-16T04:47:11.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.370+0000 7fcaf429b640 1 -- 172.21.15.46:0/699441235 shutdown_connections 2024-09-16T04:47:11.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.370+0000 7fcaf429b640 1 --2- 172.21.15.46:0/699441235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcad00036e0 0x7fcad0005b40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.370+0000 7fcaf429b640 1 -- 172.21.15.46:0/699441235 >> 172.21.15.46:0/699441235 conn(0x7fcaec0a60c0 msgr2=0x7fcaec0a64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:11.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.370+0000 7fcaf429b640 1 -- 172.21.15.46:0/699441235 shutdown_connections 2024-09-16T04:47:11.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.370+0000 7fcaf429b640 1 -- 172.21.15.46:0/699441235 wait complete. 2024-09-16T04:47:11.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.371+0000 7fcaf429b640 10 monclient: build_initial_monmap 2024-09-16T04:47:11.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.371+0000 7fcaf429b640 10 monclient: monmap: 2024-09-16T04:47:11.381 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:11.381 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:11.381 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:11.382 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:11.382 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:11.382 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:11.382 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:11.382 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:11.382 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:11.382 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:11.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.371+0000 7fcaf429b640 1 Processor -- start 2024-09-16T04:47:11.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.371+0000 7fcaf429b640 1 -- start start 2024-09-16T04:47:11.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf429b640 10 monclient: init 2024-09-16T04:47:11.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf429b640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:11.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf429b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:47:11.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf429b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcad00036e0 0x7fcaec1cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:11.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf429b640 10 monclient(hunting): picked mon.noname-a con 0x7fcad00036e0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:11.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf429b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec1cc5e0 0x7fcaec122400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:11.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf429b640 10 monclient(hunting): picked mon.noname-c con 0x7fcaec1cc5e0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:11.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf429b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcaec1cca00 0x7fcaec122940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:11.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf429b640 10 monclient(hunting): picked mon.noname-b con 0x7fcaec1cca00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:11.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf429b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:11.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf429b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec136060 con 0x7fcad00036e0 2024-09-16T04:47:11.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf429b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:11.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf429b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec0a49d0 con 0x7fcaec1cca00 2024-09-16T04:47:11.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf429b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:11.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf429b640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fcaec1361e0 con 0x7fcaec1cc5e0 2024-09-16T04:47:11.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf429b640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:11.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf429b640 10 monclient(hunting): authenticate will time out at 1229.787538s 2024-09-16T04:47:11.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf2037640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcad00036e0 0x7fcaec1cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:11.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf1836640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec1cc5e0 0x7fcaec122400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:11.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf2838640 1 -- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcaec1cca00 msgr2=0x7fcaec122940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T04:47:11.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf1836640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec1cc5e0 0x7fcaec122400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:47682/0 (socket says 172.21.15.46:47682) 2024-09-16T04:47:11.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf2037640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcad00036e0 0x7fcaec1cc0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52880/0 (socket says 172.21.15.46:52880) 2024-09-16T04:47:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf1836640 1 -- 172.21.15.46:0/1993335237 learned_addr learned my addr 172.21.15.46:0/1993335237 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf1836640 10 monclient(hunting): get_auth_request con 0x7fcaec1cc5e0 auth_method 0 2024-09-16T04:47:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf1836640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf1836640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf1836640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf2838640 1 --2- 172.21.15.46:0/1993335237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcaec1cca00 0x7fcaec122940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:47:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf2037640 10 monclient(hunting): get_auth_request con 0x7fcad00036e0 auth_method 0 2024-09-16T04:47:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf2037640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf2037640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf2037640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf1836640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf1836640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf1836640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf2037640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf2037640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.372+0000 7fcaf2037640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcaf1836640 10 monclient(hunting): handle_auth_done global_id 94397 payload 824 2024-09-16T04:47:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcaf1836640 1 -- 172.21.15.46:0/1993335237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcaec1cca00 msgr2=0x7fcaec122940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:47:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcaf1836640 1 --2- 172.21.15.46:0/1993335237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcaec1cca00 0x7fcaec122940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcaf1836640 1 -- 172.21.15.46:0/1993335237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcad00036e0 msgr2=0x7fcaec1cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcaf1836640 1 --2- 172.21.15.46:0/1993335237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcad00036e0 0x7fcaec1cc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcaf1836640 10 monclient: _finish_hunting 0 2024-09-16T04:47:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcaf1836640 1 monclient: found mon.noname-c 2024-09-16T04:47:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcaf1836640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcaf1836640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:47:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcaf1836640 1 -- 172.21.15.46:0/1993335237 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaec126070 con 0x7fcaec1cc5e0 2024-09-16T04:47:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcaf1836640 1 --2- 172.21.15.46:0/1993335237 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec1cc5e0 0x7fcaec122400 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcadc00b920 tx=0x7fcadc00bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcae2ffd640 1 -- 172.21.15.46:0/1993335237 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc00cbd0 con 0x7fcaec1cc5e0 2024-09-16T04:47:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcae2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcae2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:47:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcae2ffd640 10 monclient: dump: 2024-09-16T04:47:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:11.388 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcae2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:47:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcae2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:47:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcae2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcae2ffd640 1 -- 172.21.15.46:0/1993335237 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcadc010030 con 0x7fcaec1cc5e0 2024-09-16T04:47:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcaf429b640 5 monclient: authenticate success, global_id 94397 2024-09-16T04:47:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcaf429b640 10 monclient: _renew_subs 2024-09-16T04:47:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcaf429b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcaf429b640 1 -- 172.21.15.46:0/1993335237 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaec126350 con 0x7fcaec1cc5e0 2024-09-16T04:47:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcae2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcae2ffd640 1 -- 172.21.15.46:0/1993335237 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc015420 con 0x7fcaec1cc5e0 2024-09-16T04:47:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcae2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcae2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcae2ffd640 10 monclient: dump: 2024-09-16T04:47:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:11.390 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcaf429b640 10 monclient: _renew_subs 2024-09-16T04:47:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcaf0834640 4 set_mon_vals no callback set 2024-09-16T04:47:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcaf429b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:11.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.373+0000 7fcaf429b640 1 -- 172.21.15.46:0/1993335237 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaec0a82f0 con 0x7fcaec1cc5e0 2024-09-16T04:47:11.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.374+0000 7fcae2ffd640 1 -- 172.21.15.46:0/1993335237 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fcadc0143d0 con 0x7fcaec1cc5e0 2024-09-16T04:47:11.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.374+0000 7fcaf429b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:11.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.374+0000 7fcaf429b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:11.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.374+0000 7fcae2ffd640 1 --2- 172.21.15.46:0/1993335237 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcad00391c0 0x7fcad003b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:11.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.375+0000 7fcaf429b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:11.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.375+0000 7fcaf429b640 1 -- 172.21.15.46:0/1993335237 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaec136590 con 0x7fcaec1cc5e0 2024-09-16T04:47:11.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.375+0000 7fcae2ffd640 1 -- 172.21.15.46:0/1993335237 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7fcadc04b130 con 0x7fcaec1cc5e0 2024-09-16T04:47:11.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.375+0000 7fcaf2037640 1 --2- 172.21.15.46:0/1993335237 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcad00391c0 0x7fcad003b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:11.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.375+0000 7fcaf2037640 10 monclient: get_auth_request con 0x7fcad00391c0 auth_method 0 2024-09-16T04:47:11.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.375+0000 7fcaf2037640 1 --2- 172.21.15.46:0/1993335237 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcad00391c0 0x7fcad003b680 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fcae8035000 tx=0x7fcae8039040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:11.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.380+0000 7fcae2ffd640 1 -- 172.21.15.46:0/1993335237 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fcadc012490 con 0x7fcaec1cc5e0 2024-09-16T04:47:11.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.380+0000 7fcae2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:11.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.380+0000 7fcae2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:11.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.627+0000 7fcaf429b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:47:11.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.627+0000 7fcaf429b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:47:11.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.627+0000 7fcaf429b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:11.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.627+0000 7fcaf429b640 1 -- 172.21.15.46:0/1993335237 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcaec126d50 con 0x7fcaec1cc5e0 2024-09-16T04:47:11.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.627+0000 7fcae2ffd640 1 -- 172.21.15.46:0/1993335237 <== mon.2 v2:172.21.15.46:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fcadc013a80 con 0x7fcaec1cc5e0 2024-09-16T04:47:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.627+0000 7fcae2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:47:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.627+0000 7fcae2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:47:11.629 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":98,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:47:11.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.629+0000 7fcaf429b640 1 -- 172.21.15.46:0/1993335237 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcad00391c0 msgr2=0x7fcad003b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:11.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.629+0000 7fcaf429b640 1 --2- 172.21.15.46:0/1993335237 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcad00391c0 0x7fcad003b680 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fcae8035000 tx=0x7fcae8039040 comp rx=0 tx=0).stop 2024-09-16T04:47:11.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.629+0000 7fcaf429b640 10 monclient: shutdown 2024-09-16T04:47:11.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.629+0000 7fcaf429b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:11.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.629+0000 7fcaf429b640 1 -- 172.21.15.46:0/1993335237 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec1cc5e0 msgr2=0x7fcaec122400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:11.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.629+0000 7fcaf429b640 1 --2- 172.21.15.46:0/1993335237 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec1cc5e0 0x7fcaec122400 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcadc00b920 tx=0x7fcadc00bde0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.629+0000 7fcaf429b640 1 -- 172.21.15.46:0/1993335237 shutdown_connections 2024-09-16T04:47:11.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.629+0000 7fcaf429b640 1 --2- 172.21.15.46:0/1993335237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcaec1cca00 0x7fcaec122940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.629+0000 7fcaf429b640 1 --2- 172.21.15.46:0/1993335237 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fcaec1cc5e0 0x7fcaec122400 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.629+0000 7fcaf429b640 1 --2- 172.21.15.46:0/1993335237 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fcad00391c0 0x7fcad003b680 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.629+0000 7fcaf429b640 1 --2- 172.21.15.46:0/1993335237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcad00036e0 0x7fcaec1cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.629+0000 7fcaf429b640 1 -- 172.21.15.46:0/1993335237 >> 172.21.15.46:0/1993335237 conn(0x7fcaec0a60c0 msgr2=0x7fcaec0a7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:11.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.629+0000 7fcaf429b640 1 -- 172.21.15.46:0/1993335237 shutdown_connections 2024-09-16T04:47:11.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.629+0000 7fcaf429b640 1 -- 172.21.15.46:0/1993335237 wait complete. 2024-09-16T04:47:11.641 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:47:11.641 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:47:11.642 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:47:11.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.742+0000 7f9812223640 10 monclient: get_monmap_and_config 2024-09-16T04:47:11.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.742+0000 7f9812223640 10 monclient: build_initial_monmap 2024-09-16T04:47:11.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.742+0000 7f9812223640 10 monclient: monmap: 2024-09-16T04:47:11.744 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:11.744 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:11.744 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:11.743145+0000 2024-09-16T04:47:11.744 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:11.743145+0000 2024-09-16T04:47:11.744 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:11.744 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:11.744 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:11.744 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:11.744 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:11.744 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:11.744 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:11.744 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:11.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.743+0000 7f9812223640 1 Processor -- start 2024-09-16T04:47:11.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.743+0000 7f9812223640 1 -- start start 2024-09-16T04:47:11.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.743+0000 7f9812223640 10 monclient: init 2024-09-16T04:47:11.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.743+0000 7f9812223640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:11.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.743+0000 7f9812223640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-16T04:47:11.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.743+0000 7f9812223640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f980c06e160 0x7f980c0a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:11.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.743+0000 7f9812223640 10 monclient(hunting): picked mon.noname-c con 0x7f980c06e160 addr v2:172.21.15.114:3300/0 2024-09-16T04:47:11.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.743+0000 7f9812223640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f980c0a2ee0 addr v1:172.21.15.114:6789/0 2024-09-16T04:47:11.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.743+0000 7f9812223640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f980c137050 0x7f980c13ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:11.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.743+0000 7f9812223640 10 monclient(hunting): picked mon.noname-a con 0x7f980c137050 addr v2:172.21.15.46:3300/0 2024-09-16T04:47:11.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.743+0000 7f9812223640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f980c0a3fd0 con 0x7f980c0a2ee0 2024-09-16T04:47:11.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.743+0000 7f9812223640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:11.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.743+0000 7f9812223640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f980c0a3300 con 0x7f980c137050 2024-09-16T04:47:11.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.743+0000 7f9812223640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:11.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.743+0000 7f9812223640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f980c06e540 con 0x7f980c06e160 2024-09-16T04:47:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.743+0000 7f9812223640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.743+0000 7f9812223640 10 monclient(hunting): authenticate will time out at 1230.158965s 2024-09-16T04:47:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f980c137050 0x7f980c13ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980affd640 1 -- >> v1:172.21.15.114:6789/0 conn(0x7f980c0a2ee0 legacy=0x7f980c136940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T04:47:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980b7fe640 1 -- >> v2:172.21.15.114:3300/0 conn(0x7f980c06e160 msgr2=0x7f980c0a29a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T04:47:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f980c137050 0x7f980c13ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52886/0 (socket says 172.21.15.46:52886) 2024-09-16T04:47:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980bfff640 1 -- 172.21.15.46:0/1772230549 learned_addr learned my addr 172.21.15.46:0/1772230549 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980bfff640 10 monclient(hunting): get_auth_request con 0x7f980c137050 auth_method 0 2024-09-16T04:47:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980b7fe640 1 --2- 172.21.15.46:0/1772230549 >> v2:172.21.15.114:3300/0 conn(0x7f980c06e160 0x7f980c0a29a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:47:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980bfff640 10 monclient(hunting): handle_auth_done global_id 104205 payload 274 2024-09-16T04:47:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980bfff640 1 -- 172.21.15.46:0/1772230549 >> v2:172.21.15.114:3300/0 conn(0x7f980c06e160 msgr2=0x7f980c0a29a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:47:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980bfff640 1 --2- 172.21.15.46:0/1772230549 >> v2:172.21.15.114:3300/0 conn(0x7f980c06e160 0x7f980c0a29a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980bfff640 1 -- 172.21.15.46:0/1772230549 >> v1:172.21.15.114:6789/0 conn(0x7f980c0a2ee0 legacy=0x7f980c136940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:47:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:47:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980bfff640 1 monclient: found mon.noname-a 2024-09-16T04:47:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.744+0000 7f980bfff640 1 -- 172.21.15.46:0/1772230549 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f980c0a0bd0 con 0x7f980c137050 2024-09-16T04:47:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f980bfff640 1 --2- 172.21.15.46:0/1772230549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c137050 0x7f980c13ec80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f980c068700 tx=0x7f980402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a22ecc10d3fe14f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f980a7fc640 1 -- 172.21.15.46:0/1772230549 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9804036070 con 0x7f980c137050 2024-09-16T04:47:11.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f980a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:11.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f980a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:11.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f980a7fc640 10 monclient: dump: 2024-09-16T04:47:11.756 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:11.756 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:11.756 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.756 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.756 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:11.756 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:11.756 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:11.756 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:11.756 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:11.757 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:11.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f980a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:47:11.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f980a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:47:11.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f980a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:11.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f980a7fc640 1 -- 172.21.15.46:0/1772230549 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9804031400 con 0x7f980c137050 2024-09-16T04:47:11.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f9812223640 5 monclient: authenticate success, global_id 104205 2024-09-16T04:47:11.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f9812223640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:11.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f980a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:11.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f9812223640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:11.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f980a7fc640 1 -- 172.21.15.46:0/1772230549 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98040317d0 con 0x7f980c137050 2024-09-16T04:47:11.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f980a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:11.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f9812223640 4 set_mon_vals no callback set 2024-09-16T04:47:11.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f980a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:11.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f980a7fc640 10 monclient: dump: 2024-09-16T04:47:11.758 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:11.758 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:11.758 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.758 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.758 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:11.758 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:11.758 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:11.758 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:11.758 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:11.758 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:11.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f9812223640 10 monclient: shutdown 2024-09-16T04:47:11.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f9812223640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:11.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f9812223640 1 -- 172.21.15.46:0/1772230549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c137050 msgr2=0x7f980c13ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:11.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f9812223640 1 --2- 172.21.15.46:0/1772230549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c137050 0x7f980c13ec80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f980c068700 tx=0x7f980402a610 comp rx=0 tx=0).stop 2024-09-16T04:47:11.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f9812223640 1 -- 172.21.15.46:0/1772230549 shutdown_connections 2024-09-16T04:47:11.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f9812223640 1 --2- 172.21.15.46:0/1772230549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c137050 0x7f980c13ec80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f9812223640 1 --2- 172.21.15.46:0/1772230549 >> v2:172.21.15.114:3300/0 conn(0x7f980c06e160 0x7f980c0a29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f9812223640 1 -- 172.21.15.46:0/1772230549 >> 172.21.15.46:0/1772230549 conn(0x7f980c0a5c10 msgr2=0x7f980c0a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:11.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f9812223640 1 -- 172.21.15.46:0/1772230549 shutdown_connections 2024-09-16T04:47:11.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.745+0000 7f9812223640 1 -- 172.21.15.46:0/1772230549 wait complete. 2024-09-16T04:47:11.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.746+0000 7f9812223640 10 monclient: build_initial_monmap 2024-09-16T04:47:11.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.746+0000 7f9812223640 10 monclient: monmap: 2024-09-16T04:47:11.759 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:11.759 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:11.759 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:11.759 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:11.759 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:11.759 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:11.760 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:11.760 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:11.760 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:11.760 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:11.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.746+0000 7f9812223640 1 Processor -- start 2024-09-16T04:47:11.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.746+0000 7f9812223640 1 -- start start 2024-09-16T04:47:11.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.746+0000 7f9812223640 10 monclient: init 2024-09-16T04:47:11.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f9812223640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:11.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f9812223640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:47:11.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f9812223640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f980c06e160 0x7f980c1cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:11.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f9812223640 10 monclient(hunting): picked mon.noname-c con 0x7f980c06e160 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:11.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f9812223640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f980c0a2ee0 0x7f980c135db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:11.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f9812223640 10 monclient(hunting): picked mon.noname-b con 0x7f980c0a2ee0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:11.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f9812223640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c137050 0x7f980c1362f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:11.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f9812223640 10 monclient(hunting): picked mon.noname-a con 0x7f980c137050 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:11.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f9812223640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:11.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f9812223640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f980c0a34a0 con 0x7f980c137050 2024-09-16T04:47:11.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f9812223640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:11.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f9812223640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f980c0a5130 con 0x7f980c0a2ee0 2024-09-16T04:47:11.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f9812223640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:11.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f9812223640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f980c1cc180 con 0x7f980c06e160 2024-09-16T04:47:11.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f9812223640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:11.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f9812223640 10 monclient(hunting): authenticate will time out at 1230.162382s 2024-09-16T04:47:11.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980b7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f980c06e160 0x7f980c1cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:11.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c137050 0x7f980c1362f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:11.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980affd640 1 -- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f980c0a2ee0 msgr2=0x7f980c135db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T04:47:11.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980b7fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f980c06e160 0x7f980c1cbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:47686/0 (socket says 172.21.15.46:47686) 2024-09-16T04:47:11.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980bfff640 10 monclient(hunting): get_auth_request con 0x7f980c137050 auth_method 0 2024-09-16T04:47:11.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980b7fe640 1 -- 172.21.15.46:0/1485604209 learned_addr learned my addr 172.21.15.46:0/1485604209 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:11.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:11.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:11.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:11.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980affd640 1 --2- 172.21.15.46:0/1485604209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f980c0a2ee0 0x7f980c135db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:47:11.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980b7fe640 10 monclient(hunting): get_auth_request con 0x7f980c06e160 auth_method 0 2024-09-16T04:47:11.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:11.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:11.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:11.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:11.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:11.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:11.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:11.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980bfff640 10 monclient(hunting): handle_auth_done global_id 104208 payload 824 2024-09-16T04:47:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980bfff640 1 -- 172.21.15.46:0/1485604209 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f980c06e160 msgr2=0x7f980c1cbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980bfff640 1 --2- 172.21.15.46:0/1485604209 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f980c06e160 0x7f980c1cbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980bfff640 1 -- 172.21.15.46:0/1485604209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f980c0a2ee0 msgr2=0x7f980c135db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:47:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980bfff640 1 --2- 172.21.15.46:0/1485604209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f980c0a2ee0 0x7f980c135db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:47:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980bfff640 1 monclient: found mon.noname-a 2024-09-16T04:47:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980bfff640 1 -- 172.21.15.46:0/1485604209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f980c125960 con 0x7f980c137050 2024-09-16T04:47:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.747+0000 7f980b7fe640 1 --2- 172.21.15.46:0/1485604209 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f980c06e160 0x7f980c1cbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f980bfff640 1 --2- 172.21.15.46:0/1485604209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c137050 0x7f980c1362f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9804030000 tx=0x7f9804004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f97f3fff640 1 -- 172.21.15.46:0/1485604209 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9804036040 con 0x7f980c137050 2024-09-16T04:47:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f97f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f97f3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f97f3fff640 10 monclient: dump: 2024-09-16T04:47:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:11.764 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f97f3fff640 10 monclient: _finish_auth 0 2024-09-16T04:47:11.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f97f3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:47:11.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f97f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:11.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f9812223640 5 monclient: authenticate success, global_id 104208 2024-09-16T04:47:11.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f97f3fff640 1 -- 172.21.15.46:0/1485604209 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9804003b90 con 0x7f980c137050 2024-09-16T04:47:11.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f97f3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:11.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f97f3fff640 1 -- 172.21.15.46:0/1485604209 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9804031e40 con 0x7f980c137050 2024-09-16T04:47:11.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f9812223640 10 monclient: _renew_subs 2024-09-16T04:47:11.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f9812223640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:11.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f9812223640 1 -- 172.21.15.46:0/1485604209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f980c0a8200 con 0x7f980c137050 2024-09-16T04:47:11.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f9809ffb640 4 set_mon_vals no callback set 2024-09-16T04:47:11.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f97f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:11.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f97f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:11.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f97f3fff640 10 monclient: dump: 2024-09-16T04:47:11.765 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:11.766 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f9812223640 10 monclient: _renew_subs 2024-09-16T04:47:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f9812223640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f9812223640 1 -- 172.21.15.46:0/1485604209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f980c1271f0 con 0x7f980c137050 2024-09-16T04:47:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f9812223640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f97f3fff640 1 -- 172.21.15.46:0/1485604209 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f9804004030 con 0x7f980c137050 2024-09-16T04:47:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.748+0000 7f9812223640 1 --2- 172.21.15.46:0/1485604209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c123820 0x7f980c123c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f9812223640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f9812223640 1 -- 172.21.15.46:0/1485604209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f980c0a3620 con 0x7f980c123820 2024-09-16T04:47:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f980b7fe640 1 --2- 172.21.15.46:0/1485604209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c123820 0x7f980c123c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f980b7fe640 10 monclient: get_auth_request con 0x7f980c123820 auth_method 0 2024-09-16T04:47:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f980b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f980b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f980b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f980b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f980b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f980b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f97f3fff640 1 --2- 172.21.15.46:0/1485604209 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f97f8038390 0x7f97f803a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f97f3fff640 1 -- 172.21.15.46:0/1485604209 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7f980406c690 con 0x7f980c137050 2024-09-16T04:47:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f980b7fe640 10 monclient(hunting): handle_auth_done global_id 104211 payload 824 2024-09-16T04:47:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f980b7fe640 1 -- 172.21.15.46:0/1485604209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f980c0a8930 con 0x7f980c123820 2024-09-16T04:47:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f980affd640 1 --2- 172.21.15.46:0/1485604209 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f97f8038390 0x7f97f803a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f980affd640 10 monclient: get_auth_request con 0x7f97f8038390 auth_method 0 2024-09-16T04:47:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f980b7fe640 1 --2- 172.21.15.46:0/1485604209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c123820 0x7f980c123c00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0053f0 tx=0x7f97fc0058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f97f3fff640 1 -- 172.21.15.46:0/1485604209 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97fc016070 con 0x7f980c123820 2024-09-16T04:47:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f97f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f97f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f97f3fff640 10 monclient: dump: 2024-09-16T04:47:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:11.769 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f980affd640 1 --2- 172.21.15.46:0/1485604209 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f97f8038390 0x7f97f803a850 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9800006fd0 tx=0x7f9800006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f97f3fff640 1 -- 172.21.15.46:0/1485604209 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f97fc00eb30 con 0x7f980c123820 2024-09-16T04:47:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f97f3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.749+0000 7f97f3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.750+0000 7f97f3fff640 1 -- 172.21.15.46:0/1485604209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c123820 msgr2=0x7f980c123c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.750+0000 7f97f3fff640 1 --2- 172.21.15.46:0/1485604209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c123820 0x7f980c123c00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0053f0 tx=0x7f97fc0058b0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.750+0000 7f97f3fff640 1 -- 172.21.15.46:0/1485604209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c123820 msgr2=0x7f980c123c00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:47:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.750+0000 7f97f3fff640 1 --2- 172.21.15.46:0/1485604209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c123820 0x7f980c123c00 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0053f0 tx=0x7f97fc0058b0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f9812223640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:47:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f9812223640 1 --2- 172.21.15.46:0/1485604209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c124140 0x7f980c11f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f9812223640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f9812223640 1 -- 172.21.15.46:0/1485604209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f980c127010 con 0x7f980c124140 2024-09-16T04:47:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f980b7fe640 1 --2- 172.21.15.46:0/1485604209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c124140 0x7f980c11f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f980b7fe640 10 monclient: get_auth_request con 0x7f980c124140 auth_method 0 2024-09-16T04:47:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f980b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f980b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f980b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f980b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f980b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f980b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f980b7fe640 10 monclient(hunting): handle_auth_done global_id 104214 payload 824 2024-09-16T04:47:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f980b7fe640 1 -- 172.21.15.46:0/1485604209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f980c136880 con 0x7f980c124140 2024-09-16T04:47:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f980b7fe640 1 --2- 172.21.15.46:0/1485604209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c124140 0x7f980c11f760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0053f0 tx=0x7f97fc0059d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f97f3fff640 1 -- 172.21.15.46:0/1485604209 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97fc016040 con 0x7f980c124140 2024-09-16T04:47:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f97f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f97f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f97f3fff640 10 monclient: dump: 2024-09-16T04:47:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:11.773 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f97f3fff640 1 -- 172.21.15.46:0/1485604209 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4869 (secure 0 0 0) 0x7f97fc005a40 con 0x7f980c124140 2024-09-16T04:47:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f97f3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:47:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f97f3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:47:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f97f3fff640 1 -- 172.21.15.46:0/1485604209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c124140 msgr2=0x7f980c11f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f97f3fff640 1 --2- 172.21.15.46:0/1485604209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c124140 0x7f980c11f760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0053f0 tx=0x7f97fc0059d0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f97f3fff640 1 -- 172.21.15.46:0/1485604209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c124140 msgr2=0x7f980c11f760 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:47:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.756+0000 7f97f3fff640 1 --2- 172.21.15.46:0/1485604209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c124140 0x7f980c11f760 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0053f0 tx=0x7f97fc0059d0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.774 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:47:11.774 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:47:11.774 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:47:11.774 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:47:11.774 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 98, 2024-09-16T04:47:11.774 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 24813, 2024-09-16T04:47:11.774 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:47:11.774 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:47:11.774 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:47:11.774 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:11.775 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:47:11.775 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:47:11.775 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:47:11.775 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:47:11.775 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:47:11.775 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:47:11.775 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:47:11.775 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:47:11.775 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:47:11.775 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:47:11.775 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:47:11.775 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:47:11.775 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:47:11.775 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:47:11.776 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:47:11.776 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:11.776 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:47:11.776 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:47:11.776 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:47:11.776 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:47:11.776 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:47:11.776 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:47:11.776 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:47:11.776 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:47:11.776 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:47:11.776 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:47:11.776 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:47:11.776 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:47:11.777 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:47:11.777 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:11.777 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.777 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:47:11.777 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:47:11.777 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:47:11.777 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:47:11.777 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:47:11.777 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:47:11.777 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:47:11.777 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:47:11.777 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:47:11.777 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:47:11.778 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:47:11.778 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:47:11.778 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:47:11.778 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:47:11.778 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:47:11.778 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:47:11.778 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:47:11.778 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:47:11.778 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:47:11.778 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:47:11.778 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:47:11.778 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:47:11.778 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:47:11.778 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:47:11.779 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:47:11.779 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:47:11.779 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:47:11.779 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:47:11.779 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:11.779 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:47:11.779 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.779 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:47:11.779 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.779 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:47:11.779 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:47:11.779 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:47:11.779 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:47:11.780 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.780 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:47:11.780 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:47:11.780 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:11.780 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.780 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.780 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:47:11.780 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:47:11.780 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:11.780 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:11.780 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:11.780 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.780 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:47:11.781 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:47:11.781 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:47:11.781 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:47:11.781 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:47:11.781 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.781 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.781 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:47:11.781 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:47:11.781 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:47:11.781 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:47:11.781 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.781 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:47:11.781 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:47:11.781 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:11.782 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.782 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.782 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:47:11.782 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:47:11.782 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:11.782 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:11.782 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:11.782 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.782 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:47:11.782 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:47:11.782 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:47:11.782 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:47:11.782 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:47:11.782 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.782 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.783 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:47:11.783 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:47:11.783 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:47:11.783 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:47:11.783 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.783 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:47:11.783 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:47:11.783 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:11.783 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.783 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.783 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:47:11.783 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:47:11.783 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:11.784 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:11.784 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:11.784 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.784 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:47:11.784 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:47:11.784 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:47:11.784 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:47:11.784 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:47:11.784 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:11.784 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:11.784 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.784 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:47:11.784 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:47:11.785 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.785 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:47:11.785 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:47:11.785 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:47:11.785 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:11.785 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:11.785 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:47:11.785 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.785 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:47:11.785 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:47:11.785 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:47:11.785 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:11.785 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:11.785 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:47:11.786 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.786 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:47:11.786 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:47:11.786 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:47:11.786 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:11.786 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:11.786 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:47:11.786 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.786 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:47:11.786 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:47:11.786 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:47:11.786 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:11.786 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:11.787 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.787 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:47:11.787 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:47:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.757+0000 7f9812223640 1 -- 172.21.15.46:0/1485604209 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f97f8038390 msgr2=0x7f97f803a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.757+0000 7f9812223640 1 --2- 172.21.15.46:0/1485604209 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f97f8038390 0x7f97f803a850 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9800006fd0 tx=0x7f9800006d80 comp rx=0 tx=0).stop 2024-09-16T04:47:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.757+0000 7f9812223640 10 monclient: shutdown 2024-09-16T04:47:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.757+0000 7f9812223640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.757+0000 7f9812223640 1 -- 172.21.15.46:0/1485604209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c137050 msgr2=0x7f980c1362f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.757+0000 7f9812223640 1 --2- 172.21.15.46:0/1485604209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f980c137050 0x7f980c1362f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9804030000 tx=0x7f9804004a40 comp rx=0 tx=0).stop 2024-09-16T04:47:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.757+0000 7f980bfff640 1 -- 172.21.15.46:0/1485604209 reap_dead start 2024-09-16T04:47:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.757+0000 7f980bfff640 1 -- 172.21.15.46:0/1485604209 reap_dead start 2024-09-16T04:47:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.757+0000 7f9812223640 1 -- 172.21.15.46:0/1485604209 shutdown_connections 2024-09-16T04:47:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.757+0000 7f9812223640 1 -- 172.21.15.46:0/1485604209 >> 172.21.15.46:0/1485604209 conn(0x7f980c0a5c10 msgr2=0x7f980c0a77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.757+0000 7f9812223640 1 -- 172.21.15.46:0/1485604209 shutdown_connections 2024-09-16T04:47:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.757+0000 7f9812223640 1 -- 172.21.15.46:0/1485604209 wait complete. 2024-09-16T04:47:11.789 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:47:11.790 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T04:47:11.790 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:47:11.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.890+0000 7f7ad00f1640 10 monclient: get_monmap_and_config 2024-09-16T04:47:11.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.890+0000 7f7ad00f1640 10 monclient: build_initial_monmap 2024-09-16T04:47:11.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.890+0000 7f7ad00f1640 10 monclient: monmap: 2024-09-16T04:47:11.891 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:11.891 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:11.892 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:11.891011+0000 2024-09-16T04:47:11.892 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:11.891011+0000 2024-09-16T04:47:11.892 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:11.892 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:11.892 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:11.892 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:11.893 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:11.893 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:11.893 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:11.893 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:11.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7ad00f1640 1 Processor -- start 2024-09-16T04:47:11.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7ad00f1640 1 -- start start 2024-09-16T04:47:11.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7ad00f1640 10 monclient: init 2024-09-16T04:47:11.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7ad00f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:11.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7ad00f1640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-16T04:47:11.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7ad00f1640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac809f810 0x7f7ac809e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:11.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7ad00f1640 10 monclient(hunting): picked mon.noname-b con 0x7f7ac809f810 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:11.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7ad00f1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7ac809fbf0 0x7f7ac809ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:11.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7ad00f1640 10 monclient(hunting): picked mon.noname-a con 0x7f7ac809fbf0 addr v2:172.21.15.46:3300/0 2024-09-16T04:47:11.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7ad00f1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7ac8137600 addr v1:172.21.15.114:6789/0 2024-09-16T04:47:11.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7ad00f1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ac809c310 con 0x7f7ac8137600 2024-09-16T04:47:11.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7ad00f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:11.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7ad00f1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f7ac809d290 con 0x7f7ac809fbf0 2024-09-16T04:47:11.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7ad00f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:11.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7ad00f1640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f7ac809c840 con 0x7f7ac809f810 2024-09-16T04:47:11.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7ad00f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:11.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7ad00f1640 10 monclient(hunting): authenticate will time out at 1230.306844s 2024-09-16T04:47:11.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7acd68c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7ac809fbf0 0x7f7ac809ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:11.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7acde8d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac809f810 0x7f7ac809e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:11.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7acd68c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7ac809fbf0 0x7f7ac809ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52932/0 (socket says 172.21.15.46:52932) 2024-09-16T04:47:11.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7acd68c640 1 -- 172.21.15.46:0/287849970 learned_addr learned my addr 172.21.15.46:0/287849970 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:11.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7acd68c640 10 monclient(hunting): get_auth_request con 0x7f7ac809fbf0 auth_method 0 2024-09-16T04:47:11.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acd68c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:11.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acd68c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:11.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acd68c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:11.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.891+0000 7f7ace68e640 1 -- 172.21.15.46:0/287849970 >> v1:172.21.15.114:6789/0 conn(0x7f7ac8137600 legacy=0x7f7ac81399f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T04:47:11.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acde8d640 10 monclient(hunting): get_auth_request con 0x7f7ac809f810 auth_method 0 2024-09-16T04:47:11.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acde8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:11.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acde8d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:11.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acde8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:11.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acd68c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:11.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acd68c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:11.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acd68c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:11.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acde8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:11.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acde8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:11.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acde8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:11.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acde8d640 10 monclient(hunting): handle_auth_done global_id 94403 payload 274 2024-09-16T04:47:11.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acde8d640 1 -- 172.21.15.46:0/287849970 >> v2:172.21.15.46:3300/0 conn(0x7f7ac809fbf0 msgr2=0x7f7ac809ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:11.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acde8d640 1 --2- 172.21.15.46:0/287849970 >> v2:172.21.15.46:3300/0 conn(0x7f7ac809fbf0 0x7f7ac809ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acde8d640 1 -- 172.21.15.46:0/287849970 >> v1:172.21.15.114:6789/0 conn(0x7f7ac8137600 legacy=0x7f7ac81399f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:47:11.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acde8d640 10 monclient: _finish_hunting 0 2024-09-16T04:47:11.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acde8d640 1 monclient: found mon.noname-b 2024-09-16T04:47:11.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acde8d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:11.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acde8d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:47:11.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acde8d640 1 -- 172.21.15.46:0/287849970 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ac809d850 con 0x7f7ac809f810 2024-09-16T04:47:11.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acd68c640 1 --2- 172.21.15.46:0/287849970 >> v2:172.21.15.46:3300/0 conn(0x7f7ac809fbf0 0x7f7ac809ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:11.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.892+0000 7f7acde8d640 1 --2- 172.21.15.46:0/287849970 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac809f810 0x7f7ac809e2c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400a940 tx=0x7f7ac402b460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=facd3e53863c9208 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:11.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7acce8b640 1 -- 172.21.15.46:0/287849970 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac400b040 con 0x7f7ac809f810 2024-09-16T04:47:11.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7acce8b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:11.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7acce8b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:47:11.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7acce8b640 10 monclient: dump: 2024-09-16T04:47:11.912 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:11.912 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:11.912 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.912 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.912 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:11.912 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:11.912 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:11.912 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:11.912 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:11.912 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:11.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7acce8b640 10 monclient: _finish_auth 0 2024-09-16T04:47:11.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7acce8b640 10 monclient: _check_auth_tickets 2024-09-16T04:47:11.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7acce8b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:11.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7ad00f1640 5 monclient: authenticate success, global_id 94403 2024-09-16T04:47:11.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7ad00f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:11.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7acce8b640 1 -- 172.21.15.46:0/287849970 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ac4030710 con 0x7f7ac809f810 2024-09-16T04:47:11.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7acce8b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:11.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7ad00f1640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:11.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7acce8b640 1 -- 172.21.15.46:0/287849970 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4030ae0 con 0x7f7ac809f810 2024-09-16T04:47:11.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7acce8b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:11.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7ad00f1640 4 set_mon_vals no callback set 2024-09-16T04:47:11.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7acce8b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:11.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7acce8b640 10 monclient: dump: 2024-09-16T04:47:11.913 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:11.914 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:11.914 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.914 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.914 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:11.914 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:11.914 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:11.914 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:11.914 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:11.914 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:11.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7ad00f1640 10 monclient: shutdown 2024-09-16T04:47:11.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7ad00f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:11.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7ad00f1640 1 -- 172.21.15.46:0/287849970 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac809f810 msgr2=0x7f7ac809e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:11.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7ad00f1640 1 --2- 172.21.15.46:0/287849970 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac809f810 0x7f7ac809e2c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400a940 tx=0x7f7ac402b460 comp rx=0 tx=0).stop 2024-09-16T04:47:11.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7ad00f1640 1 -- 172.21.15.46:0/287849970 shutdown_connections 2024-09-16T04:47:11.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7ad00f1640 1 --2- 172.21.15.46:0/287849970 >> v2:172.21.15.46:3300/0 conn(0x7f7ac809fbf0 0x7f7ac809ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7ad00f1640 1 --2- 172.21.15.46:0/287849970 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac809f810 0x7f7ac809e2c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7ad00f1640 1 -- 172.21.15.46:0/287849970 >> 172.21.15.46:0/287849970 conn(0x7f7ac80a5960 msgr2=0x7f7ac80a1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:11.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7ad00f1640 1 -- 172.21.15.46:0/287849970 shutdown_connections 2024-09-16T04:47:11.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.893+0000 7f7ad00f1640 1 -- 172.21.15.46:0/287849970 wait complete. 2024-09-16T04:47:11.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.894+0000 7f7ad00f1640 10 monclient: build_initial_monmap 2024-09-16T04:47:11.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.894+0000 7f7ad00f1640 10 monclient: monmap: 2024-09-16T04:47:11.915 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:11.915 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:11.916 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:11.916 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:11.916 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:11.916 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:11.916 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:11.916 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:11.916 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:11.916 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:11.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.894+0000 7f7ad00f1640 1 Processor -- start 2024-09-16T04:47:11.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.894+0000 7f7ad00f1640 1 -- start start 2024-09-16T04:47:11.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.894+0000 7f7ad00f1640 10 monclient: init 2024-09-16T04:47:11.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ad00f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:11.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ad00f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:47:11.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ad00f1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac809f810 0x7f7ac813b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:11.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ad00f1640 10 monclient(hunting): picked mon.noname-b con 0x7f7ac809f810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:11.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ad00f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ac809fbf0 0x7f7ac813b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:11.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ad00f1640 10 monclient(hunting): picked mon.noname-a con 0x7f7ac809fbf0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:11.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ad00f1640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac8137600 0x7f7ac813bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:11.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ad00f1640 10 monclient(hunting): picked mon.noname-c con 0x7f7ac8137600 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:11.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ad00f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:11.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ad00f1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac80a0510 con 0x7f7ac809fbf0 2024-09-16T04:47:11.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ad00f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:11.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ad00f1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac809f090 con 0x7f7ac809f810 2024-09-16T04:47:11.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ad00f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:11.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ad00f1640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f7ac81308c0 con 0x7f7ac8137600 2024-09-16T04:47:11.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ad00f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:11.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ad00f1640 10 monclient(hunting): authenticate will time out at 1230.310274s 2024-09-16T04:47:11.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acd68c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ac809fbf0 0x7f7ac813b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:11.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ace68e640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac8137600 0x7f7ac813bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:11.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acd68c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ac809fbf0 0x7f7ac813b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52940/0 (socket says 172.21.15.46:52940) 2024-09-16T04:47:11.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acd68c640 1 -- 172.21.15.46:0/3848395323 learned_addr learned my addr 172.21.15.46:0/3848395323 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:11.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acd68c640 10 monclient(hunting): get_auth_request con 0x7f7ac809fbf0 auth_method 0 2024-09-16T04:47:11.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acd68c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:11.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acd68c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:11.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acd68c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:11.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ace68e640 10 monclient(hunting): get_auth_request con 0x7f7ac8137600 auth_method 0 2024-09-16T04:47:11.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ace68e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:11.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ace68e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:11.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ace68e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:11.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acde8d640 1 -- 172.21.15.46:0/3848395323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac809f810 msgr2=0x7f7ac813b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T04:47:11.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acde8d640 1 --2- 172.21.15.46:0/3848395323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac809f810 0x7f7ac813b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:47:11.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acd68c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:11.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acd68c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:11.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acd68c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:11.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ace68e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:11.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ace68e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:11.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ace68e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:11.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acd68c640 10 monclient(hunting): handle_auth_done global_id 104220 payload 824 2024-09-16T04:47:11.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acd68c640 1 -- 172.21.15.46:0/3848395323 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac8137600 msgr2=0x7f7ac813bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:11.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acd68c640 1 --2- 172.21.15.46:0/3848395323 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac8137600 0x7f7ac813bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acd68c640 1 -- 172.21.15.46:0/3848395323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac809f810 msgr2=0x7f7ac813b110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:47:11.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acd68c640 1 --2- 172.21.15.46:0/3848395323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac809f810 0x7f7ac813b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acd68c640 10 monclient: _finish_hunting 0 2024-09-16T04:47:11.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acd68c640 1 monclient: found mon.noname-a 2024-09-16T04:47:11.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acd68c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acd68c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acd68c640 1 -- 172.21.15.46:0/3848395323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ac811c560 con 0x7f7ac809fbf0 2024-09-16T04:47:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7acd68c640 1 --2- 172.21.15.46:0/3848395323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ac809fbf0 0x7f7ac813b650 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7ab800e9d0 tx=0x7f7ab800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7abe7fc640 1 -- 172.21.15.46:0/3848395323 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8017070 con 0x7f7ac809fbf0 2024-09-16T04:47:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7abe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7abe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7abe7fc640 10 monclient: dump: 2024-09-16T04:47:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:11.921 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7abe7fc640 10 monclient: _finish_auth 0 2024-09-16T04:47:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7abe7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:47:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7abe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7abe7fc640 1 -- 172.21.15.46:0/3848395323 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ab800ca60 con 0x7f7ac809fbf0 2024-09-16T04:47:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ad00f1640 5 monclient: authenticate success, global_id 104220 2024-09-16T04:47:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ad00f1640 10 monclient: _renew_subs 2024-09-16T04:47:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ad00f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7ad00f1640 1 -- 172.21.15.46:0/3848395323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ac811c840 con 0x7f7ac809fbf0 2024-09-16T04:47:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7abe7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7abe7fc640 1 -- 172.21.15.46:0/3848395323 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab800ce10 con 0x7f7ac809fbf0 2024-09-16T04:47:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7abe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.896+0000 7f7abe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.896+0000 7f7abe7fc640 10 monclient: dump: 2024-09-16T04:47:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:11.923 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:11.923 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:11.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.895+0000 7f7abffff640 4 set_mon_vals no callback set 2024-09-16T04:47:11.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.896+0000 7f7ad00f1640 10 monclient: _renew_subs 2024-09-16T04:47:11.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.896+0000 7f7ad00f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:11.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.896+0000 7f7ad00f1640 1 -- 172.21.15.46:0/3848395323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ac80a4410 con 0x7f7ac809fbf0 2024-09-16T04:47:11.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.896+0000 7f7ad00f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:11.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.896+0000 7f7ad00f1640 1 --2- 172.21.15.46:0/3848395323 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac81cc0d0 0x7f7ac81cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:11.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.896+0000 7f7ad00f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:11.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.896+0000 7f7ad00f1640 1 -- 172.21.15.46:0/3848395323 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f7ac80a0690 con 0x7f7ac81cc0d0 2024-09-16T04:47:11.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.896+0000 7f7acde8d640 1 --2- 172.21.15.46:0/3848395323 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac81cc0d0 0x7f7ac81cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:11.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7abe7fc640 1 -- 172.21.15.46:0/3848395323 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f7ab801a070 con 0x7f7ac809fbf0 2024-09-16T04:47:11.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7acde8d640 10 monclient: get_auth_request con 0x7f7ac81cc0d0 auth_method 0 2024-09-16T04:47:11.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7acde8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7acde8d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7acde8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7acde8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7acde8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7acde8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7acde8d640 10 monclient(hunting): handle_auth_done global_id 94409 payload 824 2024-09-16T04:47:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7acde8d640 1 -- 172.21.15.46:0/3848395323 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7ac809e2c0 con 0x7f7ac81cc0d0 2024-09-16T04:47:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7abe7fc640 1 --2- 172.21.15.46:0/3848395323 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7aac038360 0x7f7aac03a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7abe7fc640 1 -- 172.21.15.46:0/3848395323 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7f7ab804b8b0 con 0x7f7ac809fbf0 2024-09-16T04:47:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7acde8d640 1 --2- 172.21.15.46:0/3848395323 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac81cc0d0 0x7f7ac81cc4b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7ac80a0e90 tx=0x7f7ac4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7abe7fc640 1 -- 172.21.15.46:0/3848395323 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4006e10 con 0x7f7ac81cc0d0 2024-09-16T04:47:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7abe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7abe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7abe7fc640 10 monclient: dump: 2024-09-16T04:47:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:11.925 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7ace68e640 1 --2- 172.21.15.46:0/3848395323 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7aac038360 0x7f7aac03a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7ace68e640 10 monclient: get_auth_request con 0x7f7aac038360 auth_method 0 2024-09-16T04:47:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7abe7fc640 1 -- 172.21.15.46:0/3848395323 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7ac4004220 con 0x7f7ac81cc0d0 2024-09-16T04:47:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7abe7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7abe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7abe7fc640 1 -- 172.21.15.46:0/3848395323 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac81cc0d0 msgr2=0x7f7ac81cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7abe7fc640 1 --2- 172.21.15.46:0/3848395323 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac81cc0d0 0x7f7ac81cc4b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7ac80a0e90 tx=0x7f7ac4004730 comp rx=0 tx=0).stop 2024-09-16T04:47:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7abe7fc640 1 -- 172.21.15.46:0/3848395323 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac81cc0d0 msgr2=0x7f7ac81cc4b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:47:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.897+0000 7f7abe7fc640 1 --2- 172.21.15.46:0/3848395323 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac81cc0d0 0x7f7ac81cc4b0 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7ac80a0e90 tx=0x7f7ac4004730 comp rx=0 tx=0).stop 2024-09-16T04:47:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.898+0000 7f7ace68e640 1 --2- 172.21.15.46:0/3848395323 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7aac038360 0x7f7aac03a820 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7ac813cd60 tx=0x7f7ac000d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.903+0000 7f7ad00f1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:47:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.903+0000 7f7ad00f1640 1 --2- 172.21.15.46:0/3848395323 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac81cc9f0 0x7f7ac8133870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7ad00f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7ad00f1640 1 -- 172.21.15.46:0/3848395323 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f7ac81314d0 con 0x7f7ac81cc9f0 2024-09-16T04:47:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7acde8d640 1 --2- 172.21.15.46:0/3848395323 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac81cc9f0 0x7f7ac8133870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7acde8d640 10 monclient: get_auth_request con 0x7f7ac81cc9f0 auth_method 0 2024-09-16T04:47:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7acde8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7acde8d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7acde8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7acde8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7acde8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7acde8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7acde8d640 10 monclient(hunting): handle_auth_done global_id 94412 payload 824 2024-09-16T04:47:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7acde8d640 1 -- 172.21.15.46:0/3848395323 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7ac806c020 con 0x7f7ac81cc9f0 2024-09-16T04:47:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7acde8d640 1 --2- 172.21.15.46:0/3848395323 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac81cc9f0 0x7f7ac8133870 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4000c00 tx=0x7f7ac40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7abe7fc640 1 -- 172.21.15.46:0/3848395323 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4052020 con 0x7f7ac81cc9f0 2024-09-16T04:47:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7abe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7abe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7abe7fc640 10 monclient: dump: 2024-09-16T04:47:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:11.929 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7abe7fc640 1 -- 172.21.15.46:0/3848395323 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4700 (secure 0 0 0) 0x7f7ac4003a20 con 0x7f7ac81cc9f0 2024-09-16T04:47:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7abe7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:47:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7abe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:47:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7abe7fc640 1 -- 172.21.15.46:0/3848395323 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac81cc9f0 msgr2=0x7f7ac8133870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7abe7fc640 1 --2- 172.21.15.46:0/3848395323 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac81cc9f0 0x7f7ac8133870 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4000c00 tx=0x7f7ac40046c0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7abe7fc640 1 -- 172.21.15.46:0/3848395323 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac81cc9f0 msgr2=0x7f7ac8133870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:47:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.904+0000 7f7abe7fc640 1 --2- 172.21.15.46:0/3848395323 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7ac81cc9f0 0x7f7ac8133870 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4000c00 tx=0x7f7ac40046c0 comp rx=0 tx=0).stop 2024-09-16T04:47:11.930 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:47:11.930 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:47:11.930 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:47:11.930 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:47:11.930 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 98, 2024-09-16T04:47:11.931 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 73159, 2024-09-16T04:47:11.931 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:47:11.931 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:47:11.931 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:47:11.931 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:11.931 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 0, 2024-09-16T04:47:11.931 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:47:11.932 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:47:11.932 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:47:11.932 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:47:11.932 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:47:11.932 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:47:11.932 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:47:11.932 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:47:11.933 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:47:11.933 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:47:11.933 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:47:11.933 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:47:11.933 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:47:11.933 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:47:11.933 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:11.933 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:47:11.933 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:47:11.933 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:47:11.933 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:47:11.933 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:47:11.933 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:47:11.933 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:47:11.934 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:47:11.934 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:47:11.934 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:47:11.934 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:47:11.934 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:47:11.934 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:47:11.934 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:11.934 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.934 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:47:11.934 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:47:11.934 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:47:11.935 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:47:11.935 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:47:11.935 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:47:11.935 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:47:11.935 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:47:11.935 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:47:11.935 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:47:11.935 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:47:11.935 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:47:11.935 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:47:11.935 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:47:11.935 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:47:11.935 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:47:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:47:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:47:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:47:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:47:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:47:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:47:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:47:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:47:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:47:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:47:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:47:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:47:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:47:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:47:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:47:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:47:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:47:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:47:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:47:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:47:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:47:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:47:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:47:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:47:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:47:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:47:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:47:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:47:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:47:11.939 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:47:11.939 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:47:11.939 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.939 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:47:11.939 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:47:11.939 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:11.939 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.939 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.939 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:47:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:47:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:47:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:47:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:47:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:47:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:47:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:47:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:47:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:47:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:47:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:47:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:47:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:47:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:47:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:47:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:47:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:47:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:47:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:47:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:47:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:47:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:47:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:47:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:47:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:47:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:47:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:47:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:47:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:47:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:47:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:47:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4 2024-09-16T04:47:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:11.944 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:11.944 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:11.944 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:47:11.944 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:47:11.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.905+0000 7f7ad00f1640 1 -- 172.21.15.46:0/3848395323 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7aac038360 msgr2=0x7f7aac03a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:11.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.905+0000 7f7ad00f1640 1 --2- 172.21.15.46:0/3848395323 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7aac038360 0x7f7aac03a820 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7ac813cd60 tx=0x7f7ac000d040 comp rx=0 tx=0).stop 2024-09-16T04:47:11.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.905+0000 7f7ad00f1640 10 monclient: shutdown 2024-09-16T04:47:11.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.905+0000 7f7ad00f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:11.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.905+0000 7f7ad00f1640 1 -- 172.21.15.46:0/3848395323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ac809fbf0 msgr2=0x7f7ac813b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:11.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.905+0000 7f7ad00f1640 1 --2- 172.21.15.46:0/3848395323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ac809fbf0 0x7f7ac813b650 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7ab800e9d0 tx=0x7f7ab800ee90 comp rx=0 tx=0).stop 2024-09-16T04:47:11.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.905+0000 7f7ace68e640 1 -- 172.21.15.46:0/3848395323 reap_dead start 2024-09-16T04:47:11.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.905+0000 7f7ace68e640 1 -- 172.21.15.46:0/3848395323 reap_dead start 2024-09-16T04:47:11.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.905+0000 7f7ad00f1640 1 -- 172.21.15.46:0/3848395323 shutdown_connections 2024-09-16T04:47:11.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.905+0000 7f7ad00f1640 1 -- 172.21.15.46:0/3848395323 >> 172.21.15.46:0/3848395323 conn(0x7f7ac80a5960 msgr2=0x7f7ac80a2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:11.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.905+0000 7f7ad00f1640 1 -- 172.21.15.46:0/3848395323 shutdown_connections 2024-09-16T04:47:11.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:11.905+0000 7f7ad00f1640 1 -- 172.21.15.46:0/3848395323 wait complete. 2024-09-16T04:47:11.947 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:47:11.947 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:47:31.948 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T04:47:31.948 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T04:47:31.948 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T04:47:31.949 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T04:47:31.952 INFO:tasks.ceph.mon.b:Started 2024-09-16T04:47:31.952 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:47:31.953 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:47:32.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.067+0000 7f8268cf8640 10 monclient: get_monmap_and_config 2024-09-16T04:47:32.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.067+0000 7f8268cf8640 10 monclient: build_initial_monmap 2024-09-16T04:47:32.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.067+0000 7f8268cf8640 10 monclient: monmap: 2024-09-16T04:47:32.069 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:32.069 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:32.069 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:32.068790+0000 2024-09-16T04:47:32.069 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:32.068790+0000 2024-09-16T04:47:32.070 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:32.070 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:32.070 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:32.070 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:32.070 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:32.070 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:32.070 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:32.070 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:32.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8268cf8640 1 Processor -- start 2024-09-16T04:47:32.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8268cf8640 1 -- start start 2024-09-16T04:47:32.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8268cf8640 10 monclient: init 2024-09-16T04:47:32.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8268cf8640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:32.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8268cf8640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-16T04:47:32.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8268cf8640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f826406e160 0x7f82640a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:32.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8268cf8640 10 monclient(hunting): picked mon.noname-c con 0x7f826406e160 addr v2:172.21.15.114:3300/0 2024-09-16T04:47:32.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8268cf8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f82640a2ee0 addr v1:172.21.15.46:6789/0 2024-09-16T04:47:32.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8268cf8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8264137050 0x7f826413ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:32.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8268cf8640 10 monclient(hunting): picked mon.noname-a con 0x7f8264137050 addr v2:172.21.15.46:3300/0 2024-09-16T04:47:32.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8268cf8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82640a3fd0 con 0x7f82640a2ee0 2024-09-16T04:47:32.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8268cf8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:32.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8268cf8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f82640a3300 con 0x7f8264137050 2024-09-16T04:47:32.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8268cf8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:32.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8268cf8640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f826406e540 con 0x7f826406e160 2024-09-16T04:47:32.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8268cf8640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:32.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8268cf8640 10 monclient(hunting): authenticate will time out at 1250.484662s 2024-09-16T04:47:32.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8262d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8264137050 0x7f826413ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:32.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f825bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f82640a2ee0 0x7f8264136940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38040/0 (socket says 172.21.15.46:38040) 2024-09-16T04:47:32.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f825bfff640 1 -- 172.21.15.46:0/1204565482 learned_addr learned my addr 172.21.15.46:0/1204565482 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:47:32.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8262d9d640 10 monclient(hunting): get_auth_request con 0x7f8264137050 auth_method 0 2024-09-16T04:47:32.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8262d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:32.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8262d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:32.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8262d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:32.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8262d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:32.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8262d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:32.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.068+0000 7f8262d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:32.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.069+0000 7f826259c640 1 --2- 172.21.15.46:0/1204565482 >> v2:172.21.15.114:3300/0 conn(0x7f826406e160 0x7f82640a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:32.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.069+0000 7f826259c640 10 monclient(hunting): get_auth_request con 0x7f826406e160 auth_method 0 2024-09-16T04:47:32.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.069+0000 7f826259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:32.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.069+0000 7f826259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:32.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.069+0000 7f826259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:32.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.069+0000 7f8262d9d640 10 monclient(hunting): handle_auth_done global_id 104223 payload 274 2024-09-16T04:47:32.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.069+0000 7f8262d9d640 1 -- 172.21.15.46:0/1204565482 >> v2:172.21.15.114:3300/0 conn(0x7f826406e160 msgr2=0x7f82640a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:32.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.069+0000 7f8262d9d640 1 --2- 172.21.15.46:0/1204565482 >> v2:172.21.15.114:3300/0 conn(0x7f826406e160 0x7f82640a29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:32.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.069+0000 7f8262d9d640 1 -- 172.21.15.46:0/1204565482 >> v1:172.21.15.46:6789/0 conn(0x7f82640a2ee0 legacy=0x7f8264136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:32.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.069+0000 7f8262d9d640 10 monclient: _finish_hunting 0 2024-09-16T04:47:32.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.069+0000 7f8262d9d640 1 monclient: found mon.noname-a 2024-09-16T04:47:32.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.069+0000 7f8262d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:32.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.069+0000 7f8262d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:32.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.069+0000 7f8262d9d640 1 -- 172.21.15.46:0/1204565482 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82640a0bd0 con 0x7f8264137050 2024-09-16T04:47:32.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.069+0000 7f8261d9b640 1 -- 172.21.15.46:0/1204565482 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1978515646 0 0) 0x7f824c001110 con 0x7f82640a2ee0 2024-09-16T04:47:32.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.069+0000 7f8261d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:47:32.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.069+0000 7f8262d9d640 1 --2- 172.21.15.46:0/1204565482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8264137050 0x7f826413ec80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8264068700 tx=0x7f825c02a580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6990d617a16d9dd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:32.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8261d9b640 1 -- 172.21.15.46:0/1204565482 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c007740 con 0x7f8264137050 2024-09-16T04:47:32.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8261d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:32.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8261d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:32.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8261d9b640 10 monclient: dump: 2024-09-16T04:47:32.081 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:32.081 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:32.081 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:32.081 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:32.081 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:32.081 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:32.082 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:32.082 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:32.083 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:32.083 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:32.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8261d9b640 10 monclient: _finish_auth 0 2024-09-16T04:47:32.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8261d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:47:32.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8261d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:32.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8268cf8640 5 monclient: authenticate success, global_id 104223 2024-09-16T04:47:32.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8261d9b640 1 -- 172.21.15.46:0/1204565482 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f825c0078e0 con 0x7f8264137050 2024-09-16T04:47:32.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8268cf8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:32.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8261d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:32.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8268cf8640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:32.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8268cf8640 4 set_mon_vals no callback set 2024-09-16T04:47:32.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8261d9b640 1 -- 172.21.15.46:0/1204565482 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c007cb0 con 0x7f8264137050 2024-09-16T04:47:32.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8261d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:32.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8261d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:32.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8261d9b640 10 monclient: dump: 2024-09-16T04:47:32.084 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:32.084 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:32.084 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:32.084 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:32.084 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:32.084 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:32.084 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:32.084 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:32.084 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:32.084 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8268cf8640 10 monclient: shutdown 2024-09-16T04:47:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8268cf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8268cf8640 1 -- 172.21.15.46:0/1204565482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8264137050 msgr2=0x7f826413ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8268cf8640 1 --2- 172.21.15.46:0/1204565482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8264137050 0x7f826413ec80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8264068700 tx=0x7f825c02a580 comp rx=0 tx=0).stop 2024-09-16T04:47:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8268cf8640 1 -- 172.21.15.46:0/1204565482 shutdown_connections 2024-09-16T04:47:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8268cf8640 1 --2- 172.21.15.46:0/1204565482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8264137050 0x7f826413ec80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8268cf8640 1 --2- 172.21.15.46:0/1204565482 >> v2:172.21.15.114:3300/0 conn(0x7f826406e160 0x7f82640a29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8268cf8640 1 -- 172.21.15.46:0/1204565482 >> 172.21.15.46:0/1204565482 conn(0x7f82640a5c10 msgr2=0x7f82640a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8268cf8640 1 -- 172.21.15.46:0/1204565482 shutdown_connections 2024-09-16T04:47:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.070+0000 7f8268cf8640 1 -- 172.21.15.46:0/1204565482 wait complete. 2024-09-16T04:47:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.071+0000 7f8268cf8640 10 monclient: build_initial_monmap 2024-09-16T04:47:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.071+0000 7f8268cf8640 10 monclient: monmap: 2024-09-16T04:47:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:32.086 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:32.086 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:32.086 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:32.086 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:32.086 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:32.086 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:32.086 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:32.086 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:32.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.071+0000 7f8268cf8640 1 Processor -- start 2024-09-16T04:47:32.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.071+0000 7f8268cf8640 1 -- start start 2024-09-16T04:47:32.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.071+0000 7f8268cf8640 10 monclient: init 2024-09-16T04:47:32.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8268cf8640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:32.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8268cf8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:47:32.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8268cf8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f826406e160 0x7f82641cba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:32.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8268cf8640 10 monclient(hunting): picked mon.noname-a con 0x7f826406e160 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:32.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8268cf8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82640a2ee0 0x7f8264135b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:32.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8268cf8640 10 monclient(hunting): picked mon.noname-b con 0x7f82640a2ee0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:32.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8268cf8640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8264137050 0x7f8264136040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:32.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8268cf8640 10 monclient(hunting): picked mon.noname-c con 0x7f8264137050 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:32.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8268cf8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:32.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8268cf8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f82640a34a0 con 0x7f826406e160 2024-09-16T04:47:32.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8268cf8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:32.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8268cf8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f82640a5130 con 0x7f82640a2ee0 2024-09-16T04:47:32.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8268cf8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:32.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8268cf8640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f82640a0300 con 0x7f8264137050 2024-09-16T04:47:32.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8268cf8640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:32.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8268cf8640 10 monclient(hunting): authenticate will time out at 1250.488510s 2024-09-16T04:47:32.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f826259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f826406e160 0x7f82641cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f826259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f826406e160 0x7f82641cba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58984/0 (socket says 172.21.15.46:58984) 2024-09-16T04:47:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f826259c640 1 -- 172.21.15.46:0/2954980839 learned_addr learned my addr 172.21.15.46:0/2954980839 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f826259c640 10 monclient(hunting): get_auth_request con 0x7f826406e160 auth_method 0 2024-09-16T04:47:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f826259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f826259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f826259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f826259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f826259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8262d9d640 1 --2- 172.21.15.46:0/2954980839 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8264137050 0x7f8264136040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f826259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8262d9d640 10 monclient(hunting): get_auth_request con 0x7f8264137050 auth_method 0 2024-09-16T04:47:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8262d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8262d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8262d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8262d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8262d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f8262d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f826259c640 10 monclient(hunting): handle_auth_done global_id 104229 payload 824 2024-09-16T04:47:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f825bfff640 1 --2- 172.21.15.46:0/2954980839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82640a2ee0 0x7f8264135b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f826259c640 1 -- 172.21.15.46:0/2954980839 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8264137050 msgr2=0x7f8264136040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f826259c640 1 --2- 172.21.15.46:0/2954980839 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8264137050 0x7f8264136040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f826259c640 1 -- 172.21.15.46:0/2954980839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82640a2ee0 msgr2=0x7f8264135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f826259c640 1 --2- 172.21.15.46:0/2954980839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82640a2ee0 0x7f8264135b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f826259c640 10 monclient: _finish_hunting 0 2024-09-16T04:47:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f826259c640 1 monclient: found mon.noname-a 2024-09-16T04:47:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f826259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f826259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f826259c640 1 -- 172.21.15.46:0/2954980839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8264127220 con 0x7f826406e160 2024-09-16T04:47:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f825bfff640 10 monclient: get_auth_request con 0x7f82640a2ee0 auth_method 0 2024-09-16T04:47:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.072+0000 7f825bfff640 1 --2- 172.21.15.46:0/2954980839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82640a2ee0 0x7f8264135b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:47:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f826259c640 1 --2- 172.21.15.46:0/2954980839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f826406e160 0x7f82641cba10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f825400d5f0 tx=0x7f825400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f825affd640 1 -- 172.21.15.46:0/2954980839 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82540125f0 con 0x7f826406e160 2024-09-16T04:47:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f825affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f825affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f825affd640 10 monclient: dump: 2024-09-16T04:47:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:32.091 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:32.091 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:32.091 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:32.091 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:32.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f825affd640 10 monclient: _finish_auth 0 2024-09-16T04:47:32.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f825affd640 10 monclient: _check_auth_tickets 2024-09-16T04:47:32.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f825affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:32.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f825affd640 1 -- 172.21.15.46:0/2954980839 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8254019070 con 0x7f826406e160 2024-09-16T04:47:32.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f8268cf8640 5 monclient: authenticate success, global_id 104229 2024-09-16T04:47:32.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f825affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:32.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f825affd640 1 -- 172.21.15.46:0/2954980839 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82540129d0 con 0x7f826406e160 2024-09-16T04:47:32.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f8268cf8640 10 monclient: _renew_subs 2024-09-16T04:47:32.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f8268cf8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:32.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f8268cf8640 1 -- 172.21.15.46:0/2954980839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8264125d70 con 0x7f826406e160 2024-09-16T04:47:32.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f825affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:32.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f825affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:32.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f825affd640 10 monclient: dump: 2024-09-16T04:47:32.092 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:32.092 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:32.092 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:32.092 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:32.092 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:32.092 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:32.092 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:32.092 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:32.092 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:32.092 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:32.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f826159a640 4 set_mon_vals no callback set 2024-09-16T04:47:32.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f8268cf8640 10 monclient: _renew_subs 2024-09-16T04:47:32.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f8268cf8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:32.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.073+0000 7f8268cf8640 1 -- 172.21.15.46:0/2954980839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82640a81b0 con 0x7f826406e160 2024-09-16T04:47:32.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.074+0000 7f825affd640 1 -- 172.21.15.46:0/2954980839 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f8254012b70 con 0x7f826406e160 2024-09-16T04:47:32.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.074+0000 7f8268cf8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:32.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.074+0000 7f8268cf8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:32.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.074+0000 7f8268cf8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:32.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.074+0000 7f8268cf8640 1 -- 172.21.15.46:0/2954980839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82641374a0 con 0x7f826406e160 2024-09-16T04:47:32.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.074+0000 7f825affd640 1 --2- 172.21.15.46:0/2954980839 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f82440383b0 0x7f824403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:32.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.074+0000 7f825affd640 1 -- 172.21.15.46:0/2954980839 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7f825404b9e0 con 0x7f826406e160 2024-09-16T04:47:32.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.074+0000 7f825bfff640 1 --2- 172.21.15.46:0/2954980839 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f82440383b0 0x7f824403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:32.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.074+0000 7f825bfff640 10 monclient: get_auth_request con 0x7f82440383b0 auth_method 0 2024-09-16T04:47:32.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.075+0000 7f825bfff640 1 --2- 172.21.15.46:0/2954980839 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f82440383b0 0x7f824403a870 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f824c0013b0 tx=0x7f824c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:32.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.079+0000 7f825affd640 1 -- 172.21.15.46:0/2954980839 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8254013390 con 0x7f826406e160 2024-09-16T04:47:32.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.079+0000 7f825affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:32.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.079+0000 7f825affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:32.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.323+0000 7f8268cf8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:47:32.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.323+0000 7f8268cf8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:47:32.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.323+0000 7f8268cf8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:32.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.323+0000 7f8268cf8640 1 -- 172.21.15.46:0/2954980839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8264067e00 con 0x7f826406e160 2024-09-16T04:47:32.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.324+0000 7f825affd640 1 -- 172.21.15.46:0/2954980839 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f825401ebc0 con 0x7f826406e160 2024-09-16T04:47:32.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.324+0000 7f825affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:47:32.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.324+0000 7f825affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:47:32.326 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":98,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:47:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.326+0000 7f8268cf8640 1 -- 172.21.15.46:0/2954980839 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f82440383b0 msgr2=0x7f824403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.326+0000 7f8268cf8640 1 --2- 172.21.15.46:0/2954980839 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f82440383b0 0x7f824403a870 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f824c0013b0 tx=0x7f824c009040 comp rx=0 tx=0).stop 2024-09-16T04:47:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.326+0000 7f8268cf8640 10 monclient: shutdown 2024-09-16T04:47:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.326+0000 7f8268cf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.326+0000 7f8268cf8640 1 -- 172.21.15.46:0/2954980839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f826406e160 msgr2=0x7f82641cba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.326+0000 7f8268cf8640 1 --2- 172.21.15.46:0/2954980839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f826406e160 0x7f82641cba10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f825400d5f0 tx=0x7f825400dab0 comp rx=0 tx=0).stop 2024-09-16T04:47:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.326+0000 7f8268cf8640 1 -- 172.21.15.46:0/2954980839 shutdown_connections 2024-09-16T04:47:32.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.326+0000 7f8268cf8640 1 --2- 172.21.15.46:0/2954980839 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f8264137050 0x7f8264136040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:32.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.326+0000 7f8268cf8640 1 --2- 172.21.15.46:0/2954980839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82640a2ee0 0x7f8264135b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:32.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.326+0000 7f8268cf8640 1 --2- 172.21.15.46:0/2954980839 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f82440383b0 0x7f824403a870 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:32.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.326+0000 7f8268cf8640 1 --2- 172.21.15.46:0/2954980839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f826406e160 0x7f82641cba10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:32.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.326+0000 7f8268cf8640 1 -- 172.21.15.46:0/2954980839 >> 172.21.15.46:0/2954980839 conn(0x7f82640a5c10 msgr2=0x7f82640a77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:32.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.326+0000 7f8268cf8640 1 -- 172.21.15.46:0/2954980839 shutdown_connections 2024-09-16T04:47:32.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:32.326+0000 7f8268cf8640 1 -- 172.21.15.46:0/2954980839 wait complete. 2024-09-16T04:47:35.341 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:47:35.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.452+0000 7f5acef81640 10 monclient: get_monmap_and_config 2024-09-16T04:47:35.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.452+0000 7f5acef81640 10 monclient: build_initial_monmap 2024-09-16T04:47:35.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.452+0000 7f5acef81640 10 monclient: monmap: 2024-09-16T04:47:35.454 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:35.455 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:35.455 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:35.454130+0000 2024-09-16T04:47:35.455 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:35.454130+0000 2024-09-16T04:47:35.455 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:35.455 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:35.455 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:35.455 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:35.455 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:35.455 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:35.455 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:35.455 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.453+0000 7f5acef81640 1 Processor -- start 2024-09-16T04:47:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.453+0000 7f5acef81640 1 -- start start 2024-09-16T04:47:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.453+0000 7f5acef81640 10 monclient: init 2024-09-16T04:47:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.453+0000 7f5acef81640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.453+0000 7f5acef81640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-16T04:47:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5acef81640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5ac80a04f0 addr v1:172.21.15.114:6789/0 2024-09-16T04:47:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5acef81640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5ac8073b60 0x7f5ac813bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5acef81640 10 monclient(hunting): picked mon.noname-a con 0x7f5ac8073b60 addr v2:172.21.15.46:3300/0 2024-09-16T04:47:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5acef81640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5ac813c130 addr v1:172.21.15.46:6789/0 2024-09-16T04:47:35.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5acef81640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ac806e090 con 0x7f5ac813c130 2024-09-16T04:47:35.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5acef81640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ac809e720 con 0x7f5ac80a04f0 2024-09-16T04:47:35.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5acef81640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:35.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5acef81640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f5ac809ea00 con 0x7f5ac8073b60 2024-09-16T04:47:35.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5acef81640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:35.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5acef81640 10 monclient(hunting): authenticate will time out at 1253.870134s 2024-09-16T04:47:35.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5ac7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5ac8073b60 0x7f5ac813bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:35.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5acd51e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5ac813c130 0x7f5ac813e520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33272/0 (socket says 172.21.15.46:33272) 2024-09-16T04:47:35.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5ac7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5ac8073b60 0x7f5ac813bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41700/0 (socket says 172.21.15.46:41700) 2024-09-16T04:47:35.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5acd51e640 1 -- 172.21.15.46:0/1208976490 learned_addr learned my addr 172.21.15.46:0/1208976490 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:47:35.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5ac7fff640 10 monclient(hunting): get_auth_request con 0x7f5ac8073b60 auth_method 0 2024-09-16T04:47:35.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5ac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:35.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5ac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5ac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5ac7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5ac7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5ac7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5ac7fff640 10 monclient(hunting): handle_auth_done global_id 104232 payload 274 2024-09-16T04:47:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5ac7fff640 1 -- 172.21.15.46:0/1208976490 >> v1:172.21.15.114:6789/0 conn(0x7f5ac80a04f0 legacy=0x7f5ac80a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5ac7fff640 1 -- 172.21.15.46:0/1208976490 >> v1:172.21.15.46:6789/0 conn(0x7f5ac813c130 legacy=0x7f5ac813e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5ac7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:47:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5ac7fff640 1 monclient: found mon.noname-a 2024-09-16T04:47:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5ac7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5ac7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5ac7fff640 1 -- 172.21.15.46:0/1208976490 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac809ece0 con 0x7f5ac8073b60 2024-09-16T04:47:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5ac77fe640 1 -- 172.21.15.46:0/1208976490 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1818704830 0 0) 0x7f5ac806e090 con 0x7f5ac813c130 2024-09-16T04:47:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.454+0000 7f5ac77fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:47:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.455+0000 7f5ac7fff640 1 --2- 172.21.15.46:0/1208976490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ac8073b60 0x7f5ac813bbf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5ab40097a0 tx=0x7f5ab402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ba77301ee5605b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.455+0000 7f5ac77fe640 1 -- 172.21.15.46:0/1208976490 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab4037070 con 0x7f5ac8073b60 2024-09-16T04:47:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.455+0000 7f5ac77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.455+0000 7f5ac77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.455+0000 7f5ac77fe640 10 monclient: dump: 2024-09-16T04:47:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:35.466 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.455+0000 7f5ac77fe640 10 monclient: _finish_auth 0 2024-09-16T04:47:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.455+0000 7f5ac77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:47:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.455+0000 7f5ac77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.455+0000 7f5acef81640 5 monclient: authenticate success, global_id 104232 2024-09-16T04:47:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.455+0000 7f5acef81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.455+0000 7f5ac77fe640 1 -- 172.21.15.46:0/1208976490 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab4032440 con 0x7f5ac8073b60 2024-09-16T04:47:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.455+0000 7f5ac77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.455+0000 7f5acef81640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.455+0000 7f5ac77fe640 1 -- 172.21.15.46:0/1208976490 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab402ae60 con 0x7f5ac8073b60 2024-09-16T04:47:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.455+0000 7f5acef81640 4 set_mon_vals no callback set 2024-09-16T04:47:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.455+0000 7f5ac77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.455+0000 7f5ac77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.455+0000 7f5ac77fe640 10 monclient: dump: 2024-09-16T04:47:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:35.468 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.456+0000 7f5acef81640 10 monclient: shutdown 2024-09-16T04:47:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.456+0000 7f5acef81640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.456+0000 7f5acef81640 1 -- 172.21.15.46:0/1208976490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ac8073b60 msgr2=0x7f5ac813bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.456+0000 7f5acef81640 1 --2- 172.21.15.46:0/1208976490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ac8073b60 0x7f5ac813bbf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5ab40097a0 tx=0x7f5ab402a110 comp rx=0 tx=0).stop 2024-09-16T04:47:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.456+0000 7f5acef81640 1 -- 172.21.15.46:0/1208976490 shutdown_connections 2024-09-16T04:47:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.456+0000 7f5acef81640 1 --2- 172.21.15.46:0/1208976490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ac8073b60 0x7f5ac813bbf0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.456+0000 7f5acef81640 1 -- 172.21.15.46:0/1208976490 >> 172.21.15.46:0/1208976490 conn(0x7f5ac80a3b10 msgr2=0x7f5ac80a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.456+0000 7f5acef81640 1 -- 172.21.15.46:0/1208976490 shutdown_connections 2024-09-16T04:47:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.456+0000 7f5acef81640 1 -- 172.21.15.46:0/1208976490 wait complete. 2024-09-16T04:47:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.456+0000 7f5acef81640 10 monclient: build_initial_monmap 2024-09-16T04:47:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.456+0000 7f5acef81640 10 monclient: monmap: 2024-09-16T04:47:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:35.470 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.457+0000 7f5acef81640 1 Processor -- start 2024-09-16T04:47:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.457+0000 7f5acef81640 1 -- start start 2024-09-16T04:47:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.457+0000 7f5acef81640 10 monclient: init 2024-09-16T04:47:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.457+0000 7f5acef81640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.457+0000 7f5acef81640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:47:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.457+0000 7f5acef81640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5ac8073b60 0x7f5ac8136f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.457+0000 7f5acef81640 10 monclient(hunting): picked mon.noname-c con 0x7f5ac8073b60 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.457+0000 7f5acef81640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ac80a04f0 0x7f5ac81374c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.457+0000 7f5acef81640 10 monclient(hunting): picked mon.noname-a con 0x7f5ac80a04f0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.457+0000 7f5acef81640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ac813c130 0x7f5ac8137de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.457+0000 7f5acef81640 10 monclient(hunting): picked mon.noname-b con 0x7f5ac813c130 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.457+0000 7f5acef81640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.457+0000 7f5acef81640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac80a2c90 con 0x7f5ac80a04f0 2024-09-16T04:47:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.457+0000 7f5acef81640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.457+0000 7f5acef81640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac806de50 con 0x7f5ac813c130 2024-09-16T04:47:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.457+0000 7f5acef81640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.457+0000 7f5acef81640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f5ac80a2e10 con 0x7f5ac8073b60 2024-09-16T04:47:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.457+0000 7f5acef81640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.457+0000 7f5acef81640 10 monclient(hunting): authenticate will time out at 1253.873849s 2024-09-16T04:47:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.457+0000 7f5accd1d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5ac8073b60 0x7f5ac8136f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.457+0000 7f5ac7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ac80a04f0 0x7f5ac81374c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5accd1d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5ac8073b60 0x7f5ac8136f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38680/0 (socket says 172.21.15.46:38680) 2024-09-16T04:47:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5accd1d640 1 -- 172.21.15.46:0/1972658160 learned_addr learned my addr 172.21.15.46:0/1972658160 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5accd1d640 10 monclient(hunting): get_auth_request con 0x7f5ac8073b60 auth_method 0 2024-09-16T04:47:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5accd1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5accd1d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5accd1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac7fff640 10 monclient(hunting): get_auth_request con 0x7f5ac80a04f0 auth_method 0 2024-09-16T04:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5acd51e640 1 --2- 172.21.15.46:0/1972658160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ac813c130 0x7f5ac8137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5acd51e640 10 monclient(hunting): get_auth_request con 0x7f5ac813c130 auth_method 0 2024-09-16T04:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5acd51e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5acd51e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5acd51e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:35.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5accd1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:35.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5accd1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:35.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5accd1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:35.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5acd51e640 1 -- 172.21.15.46:0/1972658160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ac813c130 msgr2=0x7f5ac8137de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:47:35.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5acd51e640 1 -- 172.21.15.46:0/1972658160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ac813c130 msgr2=0x7f5ac8137de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:47:35.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5acd51e640 1 --2- 172.21.15.46:0/1972658160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ac813c130 0x7f5ac8137de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:47:35.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5acd51e640 1 --2- 172.21.15.46:0/1972658160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ac813c130 0x7f5ac8137de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:47:35.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac7fff640 10 monclient(hunting): handle_auth_done global_id 104238 payload 824 2024-09-16T04:47:35.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac7fff640 1 -- 172.21.15.46:0/1972658160 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5ac8073b60 msgr2=0x7f5ac8136f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:35.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac7fff640 1 --2- 172.21.15.46:0/1972658160 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5ac8073b60 0x7f5ac8136f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:35.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac7fff640 1 -- 172.21.15.46:0/1972658160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ac813c130 msgr2=0x7f5ac8137de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:47:35.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac7fff640 1 --2- 172.21.15.46:0/1972658160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ac813c130 0x7f5ac8137de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:35.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:47:35.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac7fff640 1 monclient: found mon.noname-a 2024-09-16T04:47:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac7fff640 1 -- 172.21.15.46:0/1972658160 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac8137a00 con 0x7f5ac80a04f0 2024-09-16T04:47:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac7fff640 1 --2- 172.21.15.46:0/1972658160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ac80a04f0 0x7f5ac81374c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4031000 tx=0x7f5ab40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac57fa640 1 -- 172.21.15.46:0/1972658160 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab4037050 con 0x7f5ac80a04f0 2024-09-16T04:47:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac57fa640 10 monclient: dump: 2024-09-16T04:47:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:35.475 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac57fa640 10 monclient: _finish_auth 0 2024-09-16T04:47:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac57fa640 10 monclient: _check_auth_tickets 2024-09-16T04:47:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac57fa640 1 -- 172.21.15.46:0/1972658160 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab4038040 con 0x7f5ac80a04f0 2024-09-16T04:47:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5acef81640 5 monclient: authenticate success, global_id 104238 2024-09-16T04:47:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5acef81640 10 monclient: _renew_subs 2024-09-16T04:47:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5acef81640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5acef81640 1 -- 172.21.15.46:0/1972658160 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ac813acb0 con 0x7f5ac80a04f0 2024-09-16T04:47:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac57fa640 1 -- 172.21.15.46:0/1972658160 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab4004490 con 0x7f5ac80a04f0 2024-09-16T04:47:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:35.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:35.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac57fa640 10 monclient: dump: 2024-09-16T04:47:35.476 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:35.476 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:35.476 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:35.476 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:35.476 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:35.476 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:35.477 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:35.477 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:35.477 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:35.477 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:35.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5acef81640 10 monclient: _renew_subs 2024-09-16T04:47:35.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5ac6ffd640 4 set_mon_vals no callback set 2024-09-16T04:47:35.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5acef81640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:35.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.458+0000 7f5acef81640 1 -- 172.21.15.46:0/1972658160 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ac813f940 con 0x7f5ac80a04f0 2024-09-16T04:47:35.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.459+0000 7f5ac57fa640 1 -- 172.21.15.46:0/1972658160 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f5ab4030900 con 0x7f5ac80a04f0 2024-09-16T04:47:35.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.459+0000 7f5acef81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:35.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.459+0000 7f5acef81640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:35.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.459+0000 7f5acef81640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:35.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.459+0000 7f5acef81640 1 -- 172.21.15.46:0/1972658160 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ac809d290 con 0x7f5ac80a04f0 2024-09-16T04:47:35.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.460+0000 7f5ac57fa640 1 --2- 172.21.15.46:0/1972658160 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5aac0383e0 0x7f5aac03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:35.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.460+0000 7f5ac57fa640 1 -- 172.21.15.46:0/1972658160 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7f5ab406c2a0 con 0x7f5ac80a04f0 2024-09-16T04:47:35.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.460+0000 7f5accd1d640 1 --2- 172.21.15.46:0/1972658160 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5aac0383e0 0x7f5aac03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:35.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.460+0000 7f5accd1d640 10 monclient: get_auth_request con 0x7f5aac0383e0 auth_method 0 2024-09-16T04:47:35.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.460+0000 7f5accd1d640 1 --2- 172.21.15.46:0/1972658160 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5aac0383e0 0x7f5aac03a8a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f5abc007fd0 tx=0x7f5abc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:35.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.464+0000 7f5ac57fa640 1 -- 172.21.15.46:0/1972658160 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5ac809d290 con 0x7f5ac80a04f0 2024-09-16T04:47:35.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.464+0000 7f5ac57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:35.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.464+0000 7f5ac57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:35.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.706+0000 7f5acef81640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:47:35.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.706+0000 7f5acef81640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:47:35.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.706+0000 7f5acef81640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:35.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.706+0000 7f5acef81640 1 -- 172.21.15.46:0/1972658160 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5ac813a410 con 0x7f5ac80a04f0 2024-09-16T04:47:35.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.706+0000 7f5ac57fa640 1 -- 172.21.15.46:0/1972658160 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f5ab406c080 con 0x7f5ac80a04f0 2024-09-16T04:47:35.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.706+0000 7f5ac57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:47:35.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.706+0000 7f5ac57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:47:35.709 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":98,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:47:35.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.709+0000 7f5acef81640 1 -- 172.21.15.46:0/1972658160 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5aac0383e0 msgr2=0x7f5aac03a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:35.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.709+0000 7f5acef81640 1 --2- 172.21.15.46:0/1972658160 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5aac0383e0 0x7f5aac03a8a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f5abc007fd0 tx=0x7f5abc009040 comp rx=0 tx=0).stop 2024-09-16T04:47:35.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.709+0000 7f5acef81640 10 monclient: shutdown 2024-09-16T04:47:35.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.709+0000 7f5acef81640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:35.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.709+0000 7f5acef81640 1 -- 172.21.15.46:0/1972658160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ac80a04f0 msgr2=0x7f5ac81374c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:35.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.709+0000 7f5acef81640 1 --2- 172.21.15.46:0/1972658160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ac80a04f0 0x7f5ac81374c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4031000 tx=0x7f5ab40046c0 comp rx=0 tx=0).stop 2024-09-16T04:47:35.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.709+0000 7f5acef81640 1 -- 172.21.15.46:0/1972658160 shutdown_connections 2024-09-16T04:47:35.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.709+0000 7f5acef81640 1 --2- 172.21.15.46:0/1972658160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ac813c130 0x7f5ac8137de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:35.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.709+0000 7f5acef81640 1 --2- 172.21.15.46:0/1972658160 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f5aac0383e0 0x7f5aac03a8a0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:35.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.709+0000 7f5acef81640 1 --2- 172.21.15.46:0/1972658160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ac80a04f0 0x7f5ac81374c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:35.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.709+0000 7f5acef81640 1 --2- 172.21.15.46:0/1972658160 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f5ac8073b60 0x7f5ac8136f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:35.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.709+0000 7f5acef81640 1 -- 172.21.15.46:0/1972658160 >> 172.21.15.46:0/1972658160 conn(0x7f5ac80a3b10 msgr2=0x7f5ac809cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:35.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.709+0000 7f5acef81640 1 -- 172.21.15.46:0/1972658160 shutdown_connections 2024-09-16T04:47:35.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:35.709+0000 7f5acef81640 1 -- 172.21.15.46:0/1972658160 wait complete. 2024-09-16T04:47:38.722 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:47:38.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.834+0000 7fe74a042640 10 monclient: get_monmap_and_config 2024-09-16T04:47:38.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.834+0000 7fe74a042640 10 monclient: build_initial_monmap 2024-09-16T04:47:38.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.834+0000 7fe74a042640 10 monclient: monmap: 2024-09-16T04:47:38.836 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:38.836 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:38.836 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:38.835611+0000 2024-09-16T04:47:38.836 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:38.835611+0000 2024-09-16T04:47:38.836 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:38.837 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:38.837 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:38.837 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:38.837 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:38.837 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:38.837 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:38.837 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.834+0000 7fe74a042640 1 Processor -- start 2024-09-16T04:47:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe74a042640 1 -- start start 2024-09-16T04:47:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe74a042640 10 monclient: init 2024-09-16T04:47:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe74a042640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe74a042640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-16T04:47:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe74a042640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe7440a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:47:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe74a042640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe7440a3ff0 0x7fe7440a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe74a042640 10 monclient(hunting): picked mon.noname-c con 0x7fe7440a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:47:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe74a042640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe74413be90 0x7fe74413e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe74a042640 10 monclient(hunting): picked mon.noname-a con 0x7fe74413be90 addr v2:172.21.15.46:3300/0 2024-09-16T04:47:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe74a042640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe74409b940 con 0x7fe7440a3c10 2024-09-16T04:47:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe74a042640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe74a042640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe74409ab00 con 0x7fe74413be90 2024-09-16T04:47:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe74a042640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:38.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe74a042640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fe74409b030 con 0x7fe7440a3ff0 2024-09-16T04:47:38.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe74a042640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:38.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe74a042640 10 monclient(hunting): authenticate will time out at 1257.251555s 2024-09-16T04:47:38.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe743fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe74413be90 0x7fe74413e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:38.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe7437fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe7440a3c10 0x7fe74409bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33278/0 (socket says 172.21.15.46:33278) 2024-09-16T04:47:38.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe7437fe640 1 -- 172.21.15.46:0/3965181523 learned_addr learned my addr 172.21.15.46:0/3965181523 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:47:38.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe743fff640 10 monclient(hunting): get_auth_request con 0x7fe74413be90 auth_method 0 2024-09-16T04:47:38.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe743fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:38.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe743fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:38.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe743fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:38.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe743fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:38.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe743fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:38.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe743fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:38.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe742ffd640 1 --2- 172.21.15.46:0/3965181523 >> v2:172.21.15.114:3300/0 conn(0x7fe7440a3ff0 0x7fe7440a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:38.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe742ffd640 10 monclient(hunting): get_auth_request con 0x7fe7440a3ff0 auth_method 0 2024-09-16T04:47:38.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe742ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:38.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe742ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:38.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe742ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:38.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe7427fc640 1 -- 172.21.15.46:0/3965181523 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 870911769 0 0) 0x7fe74409b940 con 0x7fe7440a3c10 2024-09-16T04:47:38.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe7427fc640 10 monclient(hunting): _negotiate 2024-09-16T04:47:38.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe7427fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:38.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe7427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:38.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe7427fc640 10 monclient(hunting): my global_id is 104244 2024-09-16T04:47:38.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.835+0000 7fe7427fc640 1 -- 172.21.15.46:0/3965181523 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe72c003520 con 0x7fe7440a3c10 2024-09-16T04:47:38.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe743fff640 10 monclient(hunting): handle_auth_done global_id 104241 payload 274 2024-09-16T04:47:38.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe743fff640 1 -- 172.21.15.46:0/3965181523 >> v2:172.21.15.114:3300/0 conn(0x7fe7440a3ff0 msgr2=0x7fe7440a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:38.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe743fff640 1 --2- 172.21.15.46:0/3965181523 >> v2:172.21.15.114:3300/0 conn(0x7fe7440a3ff0 0x7fe7440a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe743fff640 1 -- 172.21.15.46:0/3965181523 >> v1:172.21.15.46:6789/0 conn(0x7fe7440a3c10 legacy=0x7fe74409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe743fff640 10 monclient: _finish_hunting 0 2024-09-16T04:47:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe743fff640 1 monclient: found mon.noname-a 2024-09-16T04:47:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe743fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe743fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe743fff640 1 -- 172.21.15.46:0/3965181523 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe74409b310 con 0x7fe74413be90 2024-09-16T04:47:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe743fff640 1 --2- 172.21.15.46:0/3965181523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe74413be90 0x7fe74413e280 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe7440a38d0 tx=0x7fe73c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=389c0855b55f283a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe7427fc640 1 -- 172.21.15.46:0/3965181523 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c037070 con 0x7fe74413be90 2024-09-16T04:47:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe7427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe7427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe7427fc640 10 monclient: dump: 2024-09-16T04:47:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:38.850 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe7427fc640 10 monclient: _finish_auth 0 2024-09-16T04:47:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe7427fc640 10 monclient: _check_auth_tickets 2024-09-16T04:47:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe7427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe7427fc640 1 -- 172.21.15.46:0/3965181523 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe73c032440 con 0x7fe74413be90 2024-09-16T04:47:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe7427fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe7427fc640 1 -- 172.21.15.46:0/3965181523 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c02ae60 con 0x7fe74413be90 2024-09-16T04:47:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe74a042640 5 monclient: authenticate success, global_id 104241 2024-09-16T04:47:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe74a042640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe74a042640 4 set_mon_vals no callback set 2024-09-16T04:47:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe7427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe7427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.836+0000 7fe7427fc640 10 monclient: dump: 2024-09-16T04:47:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:38.851 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:38.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.837+0000 7fe74a042640 10 monclient: shutdown 2024-09-16T04:47:38.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.837+0000 7fe74a042640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:38.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.837+0000 7fe74a042640 1 -- 172.21.15.46:0/3965181523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe74413be90 msgr2=0x7fe74413e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:38.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.837+0000 7fe74a042640 1 --2- 172.21.15.46:0/3965181523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe74413be90 0x7fe74413e280 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe7440a38d0 tx=0x7fe73c02a110 comp rx=0 tx=0).stop 2024-09-16T04:47:38.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.837+0000 7fe74a042640 1 -- 172.21.15.46:0/3965181523 shutdown_connections 2024-09-16T04:47:38.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.837+0000 7fe74a042640 1 --2- 172.21.15.46:0/3965181523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe74413be90 0x7fe74413e280 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:38.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.837+0000 7fe74a042640 1 --2- 172.21.15.46:0/3965181523 >> v2:172.21.15.114:3300/0 conn(0x7fe7440a3ff0 0x7fe7440a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:38.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.837+0000 7fe74a042640 1 -- 172.21.15.46:0/3965181523 >> 172.21.15.46:0/3965181523 conn(0x7fe7440a6340 msgr2=0x7fe74406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:38.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.837+0000 7fe74a042640 1 -- 172.21.15.46:0/3965181523 shutdown_connections 2024-09-16T04:47:38.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.837+0000 7fe74a042640 1 -- 172.21.15.46:0/3965181523 wait complete. 2024-09-16T04:47:38.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.838+0000 7fe74a042640 10 monclient: build_initial_monmap 2024-09-16T04:47:38.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.838+0000 7fe74a042640 10 monclient: monmap: 2024-09-16T04:47:38.852 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:38.852 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:38.853 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.838+0000 7fe74a042640 1 Processor -- start 2024-09-16T04:47:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.838+0000 7fe74a042640 1 -- start start 2024-09-16T04:47:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.838+0000 7fe74a042640 10 monclient: init 2024-09-16T04:47:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe74a042640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe74a042640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:47:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe74a042640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7440a3c10 0x7fe7441cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe74a042640 10 monclient(hunting): picked mon.noname-b con 0x7fe7440a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe74a042640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7440a3ff0 0x7fe744136f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe74a042640 10 monclient(hunting): picked mon.noname-a con 0x7fe7440a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe74a042640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74413be90 0x7fe744139f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe74a042640 10 monclient(hunting): picked mon.noname-c con 0x7fe74413be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe74a042640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe74a042640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe7440a6b40 con 0x7fe7440a3ff0 2024-09-16T04:47:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe74a042640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe74a042640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe7440a69c0 con 0x7fe7440a3c10 2024-09-16T04:47:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe74a042640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe74a042640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe74413e7c0 con 0x7fe74413be90 2024-09-16T04:47:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe74a042640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe74a042640 10 monclient(hunting): authenticate will time out at 1257.255226s 2024-09-16T04:47:38.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe742ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7440a3ff0 0x7fe744136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:38.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe742ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7440a3ff0 0x7fe744136f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41726/0 (socket says 172.21.15.46:41726) 2024-09-16T04:47:38.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe743fff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74413be90 0x7fe744139f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:38.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe742ffd640 1 -- 172.21.15.46:0/2632355967 learned_addr learned my addr 172.21.15.46:0/2632355967 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:38.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe742ffd640 10 monclient(hunting): get_auth_request con 0x7fe7440a3ff0 auth_method 0 2024-09-16T04:47:38.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe742ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:38.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe742ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:38.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe742ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:38.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe743fff640 10 monclient(hunting): get_auth_request con 0x7fe74413be90 auth_method 0 2024-09-16T04:47:38.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe743fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:38.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe743fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:38.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe743fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:38.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe7437fe640 1 --2- 172.21.15.46:0/2632355967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7440a3c10 0x7fe7441cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe7437fe640 10 monclient(hunting): get_auth_request con 0x7fe7440a3c10 auth_method 0 2024-09-16T04:47:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe7437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe7437fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe7437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe742ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe742ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe742ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe743fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe743fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe743fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe7437fe640 1 -- 172.21.15.46:0/2632355967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7440a3c10 msgr2=0x7fe7441cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:47:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe7437fe640 1 -- 172.21.15.46:0/2632355967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7440a3c10 msgr2=0x7fe7441cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:47:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe7437fe640 1 --2- 172.21.15.46:0/2632355967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7440a3c10 0x7fe7441cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:47:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe7437fe640 1 --2- 172.21.15.46:0/2632355967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7440a3c10 0x7fe7441cb9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:47:38.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe742ffd640 10 monclient(hunting): handle_auth_done global_id 104247 payload 824 2024-09-16T04:47:38.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe742ffd640 1 -- 172.21.15.46:0/2632355967 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74413be90 msgr2=0x7fe744139f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:38.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe742ffd640 1 --2- 172.21.15.46:0/2632355967 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74413be90 0x7fe744139f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:38.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe742ffd640 1 -- 172.21.15.46:0/2632355967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7440a3c10 msgr2=0x7fe7441cb9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:47:38.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe742ffd640 1 --2- 172.21.15.46:0/2632355967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7440a3c10 0x7fe7441cb9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:38.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe742ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:47:38.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe742ffd640 1 monclient: found mon.noname-a 2024-09-16T04:47:38.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe742ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:38.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe742ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:38.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe742ffd640 1 -- 172.21.15.46:0/2632355967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe74413bba0 con 0x7fe7440a3ff0 2024-09-16T04:47:38.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe743fff640 1 --2- 172.21.15.46:0/2632355967 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74413be90 0x7fe744139f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:38.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.839+0000 7fe742ffd640 1 --2- 172.21.15.46:0/2632355967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7440a3ff0 0x7fe744136f20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe73800d830 tx=0x7fe73800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:38.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.840+0000 7fe72bfff640 1 -- 172.21.15.46:0/2632355967 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73800b660 con 0x7fe7440a3ff0 2024-09-16T04:47:38.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.840+0000 7fe72bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:38.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.840+0000 7fe72bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:38.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.840+0000 7fe72bfff640 10 monclient: dump: 2024-09-16T04:47:38.858 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:38.858 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:38.858 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:38.858 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:38.858 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:38.858 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:38.858 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:38.858 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:38.858 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:38.858 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.840+0000 7fe72bfff640 10 monclient: _finish_auth 0 2024-09-16T04:47:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.840+0000 7fe72bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:47:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.840+0000 7fe72bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.840+0000 7fe74a042640 5 monclient: authenticate success, global_id 104247 2024-09-16T04:47:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.840+0000 7fe72bfff640 1 -- 172.21.15.46:0/2632355967 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe738019070 con 0x7fe7440a3ff0 2024-09-16T04:47:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.840+0000 7fe74a042640 10 monclient: _renew_subs 2024-09-16T04:47:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.840+0000 7fe74a042640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.840+0000 7fe74a042640 1 -- 172.21.15.46:0/2632355967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7440a0400 con 0x7fe7440a3ff0 2024-09-16T04:47:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.840+0000 7fe72bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.840+0000 7fe72bfff640 1 -- 172.21.15.46:0/2632355967 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73800ba20 con 0x7fe7440a3ff0 2024-09-16T04:47:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.840+0000 7fe72bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.840+0000 7fe72bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.840+0000 7fe72bfff640 10 monclient: dump: 2024-09-16T04:47:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:38.860 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:38.860 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:38.860 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:38.860 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:38.860 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:38.860 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:38.860 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:38.860 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:38.860 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:38.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.840+0000 7fe74a042640 10 monclient: _renew_subs 2024-09-16T04:47:38.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.840+0000 7fe74a042640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:38.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.840+0000 7fe74a042640 1 -- 172.21.15.46:0/2632355967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7441215d0 con 0x7fe7440a3ff0 2024-09-16T04:47:38.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.840+0000 7fe741ffb640 4 set_mon_vals no callback set 2024-09-16T04:47:38.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.841+0000 7fe72bfff640 1 -- 172.21.15.46:0/2632355967 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fe73800bbc0 con 0x7fe7440a3ff0 2024-09-16T04:47:38.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.841+0000 7fe74a042640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.841+0000 7fe74a042640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.841+0000 7fe74a042640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.841+0000 7fe74a042640 1 -- 172.21.15.46:0/2632355967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7440a43d0 con 0x7fe7440a3ff0 2024-09-16T04:47:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.841+0000 7fe72bfff640 1 --2- 172.21.15.46:0/2632355967 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe7300383b0 0x7fe73003a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.841+0000 7fe72bfff640 1 -- 172.21.15.46:0/2632355967 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7fe73804bb10 con 0x7fe7440a3ff0 2024-09-16T04:47:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.841+0000 7fe7437fe640 1 --2- 172.21.15.46:0/2632355967 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe7300383b0 0x7fe73003a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.841+0000 7fe7437fe640 10 monclient: get_auth_request con 0x7fe7300383b0 auth_method 0 2024-09-16T04:47:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.842+0000 7fe7437fe640 1 --2- 172.21.15.46:0/2632355967 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe7300383b0 0x7fe73003a870 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe734005c00 tx=0x7fe734009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.846+0000 7fe72bfff640 1 -- 172.21.15.46:0/2632355967 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe73801d3c0 con 0x7fe7440a3ff0 2024-09-16T04:47:38.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.846+0000 7fe72bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:38.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:38.846+0000 7fe72bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:39.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.089+0000 7fe74a042640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:47:39.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.089+0000 7fe74a042640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:47:39.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.089+0000 7fe74a042640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:39.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.089+0000 7fe74a042640 1 -- 172.21.15.46:0/2632355967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe7441204c0 con 0x7fe7440a3ff0 2024-09-16T04:47:39.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.089+0000 7fe72bfff640 1 -- 172.21.15.46:0/2632355967 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fe738013ae0 con 0x7fe7440a3ff0 2024-09-16T04:47:39.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.089+0000 7fe72bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:47:39.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.089+0000 7fe72bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:47:39.092 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":100,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:47:39.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.091+0000 7fe74a042640 1 -- 172.21.15.46:0/2632355967 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe7300383b0 msgr2=0x7fe73003a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.091+0000 7fe74a042640 1 --2- 172.21.15.46:0/2632355967 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe7300383b0 0x7fe73003a870 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe734005c00 tx=0x7fe734009040 comp rx=0 tx=0).stop 2024-09-16T04:47:39.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.092+0000 7fe74a042640 10 monclient: shutdown 2024-09-16T04:47:39.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.092+0000 7fe74a042640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:39.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.092+0000 7fe74a042640 1 -- 172.21.15.46:0/2632355967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7440a3ff0 msgr2=0x7fe744136f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.092+0000 7fe74a042640 1 --2- 172.21.15.46:0/2632355967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7440a3ff0 0x7fe744136f20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe73800d830 tx=0x7fe73800dcf0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.092+0000 7fe74a042640 1 -- 172.21.15.46:0/2632355967 shutdown_connections 2024-09-16T04:47:39.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.092+0000 7fe74a042640 1 --2- 172.21.15.46:0/2632355967 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe74413be90 0x7fe744139f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.092+0000 7fe74a042640 1 --2- 172.21.15.46:0/2632355967 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe7300383b0 0x7fe73003a870 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.092+0000 7fe74a042640 1 --2- 172.21.15.46:0/2632355967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7440a3ff0 0x7fe744136f20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.092+0000 7fe74a042640 1 --2- 172.21.15.46:0/2632355967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7440a3c10 0x7fe7441cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.092+0000 7fe74a042640 1 -- 172.21.15.46:0/2632355967 >> 172.21.15.46:0/2632355967 conn(0x7fe7440a6340 msgr2=0x7fe7440a2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:39.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.092+0000 7fe74a042640 1 -- 172.21.15.46:0/2632355967 shutdown_connections 2024-09-16T04:47:39.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.092+0000 7fe74a042640 1 -- 172.21.15.46:0/2632355967 wait complete. 2024-09-16T04:47:39.104 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:47:39.105 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:47:39.105 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:47:39.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.205+0000 7f35130e6640 10 monclient: get_monmap_and_config 2024-09-16T04:47:39.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.205+0000 7f35130e6640 10 monclient: build_initial_monmap 2024-09-16T04:47:39.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.205+0000 7f35130e6640 10 monclient: monmap: 2024-09-16T04:47:39.207 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:39.207080+0000 2024-09-16T04:47:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:39.207080+0000 2024-09-16T04:47:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:39.208 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.206+0000 7f35130e6640 1 Processor -- start 2024-09-16T04:47:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.206+0000 7f35130e6640 1 -- start start 2024-09-16T04:47:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.206+0000 7f35130e6640 10 monclient: init 2024-09-16T04:47:39.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.206+0000 7f35130e6640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:39.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.206+0000 7f35130e6640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-16T04:47:39.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.206+0000 7f35130e6640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f350c0a3ba0 0x7f350c09bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.206+0000 7f35130e6640 10 monclient(hunting): picked mon.noname-c con 0x7f350c0a3ba0 addr v2:172.21.15.114:3300/0 2024-09-16T04:47:39.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.206+0000 7f35130e6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f350c09c1c0 addr v1:172.21.15.46:6789/0 2024-09-16T04:47:39.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.206+0000 7f35130e6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f350c0a4690 addr v1:172.21.15.114:6789/0 2024-09-16T04:47:39.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.206+0000 7f35130e6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f350c09afc0 con 0x7f350c09c1c0 2024-09-16T04:47:39.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.206+0000 7f35130e6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f350c09b580 con 0x7f350c0a4690 2024-09-16T04:47:39.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.206+0000 7f35130e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.206+0000 7f35130e6640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f350c09b860 con 0x7f350c0a3ba0 2024-09-16T04:47:39.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.206+0000 7f35130e6640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:39.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.206+0000 7f35130e6640 10 monclient(hunting): authenticate will time out at 1257.622943s 2024-09-16T04:47:39.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f350c09c1c0 0x7f350c0a3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33284/0 (socket says 172.21.15.46:33284) 2024-09-16T04:47:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350bfff640 1 -- 172.21.15.46:0/1139280137 learned_addr learned my addr 172.21.15.46:0/1139280137 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:47:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 1 -- 172.21.15.46:0/1139280137 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3341672288 0 0) 0x7f350c09afc0 con 0x7f350c09c1c0 2024-09-16T04:47:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:47:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 10 monclient(hunting): my global_id is 104250 2024-09-16T04:47:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 1 -- 172.21.15.46:0/1139280137 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34fc003520 con 0x7f350c09c1c0 2024-09-16T04:47:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f3510e82640 1 --2- 172.21.15.46:0/1139280137 >> v2:172.21.15.114:3300/0 conn(0x7f350c0a3ba0 0x7f350c09bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f3510e82640 10 monclient(hunting): get_auth_request con 0x7f350c0a3ba0 auth_method 0 2024-09-16T04:47:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f3510e82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f3510e82640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f3510e82640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f3510e82640 1 -- 172.21.15.46:0/1139280137 >> v2:172.21.15.114:3300/0 conn(0x7f350c0a3ba0 msgr2=0x7f350c09bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:47:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f3510e82640 1 -- 172.21.15.46:0/1139280137 >> v2:172.21.15.114:3300/0 conn(0x7f350c0a3ba0 msgr2=0x7f350c09bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:47:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f3510e82640 1 --2- 172.21.15.46:0/1139280137 >> v2:172.21.15.114:3300/0 conn(0x7f350c0a3ba0 0x7f350c09bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:47:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 1 -- 172.21.15.46:0/1139280137 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 438199963 0 0) 0x7f34fc003520 con 0x7f350c09c1c0 2024-09-16T04:47:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f3510e82640 1 --2- 172.21.15.46:0/1139280137 >> v2:172.21.15.114:3300/0 conn(0x7f350c0a3ba0 0x7f350c09bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:47:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 1 -- 172.21.15.46:0/1139280137 >> v2:172.21.15.114:3300/0 conn(0x7f350c0a3ba0 msgr2=0x7f350c09bc80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:47:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 1 --2- 172.21.15.46:0/1139280137 >> v2:172.21.15.114:3300/0 conn(0x7f350c0a3ba0 0x7f350c09bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 1 -- 172.21.15.46:0/1139280137 >> v1:172.21.15.114:6789/0 conn(0x7f350c0a4690 legacy=0x7f350c13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:47:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:47:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:47:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 1 -- 172.21.15.46:0/1139280137 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f350c0a53c0 con 0x7f350c09c1c0 2024-09-16T04:47:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:47:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:47:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 1 -- 172.21.15.46:0/1139280137 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f34f8001670 con 0x7f350c09c1c0 2024-09-16T04:47:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f35130e6640 5 monclient: authenticate success, global_id 104250 2024-09-16T04:47:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f35130e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:47:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 10 monclient: dump: 2024-09-16T04:47:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.224 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:47:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 1 -- 172.21.15.46:0/1139280137 >> v1:172.21.15.46:6789/0 conn(0x7f350c09c1c0 legacy=0x7f350c0a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:47:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 1 --2- 172.21.15.46:0/1139280137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34fc0039b0 0x7f34fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 10 monclient(hunting): picked mon.b con 0x7f34fc0039b0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 1 --2- 172.21.15.46:0/1139280137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34fc004800 0x7f34fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 10 monclient(hunting): picked mon.a con 0x7f34fc004800 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 1 --2- 172.21.15.46:0/1139280137 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f34fc0052e0 0x7f34fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 10 monclient(hunting): picked mon.c con 0x7f34fc0052e0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:39.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 1 -- 172.21.15.46:0/1139280137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f350c09b860 con 0x7f34fc004800 2024-09-16T04:47:39.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 1 -- 172.21.15.46:0/1139280137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f34fc00c1b0 con 0x7f34fc0039b0 2024-09-16T04:47:39.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 1 -- 172.21.15.46:0/1139280137 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f34fc00c3b0 con 0x7f34fc0052e0 2024-09-16T04:47:39.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:39.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:47:39.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f35130e6640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:39.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350bfff640 1 --2- 172.21.15.46:0/1139280137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34fc004800 0x7f34fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350bfff640 10 monclient(hunting): get_auth_request con 0x7f34fc004800 auth_method 0 2024-09-16T04:47:39.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:47:39.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f3511683640 1 --2- 172.21.15.46:0/1139280137 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f34fc0052e0 0x7f34fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f3510e82640 1 --2- 172.21.15.46:0/1139280137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34fc0039b0 0x7f34fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f3510e82640 10 monclient(hunting): get_auth_request con 0x7f34fc0039b0 auth_method 0 2024-09-16T04:47:39.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f3510e82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f3510e82640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f3510e82640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:47:39.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f3511683640 10 monclient(hunting): get_auth_request con 0x7f34fc0052e0 auth_method 0 2024-09-16T04:47:39.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f3511683640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f3511683640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f3511683640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:47:39.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.207+0000 7f350bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:47:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f3511683640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f3511683640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f3511683640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:47:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f3510e82640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f3510e82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f3510e82640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:47:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f350bfff640 10 monclient(hunting): handle_auth_done global_id 104250 payload 293 2024-09-16T04:47:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f350bfff640 1 -- 172.21.15.46:0/1139280137 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f34fc0052e0 msgr2=0x7f34fc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f350bfff640 1 --2- 172.21.15.46:0/1139280137 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f34fc0052e0 0x7f34fc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f350bfff640 1 -- 172.21.15.46:0/1139280137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34fc0039b0 msgr2=0x7f34fc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f350bfff640 1 --2- 172.21.15.46:0/1139280137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34fc0039b0 0x7f34fc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f350bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:47:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f350bfff640 1 monclient: found mon.a 2024-09-16T04:47:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f350bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f350bfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f350bfff640 1 -- 172.21.15.46:0/1139280137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34fc003520 con 0x7f34fc004800 2024-09-16T04:47:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f350bfff640 10 monclient: _finish_auth 0 2024-09-16T04:47:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f350bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:47:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f350bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f3511683640 1 --2- 172.21.15.46:0/1139280137 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f34fc0052e0 0x7f34fc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f3511683640 1 -- 172.21.15.46:0/1139280137 reap_dead start 2024-09-16T04:47:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.208+0000 7f350bfff640 1 --2- 172.21.15.46:0/1139280137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34fc004800 0x7f34fc004c80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f34f8001eb0 tx=0x7f34fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7714164033333289 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f350b7fe640 1 -- 172.21.15.46:0/1139280137 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f8007d30 con 0x7f34fc004800 2024-09-16T04:47:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f350b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f350b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f350b7fe640 10 monclient: dump: 2024-09-16T04:47:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.229 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f35130e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f350b7fe640 1 -- 172.21.15.46:0/1139280137 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34f8005af0 con 0x7f34fc004800 2024-09-16T04:47:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f350b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f35130e6640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f35130e6640 4 set_mon_vals no callback set 2024-09-16T04:47:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f350b7fe640 1 -- 172.21.15.46:0/1139280137 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f8007070 con 0x7f34fc004800 2024-09-16T04:47:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f350b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f350b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f350b7fe640 10 monclient: dump: 2024-09-16T04:47:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.231 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f35130e6640 10 monclient: shutdown 2024-09-16T04:47:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f35130e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f35130e6640 1 -- 172.21.15.46:0/1139280137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34fc004800 msgr2=0x7f34fc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f35130e6640 1 --2- 172.21.15.46:0/1139280137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34fc004800 0x7f34fc004c80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f34f8001eb0 tx=0x7f34fc005830 comp rx=0 tx=0).stop 2024-09-16T04:47:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f35130e6640 1 -- 172.21.15.46:0/1139280137 shutdown_connections 2024-09-16T04:47:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f35130e6640 1 --2- 172.21.15.46:0/1139280137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34fc004800 0x7f34fc004c80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f35130e6640 1 -- 172.21.15.46:0/1139280137 >> 172.21.15.46:0/1139280137 conn(0x7f350c0a60c0 msgr2=0x7f350c0a64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f35130e6640 1 -- 172.21.15.46:0/1139280137 shutdown_connections 2024-09-16T04:47:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.209+0000 7f35130e6640 1 -- 172.21.15.46:0/1139280137 wait complete. 2024-09-16T04:47:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.210+0000 7f35130e6640 10 monclient: build_initial_monmap 2024-09-16T04:47:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.210+0000 7f35130e6640 10 monclient: monmap: 2024-09-16T04:47:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:39.232 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.210+0000 7f35130e6640 1 Processor -- start 2024-09-16T04:47:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.210+0000 7f35130e6640 1 -- start start 2024-09-16T04:47:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.210+0000 7f35130e6640 10 monclient: init 2024-09-16T04:47:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:47:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f34fc004800 0x7f350c1cb730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 10 monclient(hunting): picked mon.noname-c con 0x7f34fc004800 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f350c136e70 0x7f350c1372d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 10 monclient(hunting): picked mon.noname-b con 0x7f350c136e70 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f350c1208a0 0x7f350c120d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 10 monclient(hunting): picked mon.noname-a con 0x7f350c1208a0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f350c09c010 con 0x7f350c1208a0 2024-09-16T04:47:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f350c09be90 con 0x7f350c136e70 2024-09-16T04:47:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f350c09b2a0 con 0x7f34fc004800 2024-09-16T04:47:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 10 monclient(hunting): authenticate will time out at 1257.627187s 2024-09-16T04:47:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3510e82640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f34fc004800 0x7f350c1cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3511683640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f350c1208a0 0x7f350c120d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3511683640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f350c1208a0 0x7f350c120d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41756/0 (socket says 172.21.15.46:41756) 2024-09-16T04:47:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3510e82640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f34fc004800 0x7f350c1cb730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38696/0 (socket says 172.21.15.46:38696) 2024-09-16T04:47:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3511683640 1 -- 172.21.15.46:0/3171996466 learned_addr learned my addr 172.21.15.46:0/3171996466 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3511683640 10 monclient(hunting): get_auth_request con 0x7f350c1208a0 auth_method 0 2024-09-16T04:47:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3511683640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3511683640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3511683640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3510e82640 10 monclient(hunting): get_auth_request con 0x7f34fc004800 auth_method 0 2024-09-16T04:47:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3510e82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3510e82640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3510e82640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3511683640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3511683640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3511683640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3510e82640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3510e82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3510e82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f350bfff640 1 --2- 172.21.15.46:0/3171996466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f350c136e70 0x7f350c1372d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3511683640 10 monclient(hunting): handle_auth_done global_id 104253 payload 824 2024-09-16T04:47:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3511683640 1 -- 172.21.15.46:0/3171996466 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f34fc004800 msgr2=0x7f350c1cb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3511683640 1 --2- 172.21.15.46:0/3171996466 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f34fc004800 0x7f350c1cb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3511683640 1 -- 172.21.15.46:0/3171996466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f350c136e70 msgr2=0x7f350c1372d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3511683640 1 --2- 172.21.15.46:0/3171996466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f350c136e70 0x7f350c1372d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3511683640 10 monclient: _finish_hunting 0 2024-09-16T04:47:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3511683640 1 monclient: found mon.noname-a 2024-09-16T04:47:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3511683640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3511683640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3511683640 1 -- 172.21.15.46:0/3171996466 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f350c137810 con 0x7f350c1208a0 2024-09-16T04:47:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f350bfff640 10 monclient: get_auth_request con 0x7f350c136e70 auth_method 0 2024-09-16T04:47:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f350bfff640 1 --2- 172.21.15.46:0/3171996466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f350c136e70 0x7f350c1372d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:47:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f3511683640 1 --2- 172.21.15.46:0/3171996466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f350c1208a0 0x7f350c120d20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f350c09ccf0 tx=0x7f350400ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35097fa640 1 -- 172.21.15.46:0/3171996466 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3504001a70 con 0x7f350c1208a0 2024-09-16T04:47:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35097fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35097fa640 10 monclient: dump: 2024-09-16T04:47:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.237 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35097fa640 10 monclient: _finish_auth 0 2024-09-16T04:47:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35097fa640 10 monclient: _check_auth_tickets 2024-09-16T04:47:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35097fa640 1 -- 172.21.15.46:0/3171996466 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3504018070 con 0x7f350c1208a0 2024-09-16T04:47:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 5 monclient: authenticate success, global_id 104253 2024-09-16T04:47:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35097fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 10 monclient: _renew_subs 2024-09-16T04:47:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35097fa640 1 -- 172.21.15.46:0/3171996466 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3504003da0 con 0x7f350c1208a0 2024-09-16T04:47:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 1 -- 172.21.15.46:0/3171996466 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f350c0a0390 con 0x7f350c1208a0 2024-09-16T04:47:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35097fa640 10 monclient: dump: 2024-09-16T04:47:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.239 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f350affd640 4 set_mon_vals no callback set 2024-09-16T04:47:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 10 monclient: _renew_subs 2024-09-16T04:47:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.211+0000 7f35130e6640 1 -- 172.21.15.46:0/3171996466 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f350c0a1320 con 0x7f350c1208a0 2024-09-16T04:47:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.212+0000 7f35097fa640 1 -- 172.21.15.46:0/3171996466 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f35040050d0 con 0x7f350c1208a0 2024-09-16T04:47:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.212+0000 7f35130e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.212+0000 7f35130e6640 1 --2- 172.21.15.46:0/3171996466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f350c121260 0x7f350c09cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.212+0000 7f35130e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.212+0000 7f35130e6640 1 -- 172.21.15.46:0/3171996466 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f350c0a0e60 con 0x7f350c121260 2024-09-16T04:47:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.212+0000 7f3510e82640 1 --2- 172.21.15.46:0/3171996466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f350c121260 0x7f350c09cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.212+0000 7f3510e82640 10 monclient: get_auth_request con 0x7f350c121260 auth_method 0 2024-09-16T04:47:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.212+0000 7f3510e82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.212+0000 7f3510e82640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.212+0000 7f3510e82640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.212+0000 7f35097fa640 1 --2- 172.21.15.46:0/3171996466 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f34f00383b0 0x7f34f003a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.212+0000 7f35097fa640 1 -- 172.21.15.46:0/3171996466 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7f350404b480 con 0x7f350c1208a0 2024-09-16T04:47:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.212+0000 7f3510e82640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.212+0000 7f3510e82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.213+0000 7f3510e82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.213+0000 7f3510e82640 10 monclient(hunting): handle_auth_done global_id 104256 payload 824 2024-09-16T04:47:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.213+0000 7f3510e82640 1 -- 172.21.15.46:0/3171996466 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f350c0a4130 con 0x7f350c121260 2024-09-16T04:47:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.213+0000 7f350bfff640 1 --2- 172.21.15.46:0/3171996466 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f34f00383b0 0x7f34f003a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.213+0000 7f350bfff640 10 monclient: get_auth_request con 0x7f34f00383b0 auth_method 0 2024-09-16T04:47:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.213+0000 7f3510e82640 1 --2- 172.21.15.46:0/3171996466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f350c121260 0x7f350c09cd60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3500003840 tx=0x7f3500003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.213+0000 7f35097fa640 1 -- 172.21.15.46:0/3171996466 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3500019070 con 0x7f350c121260 2024-09-16T04:47:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.213+0000 7f35097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.213+0000 7f35097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.213+0000 7f35097fa640 10 monclient: dump: 2024-09-16T04:47:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.241 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.213+0000 7f35097fa640 1 -- 172.21.15.46:0/3171996466 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3500013b40 con 0x7f350c121260 2024-09-16T04:47:39.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.213+0000 7f35097fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:39.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.213+0000 7f35097fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:39.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.213+0000 7f35097fa640 1 -- 172.21.15.46:0/3171996466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f350c121260 msgr2=0x7f350c09cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.213+0000 7f35097fa640 1 --2- 172.21.15.46:0/3171996466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f350c121260 0x7f350c09cd60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3500003840 tx=0x7f3500003d00 comp rx=0 tx=0).stop 2024-09-16T04:47:39.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.213+0000 7f35097fa640 1 -- 172.21.15.46:0/3171996466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f350c121260 msgr2=0x7f350c09cd60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:47:39.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.213+0000 7f35097fa640 1 --2- 172.21.15.46:0/3171996466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f350c121260 0x7f350c09cd60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.213+0000 7f350bfff640 1 --2- 172.21.15.46:0/3171996466 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f34f00383b0 0x7f34f003a870 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f34fc004780 tx=0x7f34f802d000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:39.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.219+0000 7f35130e6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:47:39.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.219+0000 7f35130e6640 1 --2- 172.21.15.46:0/3171996466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f350c09d2a0 0x7f350c09d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.219+0000 7f35130e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.219+0000 7f35130e6640 1 -- 172.21.15.46:0/3171996466 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f350c122e20 con 0x7f350c09d2a0 2024-09-16T04:47:39.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f3510e82640 1 --2- 172.21.15.46:0/3171996466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f350c09d2a0 0x7f350c09d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f3510e82640 10 monclient: get_auth_request con 0x7f350c09d2a0 auth_method 0 2024-09-16T04:47:39.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f3510e82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f3510e82640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f3510e82640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f3510e82640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f3510e82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f3510e82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:39.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f3510e82640 10 monclient(hunting): handle_auth_done global_id 104259 payload 824 2024-09-16T04:47:39.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f3510e82640 1 -- 172.21.15.46:0/3171996466 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f350c1cbc70 con 0x7f350c09d2a0 2024-09-16T04:47:39.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f3510e82640 1 --2- 172.21.15.46:0/3171996466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f350c09d2a0 0x7f350c09d6b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3500003840 tx=0x7f35000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:39.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f35097fa640 1 -- 172.21.15.46:0/3171996466 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3500023770 con 0x7f350c09d2a0 2024-09-16T04:47:39.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f35097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f35097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:39.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f35097fa640 10 monclient: dump: 2024-09-16T04:47:39.245 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.245 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.245 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.246 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.246 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.246 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.246 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.246 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.246 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.246 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f35097fa640 1 -- 172.21.15.46:0/3171996466 <== mon.0 v2:172.21.15.46:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7f3500023770 con 0x7f350c09d2a0 2024-09-16T04:47:39.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f35097fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:47:39.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f35097fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:47:39.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f35097fa640 1 -- 172.21.15.46:0/3171996466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f350c09d2a0 msgr2=0x7f350c09d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f35097fa640 1 --2- 172.21.15.46:0/3171996466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f350c09d2a0 0x7f350c09d6b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3500003840 tx=0x7f35000046c0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f35097fa640 1 -- 172.21.15.46:0/3171996466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f350c09d2a0 msgr2=0x7f350c09d6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:47:39.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.220+0000 7f35097fa640 1 --2- 172.21.15.46:0/3171996466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f350c09d2a0 0x7f350c09d6b0 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3500003840 tx=0x7f35000046c0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.247 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:47:39.247 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:47:39.247 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:47:39.247 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "leader", 2024-09-16T04:47:39.247 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 100, 2024-09-16T04:47:39.247 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 52278, 2024-09-16T04:47:39.247 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:47:39.247 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:47:39.247 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:47:39.247 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:47:39.248 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:39.248 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 1, 2024-09-16T04:47:39.248 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:47:39.248 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:47:39.248 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:47:39.248 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:47:39.248 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:47:39.248 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:47:39.248 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:47:39.248 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:47:39.248 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:47:39.248 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:47:39.248 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:47:39.248 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:47:39.249 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:47:39.249 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:47:39.249 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:39.249 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:47:39.249 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:47:39.249 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:47:39.249 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:47:39.249 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:47:39.249 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:47:39.249 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:47:39.249 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:47:39.249 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:47:39.249 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:47:39.249 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:47:39.250 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:47:39.250 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:47:39.250 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:39.250 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.250 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:47:39.250 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:47:39.250 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:47:39.250 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:47:39.250 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:47:39.250 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:47:39.250 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:47:39.250 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:47:39.250 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:47:39.250 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:47:39.251 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:47:39.251 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:47:39.251 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:47:39.251 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:47:39.251 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:47:39.251 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:47:39.251 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:47:39.251 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:47:39.251 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:47:39.251 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:47:39.251 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:47:39.251 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:47:39.251 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:47:39.251 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:47:39.252 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:47:39.252 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:47:39.252 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:47:39.252 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:47:39.252 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:39.252 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:47:39.252 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.252 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:47:39.252 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.252 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:47:39.252 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:47:39.252 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:47:39.252 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:47:39.252 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.253 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:47:39.253 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:47:39.253 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:39.253 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.253 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.253 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:47:39.253 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:47:39.253 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:39.253 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:39.253 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:39.253 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.253 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:47:39.253 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:47:39.254 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:47:39.254 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:47:39.254 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:47:39.254 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.254 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.254 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:47:39.254 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:47:39.254 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:47:39.254 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:47:39.254 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.254 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:47:39.254 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:47:39.254 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:39.254 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.255 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.255 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:47:39.255 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:47:39.255 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:39.255 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:39.255 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:39.255 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.255 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:47:39.255 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:47:39.255 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:47:39.255 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:47:39.255 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:47:39.255 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.255 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.256 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:47:39.256 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:47:39.256 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:47:39.256 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:47:39.256 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.256 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:47:39.256 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:47:39.256 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:39.256 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.256 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.256 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:47:39.256 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:47:39.256 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:39.256 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:39.257 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:39.257 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.257 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:47:39.257 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:47:39.257 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:47:39.257 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:47:39.257 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:47:39.257 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:39.257 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:39.257 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.257 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:47:39.257 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:47:39.257 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.257 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:47:39.258 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:47:39.258 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:47:39.258 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:39.258 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:39.258 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:47:39.258 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.258 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:47:39.258 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:47:39.258 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:47:39.258 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:39.258 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:39.258 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:47:39.258 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.258 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:47:39.259 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:47:39.259 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:47:39.259 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:39.259 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:39.259 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": [ 2024-09-16T04:47:39.259 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.259 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:47:39.259 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:47:39.259 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:47:39.259 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:39.259 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:39.259 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.259 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:47:39.259 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:47:39.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.221+0000 7f35130e6640 1 -- 172.21.15.46:0/3171996466 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f34f00383b0 msgr2=0x7f34f003a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.221+0000 7f35130e6640 1 --2- 172.21.15.46:0/3171996466 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f34f00383b0 0x7f34f003a870 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f34fc004780 tx=0x7f34f802d000 comp rx=0 tx=0).stop 2024-09-16T04:47:39.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.221+0000 7f35130e6640 10 monclient: shutdown 2024-09-16T04:47:39.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.221+0000 7f35130e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:39.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.221+0000 7f35130e6640 1 -- 172.21.15.46:0/3171996466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f350c1208a0 msgr2=0x7f350c120d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.221+0000 7f3511683640 1 -- 172.21.15.46:0/3171996466 reap_dead start 2024-09-16T04:47:39.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.221+0000 7f35130e6640 1 --2- 172.21.15.46:0/3171996466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f350c1208a0 0x7f350c120d20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f350c09ccf0 tx=0x7f350400ff70 comp rx=0 tx=0).stop 2024-09-16T04:47:39.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.221+0000 7f35130e6640 1 -- 172.21.15.46:0/3171996466 shutdown_connections 2024-09-16T04:47:39.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.221+0000 7f35130e6640 1 --2- 172.21.15.46:0/3171996466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f350c1208a0 0x7f350c120d20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.221+0000 7f35130e6640 1 -- 172.21.15.46:0/3171996466 >> 172.21.15.46:0/3171996466 conn(0x7f350c0a60c0 msgr2=0x7f350c0a2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:39.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.221+0000 7f35130e6640 1 -- 172.21.15.46:0/3171996466 shutdown_connections 2024-09-16T04:47:39.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.221+0000 7f35130e6640 1 -- 172.21.15.46:0/3171996466 wait complete. 2024-09-16T04:47:39.262 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:47:39.262 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:47:39.262 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:47:39.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.363+0000 7fe6c14af640 10 monclient: get_monmap_and_config 2024-09-16T04:47:39.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.363+0000 7fe6c14af640 10 monclient: build_initial_monmap 2024-09-16T04:47:39.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.363+0000 7fe6c14af640 10 monclient: monmap: 2024-09-16T04:47:39.366 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:39.366 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.366 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:39.364591+0000 2024-09-16T04:47:39.366 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:39.364591+0000 2024-09-16T04:47:39.366 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:39.366 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.366 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:39.366 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:39.366 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:39.367 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:39.367 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:39.367 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.363+0000 7fe6c14af640 1 Processor -- start 2024-09-16T04:47:39.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.363+0000 7fe6c14af640 1 -- start start 2024-09-16T04:47:39.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.363+0000 7fe6c14af640 10 monclient: init 2024-09-16T04:47:39.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6c14af640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:39.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6c14af640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-16T04:47:39.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6c14af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6bc0a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:47:39.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6c14af640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe6bc0a3ff0 0x7fe6bc0a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6c14af640 10 monclient(hunting): picked mon.noname-a con 0x7fe6bc0a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:47:39.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6c14af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe6bc13be90 addr v1:172.21.15.46:6789/0 2024-09-16T04:47:39.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6c14af640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6bc09b940 con 0x7fe6bc13be90 2024-09-16T04:47:39.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6c14af640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6bc09ab00 con 0x7fe6bc0a3c10 2024-09-16T04:47:39.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6c14af640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6c14af640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe6bc09b030 con 0x7fe6bc0a3ff0 2024-09-16T04:47:39.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6c14af640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:39.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6c14af640 10 monclient(hunting): authenticate will time out at 1257.780320s 2024-09-16T04:47:39.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6ba7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe6bc0a3ff0 0x7fe6bc0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6ba7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe6bc0a3ff0 0x7fe6bc0a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41780/0 (socket says 172.21.15.46:41780) 2024-09-16T04:47:39.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6ba7fc640 1 -- 172.21.15.46:0/3554929039 learned_addr learned my addr 172.21.15.46:0/3554929039 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:39.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6bb7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe6bc13be90 0x7fe6bc13e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33286/0 (socket says 172.21.15.46:33286) 2024-09-16T04:47:39.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6ba7fc640 10 monclient(hunting): get_auth_request con 0x7fe6bc0a3ff0 auth_method 0 2024-09-16T04:47:39.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6ba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6ba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6ba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:39.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6ba7fc640 10 monclient(hunting): handle_auth_done global_id 104262 payload 274 2024-09-16T04:47:39.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6ba7fc640 1 -- 172.21.15.46:0/3554929039 >> v1:172.21.15.114:6789/0 conn(0x7fe6bc0a3c10 legacy=0x7fe6bc09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6ba7fc640 1 -- 172.21.15.46:0/3554929039 >> v1:172.21.15.46:6789/0 conn(0x7fe6bc13be90 legacy=0x7fe6bc13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6ba7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:47:39.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6ba7fc640 1 monclient: found mon.noname-a 2024-09-16T04:47:39.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:39.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6ba7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:39.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6ba7fc640 1 -- 172.21.15.46:0/3554929039 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6bc09b310 con 0x7fe6bc0a3ff0 2024-09-16T04:47:39.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6b9ffb640 1 -- 172.21.15.46:0/3554929039 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3812172873 0 0) 0x7fe6bc09b940 con 0x7fe6bc13be90 2024-09-16T04:47:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.364+0000 7fe6b9ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:47:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.365+0000 7fe6ba7fc640 1 --2- 172.21.15.46:0/3554929039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6bc0a3ff0 0x7fe6bc0a43d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0009870 tx=0x7fe6b002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03eac6784598e64 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.365+0000 7fe6b9ffb640 1 -- 172.21.15.46:0/3554929039 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0037070 con 0x7fe6bc0a3ff0 2024-09-16T04:47:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.365+0000 7fe6b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.365+0000 7fe6b9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.365+0000 7fe6b9ffb640 10 monclient: dump: 2024-09-16T04:47:39.386 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.386 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.386 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.386 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.386 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.386 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.387 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.387 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.387 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.387 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.365+0000 7fe6b9ffb640 10 monclient: _finish_auth 0 2024-09-16T04:47:39.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.365+0000 7fe6b9ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:47:39.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.365+0000 7fe6b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:39.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.365+0000 7fe6b9ffb640 1 -- 172.21.15.46:0/3554929039 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6b0032440 con 0x7fe6bc0a3ff0 2024-09-16T04:47:39.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.365+0000 7fe6c14af640 5 monclient: authenticate success, global_id 104262 2024-09-16T04:47:39.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.365+0000 7fe6c14af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:39.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.365+0000 7fe6b9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:39.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.365+0000 7fe6c14af640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:39.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.365+0000 7fe6c14af640 4 set_mon_vals no callback set 2024-09-16T04:47:39.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.365+0000 7fe6b9ffb640 1 -- 172.21.15.46:0/3554929039 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0032810 con 0x7fe6bc0a3ff0 2024-09-16T04:47:39.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.365+0000 7fe6b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.365+0000 7fe6b9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:39.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.365+0000 7fe6b9ffb640 10 monclient: dump: 2024-09-16T04:47:39.389 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.389 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.389 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.390 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.390 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.390 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.390 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.390 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.390 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.390 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.366+0000 7fe6c14af640 10 monclient: shutdown 2024-09-16T04:47:39.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.366+0000 7fe6c14af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:39.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.366+0000 7fe6c14af640 1 -- 172.21.15.46:0/3554929039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6bc0a3ff0 msgr2=0x7fe6bc0a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.366+0000 7fe6c14af640 1 --2- 172.21.15.46:0/3554929039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6bc0a3ff0 0x7fe6bc0a43d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0009870 tx=0x7fe6b002a270 comp rx=0 tx=0).stop 2024-09-16T04:47:39.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.366+0000 7fe6c14af640 1 -- 172.21.15.46:0/3554929039 shutdown_connections 2024-09-16T04:47:39.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.366+0000 7fe6c14af640 1 --2- 172.21.15.46:0/3554929039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6bc0a3ff0 0x7fe6bc0a43d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.366+0000 7fe6c14af640 1 -- 172.21.15.46:0/3554929039 >> 172.21.15.46:0/3554929039 conn(0x7fe6bc0a6340 msgr2=0x7fe6bc06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:39.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.366+0000 7fe6c14af640 1 -- 172.21.15.46:0/3554929039 shutdown_connections 2024-09-16T04:47:39.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.366+0000 7fe6c14af640 1 -- 172.21.15.46:0/3554929039 wait complete. 2024-09-16T04:47:39.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.366+0000 7fe6c14af640 10 monclient: build_initial_monmap 2024-09-16T04:47:39.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.367+0000 7fe6c14af640 10 monclient: monmap: 2024-09-16T04:47:39.391 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:39.391 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:39.391 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:39.391 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:39.391 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:39.391 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.391 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:39.392 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:39.392 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:39.392 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.367+0000 7fe6c14af640 1 Processor -- start 2024-09-16T04:47:39.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.367+0000 7fe6c14af640 1 -- start start 2024-09-16T04:47:39.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.367+0000 7fe6c14af640 10 monclient: init 2024-09-16T04:47:39.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6c14af640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:39.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6c14af640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:47:39.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6c14af640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6bc0a3c10 0x7fe6bc136ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6c14af640 10 monclient(hunting): picked mon.noname-b con 0x7fe6bc0a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:39.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6c14af640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe6bc0a3ff0 0x7fe6bc138d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6c14af640 10 monclient(hunting): picked mon.noname-c con 0x7fe6bc0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:39.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6c14af640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6bc13be90 0x7fe6bc139240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6c14af640 10 monclient(hunting): picked mon.noname-a con 0x7fe6bc13be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:39.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6c14af640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6c14af640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc0a6b40 con 0x7fe6bc13be90 2024-09-16T04:47:39.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6c14af640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6c14af640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc0a69c0 con 0x7fe6bc0a3c10 2024-09-16T04:47:39.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6c14af640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6c14af640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe6bc13f9a0 con 0x7fe6bc0a3ff0 2024-09-16T04:47:39.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6c14af640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:39.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6c14af640 10 monclient(hunting): authenticate will time out at 1257.784196s 2024-09-16T04:47:39.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6ba7fc640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe6bc0a3ff0 0x7fe6bc138d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6bb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6bc13be90 0x7fe6bc139240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6ba7fc640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe6bc0a3ff0 0x7fe6bc138d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38712/0 (socket says 172.21.15.46:38712) 2024-09-16T04:47:39.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6ba7fc640 1 -- 172.21.15.46:0/1871861756 learned_addr learned my addr 172.21.15.46:0/1871861756 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:39.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6ba7fc640 10 monclient(hunting): get_auth_request con 0x7fe6bc0a3ff0 auth_method 0 2024-09-16T04:47:39.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6bb7fe640 10 monclient(hunting): get_auth_request con 0x7fe6bc13be90 auth_method 0 2024-09-16T04:47:39.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6ba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6ba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6baffd640 1 --2- 172.21.15.46:0/1871861756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6bc0a3c10 0x7fe6bc136ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6ba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:39.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:39.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6baffd640 10 monclient(hunting): get_auth_request con 0x7fe6bc0a3c10 auth_method 0 2024-09-16T04:47:39.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:39.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6ba7fc640 10 monclient(hunting): handle_auth_done global_id 94427 payload 824 2024-09-16T04:47:39.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6ba7fc640 1 -- 172.21.15.46:0/1871861756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6bc0a3c10 msgr2=0x7fe6bc136ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6ba7fc640 1 --2- 172.21.15.46:0/1871861756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6bc0a3c10 0x7fe6bc136ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6ba7fc640 1 -- 172.21.15.46:0/1871861756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6bc13be90 msgr2=0x7fe6bc139240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6ba7fc640 1 --2- 172.21.15.46:0/1871861756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6bc13be90 0x7fe6bc139240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6ba7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:47:39.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6ba7fc640 1 monclient: found mon.noname-c 2024-09-16T04:47:39.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:39.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6ba7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:47:39.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6ba7fc640 1 -- 172.21.15.46:0/1871861756 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6bc1205a0 con 0x7fe6bc0a3ff0 2024-09-16T04:47:39.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6bb7fe640 1 --2- 172.21.15.46:0/1871861756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6bc13be90 0x7fe6bc139240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:39.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6ba7fc640 1 --2- 172.21.15.46:0/1871861756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe6bc0a3ff0 0x7fe6bc138d00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0009870 tx=0x7fe6b00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:39.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6a37fe640 1 -- 172.21.15.46:0/1871861756 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0041020 con 0x7fe6bc0a3ff0 2024-09-16T04:47:39.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6a37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:47:39.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.368+0000 7fe6a37fe640 10 monclient: dump: 2024-09-16T04:47:39.399 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.399 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.399 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.400 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.400 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.400 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.400 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.400 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.401 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.401 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6a37fe640 10 monclient: _finish_auth 0 2024-09-16T04:47:39.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6a37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:47:39.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:39.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6c14af640 5 monclient: authenticate success, global_id 94427 2024-09-16T04:47:39.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6a37fe640 1 -- 172.21.15.46:0/1871861756 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6b0030970 con 0x7fe6bc0a3ff0 2024-09-16T04:47:39.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6c14af640 10 monclient: _renew_subs 2024-09-16T04:47:39.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6c14af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:39.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6c14af640 1 -- 172.21.15.46:0/1871861756 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6bc13f530 con 0x7fe6bc0a3ff0 2024-09-16T04:47:39.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6a37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:39.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6a37fe640 1 -- 172.21.15.46:0/1871861756 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0037040 con 0x7fe6bc0a3ff0 2024-09-16T04:47:39.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:39.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6a37fe640 10 monclient: dump: 2024-09-16T04:47:39.402 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.402 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.402 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.402 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.402 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.402 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.402 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.402 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.402 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.402 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6c14af640 10 monclient: _renew_subs 2024-09-16T04:47:39.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6c14af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:39.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6c14af640 1 -- 172.21.15.46:0/1871861756 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6bc11ffe0 con 0x7fe6bc0a3ff0 2024-09-16T04:47:39.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6b97fa640 4 set_mon_vals no callback set 2024-09-16T04:47:39.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6c14af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:39.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6c14af640 1 --2- 172.21.15.46:0/1871861756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe6bc137000 0x7fe6bc139780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6c14af640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6c14af640 1 -- 172.21.15.46:0/1871861756 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe6bc11fe60 con 0x7fe6bc137000 2024-09-16T04:47:39.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.369+0000 7fe6a37fe640 1 -- 172.21.15.46:0/1871861756 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fe6b0044020 con 0x7fe6bc0a3ff0 2024-09-16T04:47:39.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6baffd640 1 --2- 172.21.15.46:0/1871861756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe6bc137000 0x7fe6bc139780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6baffd640 10 monclient: get_auth_request con 0x7fe6bc137000 auth_method 0 2024-09-16T04:47:39.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6a37fe640 1 --2- 172.21.15.46:0/1871861756 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe6a80383b0 0x7fe6a803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:39.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6a37fe640 1 -- 172.21.15.46:0/1871861756 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7fe6b006bf40 con 0x7fe6bc0a3ff0 2024-09-16T04:47:39.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6baffd640 10 monclient(hunting): handle_auth_done global_id 94430 payload 824 2024-09-16T04:47:39.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6baffd640 1 -- 172.21.15.46:0/1871861756 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe6bc0a4450 con 0x7fe6bc137000 2024-09-16T04:47:39.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6bb7fe640 1 --2- 172.21.15.46:0/1871861756 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe6a80383b0 0x7fe6a803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6bb7fe640 10 monclient: get_auth_request con 0x7fe6a80383b0 auth_method 0 2024-09-16T04:47:39.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6baffd640 1 --2- 172.21.15.46:0/1871861756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe6bc137000 0x7fe6bc139780 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac00f770 tx=0x7fe6ac00fc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:39.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6a37fe640 1 -- 172.21.15.46:0/1871861756 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac0050d0 con 0x7fe6bc137000 2024-09-16T04:47:39.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:39.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6a37fe640 10 monclient: dump: 2024-09-16T04:47:39.405 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.405 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.405 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.405 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.405 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.405 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.405 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.405 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.405 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.405 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6bb7fe640 1 --2- 172.21.15.46:0/1871861756 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe6a80383b0 0x7fe6a803a870 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc09c9d0 tx=0x7fe6b4004d40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:39.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6a37fe640 1 -- 172.21.15.46:0/1871861756 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe6ac018070 con 0x7fe6bc137000 2024-09-16T04:47:39.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6a37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:39.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.370+0000 7fe6a37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:39.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.371+0000 7fe6a37fe640 1 -- 172.21.15.46:0/1871861756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe6bc137000 msgr2=0x7fe6bc139780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.371+0000 7fe6a37fe640 1 --2- 172.21.15.46:0/1871861756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe6bc137000 0x7fe6bc139780 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac00f770 tx=0x7fe6ac00fc30 comp rx=0 tx=0).stop 2024-09-16T04:47:39.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.371+0000 7fe6a37fe640 1 -- 172.21.15.46:0/1871861756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe6bc137000 msgr2=0x7fe6bc139780 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:47:39.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.371+0000 7fe6a37fe640 1 --2- 172.21.15.46:0/1871861756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe6bc137000 0x7fe6bc139780 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac00f770 tx=0x7fe6ac00fc30 comp rx=0 tx=0).stop 2024-09-16T04:47:39.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.377+0000 7fe6c14af640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:47:39.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.377+0000 7fe6c14af640 1 --2- 172.21.15.46:0/1871861756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe6bc139cc0 0x7fe6bc09d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.377+0000 7fe6c14af640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.377+0000 7fe6c14af640 1 -- 172.21.15.46:0/1871861756 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fe6bc13a0d0 con 0x7fe6bc139cc0 2024-09-16T04:47:39.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.377+0000 7fe6baffd640 1 --2- 172.21.15.46:0/1871861756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe6bc139cc0 0x7fe6bc09d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.377+0000 7fe6baffd640 10 monclient: get_auth_request con 0x7fe6bc139cc0 auth_method 0 2024-09-16T04:47:39.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.377+0000 7fe6baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.377+0000 7fe6baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.377+0000 7fe6baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.377+0000 7fe6baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.377+0000 7fe6baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.377+0000 7fe6baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.378+0000 7fe6baffd640 10 monclient(hunting): handle_auth_done global_id 94433 payload 824 2024-09-16T04:47:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.378+0000 7fe6baffd640 1 -- 172.21.15.46:0/1871861756 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe6bc13b1c0 con 0x7fe6bc139cc0 2024-09-16T04:47:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.378+0000 7fe6baffd640 1 --2- 172.21.15.46:0/1871861756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe6bc139cc0 0x7fe6bc09d110 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac017000 tx=0x7fe6ac001120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.378+0000 7fe6a37fe640 1 -- 172.21.15.46:0/1871861756 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac00d960 con 0x7fe6bc139cc0 2024-09-16T04:47:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.378+0000 7fe6a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.378+0000 7fe6a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.378+0000 7fe6a37fe640 10 monclient: dump: 2024-09-16T04:47:39.408 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.408 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.408 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.408 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.408 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.408 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.408 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.408 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.408 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.408 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.378+0000 7fe6a37fe640 1 -- 172.21.15.46:0/1871861756 <== mon.2 v2:172.21.15.46:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7fe6ac00d960 con 0x7fe6bc139cc0 2024-09-16T04:47:39.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.378+0000 7fe6a37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:47:39.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.378+0000 7fe6a37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:47:39.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.378+0000 7fe6a37fe640 1 -- 172.21.15.46:0/1871861756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe6bc139cc0 msgr2=0x7fe6bc09d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.378+0000 7fe6a37fe640 1 --2- 172.21.15.46:0/1871861756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe6bc139cc0 0x7fe6bc09d110 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac017000 tx=0x7fe6ac001120 comp rx=0 tx=0).stop 2024-09-16T04:47:39.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.378+0000 7fe6a37fe640 1 -- 172.21.15.46:0/1871861756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe6bc139cc0 msgr2=0x7fe6bc09d110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:47:39.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.378+0000 7fe6a37fe640 1 --2- 172.21.15.46:0/1871861756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe6bc139cc0 0x7fe6bc09d110 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac017000 tx=0x7fe6ac001120 comp rx=0 tx=0).stop 2024-09-16T04:47:39.409 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:47:39.409 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:47:39.410 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:47:39.410 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:47:39.410 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 100, 2024-09-16T04:47:39.410 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 100633, 2024-09-16T04:47:39.410 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:47:39.410 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:47:39.410 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:47:39.410 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:47:39.410 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:39.410 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 2, 2024-09-16T04:47:39.410 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:47:39.410 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:47:39.410 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:47:39.411 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:47:39.411 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:47:39.411 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:47:39.411 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:47:39.411 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:47:39.411 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:47:39.411 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:47:39.411 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:47:39.411 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:47:39.411 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:47:39.411 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:47:39.411 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:39.411 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:47:39.411 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:47:39.412 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:47:39.412 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:47:39.412 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:47:39.412 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:47:39.412 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:47:39.412 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:47:39.412 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:47:39.412 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:47:39.412 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:47:39.412 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:47:39.412 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:47:39.412 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:39.413 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.413 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:47:39.413 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:47:39.413 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:47:39.413 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:47:39.413 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:47:39.413 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:47:39.413 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:47:39.413 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:47:39.413 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:47:39.414 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:47:39.414 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:47:39.414 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:47:39.414 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:47:39.414 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:47:39.414 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:47:39.414 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:47:39.414 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:47:39.414 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:47:39.414 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:47:39.414 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:47:39.414 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:47:39.414 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:47:39.414 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:47:39.415 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:47:39.415 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:47:39.415 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:47:39.415 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:47:39.415 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:47:39.415 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:39.415 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:47:39.415 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.415 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:47:39.415 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.415 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:47:39.415 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:47:39.416 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:47:39.416 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:47:39.416 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.416 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:47:39.416 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:47:39.416 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:39.416 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.416 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.416 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:47:39.416 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:47:39.416 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:39.417 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:39.417 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:39.417 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.417 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:47:39.417 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:47:39.417 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:47:39.417 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:47:39.417 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:47:39.418 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.418 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.418 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:47:39.418 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:47:39.418 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:47:39.418 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:47:39.418 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.418 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:47:39.418 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:47:39.419 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:39.419 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.419 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.419 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:47:39.419 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:47:39.419 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:39.419 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:39.419 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:39.420 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.420 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:47:39.420 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:47:39.420 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:47:39.420 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:47:39.420 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:47:39.420 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.420 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.421 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:47:39.421 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:47:39.421 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:47:39.421 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:47:39.421 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.421 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:47:39.421 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:47:39.421 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:39.421 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.422 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.422 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:47:39.422 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:47:39.422 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:39.422 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:39.422 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:39.422 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.422 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:47:39.423 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:47:39.423 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:47:39.423 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:47:39.423 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:47:39.423 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:39.423 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:39.423 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.423 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:47:39.424 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:47:39.424 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.424 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:47:39.424 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:47:39.424 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:47:39.424 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:39.424 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:39.424 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": [ 2024-09-16T04:47:39.425 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.425 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:47:39.425 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:47:39.425 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 3 2024-09-16T04:47:39.425 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:39.425 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:39.425 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:47:39.425 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.425 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:47:39.426 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:47:39.426 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5 2024-09-16T04:47:39.426 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:39.426 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:39.426 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.426 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:47:39.426 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:47:39.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.379+0000 7fe6c14af640 1 -- 172.21.15.46:0/1871861756 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe6a80383b0 msgr2=0x7fe6a803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.379+0000 7fe6c14af640 1 --2- 172.21.15.46:0/1871861756 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fe6a80383b0 0x7fe6a803a870 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc09c9d0 tx=0x7fe6b4004d40 comp rx=0 tx=0).stop 2024-09-16T04:47:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.379+0000 7fe6c14af640 10 monclient: shutdown 2024-09-16T04:47:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.379+0000 7fe6c14af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.379+0000 7fe6c14af640 1 -- 172.21.15.46:0/1871861756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe6bc0a3ff0 msgr2=0x7fe6bc138d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.379+0000 7fe6c14af640 1 --2- 172.21.15.46:0/1871861756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fe6bc0a3ff0 0x7fe6bc138d00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0009870 tx=0x7fe6b00046c0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.379+0000 7fe6bb7fe640 1 -- 172.21.15.46:0/1871861756 reap_dead start 2024-09-16T04:47:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.379+0000 7fe6bb7fe640 1 -- 172.21.15.46:0/1871861756 reap_dead start 2024-09-16T04:47:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.379+0000 7fe6c14af640 1 -- 172.21.15.46:0/1871861756 shutdown_connections 2024-09-16T04:47:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.379+0000 7fe6c14af640 1 -- 172.21.15.46:0/1871861756 >> 172.21.15.46:0/1871861756 conn(0x7fe6bc0a6340 msgr2=0x7fe6bc0a24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:39.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.379+0000 7fe6c14af640 1 -- 172.21.15.46:0/1871861756 shutdown_connections 2024-09-16T04:47:39.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.379+0000 7fe6c14af640 1 -- 172.21.15.46:0/1871861756 wait complete. 2024-09-16T04:47:39.430 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:47:39.430 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:47:39.431 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:47:39.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.531+0000 7f336ceba640 10 monclient: get_monmap_and_config 2024-09-16T04:47:39.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.531+0000 7f336ceba640 10 monclient: build_initial_monmap 2024-09-16T04:47:39.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.531+0000 7f336ceba640 10 monclient: monmap: 2024-09-16T04:47:39.533 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:39.534 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.534 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:39.533036+0000 2024-09-16T04:47:39.534 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:39.533036+0000 2024-09-16T04:47:39.534 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:39.534 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.534 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:39.534 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:39.534 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:39.534 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:39.534 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:39.534 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.532+0000 7f336ceba640 1 Processor -- start 2024-09-16T04:47:39.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.532+0000 7f336ceba640 1 -- start start 2024-09-16T04:47:39.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.532+0000 7f336ceba640 10 monclient: init 2024-09-16T04:47:39.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.532+0000 7f336ceba640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:39.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.532+0000 7f336ceba640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-16T04:47:39.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.532+0000 7f336ceba640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f33680a3ba0 0x7f336809bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.532+0000 7f336ceba640 10 monclient(hunting): picked mon.noname-c con 0x7f33680a3ba0 addr v2:172.21.15.114:3300/0 2024-09-16T04:47:39.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.532+0000 7f336ceba640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336809c1c0 0x7f33680a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.532+0000 7f336ceba640 10 monclient(hunting): picked mon.noname-b con 0x7f336809c1c0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:39.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.532+0000 7f336ceba640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f33680a44c0 0x7f3368135950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.532+0000 7f336ceba640 10 monclient(hunting): picked mon.noname-a con 0x7f33680a44c0 addr v2:172.21.15.46:3300/0 2024-09-16T04:47:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.532+0000 7f336ceba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.532+0000 7f336ceba640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f336809b860 con 0x7f33680a44c0 2024-09-16T04:47:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.532+0000 7f336ceba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.532+0000 7f336ceba640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f336809b2a0 con 0x7f33680a3ba0 2024-09-16T04:47:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.532+0000 7f336ceba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.532+0000 7f336ceba640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f336809afc0 con 0x7f336809c1c0 2024-09-16T04:47:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.532+0000 7f336ceba640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.532+0000 7f336ceba640 10 monclient(hunting): authenticate will time out at 1257.948994s 2024-09-16T04:47:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f335ffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336809c1c0 0x7f33680a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f3366d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f33680a44c0 0x7f3368135950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f335ffff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336809c1c0 0x7f33680a3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38742/0 (socket says 172.21.15.46:38742) 2024-09-16T04:47:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f335ffff640 1 -- 172.21.15.46:0/1617523032 learned_addr learned my addr 172.21.15.46:0/1617523032 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f335ffff640 10 monclient(hunting): get_auth_request con 0x7f336809c1c0 auth_method 0 2024-09-16T04:47:39.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f335ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f335ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f335ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f3366d9d640 10 monclient(hunting): get_auth_request con 0x7f33680a44c0 auth_method 0 2024-09-16T04:47:39.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f3366d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f3366d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f3366d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f3366d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f3366d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f3366d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:39.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f335ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f335ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f335ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:39.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f336659c640 1 --2- 172.21.15.46:0/1617523032 >> v2:172.21.15.114:3300/0 conn(0x7f33680a3ba0 0x7f336809bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f336659c640 10 monclient(hunting): get_auth_request con 0x7f33680a3ba0 auth_method 0 2024-09-16T04:47:39.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f336659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f336659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f336659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f3366d9d640 10 monclient(hunting): handle_auth_done global_id 104271 payload 274 2024-09-16T04:47:39.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f3366d9d640 1 -- 172.21.15.46:0/1617523032 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336809c1c0 msgr2=0x7f33680a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:39.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f3366d9d640 1 --2- 172.21.15.46:0/1617523032 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336809c1c0 0x7f33680a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f3366d9d640 1 -- 172.21.15.46:0/1617523032 >> v2:172.21.15.114:3300/0 conn(0x7f33680a3ba0 msgr2=0x7f336809bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:39.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f3366d9d640 1 --2- 172.21.15.46:0/1617523032 >> v2:172.21.15.114:3300/0 conn(0x7f33680a3ba0 0x7f336809bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f3366d9d640 10 monclient: _finish_hunting 0 2024-09-16T04:47:39.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f3366d9d640 1 monclient: found mon.noname-a 2024-09-16T04:47:39.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f3366d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:39.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f3366d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:39.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f3366d9d640 1 -- 172.21.15.46:0/1617523032 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f336809b580 con 0x7f33680a44c0 2024-09-16T04:47:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.533+0000 7f335ffff640 1 --2- 172.21.15.46:0/1617523032 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336809c1c0 0x7f33680a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f3366d9d640 1 --2- 172.21.15.46:0/1617523032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33680a44c0 0x7f3368135950 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3368068440 tx=0x7f336002a540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee51464d7a7bc17e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f3365d9b640 1 -- 172.21.15.46:0/1617523032 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3360007740 con 0x7f33680a44c0 2024-09-16T04:47:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f3365d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f3365d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f3365d9b640 10 monclient: dump: 2024-09-16T04:47:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.547 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f3365d9b640 10 monclient: _finish_auth 0 2024-09-16T04:47:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f3365d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:47:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f3365d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f3365d9b640 1 -- 172.21.15.46:0/1617523032 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33600078e0 con 0x7f33680a44c0 2024-09-16T04:47:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f336ceba640 5 monclient: authenticate success, global_id 104271 2024-09-16T04:47:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f336ceba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f3365d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f3365d9b640 1 -- 172.21.15.46:0/1617523032 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3360007cb0 con 0x7f33680a44c0 2024-09-16T04:47:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f336ceba640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f336ceba640 4 set_mon_vals no callback set 2024-09-16T04:47:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f3365d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f3365d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f3365d9b640 10 monclient: dump: 2024-09-16T04:47:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.548 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f336ceba640 10 monclient: shutdown 2024-09-16T04:47:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f336ceba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f336ceba640 1 -- 172.21.15.46:0/1617523032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33680a44c0 msgr2=0x7f3368135950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f336ceba640 1 --2- 172.21.15.46:0/1617523032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33680a44c0 0x7f3368135950 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3368068440 tx=0x7f336002a540 comp rx=0 tx=0).stop 2024-09-16T04:47:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f336ceba640 1 -- 172.21.15.46:0/1617523032 shutdown_connections 2024-09-16T04:47:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f336ceba640 1 --2- 172.21.15.46:0/1617523032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33680a44c0 0x7f3368135950 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f336ceba640 1 --2- 172.21.15.46:0/1617523032 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336809c1c0 0x7f33680a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f336ceba640 1 --2- 172.21.15.46:0/1617523032 >> v2:172.21.15.114:3300/0 conn(0x7f33680a3ba0 0x7f336809bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f336ceba640 1 -- 172.21.15.46:0/1617523032 >> 172.21.15.46:0/1617523032 conn(0x7f33680a60c0 msgr2=0x7f33680a64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.534+0000 7f336ceba640 1 -- 172.21.15.46:0/1617523032 shutdown_connections 2024-09-16T04:47:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.535+0000 7f336ceba640 1 -- 172.21.15.46:0/1617523032 wait complete. 2024-09-16T04:47:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.535+0000 7f336ceba640 10 monclient: build_initial_monmap 2024-09-16T04:47:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.535+0000 7f336ceba640 10 monclient: monmap: 2024-09-16T04:47:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:39.550 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336ceba640 1 Processor -- start 2024-09-16T04:47:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336ceba640 1 -- start start 2024-09-16T04:47:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336ceba640 10 monclient: init 2024-09-16T04:47:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336ceba640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336ceba640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:47:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336ceba640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336809c1c0 0x7f33681cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336ceba640 10 monclient(hunting): picked mon.noname-c con 0x7f336809c1c0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336ceba640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33680a3ba0 0x7f33681cc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336ceba640 10 monclient(hunting): picked mon.noname-b con 0x7f33680a3ba0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336ceba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33680a44c0 0x7f33681220a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336ceba640 10 monclient(hunting): picked mon.noname-a con 0x7f33680a44c0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336ceba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336ceba640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f33680a5540 con 0x7f33680a44c0 2024-09-16T04:47:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336ceba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336ceba640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f33680a53c0 con 0x7f33680a3ba0 2024-09-16T04:47:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336ceba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336ceba640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f3368135e90 con 0x7f336809c1c0 2024-09-16T04:47:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336ceba640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336ceba640 10 monclient(hunting): authenticate will time out at 1257.952636s 2024-09-16T04:47:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336659c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336809c1c0 0x7f33681cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f3366d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33680a44c0 0x7f33681220a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f3366d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33680a44c0 0x7f33681220a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41802/0 (socket says 172.21.15.46:41802) 2024-09-16T04:47:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336659c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336809c1c0 0x7f33681cbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38754/0 (socket says 172.21.15.46:38754) 2024-09-16T04:47:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f3366d9d640 1 -- 172.21.15.46:0/2614963417 learned_addr learned my addr 172.21.15.46:0/2614963417 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f3366d9d640 10 monclient(hunting): get_auth_request con 0x7f33680a44c0 auth_method 0 2024-09-16T04:47:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f3366d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f3366d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f3366d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336659c640 10 monclient(hunting): get_auth_request con 0x7f336809c1c0 auth_method 0 2024-09-16T04:47:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f336659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f3366d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f3366d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f3366d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f335ffff640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33680a3ba0 0x7f33681cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f335ffff640 10 monclient(hunting): get_auth_request con 0x7f33680a3ba0 auth_method 0 2024-09-16T04:47:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f335ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f335ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.536+0000 7f335ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f336659c640 10 monclient(hunting): handle_auth_done global_id 94439 payload 824 2024-09-16T04:47:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f336659c640 1 -- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33680a3ba0 msgr2=0x7f33681cc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f336659c640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33680a3ba0 0x7f33681cc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f336659c640 1 -- 172.21.15.46:0/2614963417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33680a44c0 msgr2=0x7f33681220a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f336659c640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33680a44c0 0x7f33681220a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f336659c640 10 monclient: _finish_hunting 0 2024-09-16T04:47:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f336659c640 1 monclient: found mon.noname-c 2024-09-16T04:47:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f336659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f336659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:47:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f336659c640 1 -- 172.21.15.46:0/2614963417 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33681247c0 con 0x7f336809c1c0 2024-09-16T04:47:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f335ffff640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33680a3ba0 0x7f33681cc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:47:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f3366d9d640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33680a44c0 0x7f33681220a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f336659c640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336809c1c0 0x7f33681cbf50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f335800d5f0 tx=0x7f335800dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f335effd640 1 -- 172.21.15.46:0/2614963417 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33580125f0 con 0x7f336809c1c0 2024-09-16T04:47:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f335effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f335effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:47:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f335effd640 10 monclient: dump: 2024-09-16T04:47:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.555 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f335effd640 10 monclient: _finish_auth 0 2024-09-16T04:47:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f335effd640 10 monclient: _check_auth_tickets 2024-09-16T04:47:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f335effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f336ceba640 5 monclient: authenticate success, global_id 94439 2024-09-16T04:47:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f335effd640 1 -- 172.21.15.46:0/2614963417 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3358019070 con 0x7f336809c1c0 2024-09-16T04:47:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f336ceba640 10 monclient: _renew_subs 2024-09-16T04:47:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f336ceba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f336ceba640 1 -- 172.21.15.46:0/2614963417 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3368125c70 con 0x7f336809c1c0 2024-09-16T04:47:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f335effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f335effd640 1 -- 172.21.15.46:0/2614963417 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33580129d0 con 0x7f336809c1c0 2024-09-16T04:47:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f335effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f335effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f335effd640 10 monclient: dump: 2024-09-16T04:47:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.556 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f336559a640 4 set_mon_vals no callback set 2024-09-16T04:47:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f336ceba640 10 monclient: _renew_subs 2024-09-16T04:47:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f336ceba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.537+0000 7f336ceba640 1 -- 172.21.15.46:0/2614963417 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3368125990 con 0x7f336809c1c0 2024-09-16T04:47:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.538+0000 7f336ceba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.538+0000 7f336ceba640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33681225e0 0x7f33681229c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.538+0000 7f336ceba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.538+0000 7f336ceba640 1 -- 172.21.15.46:0/2614963417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f336809bc80 con 0x7f33681225e0 2024-09-16T04:47:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.538+0000 7f335effd640 1 -- 172.21.15.46:0/2614963417 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f3358012b70 con 0x7f336809c1c0 2024-09-16T04:47:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.538+0000 7f335effd640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f33480383b0 0x7f334803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.538+0000 7f335effd640 1 -- 172.21.15.46:0/2614963417 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7f335804b8f0 con 0x7f336809c1c0 2024-09-16T04:47:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.538+0000 7f335ffff640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33681225e0 0x7f33681229c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.538+0000 7f335ffff640 10 monclient: get_auth_request con 0x7f33681225e0 auth_method 0 2024-09-16T04:47:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.538+0000 7f335ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.538+0000 7f335ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.538+0000 7f335ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.538+0000 7f335ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.538+0000 7f335ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.538+0000 7f335ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.538+0000 7f335ffff640 1 -- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33681225e0 msgr2=0x7f33681229c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:47:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.538+0000 7f335ffff640 1 -- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33681225e0 msgr2=0x7f33681229c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:47:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.538+0000 7f335ffff640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33681225e0 0x7f33681229c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:47:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.538+0000 7f335ffff640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33681225e0 0x7f33681229c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:47:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.538+0000 7f3366d9d640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f33480383b0 0x7f334803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.538+0000 7f3366d9d640 10 monclient: get_auth_request con 0x7f33480383b0 auth_method 0 2024-09-16T04:47:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.539+0000 7f3366d9d640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f33480383b0 0x7f334803a870 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3360009bd0 tx=0x7f336002f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:39.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.739+0000 7f335ffff640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33681225e0 0x7f33681229c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.739+0000 7f335ffff640 10 monclient: get_auth_request con 0x7f33681225e0 auth_method 0 2024-09-16T04:47:39.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.739+0000 7f335ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.739+0000 7f335ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.739+0000 7f335ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:47:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.739+0000 7f335ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.739+0000 7f335ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.739+0000 7f335ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.739+0000 7f335ffff640 10 monclient(hunting): handle_auth_done global_id 114112 payload 824 2024-09-16T04:47:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.739+0000 7f335ffff640 1 -- 172.21.15.46:0/2614963417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3368125eb0 con 0x7f33681225e0 2024-09-16T04:47:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.740+0000 7f335ffff640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33681225e0 0x7f33681229c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3368123860 tx=0x7f3350014d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.740+0000 7f335effd640 1 -- 172.21.15.46:0/2614963417 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3350004440 con 0x7f33681225e0 2024-09-16T04:47:39.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.740+0000 7f335effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.740+0000 7f335effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:47:39.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.740+0000 7f335effd640 10 monclient: dump: 2024-09-16T04:47:39.744 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.744 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.744 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.744 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.745 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.745 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.745 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.745 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.745 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.745 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.740+0000 7f335effd640 1 -- 172.21.15.46:0/2614963417 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3350009e20 con 0x7f33681225e0 2024-09-16T04:47:39.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.740+0000 7f335effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:39.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.740+0000 7f335effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:39.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.740+0000 7f335effd640 1 -- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33681225e0 msgr2=0x7f33681229c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.740+0000 7f335effd640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33681225e0 0x7f33681229c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3368123860 tx=0x7f3350014d80 comp rx=0 tx=0).stop 2024-09-16T04:47:39.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.740+0000 7f335effd640 1 -- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33681225e0 msgr2=0x7f33681229c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:47:39.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.740+0000 7f335effd640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33681225e0 0x7f33681229c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.746+0000 7f336ceba640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:47:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.746+0000 7f336ceba640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3368122f00 0x7f336809d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.746+0000 7f336ceba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.746+0000 7f336ceba640 1 -- 172.21.15.46:0/2614963417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f33681240a0 con 0x7f3368122f00 2024-09-16T04:47:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.747+0000 7f335ffff640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3368122f00 0x7f336809d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.747+0000 7f335ffff640 10 monclient: get_auth_request con 0x7f3368122f00 auth_method 0 2024-09-16T04:47:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.747+0000 7f335ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.747+0000 7f335ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.747+0000 7f335ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.747+0000 7f335ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.747+0000 7f335ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.747+0000 7f335ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:39.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.747+0000 7f335ffff640 10 monclient(hunting): handle_auth_done global_id 114115 payload 824 2024-09-16T04:47:39.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.747+0000 7f335ffff640 1 -- 172.21.15.46:0/2614963417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3368067e20 con 0x7f3368122f00 2024-09-16T04:47:39.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.747+0000 7f335ffff640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3368122f00 0x7f336809d890 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3350008010 tx=0x7f3350004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:39.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f335effd640 1 -- 172.21.15.46:0/2614963417 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335002d020 con 0x7f3368122f00 2024-09-16T04:47:39.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f335effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f335effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:47:39.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f335effd640 10 monclient: dump: 2024-09-16T04:47:39.751 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.751 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.751 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.751 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.751 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.752 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f335effd640 1 -- 172.21.15.46:0/2614963417 <== mon.1 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f3350016430 con 0x7f3368122f00 2024-09-16T04:47:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f335effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:47:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f335effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:47:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f335effd640 1 -- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3368122f00 msgr2=0x7f336809d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f335effd640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3368122f00 0x7f336809d890 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3350008010 tx=0x7f3350004730 comp rx=0 tx=0).stop 2024-09-16T04:47:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f335effd640 1 -- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3368122f00 msgr2=0x7f336809d890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:47:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f335effd640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3368122f00 0x7f336809d890 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.753 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:47:39.753 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:47:39.753 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:47:39.753 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "peon", 2024-09-16T04:47:39.753 INFO:teuthology.orchestra.run.smithi046.stdout: "election_epoch": 100, 2024-09-16T04:47:39.753 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 7710, 2024-09-16T04:47:39.753 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum": [ 2024-09-16T04:47:39.753 INFO:teuthology.orchestra.run.smithi046.stdout: 0, 2024-09-16T04:47:39.753 INFO:teuthology.orchestra.run.smithi046.stdout: 1, 2024-09-16T04:47:39.754 INFO:teuthology.orchestra.run.smithi046.stdout: 2 2024-09-16T04:47:39.754 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:39.754 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_age": 2, 2024-09-16T04:47:39.754 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:47:39.754 INFO:teuthology.orchestra.run.smithi046.stdout: "required_con": "2450521708007866372", 2024-09-16T04:47:39.754 INFO:teuthology.orchestra.run.smithi046.stdout: "required_mon": [ 2024-09-16T04:47:39.754 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:47:39.754 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:47:39.754 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:47:39.754 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:47:39.754 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:47:39.755 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:47:39.756 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:47:39.756 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:47:39.756 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:47:39.756 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:47:39.756 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:47:39.757 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:39.757 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:47:39.757 INFO:teuthology.orchestra.run.smithi046.stdout: "quorum_mon": [ 2024-09-16T04:47:39.757 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:47:39.757 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:47:39.757 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:47:39.757 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:47:39.757 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:47:39.757 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:47:39.757 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:47:39.757 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:47:39.757 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:47:39.758 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:47:39.758 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:47:39.758 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:39.758 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.758 INFO:teuthology.orchestra.run.smithi046.stdout: "outside_quorum": [], 2024-09-16T04:47:39.758 INFO:teuthology.orchestra.run.smithi046.stdout: "extra_probe_peers": [], 2024-09-16T04:47:39.758 INFO:teuthology.orchestra.run.smithi046.stdout: "sync_provider": [], 2024-09-16T04:47:39.758 INFO:teuthology.orchestra.run.smithi046.stdout: "monmap": { 2024-09-16T04:47:39.758 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 1, 2024-09-16T04:47:39.758 INFO:teuthology.orchestra.run.smithi046.stdout: "fsid": "01e423f6-9e74-4519-981b-340ad8faba4f", 2024-09-16T04:47:39.758 INFO:teuthology.orchestra.run.smithi046.stdout: "modified": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:47:39.758 INFO:teuthology.orchestra.run.smithi046.stdout: "created": "2024-09-16T04:38:10.269597Z", 2024-09-16T04:47:39.759 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release": 19, 2024-09-16T04:47:39.759 INFO:teuthology.orchestra.run.smithi046.stdout: "min_mon_release_name": "squid", 2024-09-16T04:47:39.759 INFO:teuthology.orchestra.run.smithi046.stdout: "election_strategy": 1, 2024-09-16T04:47:39.759 INFO:teuthology.orchestra.run.smithi046.stdout: "disallowed_leaders: ": "", 2024-09-16T04:47:39.759 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false, 2024-09-16T04:47:39.759 INFO:teuthology.orchestra.run.smithi046.stdout: "tiebreaker_mon": "", 2024-09-16T04:47:39.759 INFO:teuthology.orchestra.run.smithi046.stdout: "removed_ranks: ": "", 2024-09-16T04:47:39.759 INFO:teuthology.orchestra.run.smithi046.stdout: "features": { 2024-09-16T04:47:39.759 INFO:teuthology.orchestra.run.smithi046.stdout: "persistent": [ 2024-09-16T04:47:39.759 INFO:teuthology.orchestra.run.smithi046.stdout: "kraken", 2024-09-16T04:47:39.759 INFO:teuthology.orchestra.run.smithi046.stdout: "luminous", 2024-09-16T04:47:39.760 INFO:teuthology.orchestra.run.smithi046.stdout: "mimic", 2024-09-16T04:47:39.760 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap-prune", 2024-09-16T04:47:39.760 INFO:teuthology.orchestra.run.smithi046.stdout: "nautilus", 2024-09-16T04:47:39.760 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus", 2024-09-16T04:47:39.760 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific", 2024-09-16T04:47:39.760 INFO:teuthology.orchestra.run.smithi046.stdout: "elector-pinging", 2024-09-16T04:47:39.760 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy", 2024-09-16T04:47:39.760 INFO:teuthology.orchestra.run.smithi046.stdout: "reef", 2024-09-16T04:47:39.760 INFO:teuthology.orchestra.run.smithi046.stdout: "squid" 2024-09-16T04:47:39.760 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:39.760 INFO:teuthology.orchestra.run.smithi046.stdout: "optional": [] 2024-09-16T04:47:39.761 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.761 INFO:teuthology.orchestra.run.smithi046.stdout: "mons": [ 2024-09-16T04:47:39.761 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.761 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 0, 2024-09-16T04:47:39.761 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "a", 2024-09-16T04:47:39.761 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:47:39.761 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:47:39.761 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.761 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:47:39.761 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3300", 2024-09-16T04:47:39.761 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:39.762 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.762 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.762 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:47:39.762 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789", 2024-09-16T04:47:39.762 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:39.762 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:39.762 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:39.762 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.762 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6789/0", 2024-09-16T04:47:39.762 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6789/0", 2024-09-16T04:47:39.762 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:47:39.762 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:47:39.763 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:47:39.763 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.763 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.763 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 1, 2024-09-16T04:47:39.763 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "b", 2024-09-16T04:47:39.763 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:47:39.763 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:47:39.763 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.763 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:47:39.763 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:3300", 2024-09-16T04:47:39.763 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:39.764 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.764 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.764 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:47:39.764 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789", 2024-09-16T04:47:39.764 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:39.764 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:39.764 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:39.764 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.764 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T04:47:39.764 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T04:47:39.764 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:47:39.764 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:47:39.765 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:47:39.765 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.765 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.765 INFO:teuthology.orchestra.run.smithi046.stdout: "rank": 2, 2024-09-16T04:47:39.765 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "c", 2024-09-16T04:47:39.765 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addrs": { 2024-09-16T04:47:39.765 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-09-16T04:47:39.765 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.765 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-09-16T04:47:39.765 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:3301", 2024-09-16T04:47:39.765 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:39.765 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.766 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.766 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-09-16T04:47:39.766 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790", 2024-09-16T04:47:39.766 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 0 2024-09-16T04:47:39.766 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:39.766 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:39.766 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.766 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6790/0", 2024-09-16T04:47:39.766 INFO:teuthology.orchestra.run.smithi046.stdout: "public_addr": "172.21.15.46:6790/0", 2024-09-16T04:47:39.766 INFO:teuthology.orchestra.run.smithi046.stdout: "priority": 0, 2024-09-16T04:47:39.766 INFO:teuthology.orchestra.run.smithi046.stdout: "weight": 0, 2024-09-16T04:47:39.766 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_location": "{}" 2024-09-16T04:47:39.767 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:39.767 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:39.767 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.767 INFO:teuthology.orchestra.run.smithi046.stdout: "feature_map": { 2024-09-16T04:47:39.767 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": [ 2024-09-16T04:47:39.767 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.767 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:47:39.767 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:47:39.767 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:47:39.767 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:39.767 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-09-16T04:47:39.768 INFO:teuthology.orchestra.run.smithi046.stdout: "client": [ 2024-09-16T04:47:39.768 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-09-16T04:47:39.768 INFO:teuthology.orchestra.run.smithi046.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:47:39.768 INFO:teuthology.orchestra.run.smithi046.stdout: "release": "squid", 2024-09-16T04:47:39.768 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 1 2024-09-16T04:47:39.768 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-09-16T04:47:39.768 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-09-16T04:47:39.768 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-09-16T04:47:39.768 INFO:teuthology.orchestra.run.smithi046.stdout: "stretch_mode": false 2024-09-16T04:47:39.768 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:47:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f336ceba640 1 -- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f33480383b0 msgr2=0x7f334803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f336ceba640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f33480383b0 0x7f334803a870 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3360009bd0 tx=0x7f336002f000 comp rx=0 tx=0).stop 2024-09-16T04:47:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f336ceba640 10 monclient: shutdown 2024-09-16T04:47:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f336ceba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f336ceba640 1 -- 172.21.15.46:0/2614963417 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336809c1c0 msgr2=0x7f33681cbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f336ceba640 1 --2- 172.21.15.46:0/2614963417 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f336809c1c0 0x7f33681cbf50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f335800d5f0 tx=0x7f335800dab0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f3366d9d640 1 -- 172.21.15.46:0/2614963417 reap_dead start 2024-09-16T04:47:39.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f3366d9d640 1 -- 172.21.15.46:0/2614963417 reap_dead start 2024-09-16T04:47:39.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f336ceba640 1 -- 172.21.15.46:0/2614963417 shutdown_connections 2024-09-16T04:47:39.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f336ceba640 1 -- 172.21.15.46:0/2614963417 >> 172.21.15.46:0/2614963417 conn(0x7f33680a60c0 msgr2=0x7f33681361e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:39.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f336ceba640 1 -- 172.21.15.46:0/2614963417 shutdown_connections 2024-09-16T04:47:39.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.748+0000 7f336ceba640 1 -- 172.21.15.46:0/2614963417 wait complete. 2024-09-16T04:47:39.771 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:47:39.771 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:47:39.771 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:47:39.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.871+0000 7f82f04e6640 10 monclient: get_monmap_and_config 2024-09-16T04:47:39.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.871+0000 7f82f04e6640 10 monclient: build_initial_monmap 2024-09-16T04:47:39.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.871+0000 7f82f04e6640 10 monclient: monmap: 2024-09-16T04:47:39.874 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:39.874 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.874 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:39.872906+0000 2024-09-16T04:47:39.874 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:39.872906+0000 2024-09-16T04:47:39.874 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:39.874 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.874 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:39.874 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:39.874 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:39.874 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:39.874 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:39.874 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82f04e6640 1 Processor -- start 2024-09-16T04:47:39.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82f04e6640 1 -- start start 2024-09-16T04:47:39.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82f04e6640 10 monclient: init 2024-09-16T04:47:39.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82f04e6640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:39.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82f04e6640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-16T04:47:39.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82f04e6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f82e8133b30 0x7f82e8133f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82f04e6640 10 monclient(hunting): picked mon.noname-a con 0x7f82e8133b30 addr v2:172.21.15.46:3300/0 2024-09-16T04:47:39.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82f04e6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f82e809aa70 addr v1:172.21.15.46:6789/0 2024-09-16T04:47:39.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82f04e6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f82e80a2040 addr v1:172.21.15.114:6789/0 2024-09-16T04:47:39.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82f04e6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82e806c440 con 0x7f82e809aa70 2024-09-16T04:47:39.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82f04e6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82e809b1c0 con 0x7f82e80a2040 2024-09-16T04:47:39.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82f04e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82f04e6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f82e809b4a0 con 0x7f82e8133b30 2024-09-16T04:47:39.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82f04e6640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:39.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82f04e6640 10 monclient(hunting): authenticate will time out at 1258.288721s 2024-09-16T04:47:39.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82ee282640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f82e8133b30 0x7f82e8133f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82eda81640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f82e809aa70 0x7f82e80a1930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33292/0 (socket says 172.21.15.46:33292) 2024-09-16T04:47:39.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82eda81640 1 -- 172.21.15.46:0/3272919784 learned_addr learned my addr 172.21.15.46:0/3272919784 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:47:39.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82ee282640 10 monclient(hunting): get_auth_request con 0x7f82e8133b30 auth_method 0 2024-09-16T04:47:39.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82ee282640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82ee282640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82ee282640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82ee282640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.872+0000 7f82ee282640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.873+0000 7f82ee282640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:39.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.873+0000 7f82ee282640 10 monclient(hunting): handle_auth_done global_id 104277 payload 274 2024-09-16T04:47:39.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.873+0000 7f82ee282640 1 -- 172.21.15.46:0/3272919784 >> v1:172.21.15.114:6789/0 conn(0x7f82e80a2040 legacy=0x7f82e813ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.873+0000 7f82ee282640 1 -- 172.21.15.46:0/3272919784 >> v1:172.21.15.46:6789/0 conn(0x7f82e809aa70 legacy=0x7f82e80a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.873+0000 7f82ee282640 10 monclient: _finish_hunting 0 2024-09-16T04:47:39.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.873+0000 7f82ee282640 1 monclient: found mon.noname-a 2024-09-16T04:47:39.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.873+0000 7f82ee282640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:39.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.873+0000 7f82ee282640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:39.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.873+0000 7f82ee282640 1 -- 172.21.15.46:0/3272919784 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e809b780 con 0x7f82e8133b30 2024-09-16T04:47:39.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.873+0000 7f82ed280640 1 -- 172.21.15.46:0/3272919784 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 387263181 0 0) 0x7f82e806c440 con 0x7f82e809aa70 2024-09-16T04:47:39.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.873+0000 7f82ed280640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:47:39.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.873+0000 7f82ee282640 1 --2- 172.21.15.46:0/3272919784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82e8133b30 0x7f82e8133f10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f82e40097a0 tx=0x7f82e402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ad2c05c795c4396 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:39.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82ed280640 1 -- 172.21.15.46:0/3272919784 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e4037070 con 0x7f82e8133b30 2024-09-16T04:47:39.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82ed280640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82ed280640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:39.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82ed280640 10 monclient: dump: 2024-09-16T04:47:39.886 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.886 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.886 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.886 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.886 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.886 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.886 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.886 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.886 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.886 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82ed280640 10 monclient: _finish_auth 0 2024-09-16T04:47:39.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82ed280640 10 monclient: _check_auth_tickets 2024-09-16T04:47:39.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82ed280640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:39.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82ed280640 1 -- 172.21.15.46:0/3272919784 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82e4032440 con 0x7f82e8133b30 2024-09-16T04:47:39.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82f04e6640 5 monclient: authenticate success, global_id 104277 2024-09-16T04:47:39.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82f04e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:39.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82ed280640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:39.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82f04e6640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:39.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82ed280640 1 -- 172.21.15.46:0/3272919784 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e402ae60 con 0x7f82e8133b30 2024-09-16T04:47:39.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82f04e6640 4 set_mon_vals no callback set 2024-09-16T04:47:39.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82ed280640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82ed280640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82ed280640 10 monclient: dump: 2024-09-16T04:47:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.889 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.889 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.889 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82f04e6640 10 monclient: shutdown 2024-09-16T04:47:39.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82f04e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:39.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82f04e6640 1 -- 172.21.15.46:0/3272919784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82e8133b30 msgr2=0x7f82e8133f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82f04e6640 1 --2- 172.21.15.46:0/3272919784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82e8133b30 0x7f82e8133f10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f82e40097a0 tx=0x7f82e402a110 comp rx=0 tx=0).stop 2024-09-16T04:47:39.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82f04e6640 1 -- 172.21.15.46:0/3272919784 shutdown_connections 2024-09-16T04:47:39.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82f04e6640 1 --2- 172.21.15.46:0/3272919784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82e8133b30 0x7f82e8133f10 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82f04e6640 1 -- 172.21.15.46:0/3272919784 >> 172.21.15.46:0/3272919784 conn(0x7f82e80a6630 msgr2=0x7f82e80a2a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:39.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82f04e6640 1 -- 172.21.15.46:0/3272919784 shutdown_connections 2024-09-16T04:47:39.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.874+0000 7f82f04e6640 1 -- 172.21.15.46:0/3272919784 wait complete. 2024-09-16T04:47:39.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.875+0000 7f82f04e6640 10 monclient: build_initial_monmap 2024-09-16T04:47:39.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.875+0000 7f82f04e6640 10 monclient: monmap: 2024-09-16T04:47:39.892 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:39.892 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:39.892 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:39.892 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:39.892 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:39.893 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.893 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:39.893 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:39.893 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:39.893 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.875+0000 7f82f04e6640 1 Processor -- start 2024-09-16T04:47:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.875+0000 7f82f04e6640 1 -- start start 2024-09-16T04:47:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.875+0000 7f82f04e6640 10 monclient: init 2024-09-16T04:47:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.875+0000 7f82f04e6640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.875+0000 7f82f04e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:47:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.875+0000 7f82f04e6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82e809aa70 0x7f82e81328a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.875+0000 7f82f04e6640 10 monclient(hunting): picked mon.noname-a con 0x7f82e809aa70 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.875+0000 7f82f04e6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82e80a2040 0x7f82e8132de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.875+0000 7f82f04e6640 10 monclient(hunting): picked mon.noname-b con 0x7f82e80a2040 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:39.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.875+0000 7f82f04e6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82e8133b30 0x7f82e8120310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.875+0000 7f82f04e6640 10 monclient(hunting): picked mon.noname-c con 0x7f82e8133b30 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:39.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82f04e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82f04e6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f82e80a33a0 con 0x7f82e809aa70 2024-09-16T04:47:39.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82f04e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82f04e6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f82e8134450 con 0x7f82e80a2040 2024-09-16T04:47:39.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82f04e6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:39.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82f04e6640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f82e80a3520 con 0x7f82e8133b30 2024-09-16T04:47:39.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82f04e6640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:39.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82f04e6640 10 monclient(hunting): authenticate will time out at 1258.292017s 2024-09-16T04:47:39.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82ee282640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82e809aa70 0x7f82e81328a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82ee282640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82e809aa70 0x7f82e81328a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41828/0 (socket says 172.21.15.46:41828) 2024-09-16T04:47:39.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82ee282640 1 -- 172.21.15.46:0/3904411756 learned_addr learned my addr 172.21.15.46:0/3904411756 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:39.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82ee282640 10 monclient(hunting): get_auth_request con 0x7f82e809aa70 auth_method 0 2024-09-16T04:47:39.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82ee282640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82ee282640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82ee282640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82eea83640 1 --2- 172.21.15.46:0/3904411756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82e8133b30 0x7f82e8120310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82eea83640 10 monclient(hunting): get_auth_request con 0x7f82e8133b30 auth_method 0 2024-09-16T04:47:39.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82eea83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82eea83640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82eea83640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82ee282640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82ee282640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82ee282640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82eda81640 1 --2- 172.21.15.46:0/3904411756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82e80a2040 0x7f82e8132de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82eda81640 10 monclient(hunting): get_auth_request con 0x7f82e80a2040 auth_method 0 2024-09-16T04:47:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82eda81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82eda81640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82eda81640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82eea83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82eea83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82eea83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82eda81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:39.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82eda81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:39.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82eda81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:39.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82ee282640 10 monclient(hunting): handle_auth_done global_id 104283 payload 824 2024-09-16T04:47:39.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82ee282640 1 -- 172.21.15.46:0/3904411756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82e8133b30 msgr2=0x7f82e8120310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82ee282640 1 --2- 172.21.15.46:0/3904411756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82e8133b30 0x7f82e8120310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82ee282640 1 -- 172.21.15.46:0/3904411756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82e80a2040 msgr2=0x7f82e8132de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:39.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82ee282640 1 --2- 172.21.15.46:0/3904411756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82e80a2040 0x7f82e8132de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:39.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82ee282640 10 monclient: _finish_hunting 0 2024-09-16T04:47:39.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82ee282640 1 monclient: found mon.noname-a 2024-09-16T04:47:39.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82ee282640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:39.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82ee282640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:39.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82ee282640 1 -- 172.21.15.46:0/3904411756 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e8122b10 con 0x7f82e809aa70 2024-09-16T04:47:39.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82ee282640 1 --2- 172.21.15.46:0/3904411756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82e809aa70 0x7f82e81328a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f82e40098c0 tx=0x7f82e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:39.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82deffd640 1 -- 172.21.15.46:0/3904411756 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e4037040 con 0x7f82e809aa70 2024-09-16T04:47:39.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82deffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:39.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82deffd640 10 monclient: dump: 2024-09-16T04:47:39.904 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.904 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.904 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.904 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.904 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.904 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.904 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.904 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.904 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.904 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82deffd640 10 monclient: _finish_auth 0 2024-09-16T04:47:39.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82deffd640 10 monclient: _check_auth_tickets 2024-09-16T04:47:39.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:39.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82f04e6640 5 monclient: authenticate success, global_id 104283 2024-09-16T04:47:39.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82deffd640 1 -- 172.21.15.46:0/3904411756 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82e4042070 con 0x7f82e809aa70 2024-09-16T04:47:39.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82deffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:39.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82deffd640 1 -- 172.21.15.46:0/3904411756 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e4030c70 con 0x7f82e809aa70 2024-09-16T04:47:39.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82f04e6640 10 monclient: _renew_subs 2024-09-16T04:47:39.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82f04e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:39.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82f04e6640 1 -- 172.21.15.46:0/3904411756 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82e8123fc0 con 0x7f82e809aa70 2024-09-16T04:47:39.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82eca7f640 4 set_mon_vals no callback set 2024-09-16T04:47:39.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:39.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:39.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82deffd640 10 monclient: dump: 2024-09-16T04:47:39.906 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:39.906 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:39.906 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.906 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:39.907 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:39.907 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:39.907 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:39.907 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:39.907 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:39.907 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:39.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82f04e6640 10 monclient: _renew_subs 2024-09-16T04:47:39.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82f04e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:39.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.876+0000 7f82f04e6640 1 -- 172.21.15.46:0/3904411756 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82e80a5850 con 0x7f82e809aa70 2024-09-16T04:47:39.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.877+0000 7f82deffd640 1 -- 172.21.15.46:0/3904411756 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f82e4047460 con 0x7f82e809aa70 2024-09-16T04:47:39.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.877+0000 7f82f04e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:39.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.877+0000 7f82f04e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:39.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.877+0000 7f82f04e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:39.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.877+0000 7f82f04e6640 1 -- 172.21.15.46:0/3904411756 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82e8069860 con 0x7f82e809aa70 2024-09-16T04:47:39.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.877+0000 7f82deffd640 1 --2- 172.21.15.46:0/3904411756 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f82cc0383b0 0x7f82cc03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:39.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.878+0000 7f82deffd640 1 -- 172.21.15.46:0/3904411756 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7f82e406c2f0 con 0x7f82e809aa70 2024-09-16T04:47:39.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.878+0000 7f82eda81640 1 --2- 172.21.15.46:0/3904411756 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f82cc0383b0 0x7f82cc03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:39.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.878+0000 7f82eda81640 10 monclient: get_auth_request con 0x7f82cc0383b0 auth_method 0 2024-09-16T04:47:39.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.878+0000 7f82eda81640 1 --2- 172.21.15.46:0/3904411756 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f82cc0383b0 0x7f82cc03a870 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f82d8007f70 tx=0x7f82d8007f00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:39.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.882+0000 7f82deffd640 1 -- 172.21.15.46:0/3904411756 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f82e4030e10 con 0x7f82e809aa70 2024-09-16T04:47:39.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.882+0000 7f82deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:39.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:39.882+0000 7f82deffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:40.131+0000 7f82f04e6640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:47:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:40.131+0000 7f82f04e6640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:47:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:40.131+0000 7f82f04e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:40.131+0000 7f82f04e6640 1 -- 172.21.15.46:0/3904411756 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f82e806d150 con 0x7f82e809aa70 2024-09-16T04:47:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:40.132+0000 7f82deffd640 1 -- 172.21.15.46:0/3904411756 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f82e4032440 con 0x7f82e809aa70 2024-09-16T04:47:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:40.132+0000 7f82deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:47:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:40.132+0000 7f82deffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:47:40.135 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.132+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.135 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.132+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=12,kv=12,logm=55} crc {auth=1637925927,config=3541143443,health=1378089662,kv=4017288519,logm=4153715689}) 2024-09-16T04:47:40.135 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.132+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=17,config=2,health=12,kv=12,logm=57} crc {auth=938509543,config=3541143443,health=1378089662,kv=3692544301,logm=4098271403}) 2024-09-16T04:47:40.135 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.132+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.135 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.132+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=12,kv=12,logm=55} crc {auth=1637925927,config=3541143443,health=1378089662,kv=4017288519,logm=4153715689}) 2024-09-16T04:47:40.135 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.132+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=1,health=11,kv=9,logm=61} crc {auth=1903350027,config=1228700967,health=1435182105,kv=554051826,logm=4150898000}) 2024-09-16T04:47:40.136 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.134+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.136 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.134+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3690484161}) 2024-09-16T04:47:40.136 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.134+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3522170277}) 2024-09-16T04:47:40.136 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.134+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.136 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.134+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3690484161}) 2024-09-16T04:47:40.136 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.134+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3691760012}) 2024-09-16T04:47:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:40.134+0000 7f82f04e6640 1 -- 172.21.15.46:0/3904411756 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f82cc0383b0 msgr2=0x7f82cc03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:40.134+0000 7f82f04e6640 1 --2- 172.21.15.46:0/3904411756 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f82cc0383b0 0x7f82cc03a870 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f82d8007f70 tx=0x7f82d8007f00 comp rx=0 tx=0).stop 2024-09-16T04:47:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:40.134+0000 7f82f04e6640 10 monclient: shutdown 2024-09-16T04:47:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:40.134+0000 7f82f04e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:40.134+0000 7f82f04e6640 1 -- 172.21.15.46:0/3904411756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82e809aa70 msgr2=0x7f82e81328a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:40.134+0000 7f82f04e6640 1 --2- 172.21.15.46:0/3904411756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82e809aa70 0x7f82e81328a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f82e40098c0 tx=0x7f82e40046c0 comp rx=0 tx=0).stop 2024-09-16T04:47:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:40.134+0000 7f82f04e6640 1 -- 172.21.15.46:0/3904411756 shutdown_connections 2024-09-16T04:47:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:40.134+0000 7f82f04e6640 1 --2- 172.21.15.46:0/3904411756 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f82e8133b30 0x7f82e8120310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:40.134+0000 7f82f04e6640 1 --2- 172.21.15.46:0/3904411756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82e80a2040 0x7f82e8132de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:40.134+0000 7f82f04e6640 1 --2- 172.21.15.46:0/3904411756 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f82cc0383b0 0x7f82cc03a870 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:40.134+0000 7f82f04e6640 1 --2- 172.21.15.46:0/3904411756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82e809aa70 0x7f82e81328a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:40.134+0000 7f82f04e6640 1 -- 172.21.15.46:0/3904411756 >> 172.21.15.46:0/3904411756 conn(0x7f82e80a6630 msgr2=0x7f82e80a4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:40.134+0000 7f82f04e6640 1 -- 172.21.15.46:0/3904411756 shutdown_connections 2024-09-16T04:47:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:40.134+0000 7f82f04e6640 1 -- 172.21.15.46:0/3904411756 wait complete. 2024-09-16T04:47:40.141 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.135+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.141 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.135+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3287257575}) 2024-09-16T04:47:40.141 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.135+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=242328728}) 2024-09-16T04:47:40.141 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.135+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.141 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.135+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3287257575}) 2024-09-16T04:47:40.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.135+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3508118650}) 2024-09-16T04:47:40.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.136+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.136+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3190699984}) 2024-09-16T04:47:40.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.136+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3009155416}) 2024-09-16T04:47:40.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.136+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.136+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3190699984}) 2024-09-16T04:47:40.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.136+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3873265974}) 2024-09-16T04:47:40.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.137+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.137+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=990257968}) 2024-09-16T04:47:40.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.137+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=355203525}) 2024-09-16T04:47:40.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.137+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.137+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=990257968}) 2024-09-16T04:47:40.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.137+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=602402905}) 2024-09-16T04:47:40.142 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.138+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.143 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.138+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362461435}) 2024-09-16T04:47:40.143 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.138+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2079022275}) 2024-09-16T04:47:40.143 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.138+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.143 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.138+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362461435}) 2024-09-16T04:47:40.143 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.138+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1368226764}) 2024-09-16T04:47:40.143 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.139+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.143 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.139+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077599271}) 2024-09-16T04:47:40.143 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.139+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1734248936}) 2024-09-16T04:47:40.143 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.139+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.143 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.139+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077599271}) 2024-09-16T04:47:40.143 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.139+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2475447214}) 2024-09-16T04:47:40.144 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.140+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.144 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.140+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2832333862}) 2024-09-16T04:47:40.145 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.140+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3906330111}) 2024-09-16T04:47:40.145 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.140+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.145 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.140+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2832333862}) 2024-09-16T04:47:40.145 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.140+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2520804828}) 2024-09-16T04:47:40.145 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.141+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.145 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.141+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2079713083}) 2024-09-16T04:47:40.145 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.141+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2901519495}) 2024-09-16T04:47:40.145 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.141+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.145 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.141+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2079713083}) 2024-09-16T04:47:40.145 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.141+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=579371471}) 2024-09-16T04:47:40.145 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.142+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.145 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.142+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2127754956}) 2024-09-16T04:47:40.145 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.142+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1508985145}) 2024-09-16T04:47:40.145 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.142+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.146 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.142+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2127754956}) 2024-09-16T04:47:40.146 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.142+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=428733035}) 2024-09-16T04:47:40.147 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.143+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.147 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.143+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2789975709}) 2024-09-16T04:47:40.147 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.143+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1522600188}) 2024-09-16T04:47:40.147 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.143+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.147 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.143+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2789975709}) 2024-09-16T04:47:40.147 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.143+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4211009454}) 2024-09-16T04:47:40.147 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.144+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.147 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.144+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3165864072}) 2024-09-16T04:47:40.147 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.144+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2365875658}) 2024-09-16T04:47:40.147 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.144+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.147 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.144+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3165864072}) 2024-09-16T04:47:40.147 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.144+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4214357353}) 2024-09-16T04:47:40.148 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.145+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.148 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.145+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2709647097}) 2024-09-16T04:47:40.148 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.145+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1992742915}) 2024-09-16T04:47:40.148 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.145+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.148 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.145+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2709647097}) 2024-09-16T04:47:40.148 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.145+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1567683363}) 2024-09-16T04:47:40.149 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:47:40.150 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.146+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.150 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.146+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1900669231}) 2024-09-16T04:47:40.150 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.146+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3420234477}) 2024-09-16T04:47:40.150 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.146+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.150 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.146+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1900669231}) 2024-09-16T04:47:40.150 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.146+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=944248079}) 2024-09-16T04:47:40.151 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.147+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.151 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.147+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1643382523}) 2024-09-16T04:47:40.151 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.147+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=264380853}) 2024-09-16T04:47:40.151 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.147+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.151 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.147+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1643382523}) 2024-09-16T04:47:40.151 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.147+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=580730968}) 2024-09-16T04:47:40.151 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.148+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.151 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.148+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3758409096}) 2024-09-16T04:47:40.151 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.148+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3132715151}) 2024-09-16T04:47:40.151 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.148+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.151 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.148+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3758409096}) 2024-09-16T04:47:40.151 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.148+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1115480845}) 2024-09-16T04:47:40.152 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.149+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.152 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.149+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1478431274}) 2024-09-16T04:47:40.152 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.149+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=402651918}) 2024-09-16T04:47:40.152 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.149+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.152 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.149+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1478431274}) 2024-09-16T04:47:40.153 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.149+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4125229383}) 2024-09-16T04:47:40.153 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.150+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.153 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.150+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175171367}) 2024-09-16T04:47:40.153 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.150+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4249032202}) 2024-09-16T04:47:40.153 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.150+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.153 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.150+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175171367}) 2024-09-16T04:47:40.153 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.150+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3036463133}) 2024-09-16T04:47:40.154 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.151+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.154 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.151+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1568654380}) 2024-09-16T04:47:40.154 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.151+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2762969995}) 2024-09-16T04:47:40.154 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.151+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.154 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.151+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1568654380}) 2024-09-16T04:47:40.154 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.151+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2869161967}) 2024-09-16T04:47:40.155 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.152+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.155 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.152+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=421560644}) 2024-09-16T04:47:40.155 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.152+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3115307889}) 2024-09-16T04:47:40.155 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.152+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.155 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.152+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=421560644}) 2024-09-16T04:47:40.155 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.152+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3934906106}) 2024-09-16T04:47:40.156 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.153+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.156 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.153+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3680077941}) 2024-09-16T04:47:40.156 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.153+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2327143539}) 2024-09-16T04:47:40.156 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.153+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.156 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.153+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3680077941}) 2024-09-16T04:47:40.156 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.153+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1561358231}) 2024-09-16T04:47:40.157 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.154+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.157 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.154+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179433239}) 2024-09-16T04:47:40.157 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.154+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=142335104}) 2024-09-16T04:47:40.157 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.154+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.157 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.154+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179433239}) 2024-09-16T04:47:40.157 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.154+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2160076282}) 2024-09-16T04:47:40.157 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.155+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.158 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.155+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479695448}) 2024-09-16T04:47:40.158 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.155+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3591901680}) 2024-09-16T04:47:40.158 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.155+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.158 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.155+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479695448}) 2024-09-16T04:47:40.158 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.155+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3638613069}) 2024-09-16T04:47:40.158 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.156+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.159 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.156+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428191448}) 2024-09-16T04:47:40.159 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.156+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2705182800}) 2024-09-16T04:47:40.159 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.156+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.159 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.156+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428191448}) 2024-09-16T04:47:40.159 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.156+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=693952768}) 2024-09-16T04:47:40.159 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.157+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.160 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.157+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4027207868}) 2024-09-16T04:47:40.160 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.157+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4152327326}) 2024-09-16T04:47:40.160 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.157+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.160 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.157+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4027207868}) 2024-09-16T04:47:40.160 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.157+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2965388814}) 2024-09-16T04:47:40.160 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.158+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.161 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.158+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1481090884}) 2024-09-16T04:47:40.161 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.158+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2385160787}) 2024-09-16T04:47:40.161 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.158+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.161 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.158+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1481090884}) 2024-09-16T04:47:40.161 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.158+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1797076419}) 2024-09-16T04:47:40.161 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.159+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.162 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.159+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3444117237}) 2024-09-16T04:47:40.162 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.159+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=320960216}) 2024-09-16T04:47:40.162 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.159+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.162 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.159+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3444117237}) 2024-09-16T04:47:40.162 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.159+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=809484264}) 2024-09-16T04:47:40.162 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.160+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.163 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.160+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=3583782516,mdsmap=3956149183,mgr=2533529775}) 2024-09-16T04:47:40.163 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.160+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=2455714807,mdsmap=3956149183,mgr=3189685207}) 2024-09-16T04:47:40.163 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.160+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.163 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.160+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=3583782516,mdsmap=3956149183,mgr=2533529775}) 2024-09-16T04:47:40.163 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.160+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=98,mdsmap=2} crc {logm=3811965619,mdsmap=571966421}) 2024-09-16T04:47:40.164 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.162+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.164 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.162+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=61} crc {mgr=3165709797,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=2810502443,mon_config_key=3702505546,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=57929946,osd_snap=723727393,osdmap=1567027765}) 2024-09-16T04:47:40.164 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.162+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=60} crc {mgr=1189732577,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=2810502442,mon_config_key=1385660123,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=57929946,osd_snap=723727393,osdmap=2807030966}) 2024-09-16T04:47:40.164 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.162+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.164 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.162+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=61} crc {mgr=3165709797,mgr_command_descs=3961477881,mgr_metadata=4119166318,mgrstat=2810502443,mon_config_key=3702505546,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=57929946,osd_snap=723727393,osdmap=1567027765}) 2024-09-16T04:47:40.164 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.162+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=61} crc {mgr=3667558009,mgr_command_descs=3961477882,mgr_metadata=4119166318,mgrstat=3183257142,mon_config_key=3267531759,monmap=3088247691,osd_metadata=777518657,osd_pg_creating=57929946,osd_snap=723727393,osdmap=2070250905}) 2024-09-16T04:47:40.165 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.163+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.165 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.163+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3802044452}) 2024-09-16T04:47:40.165 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.163+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3624964976}) 2024-09-16T04:47:40.165 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.163+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.165 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.163+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3802044452}) 2024-09-16T04:47:40.165 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.163+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=327325807}) 2024-09-16T04:47:40.166 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.164+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.166 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.164+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3320951454}) 2024-09-16T04:47:40.166 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.164+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=553487989}) 2024-09-16T04:47:40.166 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.164+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.166 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.164+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3320951454}) 2024-09-16T04:47:40.166 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.164+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2624418001}) 2024-09-16T04:47:40.168 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.166+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.168 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.166+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2272545610}) 2024-09-16T04:47:40.168 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.166+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=506069869}) 2024-09-16T04:47:40.168 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.166+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.168 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.166+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2272545610}) 2024-09-16T04:47:40.168 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.166+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=902497543}) 2024-09-16T04:47:40.173 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.171+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.173 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.171+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=97} crc {osdmap=2348120163}) 2024-09-16T04:47:40.173 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.171+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3418679529}) 2024-09-16T04:47:40.173 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.171+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:47:40.173 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.171+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=97} crc {osdmap=2348120163}) 2024-09-16T04:47:40.173 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-09-16T04:47:40.171+0000 7fb72ee45640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=673803337}) 2024-09-16T04:47:41.150 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:47:41.151 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:47:41.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.262+0000 7f3360880640 10 monclient: get_monmap_and_config 2024-09-16T04:47:41.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.262+0000 7f3360880640 10 monclient: build_initial_monmap 2024-09-16T04:47:41.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.262+0000 7f3360880640 10 monclient: monmap: 2024-09-16T04:47:41.265 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:41.265 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:41.265 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:41.264414+0000 2024-09-16T04:47:41.265 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:41.264414+0000 2024-09-16T04:47:41.265 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:41.265 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:41.265 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:41.265 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:41.265 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:41.265 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:41.266 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:41.266 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:41.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.263+0000 7f3360880640 1 Processor -- start 2024-09-16T04:47:41.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.263+0000 7f3360880640 1 -- start start 2024-09-16T04:47:41.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.263+0000 7f3360880640 10 monclient: init 2024-09-16T04:47:41.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3360880640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:41.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3360880640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-16T04:47:41.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3360880640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f335c0a3c10 0x7f335c09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:41.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3360880640 10 monclient(hunting): picked mon.noname-a con 0x7f335c0a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:47:41.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3360880640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f335c0a3ff0 0x7f335c0a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:41.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3360880640 10 monclient(hunting): picked mon.noname-b con 0x7f335c0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:41.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3360880640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f335c1369d0 addr v1:172.21.15.46:6789/0 2024-09-16T04:47:41.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3360880640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f335c09b940 con 0x7f335c1369d0 2024-09-16T04:47:41.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3360880640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:41.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3360880640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f335c09ab00 con 0x7f335c0a3c10 2024-09-16T04:47:41.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3360880640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:41.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3360880640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f335c09b030 con 0x7f335c0a3ff0 2024-09-16T04:47:41.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3360880640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:41.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3360880640 10 monclient(hunting): authenticate will time out at 1259.680348s 2024-09-16T04:47:41.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3359d9b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f335c0a3ff0 0x7f335c0a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:41.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3359d9b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f335c0a3ff0 0x7f335c0a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38772/0 (socket says 172.21.15.46:38772) 2024-09-16T04:47:41.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3359d9b640 1 -- 172.21.15.46:0/2147536242 learned_addr learned my addr 172.21.15.46:0/2147536242 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:41.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3359d9b640 10 monclient(hunting): get_auth_request con 0x7f335c0a3ff0 auth_method 0 2024-09-16T04:47:41.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3359d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:41.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3359d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:41.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3359d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:41.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f335a59c640 1 --2- 172.21.15.46:0/2147536242 >> v2:172.21.15.46:3300/0 conn(0x7f335c0a3c10 0x7f335c09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:41.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f335a59c640 10 monclient(hunting): get_auth_request con 0x7f335c0a3c10 auth_method 0 2024-09-16T04:47:41.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f335a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:41.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f335a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:41.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f335a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:41.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3359d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:41.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3359d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:41.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3359d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:41.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f335a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:41.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f335a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:41.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f335a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:41.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3359d9b640 10 monclient(hunting): handle_auth_done global_id 94445 payload 274 2024-09-16T04:47:41.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3359d9b640 1 -- 172.21.15.46:0/2147536242 >> v2:172.21.15.46:3300/0 conn(0x7f335c0a3c10 msgr2=0x7f335c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:41.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3359d9b640 1 --2- 172.21.15.46:0/2147536242 >> v2:172.21.15.46:3300/0 conn(0x7f335c0a3c10 0x7f335c09bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3359d9b640 1 -- 172.21.15.46:0/2147536242 >> v1:172.21.15.46:6789/0 conn(0x7f335c1369d0 legacy=0x7f335c136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3359d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:47:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3359d9b640 1 monclient: found mon.noname-b 2024-09-16T04:47:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3359d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3359d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:47:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.264+0000 7f3359d9b640 1 -- 172.21.15.46:0/2147536242 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f335c09b310 con 0x7f335c0a3ff0 2024-09-16T04:47:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.265+0000 7f3359d9b640 1 --2- 172.21.15.46:0/2147536242 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f335c0a3ff0 0x7f335c0a43d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3344009870 tx=0x7f334402a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c708c37604913ec2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.265+0000 7f335959a640 1 -- 172.21.15.46:0/2147536242 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3344037070 con 0x7f335c0a3ff0 2024-09-16T04:47:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.265+0000 7f335959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.265+0000 7f335959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:47:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.265+0000 7f335959a640 10 monclient: dump: 2024-09-16T04:47:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.279 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.279 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:41.279 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:41.279 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:41.279 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:41.279 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:41.279 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:41.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.265+0000 7f335959a640 10 monclient: _finish_auth 0 2024-09-16T04:47:41.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.265+0000 7f335959a640 10 monclient: _check_auth_tickets 2024-09-16T04:47:41.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.265+0000 7f335959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:41.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.265+0000 7f3360880640 5 monclient: authenticate success, global_id 94445 2024-09-16T04:47:41.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.265+0000 7f3360880640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:41.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.265+0000 7f335959a640 1 -- 172.21.15.46:0/2147536242 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3344032440 con 0x7f335c0a3ff0 2024-09-16T04:47:41.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.265+0000 7f335959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:41.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.265+0000 7f3360880640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:41.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.265+0000 7f3360880640 4 set_mon_vals no callback set 2024-09-16T04:47:41.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.265+0000 7f335959a640 1 -- 172.21.15.46:0/2147536242 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3344032810 con 0x7f335c0a3ff0 2024-09-16T04:47:41.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.265+0000 7f335959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:41.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.265+0000 7f335959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:41.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.265+0000 7f335959a640 10 monclient: dump: 2024-09-16T04:47:41.280 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:41.280 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:41.280 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.280 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.280 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:41.280 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:41.280 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:41.281 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:41.281 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:41.281 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:41.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.266+0000 7f3360880640 10 monclient: shutdown 2024-09-16T04:47:41.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.266+0000 7f3360880640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:41.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.266+0000 7f3360880640 1 -- 172.21.15.46:0/2147536242 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f335c0a3ff0 msgr2=0x7f335c0a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:41.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.266+0000 7f3360880640 1 --2- 172.21.15.46:0/2147536242 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f335c0a3ff0 0x7f335c0a43d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3344009870 tx=0x7f334402a130 comp rx=0 tx=0).stop 2024-09-16T04:47:41.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.266+0000 7f3360880640 1 -- 172.21.15.46:0/2147536242 shutdown_connections 2024-09-16T04:47:41.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.266+0000 7f3360880640 1 --2- 172.21.15.46:0/2147536242 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f335c0a3ff0 0x7f335c0a43d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.266+0000 7f3360880640 1 --2- 172.21.15.46:0/2147536242 >> v2:172.21.15.46:3300/0 conn(0x7f335c0a3c10 0x7f335c09bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.266+0000 7f3360880640 1 -- 172.21.15.46:0/2147536242 >> 172.21.15.46:0/2147536242 conn(0x7f335c0a6340 msgr2=0x7f335c06d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:41.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.266+0000 7f3360880640 1 -- 172.21.15.46:0/2147536242 shutdown_connections 2024-09-16T04:47:41.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.266+0000 7f3360880640 1 -- 172.21.15.46:0/2147536242 wait complete. 2024-09-16T04:47:41.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.267+0000 7f3360880640 10 monclient: build_initial_monmap 2024-09-16T04:47:41.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.267+0000 7f3360880640 10 monclient: monmap: 2024-09-16T04:47:41.282 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:41.282 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:41.282 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:41.282 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:41.282 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:41.282 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:41.282 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:41.282 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:41.282 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:41.282 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:41.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.267+0000 7f3360880640 1 Processor -- start 2024-09-16T04:47:41.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.267+0000 7f3360880640 1 -- start start 2024-09-16T04:47:41.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.267+0000 7f3360880640 10 monclient: init 2024-09-16T04:47:41.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3360880640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:41.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3360880640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:47:41.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3360880640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f335c0a3c10 0x7f335c1cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:41.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3360880640 10 monclient(hunting): picked mon.noname-b con 0x7f335c0a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:41.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3360880640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f335c0a3ff0 0x7f335c1cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:41.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3360880640 10 monclient(hunting): picked mon.noname-c con 0x7f335c0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:41.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3360880640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f335c1369d0 0x7f335c1232f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:41.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3360880640 10 monclient(hunting): picked mon.noname-a con 0x7f335c1369d0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:41.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3360880640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:41.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3360880640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f335c0a4910 con 0x7f335c1369d0 2024-09-16T04:47:41.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3360880640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:41.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3360880640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f335c0a69c0 con 0x7f335c0a3c10 2024-09-16T04:47:41.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3360880640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:41.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3360880640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f335c127280 con 0x7f335c0a3ff0 2024-09-16T04:47:41.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3360880640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:41.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3360880640 10 monclient(hunting): authenticate will time out at 1259.684318s 2024-09-16T04:47:41.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3359d9b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f335c0a3ff0 0x7f335c1cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:41.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f335c1369d0 0x7f335c1232f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:41.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3359d9b640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f335c0a3ff0 0x7f335c1cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38778/0 (socket says 172.21.15.46:38778) 2024-09-16T04:47:41.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3359d9b640 1 -- 172.21.15.46:0/1659325386 learned_addr learned my addr 172.21.15.46:0/1659325386 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:41.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3359d9b640 10 monclient(hunting): get_auth_request con 0x7f335c0a3ff0 auth_method 0 2024-09-16T04:47:41.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f335c1369d0 0x7f335c1232f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41840/0 (socket says 172.21.15.46:41840) 2024-09-16T04:47:41.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3359d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:41.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3359d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:41.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3359d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:41.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335ad9d640 10 monclient(hunting): get_auth_request con 0x7f335c1369d0 auth_method 0 2024-09-16T04:47:41.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:41.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:41.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:41.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3359d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:41.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3359d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:41.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3359d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:41.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:41.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:41.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:41.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335a59c640 1 --2- 172.21.15.46:0/1659325386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f335c0a3c10 0x7f335c1cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:41.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335a59c640 10 monclient(hunting): get_auth_request con 0x7f335c0a3c10 auth_method 0 2024-09-16T04:47:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335ad9d640 10 monclient(hunting): handle_auth_done global_id 104292 payload 824 2024-09-16T04:47:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335ad9d640 1 -- 172.21.15.46:0/1659325386 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f335c0a3ff0 msgr2=0x7f335c1cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335ad9d640 1 --2- 172.21.15.46:0/1659325386 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f335c0a3ff0 0x7f335c1cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335ad9d640 1 -- 172.21.15.46:0/1659325386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f335c0a3c10 msgr2=0x7f335c1cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335ad9d640 1 --2- 172.21.15.46:0/1659325386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f335c0a3c10 0x7f335c1cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335ad9d640 10 monclient: _finish_hunting 0 2024-09-16T04:47:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335ad9d640 1 monclient: found mon.noname-a 2024-09-16T04:47:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:41.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:41.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335ad9d640 1 -- 172.21.15.46:0/1659325386 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f335c125430 con 0x7f335c1369d0 2024-09-16T04:47:41.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f3359d9b640 1 --2- 172.21.15.46:0/1659325386 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f335c0a3ff0 0x7f335c1cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:41.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.268+0000 7f335ad9d640 1 --2- 172.21.15.46:0/1659325386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f335c1369d0 0x7f335c1232f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f335400c390 tx=0x7f335400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:41.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.269+0000 7f3352ffd640 1 -- 172.21.15.46:0/1659325386 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3354005220 con 0x7f335c1369d0 2024-09-16T04:47:41.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.269+0000 7f3352ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:41.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.269+0000 7f3352ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:41.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.269+0000 7f3352ffd640 10 monclient: dump: 2024-09-16T04:47:41.290 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:41.290 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:41.290 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.291 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.291 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:41.291 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:41.291 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:41.291 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:41.291 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:41.292 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.269+0000 7f3352ffd640 10 monclient: _finish_auth 0 2024-09-16T04:47:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.269+0000 7f3352ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:47:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.269+0000 7f3352ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.269+0000 7f3352ffd640 1 -- 172.21.15.46:0/1659325386 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33540053c0 con 0x7f335c1369d0 2024-09-16T04:47:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.269+0000 7f3360880640 5 monclient: authenticate success, global_id 104292 2024-09-16T04:47:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.269+0000 7f3360880640 10 monclient: _renew_subs 2024-09-16T04:47:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.269+0000 7f3360880640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.269+0000 7f3360880640 1 -- 172.21.15.46:0/1659325386 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f335c125af0 con 0x7f335c1369d0 2024-09-16T04:47:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.269+0000 7f3352ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.269+0000 7f3352ffd640 1 -- 172.21.15.46:0/1659325386 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3354007270 con 0x7f335c1369d0 2024-09-16T04:47:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.269+0000 7f3352ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.269+0000 7f3352ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.269+0000 7f3352ffd640 10 monclient: dump: 2024-09-16T04:47:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:41.293 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.269+0000 7f3358d99640 4 set_mon_vals no callback set 2024-09-16T04:47:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.269+0000 7f3360880640 10 monclient: _renew_subs 2024-09-16T04:47:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.269+0000 7f3360880640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:41.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.269+0000 7f3360880640 1 -- 172.21.15.46:0/1659325386 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f335c126cc0 con 0x7f335c1369d0 2024-09-16T04:47:41.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.270+0000 7f3352ffd640 1 -- 172.21.15.46:0/1659325386 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f3354019020 con 0x7f335c1369d0 2024-09-16T04:47:41.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.270+0000 7f3360880640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:41.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.270+0000 7f3360880640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:41.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.270+0000 7f3360880640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:41.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.270+0000 7f3360880640 1 -- 172.21.15.46:0/1659325386 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f335c09bcf0 con 0x7f335c1369d0 2024-09-16T04:47:41.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.270+0000 7f3352ffd640 1 --2- 172.21.15.46:0/1659325386 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f333c0383b0 0x7f333c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:41.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.270+0000 7f3352ffd640 1 -- 172.21.15.46:0/1659325386 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7f335404a730 con 0x7f335c1369d0 2024-09-16T04:47:41.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.270+0000 7f335a59c640 1 --2- 172.21.15.46:0/1659325386 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f333c0383b0 0x7f333c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:41.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.270+0000 7f335a59c640 10 monclient: get_auth_request con 0x7f333c0383b0 auth_method 0 2024-09-16T04:47:41.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.271+0000 7f335a59c640 1 --2- 172.21.15.46:0/1659325386 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f333c0383b0 0x7f333c03a870 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f334c0113e0 tx=0x7f334c00b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:41.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.275+0000 7f3352ffd640 1 -- 172.21.15.46:0/1659325386 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f33540125e0 con 0x7f335c1369d0 2024-09-16T04:47:41.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.275+0000 7f3352ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:41.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.275+0000 7f3352ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:41.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.520+0000 7f3360880640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:47:41.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.521+0000 7f3360880640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:47:41.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.521+0000 7f3360880640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:41.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.521+0000 7f3360880640 1 -- 172.21.15.46:0/1659325386 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f335c067e20 con 0x7f335c1369d0 2024-09-16T04:47:41.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.521+0000 7f3352ffd640 1 -- 172.21.15.46:0/1659325386 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f3354012840 con 0x7f335c1369d0 2024-09-16T04:47:41.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.521+0000 7f3352ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:47:41.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.521+0000 7f3352ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:47:41.523 INFO:teuthology.orchestra.run.smithi046.stdout:{"election_epoch":100,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","modified":"2024-09-16T04:38:10.269597Z","created":"2024-09-16T04:38:10.269597Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3301","nonce":0},{"type":"v1","addr":"172.21.15.46:6790","nonce":0}]},"addr":"172.21.15.46:6790/0","public_addr":"172.21.15.46:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:47:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.523+0000 7f3360880640 1 -- 172.21.15.46:0/1659325386 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f333c0383b0 msgr2=0x7f333c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.523+0000 7f3360880640 1 --2- 172.21.15.46:0/1659325386 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f333c0383b0 0x7f333c03a870 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f334c0113e0 tx=0x7f334c00b040 comp rx=0 tx=0).stop 2024-09-16T04:47:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.523+0000 7f3360880640 10 monclient: shutdown 2024-09-16T04:47:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.523+0000 7f3360880640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.523+0000 7f3360880640 1 -- 172.21.15.46:0/1659325386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f335c1369d0 msgr2=0x7f335c1232f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.523+0000 7f3360880640 1 --2- 172.21.15.46:0/1659325386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f335c1369d0 0x7f335c1232f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f335400c390 tx=0x7f335400c850 comp rx=0 tx=0).stop 2024-09-16T04:47:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.523+0000 7f3360880640 1 -- 172.21.15.46:0/1659325386 shutdown_connections 2024-09-16T04:47:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.523+0000 7f3360880640 1 --2- 172.21.15.46:0/1659325386 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f333c0383b0 0x7f333c03a870 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.523+0000 7f3360880640 1 --2- 172.21.15.46:0/1659325386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f335c1369d0 0x7f335c1232f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.523+0000 7f3360880640 1 --2- 172.21.15.46:0/1659325386 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f335c0a3ff0 0x7f335c1cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.523+0000 7f3360880640 1 --2- 172.21.15.46:0/1659325386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f335c0a3c10 0x7f335c1cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.523+0000 7f3360880640 1 -- 172.21.15.46:0/1659325386 >> 172.21.15.46:0/1659325386 conn(0x7f335c0a6340 msgr2=0x7f335c0a7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.523+0000 7f3360880640 1 -- 172.21.15.46:0/1659325386 shutdown_connections 2024-09-16T04:47:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.523+0000 7f3360880640 1 -- 172.21.15.46:0/1659325386 wait complete. 2024-09-16T04:47:41.535 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:47:41.535 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T04:47:41.545 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T04:47:41.545 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:41.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.644+0000 7f4407b11640 10 monclient: get_monmap_and_config 2024-09-16T04:47:41.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.644+0000 7f4407b11640 10 monclient: build_initial_monmap 2024-09-16T04:47:41.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.644+0000 7f4407b11640 10 monclient: monmap: 2024-09-16T04:47:41.647 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:41.647 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:41.647 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:41.646423+0000 2024-09-16T04:47:41.647 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:41.646423+0000 2024-09-16T04:47:41.647 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:41.647 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:41.647 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:41.647 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:41.647 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:41.647 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:41.648 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:41.648 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:41.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.645+0000 7f4407b11640 1 Processor -- start 2024-09-16T04:47:41.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.645+0000 7f4407b11640 1 -- start start 2024-09-16T04:47:41.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.645+0000 7f4407b11640 10 monclient: init 2024-09-16T04:47:41.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f4407b11640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:41.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f4407b11640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-16T04:47:41.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f4407b11640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4400130900 0x7f4400130ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:41.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f4407b11640 10 monclient(hunting): picked mon.noname-c con 0x7f4400130900 addr v2:172.21.15.114:3300/0 2024-09-16T04:47:41.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f4407b11640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4400133750 0x7f4400133b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:41.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f4407b11640 10 monclient(hunting): picked mon.noname-a con 0x7f4400133750 addr v2:172.21.15.46:3300/0 2024-09-16T04:47:41.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f4407b11640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44001340d0 0x7f440013aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:41.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f4407b11640 10 monclient(hunting): picked mon.noname-b con 0x7f44001340d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:41.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f4407b11640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:41.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f4407b11640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f44000a16e0 con 0x7f4400133750 2024-09-16T04:47:41.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f4407b11640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:41.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f4407b11640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f4400134560 con 0x7f4400130900 2024-09-16T04:47:41.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f4407b11640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:41.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f4407b11640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f440009d5b0 con 0x7f44001340d0 2024-09-16T04:47:41.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f4407b11640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:41.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f4407b11640 10 monclient(hunting): authenticate will time out at 1260.062270s 2024-09-16T04:47:41.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44050ac640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4400133750 0x7f4400133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:41.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44050ac640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4400133750 0x7f4400133b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41846/0 (socket says 172.21.15.46:41846) 2024-09-16T04:47:41.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44050ac640 1 -- 172.21.15.46:0/1632111020 learned_addr learned my addr 172.21.15.46:0/1632111020 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:41.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44060ae640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44001340d0 0x7f440013aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:41.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44050ac640 10 monclient(hunting): get_auth_request con 0x7f4400133750 auth_method 0 2024-09-16T04:47:41.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44050ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:41.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44050ac640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:41.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44050ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:41.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44060ae640 10 monclient(hunting): get_auth_request con 0x7f44001340d0 auth_method 0 2024-09-16T04:47:41.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44060ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:41.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44060ae640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:41.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44060ae640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:41.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44058ad640 1 --2- 172.21.15.46:0/1632111020 >> v2:172.21.15.114:3300/0 conn(0x7f4400130900 0x7f4400130ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:41.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44050ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:41.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44050ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:41.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44050ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44058ad640 10 monclient(hunting): get_auth_request con 0x7f4400130900 auth_method 0 2024-09-16T04:47:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44058ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44058ad640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44058ad640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44060ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44060ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44060ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44058ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44058ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44058ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44050ac640 10 monclient(hunting): handle_auth_done global_id 104295 payload 274 2024-09-16T04:47:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44050ac640 1 -- 172.21.15.46:0/1632111020 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44001340d0 msgr2=0x7f440013aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44050ac640 1 --2- 172.21.15.46:0/1632111020 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44001340d0 0x7f440013aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44050ac640 1 -- 172.21.15.46:0/1632111020 >> v2:172.21.15.114:3300/0 conn(0x7f4400130900 msgr2=0x7f4400130ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:41.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44050ac640 1 --2- 172.21.15.46:0/1632111020 >> v2:172.21.15.114:3300/0 conn(0x7f4400130900 0x7f4400130ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44050ac640 10 monclient: _finish_hunting 0 2024-09-16T04:47:41.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44050ac640 1 monclient: found mon.noname-a 2024-09-16T04:47:41.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44050ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:41.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44050ac640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:41.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44050ac640 1 -- 172.21.15.46:0/1632111020 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f440009dae0 con 0x7f4400133750 2024-09-16T04:47:41.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.646+0000 7f44060ae640 1 --2- 172.21.15.46:0/1632111020 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44001340d0 0x7f440013aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:41.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.647+0000 7f44050ac640 1 --2- 172.21.15.46:0/1632111020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4400133750 0x7f4400133b90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f440009bee0 tx=0x7f43f002c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8198b2b39997d2dd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:41.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.647+0000 7f44048ab640 1 -- 172.21.15.46:0/1632111020 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f0038070 con 0x7f4400133750 2024-09-16T04:47:41.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.647+0000 7f44048ab640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:41.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.647+0000 7f44048ab640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:41.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.647+0000 7f44048ab640 10 monclient: dump: 2024-09-16T04:47:41.660 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:41.661 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:41.661 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.661 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.661 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:41.661 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:41.661 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:41.661 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:41.661 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:41.661 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:41.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.647+0000 7f44048ab640 10 monclient: _finish_auth 0 2024-09-16T04:47:41.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.647+0000 7f44048ab640 10 monclient: _check_auth_tickets 2024-09-16T04:47:41.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.647+0000 7f44048ab640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:41.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.647+0000 7f44048ab640 1 -- 172.21.15.46:0/1632111020 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f0031710 con 0x7f4400133750 2024-09-16T04:47:41.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.647+0000 7f4407b11640 5 monclient: authenticate success, global_id 104295 2024-09-16T04:47:41.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.647+0000 7f4407b11640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:41.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.647+0000 7f44048ab640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:41.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.647+0000 7f44048ab640 1 -- 172.21.15.46:0/1632111020 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f0031ae0 con 0x7f4400133750 2024-09-16T04:47:41.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.647+0000 7f44048ab640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:41.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.647+0000 7f44048ab640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:41.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.647+0000 7f44048ab640 10 monclient: dump: 2024-09-16T04:47:41.662 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:41.662 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:41.662 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.662 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.662 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:41.662 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:41.662 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:41.662 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:41.663 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.647+0000 7f4407b11640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.647+0000 7f4407b11640 4 set_mon_vals no callback set 2024-09-16T04:47:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.648+0000 7f4407b11640 10 monclient: shutdown 2024-09-16T04:47:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.648+0000 7f4407b11640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.648+0000 7f4407b11640 1 -- 172.21.15.46:0/1632111020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4400133750 msgr2=0x7f4400133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.648+0000 7f4407b11640 1 --2- 172.21.15.46:0/1632111020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4400133750 0x7f4400133b90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f440009bee0 tx=0x7f43f002c770 comp rx=0 tx=0).stop 2024-09-16T04:47:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.648+0000 7f4407b11640 1 -- 172.21.15.46:0/1632111020 shutdown_connections 2024-09-16T04:47:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.648+0000 7f4407b11640 1 --2- 172.21.15.46:0/1632111020 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44001340d0 0x7f440013aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.648+0000 7f4407b11640 1 --2- 172.21.15.46:0/1632111020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4400133750 0x7f4400133b90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.648+0000 7f4407b11640 1 --2- 172.21.15.46:0/1632111020 >> v2:172.21.15.114:3300/0 conn(0x7f4400130900 0x7f4400130ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.648+0000 7f4407b11640 1 -- 172.21.15.46:0/1632111020 >> 172.21.15.46:0/1632111020 conn(0x7f44000a5960 msgr2=0x7f44000a1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.648+0000 7f4407b11640 1 -- 172.21.15.46:0/1632111020 shutdown_connections 2024-09-16T04:47:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.648+0000 7f4407b11640 1 -- 172.21.15.46:0/1632111020 wait complete. 2024-09-16T04:47:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.648+0000 7f4407b11640 10 monclient: build_initial_monmap 2024-09-16T04:47:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.648+0000 7f4407b11640 10 monclient: monmap: 2024-09-16T04:47:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:41.665 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:41.665 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:41.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f4407b11640 1 Processor -- start 2024-09-16T04:47:41.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f4407b11640 1 -- start start 2024-09-16T04:47:41.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f4407b11640 10 monclient: init 2024-09-16T04:47:41.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f4407b11640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:41.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f4407b11640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:47:41.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f4407b11640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4400130900 0x7f44001cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:41.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f4407b11640 10 monclient(hunting): picked mon.noname-b con 0x7f4400130900 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:41.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f4407b11640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4400133750 0x7f440012fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:41.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f4407b11640 10 monclient(hunting): picked mon.noname-a con 0x7f4400133750 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:41.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f4407b11640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44001340d0 0x7f4400130020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:41.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f4407b11640 10 monclient(hunting): picked mon.noname-c con 0x7f44001340d0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f4407b11640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f4407b11640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f44000a2290 con 0x7f4400133750 2024-09-16T04:47:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f4407b11640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f4407b11640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f44000a2410 con 0x7f4400130900 2024-09-16T04:47:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f4407b11640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f4407b11640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f44000a0fe0 con 0x7f44001340d0 2024-09-16T04:47:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f4407b11640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f4407b11640 10 monclient(hunting): authenticate will time out at 1260.065706s 2024-09-16T04:47:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f44060ae640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44001340d0 0x7f4400130020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f44050ac640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4400133750 0x7f440012fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f44060ae640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44001340d0 0x7f4400130020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38788/0 (socket says 172.21.15.46:38788) 2024-09-16T04:47:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f44050ac640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4400133750 0x7f440012fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41852/0 (socket says 172.21.15.46:41852) 2024-09-16T04:47:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f44060ae640 1 -- 172.21.15.46:0/3846324656 learned_addr learned my addr 172.21.15.46:0/3846324656 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f44060ae640 10 monclient(hunting): get_auth_request con 0x7f44001340d0 auth_method 0 2024-09-16T04:47:41.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f44060ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:41.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f44060ae640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:41.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f44060ae640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:41.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f44050ac640 10 monclient(hunting): get_auth_request con 0x7f4400133750 auth_method 0 2024-09-16T04:47:41.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f44050ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:41.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f44050ac640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:41.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f44050ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:41.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f44050ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:41.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f44050ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:41.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f44050ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:41.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f44060ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:41.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f44060ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:41.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.649+0000 7f44060ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f44058ad640 1 --2- 172.21.15.46:0/3846324656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4400130900 0x7f44001cbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f44058ad640 10 monclient(hunting): get_auth_request con 0x7f4400130900 auth_method 0 2024-09-16T04:47:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f44058ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f44058ad640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f44058ad640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f44050ac640 10 monclient(hunting): handle_auth_done global_id 104298 payload 824 2024-09-16T04:47:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f44050ac640 1 -- 172.21.15.46:0/3846324656 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44001340d0 msgr2=0x7f4400130020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f44050ac640 1 --2- 172.21.15.46:0/3846324656 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44001340d0 0x7f4400130020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f44050ac640 1 -- 172.21.15.46:0/3846324656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4400130900 msgr2=0x7f44001cbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f44050ac640 1 --2- 172.21.15.46:0/3846324656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4400130900 0x7f44001cbad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f44050ac640 10 monclient: _finish_hunting 0 2024-09-16T04:47:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f44050ac640 1 monclient: found mon.noname-a 2024-09-16T04:47:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f44050ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f44050ac640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f44050ac640 1 -- 172.21.15.46:0/3846324656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4400124230 con 0x7f4400133750 2024-09-16T04:47:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f44060ae640 1 --2- 172.21.15.46:0/3846324656 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44001340d0 0x7f4400130020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f44050ac640 1 --2- 172.21.15.46:0/3846324656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4400133750 0x7f440012fae0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f43f0033000 tx=0x7f43f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f43f67fc640 1 -- 172.21.15.46:0/3846324656 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f0043070 con 0x7f4400133750 2024-09-16T04:47:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f43f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f43f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f43f67fc640 10 monclient: dump: 2024-09-16T04:47:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:41.670 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f43f67fc640 10 monclient: _finish_auth 0 2024-09-16T04:47:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f43f67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:47:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f43f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f43f67fc640 1 -- 172.21.15.46:0/3846324656 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f00339c0 con 0x7f4400133750 2024-09-16T04:47:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f4407b11640 5 monclient: authenticate success, global_id 104298 2024-09-16T04:47:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f43f67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f43f67fc640 1 -- 172.21.15.46:0/3846324656 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f0038070 con 0x7f4400133750 2024-09-16T04:47:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f4407b11640 10 monclient: _renew_subs 2024-09-16T04:47:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f4407b11640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f4407b11640 1 -- 172.21.15.46:0/3846324656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44001cc010 con 0x7f4400133750 2024-09-16T04:47:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f43f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f43f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f43f67fc640 10 monclient: dump: 2024-09-16T04:47:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:41.671 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:41.671 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f43f7fff640 4 set_mon_vals no callback set 2024-09-16T04:47:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f4407b11640 10 monclient: _renew_subs 2024-09-16T04:47:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f4407b11640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.650+0000 7f4407b11640 1 -- 172.21.15.46:0/3846324656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f440009c100 con 0x7f4400133750 2024-09-16T04:47:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.651+0000 7f43f67fc640 1 -- 172.21.15.46:0/3846324656 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f43f0046070 con 0x7f4400133750 2024-09-16T04:47:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.651+0000 7f4407b11640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.651+0000 7f4407b11640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.651+0000 7f4407b11640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.651+0000 7f4407b11640 1 -- 172.21.15.46:0/3846324656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f440009ee80 con 0x7f4400133750 2024-09-16T04:47:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.651+0000 7f43f67fc640 1 --2- 172.21.15.46:0/3846324656 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f43e40383b0 0x7f43e403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.651+0000 7f43f67fc640 1 -- 172.21.15.46:0/3846324656 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7f43f006b730 con 0x7f4400133750 2024-09-16T04:47:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.651+0000 7f44058ad640 1 --2- 172.21.15.46:0/3846324656 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f43e40383b0 0x7f43e403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.651+0000 7f44058ad640 10 monclient: get_auth_request con 0x7f43e40383b0 auth_method 0 2024-09-16T04:47:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.652+0000 7f44058ad640 1 --2- 172.21.15.46:0/3846324656 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f43e40383b0 0x7f43e403a870 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f43fc009ad0 tx=0x7f43fc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.656+0000 7f43f67fc640 1 -- 172.21.15.46:0/3846324656 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f43f0046360 con 0x7f4400133750 2024-09-16T04:47:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.656+0000 7f43f67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.656+0000 7f43f67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:41.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.828+0000 7f4407b11640 1 -- 172.21.15.46:0/3846324656 --> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4400124470 con 0x7f43e40383b0 2024-09-16T04:47:41.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.830+0000 7f43f67fc640 1 -- 172.21.15.46:0/3846324656 <== mgr.4099 v2:172.21.15.114:6800/3521677055 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33951 (secure 0 0 0) 0x7f43fc0119a0 con 0x7f43e40383b0 2024-09-16T04:47:41.832 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:47:41.833 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-09-16T04:47:41.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.832+0000 7f4407b11640 1 -- 172.21.15.46:0/3846324656 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f43e40383b0 msgr2=0x7f43e403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:41.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.832+0000 7f4407b11640 1 --2- 172.21.15.46:0/3846324656 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f43e40383b0 0x7f43e403a870 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f43fc009ad0 tx=0x7f43fc008040 comp rx=0 tx=0).stop 2024-09-16T04:47:41.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.832+0000 7f4407b11640 10 monclient: shutdown 2024-09-16T04:47:41.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.832+0000 7f4407b11640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:41.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.832+0000 7f4407b11640 1 -- 172.21.15.46:0/3846324656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4400133750 msgr2=0x7f440012fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:41.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.832+0000 7f4407b11640 1 --2- 172.21.15.46:0/3846324656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4400133750 0x7f440012fae0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f43f0033000 tx=0x7f43f0004730 comp rx=0 tx=0).stop 2024-09-16T04:47:41.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.832+0000 7f4407b11640 1 -- 172.21.15.46:0/3846324656 shutdown_connections 2024-09-16T04:47:41.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.832+0000 7f4407b11640 1 --2- 172.21.15.46:0/3846324656 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f44001340d0 0x7f4400130020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.832+0000 7f4407b11640 1 --2- 172.21.15.46:0/3846324656 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f43e40383b0 0x7f43e403a870 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.832+0000 7f4407b11640 1 --2- 172.21.15.46:0/3846324656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4400133750 0x7f440012fae0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.832+0000 7f4407b11640 1 --2- 172.21.15.46:0/3846324656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4400130900 0x7f44001cbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.832+0000 7f4407b11640 1 -- 172.21.15.46:0/3846324656 >> 172.21.15.46:0/3846324656 conn(0x7f44000a5960 msgr2=0x7f440009b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:41.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.832+0000 7f4407b11640 1 -- 172.21.15.46:0/3846324656 shutdown_connections 2024-09-16T04:47:41.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.832+0000 7f4407b11640 1 -- 172.21.15.46:0/3846324656 wait complete. 2024-09-16T04:47:41.844 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":576,"stamp":"2024-09-16T04:47:41.459924+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":217056,"kb_used_data":25768,"kb_used_omap":9,"kb_used_meta":191158,"kb_avail":566013984,"statfs":{"total":579820584960,"available":579598319616,"internally_reserved":0,"allocated":26386432,"data_stored":10465252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":195746464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002314"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":597,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.254605+0000","last_change":"2024-09-16T04:39:00.524013+0000","last_active":"2024-09-16T04:47:13.254605+0000","last_peered":"2024-09-16T04:47:13.254605+0000","last_clean":"2024-09-16T04:47:13.254605+0000","last_became_active":"2024-09-16T04:38:58.756778+0000","last_became_peered":"2024-09-16T04:38:58.756778+0000","last_unstale":"2024-09-16T04:47:13.254605+0000","last_undegraded":"2024-09-16T04:47:13.254605+0000","last_fullsized":"2024-09-16T04:47:13.254605+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:24:15.429766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054536700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":591,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:11.265078+0000","last_change":"2024-09-16T04:39:00.532294+0000","last_active":"2024-09-16T04:47:11.265078+0000","last_peered":"2024-09-16T04:47:11.265078+0000","last_clean":"2024-09-16T04:47:11.265078+0000","last_became_active":"2024-09-16T04:38:58.520473+0000","last_became_peered":"2024-09-16T04:38:58.520473+0000","last_unstale":"2024-09-16T04:47:11.265078+0000","last_undegraded":"2024-09-16T04:47:11.265078+0000","last_fullsized":"2024-09-16T04:47:11.265078+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:15:41.548834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053532500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":598,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.254532+0000","last_change":"2024-09-16T04:39:00.523783+0000","last_active":"2024-09-16T04:47:13.254532+0000","last_peered":"2024-09-16T04:47:13.254532+0000","last_clean":"2024-09-16T04:47:13.254532+0000","last_became_active":"2024-09-16T04:38:58.521000+0000","last_became_peered":"2024-09-16T04:38:58.521000+0000","last_unstale":"2024-09-16T04:47:13.254532+0000","last_undegraded":"2024-09-16T04:47:13.254532+0000","last_fullsized":"2024-09-16T04:47:13.254532+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:42.837022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035484999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":591,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:11.263646+0000","last_change":"2024-09-16T04:39:00.532400+0000","last_active":"2024-09-16T04:47:11.263646+0000","last_peered":"2024-09-16T04:47:11.263646+0000","last_clean":"2024-09-16T04:47:11.263646+0000","last_became_active":"2024-09-16T04:38:58.521627+0000","last_became_peered":"2024-09-16T04:38:58.521627+0000","last_unstale":"2024-09-16T04:47:11.263646+0000","last_undegraded":"2024-09-16T04:47:11.263646+0000","last_fullsized":"2024-09-16T04:47:11.263646+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:41.804897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062256500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":594,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:11.805224+0000","last_change":"2024-09-16T04:39:00.530578+0000","last_active":"2024-09-16T04:47:11.805224+0000","last_peered":"2024-09-16T04:47:11.805224+0000","last_clean":"2024-09-16T04:47:11.805224+0000","last_became_active":"2024-09-16T04:38:58.522744+0000","last_became_peered":"2024-09-16T04:38:58.522744+0000","last_unstale":"2024-09-16T04:47:11.805224+0000","last_undegraded":"2024-09-16T04:47:11.805224+0000","last_fullsized":"2024-09-16T04:47:11.805224+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:04:47.450053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000966258,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":601,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.255101+0000","last_change":"2024-09-16T04:39:00.523661+0000","last_active":"2024-09-16T04:47:13.255101+0000","last_peered":"2024-09-16T04:47:13.255101+0000","last_clean":"2024-09-16T04:47:13.255101+0000","last_became_active":"2024-09-16T04:38:58.524372+0000","last_became_peered":"2024-09-16T04:38:58.524372+0000","last_unstale":"2024-09-16T04:47:13.255101+0000","last_undegraded":"2024-09-16T04:47:13.255101+0000","last_fullsized":"2024-09-16T04:47:13.255101+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:46.534727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047533599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":598,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.254361+0000","last_change":"2024-09-16T04:39:00.523838+0000","last_active":"2024-09-16T04:47:13.254361+0000","last_peered":"2024-09-16T04:47:13.254361+0000","last_clean":"2024-09-16T04:47:13.254361+0000","last_became_active":"2024-09-16T04:38:58.522142+0000","last_became_peered":"2024-09-16T04:38:58.522142+0000","last_unstale":"2024-09-16T04:47:13.254361+0000","last_undegraded":"2024-09-16T04:47:13.254361+0000","last_fullsized":"2024-09-16T04:47:13.254361+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:35.411495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039704200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":593,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:11.805254+0000","last_change":"2024-09-16T04:39:00.530064+0000","last_active":"2024-09-16T04:47:11.805254+0000","last_peered":"2024-09-16T04:47:11.805254+0000","last_clean":"2024-09-16T04:47:11.805254+0000","last_became_active":"2024-09-16T04:38:58.523268+0000","last_became_peered":"2024-09-16T04:38:58.523268+0000","last_unstale":"2024-09-16T04:47:11.805254+0000","last_undegraded":"2024-09-16T04:47:11.805254+0000","last_fullsized":"2024-09-16T04:47:11.805254+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:12:33.958554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043992499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":749,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.254488+0000","last_change":"2024-09-16T04:38:52.501054+0000","last_active":"2024-09-16T04:47:13.254488+0000","last_peered":"2024-09-16T04:47:13.254488+0000","last_clean":"2024-09-16T04:47:13.254488+0000","last_became_active":"2024-09-16T04:38:52.500796+0000","last_became_peered":"2024-09-16T04:38:52.500796+0000","last_unstale":"2024-09-16T04:47:13.254488+0000","last_undegraded":"2024-09-16T04:47:13.254488+0000","last_fullsized":"2024-09-16T04:47:13.254488+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:51.492404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:51.492404+0000","last_clean_scrub_stamp":"2024-09-16T04:38:51.492404+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:36.122039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607661,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31148,"kb_used_data":4120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340692,"statfs":{"total":96636764160,"available":96604868608,"internally_reserved":0,"allocated":4218880,"data_stored":1567893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607661,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31668,"kb_used_data":4640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340172,"statfs":{"total":96636764160,"available":96604336128,"internally_reserved":0,"allocated":4751360,"data_stored":2096821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607661,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39988,"kb_used_data":4640,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94331852,"statfs":{"total":96636764160,"available":96595816448,"internally_reserved":0,"allocated":4751360,"data_stored":2096821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640366,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43632,"kb_used_data":4124,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94328208,"statfs":{"total":96636764160,"available":96592084992,"internally_reserved":0,"allocated":4222976,"data_stored":1567912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640367,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31148,"kb_used_data":4120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340692,"statfs":{"total":96636764160,"available":96604868608,"internally_reserved":0,"allocated":4218880,"data_stored":1567893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640365,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39472,"kb_used_data":4124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332368,"statfs":{"total":96636764160,"available":96596344832,"internally_reserved":0,"allocated":4222976,"data_stored":1567912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:41.846 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:41.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.944+0000 7f0db1598640 10 monclient: get_monmap_and_config 2024-09-16T04:47:41.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.944+0000 7f0db1598640 10 monclient: build_initial_monmap 2024-09-16T04:47:41.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.944+0000 7f0db1598640 10 monclient: monmap: 2024-09-16T04:47:41.947 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:41.947 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:41.947 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:41.946215+0000 2024-09-16T04:47:41.947 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:41.946215+0000 2024-09-16T04:47:41.947 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:41.947 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:41.947 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:41.947 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:41.947 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:41.947 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:41.947 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:41.947 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:41.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.945+0000 7f0db1598640 1 Processor -- start 2024-09-16T04:47:41.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.945+0000 7f0db1598640 1 -- start start 2024-09-16T04:47:41.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.945+0000 7f0db1598640 10 monclient: init 2024-09-16T04:47:41.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.945+0000 7f0db1598640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:41.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.945+0000 7f0db1598640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-16T04:47:41.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.945+0000 7f0db1598640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0dac0a3ba0 0x7f0dac09bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:41.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.945+0000 7f0db1598640 10 monclient(hunting): picked mon.noname-a con 0x7f0dac0a3ba0 addr v2:172.21.15.46:3300/0 2024-09-16T04:47:41.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.945+0000 7f0db1598640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0dac09c1c0 0x7f0dac0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:41.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.945+0000 7f0db1598640 10 monclient(hunting): picked mon.noname-b con 0x7f0dac09c1c0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.945+0000 7f0db1598640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0dac0a44c0 addr v1:172.21.15.114:6789/0 2024-09-16T04:47:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.945+0000 7f0db1598640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dac09afc0 con 0x7f0dac0a44c0 2024-09-16T04:47:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.945+0000 7f0db1598640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.945+0000 7f0db1598640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0dac09b860 con 0x7f0dac0a3ba0 2024-09-16T04:47:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.945+0000 7f0db1598640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.945+0000 7f0db1598640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f0dac09b2a0 con 0x7f0dac09c1c0 2024-09-16T04:47:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.945+0000 7f0db1598640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.945+0000 7f0db1598640 10 monclient(hunting): authenticate will time out at 1260.361902s 2024-09-16T04:47:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0da27fc640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0dac09c1c0 0x7f0dac0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0daaffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0dac0a3ba0 0x7f0dac09bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0daaffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0dac0a3ba0 0x7f0dac09bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41854/0 (socket says 172.21.15.46:41854) 2024-09-16T04:47:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0da27fc640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0dac09c1c0 0x7f0dac0a3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38792/0 (socket says 172.21.15.46:38792) 2024-09-16T04:47:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0daaffd640 1 -- 172.21.15.46:0/524895489 learned_addr learned my addr 172.21.15.46:0/524895489 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0daaffd640 10 monclient(hunting): get_auth_request con 0x7f0dac0a3ba0 auth_method 0 2024-09-16T04:47:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0daaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0daaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0daaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0da27fc640 10 monclient(hunting): get_auth_request con 0x7f0dac09c1c0 auth_method 0 2024-09-16T04:47:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0da27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0da27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0da27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0da27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0da27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0da27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0daaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0daaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0daaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0da27fc640 10 monclient(hunting): handle_auth_done global_id 94457 payload 274 2024-09-16T04:47:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0da27fc640 1 -- 172.21.15.46:0/524895489 >> v2:172.21.15.46:3300/0 conn(0x7f0dac0a3ba0 msgr2=0x7f0dac09bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0da27fc640 1 --2- 172.21.15.46:0/524895489 >> v2:172.21.15.46:3300/0 conn(0x7f0dac0a3ba0 0x7f0dac09bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0da27fc640 1 -- 172.21.15.46:0/524895489 >> v1:172.21.15.114:6789/0 conn(0x7f0dac0a44c0 legacy=0x7f0dac135950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0da27fc640 10 monclient: _finish_hunting 0 2024-09-16T04:47:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0da27fc640 1 monclient: found mon.noname-b 2024-09-16T04:47:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0da27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0da27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:47:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0da27fc640 1 -- 172.21.15.46:0/524895489 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dac09b580 con 0x7f0dac09c1c0 2024-09-16T04:47:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.946+0000 7f0daaffd640 1 --2- 172.21.15.46:0/524895489 >> v2:172.21.15.46:3300/0 conn(0x7f0dac0a3ba0 0x7f0dac09bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0da27fc640 1 --2- 172.21.15.46:0/524895489 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0dac09c1c0 0x7f0dac0a3f80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0d940097a0 tx=0x7f0d9402a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba40bddbf7e141d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0daa7fc640 1 -- 172.21.15.46:0/524895489 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d94037070 con 0x7f0dac09c1c0 2024-09-16T04:47:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0daa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0daa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:47:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0daa7fc640 10 monclient: dump: 2024-09-16T04:47:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:41.959 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0daa7fc640 10 monclient: _finish_auth 0 2024-09-16T04:47:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0daa7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:47:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0daa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0daa7fc640 1 -- 172.21.15.46:0/524895489 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d9402ae50 con 0x7f0dac09c1c0 2024-09-16T04:47:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0db1598640 5 monclient: authenticate success, global_id 94457 2024-09-16T04:47:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0db1598640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0daa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0db1598640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0daa7fc640 1 -- 172.21.15.46:0/524895489 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d94032660 con 0x7f0dac09c1c0 2024-09-16T04:47:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0db1598640 4 set_mon_vals no callback set 2024-09-16T04:47:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0daa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0daa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0daa7fc640 10 monclient: dump: 2024-09-16T04:47:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:41.961 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0db1598640 10 monclient: shutdown 2024-09-16T04:47:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0db1598640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0db1598640 1 -- 172.21.15.46:0/524895489 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0dac09c1c0 msgr2=0x7f0dac0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0db1598640 1 --2- 172.21.15.46:0/524895489 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0dac09c1c0 0x7f0dac0a3f80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0d940097a0 tx=0x7f0d9402a100 comp rx=0 tx=0).stop 2024-09-16T04:47:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0db1598640 1 -- 172.21.15.46:0/524895489 shutdown_connections 2024-09-16T04:47:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0db1598640 1 --2- 172.21.15.46:0/524895489 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0dac09c1c0 0x7f0dac0a3f80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0db1598640 1 --2- 172.21.15.46:0/524895489 >> v2:172.21.15.46:3300/0 conn(0x7f0dac0a3ba0 0x7f0dac09bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0db1598640 1 -- 172.21.15.46:0/524895489 >> 172.21.15.46:0/524895489 conn(0x7f0dac0a60c0 msgr2=0x7f0dac0a64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0db1598640 1 -- 172.21.15.46:0/524895489 shutdown_connections 2024-09-16T04:47:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.947+0000 7f0db1598640 1 -- 172.21.15.46:0/524895489 wait complete. 2024-09-16T04:47:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.948+0000 7f0db1598640 10 monclient: build_initial_monmap 2024-09-16T04:47:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.948+0000 7f0db1598640 10 monclient: monmap: 2024-09-16T04:47:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:41.963 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.948+0000 7f0db1598640 1 Processor -- start 2024-09-16T04:47:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0db1598640 1 -- start start 2024-09-16T04:47:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0db1598640 10 monclient: init 2024-09-16T04:47:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0db1598640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0db1598640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:47:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0db1598640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dac09c1c0 0x7f0dac1cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0db1598640 10 monclient(hunting): picked mon.noname-a con 0x7f0dac09c1c0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0db1598640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0dac0a3ba0 0x7f0dac1cc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0db1598640 10 monclient(hunting): picked mon.noname-c con 0x7f0dac0a3ba0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0db1598640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dac0a44c0 0x7f0dac122770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0db1598640 10 monclient(hunting): picked mon.noname-b con 0x7f0dac0a44c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0db1598640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0db1598640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0dac136060 con 0x7f0dac09c1c0 2024-09-16T04:47:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0db1598640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0db1598640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0dac0a49d0 con 0x7f0dac0a44c0 2024-09-16T04:47:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0db1598640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0db1598640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f0dac1361e0 con 0x7f0dac0a3ba0 2024-09-16T04:47:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0db1598640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0db1598640 10 monclient(hunting): authenticate will time out at 1260.365508s 2024-09-16T04:47:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0daaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dac09c1c0 0x7f0dac1cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0daaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dac09c1c0 0x7f0dac1cbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41856/0 (socket says 172.21.15.46:41856) 2024-09-16T04:47:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0daaffd640 1 -- 172.21.15.46:0/2709470096 learned_addr learned my addr 172.21.15.46:0/2709470096 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0daaffd640 10 monclient(hunting): get_auth_request con 0x7f0dac09c1c0 auth_method 0 2024-09-16T04:47:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0da27fc640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0dac0a3ba0 0x7f0dac1cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0daaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0daaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0daaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0da27fc640 10 monclient(hunting): get_auth_request con 0x7f0dac0a3ba0 auth_method 0 2024-09-16T04:47:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0da27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0da27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0da27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0daaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0daaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0daaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0da27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0da27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0da27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0dab7fe640 1 --2- 172.21.15.46:0/2709470096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dac0a44c0 0x7f0dac122770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0dab7fe640 10 monclient(hunting): get_auth_request con 0x7f0dac0a44c0 auth_method 0 2024-09-16T04:47:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0dab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0dab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0dab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0daaffd640 10 monclient(hunting): handle_auth_done global_id 104304 payload 824 2024-09-16T04:47:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0daaffd640 1 -- 172.21.15.46:0/2709470096 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0dac0a3ba0 msgr2=0x7f0dac1cc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0daaffd640 1 --2- 172.21.15.46:0/2709470096 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0dac0a3ba0 0x7f0dac1cc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0daaffd640 1 -- 172.21.15.46:0/2709470096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dac0a44c0 msgr2=0x7f0dac122770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0daaffd640 1 --2- 172.21.15.46:0/2709470096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dac0a44c0 0x7f0dac122770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0daaffd640 10 monclient: _finish_hunting 0 2024-09-16T04:47:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0daaffd640 1 monclient: found mon.noname-a 2024-09-16T04:47:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0daaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0daaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0daaffd640 1 -- 172.21.15.46:0/2709470096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dac1270e0 con 0x7f0dac09c1c0 2024-09-16T04:47:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.949+0000 7f0da27fc640 1 --2- 172.21.15.46:0/2709470096 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0dac0a3ba0 0x7f0dac1cc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0daaffd640 1 --2- 172.21.15.46:0/2709470096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dac09c1c0 0x7f0dac1cbf50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c00e9d0 tx=0x7f0d9c00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0da3fff640 1 -- 172.21.15.46:0/2709470096 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c017070 con 0x7f0dac09c1c0 2024-09-16T04:47:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0da3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0da3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0da3fff640 10 monclient: dump: 2024-09-16T04:47:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:41.968 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:41.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0da3fff640 10 monclient: _finish_auth 0 2024-09-16T04:47:41.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0da3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:47:41.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0da3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:41.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0da3fff640 1 -- 172.21.15.46:0/2709470096 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d9c00ca60 con 0x7f0dac09c1c0 2024-09-16T04:47:41.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0db1598640 5 monclient: authenticate success, global_id 104304 2024-09-16T04:47:41.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0db1598640 10 monclient: _renew_subs 2024-09-16T04:47:41.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0db1598640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:41.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0db1598640 1 -- 172.21.15.46:0/2709470096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dac1248b0 con 0x7f0dac09c1c0 2024-09-16T04:47:41.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0da3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0da3fff640 1 -- 172.21.15.46:0/2709470096 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c00ce10 con 0x7f0dac09c1c0 2024-09-16T04:47:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0da3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0da3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0da3fff640 10 monclient: dump: 2024-09-16T04:47:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:41.970 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0da9ffb640 4 set_mon_vals no callback set 2024-09-16T04:47:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0db1598640 10 monclient: _renew_subs 2024-09-16T04:47:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0db1598640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.950+0000 7f0db1598640 1 -- 172.21.15.46:0/2709470096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dac0a83b0 con 0x7f0dac09c1c0 2024-09-16T04:47:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.951+0000 7f0da3fff640 1 -- 172.21.15.46:0/2709470096 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f0d9c01a070 con 0x7f0dac09c1c0 2024-09-16T04:47:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.951+0000 7f0db1598640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.951+0000 7f0db1598640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.951+0000 7f0db1598640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.951+0000 7f0db1598640 1 -- 172.21.15.46:0/2709470096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dac09cd60 con 0x7f0dac09c1c0 2024-09-16T04:47:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.951+0000 7f0da3fff640 1 --2- 172.21.15.46:0/2709470096 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0d8c0383b0 0x7f0d8c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.951+0000 7f0da3fff640 1 -- 172.21.15.46:0/2709470096 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7f0d9c04b580 con 0x7f0dac09c1c0 2024-09-16T04:47:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.951+0000 7f0da27fc640 1 --2- 172.21.15.46:0/2709470096 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0d8c0383b0 0x7f0d8c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:41.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.951+0000 7f0da27fc640 10 monclient: get_auth_request con 0x7f0d8c0383b0 auth_method 0 2024-09-16T04:47:41.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.952+0000 7f0da27fc640 1 --2- 172.21.15.46:0/2709470096 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0d8c0383b0 0x7f0d8c03a870 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f0d94034000 tx=0x7f0d9402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:41.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.956+0000 7f0da3fff640 1 -- 172.21.15.46:0/2709470096 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0dac09cd60 con 0x7f0dac09c1c0 2024-09-16T04:47:41.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.956+0000 7f0da3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:41.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:41.956+0000 7f0da3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:42.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.127+0000 7f0db1598640 1 -- 172.21.15.46:0/2709470096 --> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0dac067e20 con 0x7f0d8c0383b0 2024-09-16T04:47:42.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.129+0000 7f0da3fff640 1 -- 172.21.15.46:0/2709470096 <== mgr.4099 v2:172.21.15.114:6800/3521677055 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33951 (secure 0 0 0) 0x7f0dac067e20 con 0x7f0d8c0383b0 2024-09-16T04:47:42.131 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:47:42.131 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-09-16T04:47:42.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.130+0000 7f0db1598640 1 -- 172.21.15.46:0/2709470096 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0d8c0383b0 msgr2=0x7f0d8c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:42.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.130+0000 7f0db1598640 1 --2- 172.21.15.46:0/2709470096 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0d8c0383b0 0x7f0d8c03a870 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f0d94034000 tx=0x7f0d9402e000 comp rx=0 tx=0).stop 2024-09-16T04:47:42.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.130+0000 7f0db1598640 10 monclient: shutdown 2024-09-16T04:47:42.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.130+0000 7f0db1598640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:42.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.130+0000 7f0db1598640 1 -- 172.21.15.46:0/2709470096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dac09c1c0 msgr2=0x7f0dac1cbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:42.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.130+0000 7f0db1598640 1 --2- 172.21.15.46:0/2709470096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dac09c1c0 0x7f0dac1cbf50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c00e9d0 tx=0x7f0d9c00ee90 comp rx=0 tx=0).stop 2024-09-16T04:47:42.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.131+0000 7f0db1598640 1 -- 172.21.15.46:0/2709470096 shutdown_connections 2024-09-16T04:47:42.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.131+0000 7f0db1598640 1 --2- 172.21.15.46:0/2709470096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dac0a44c0 0x7f0dac122770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.131+0000 7f0db1598640 1 --2- 172.21.15.46:0/2709470096 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f0dac0a3ba0 0x7f0dac1cc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.131+0000 7f0db1598640 1 --2- 172.21.15.46:0/2709470096 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f0d8c0383b0 0x7f0d8c03a870 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.131+0000 7f0db1598640 1 --2- 172.21.15.46:0/2709470096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dac09c1c0 0x7f0dac1cbf50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.131+0000 7f0db1598640 1 -- 172.21.15.46:0/2709470096 >> 172.21.15.46:0/2709470096 conn(0x7f0dac0a60c0 msgr2=0x7f0dac0a6fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:42.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.131+0000 7f0db1598640 1 -- 172.21.15.46:0/2709470096 shutdown_connections 2024-09-16T04:47:42.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.131+0000 7f0db1598640 1 -- 172.21.15.46:0/2709470096 wait complete. 2024-09-16T04:47:42.143 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":576,"stamp":"2024-09-16T04:47:41.459924+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":217056,"kb_used_data":25768,"kb_used_omap":9,"kb_used_meta":191158,"kb_avail":566013984,"statfs":{"total":579820584960,"available":579598319616,"internally_reserved":0,"allocated":26386432,"data_stored":10465252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":195746464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002314"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":597,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.254605+0000","last_change":"2024-09-16T04:39:00.524013+0000","last_active":"2024-09-16T04:47:13.254605+0000","last_peered":"2024-09-16T04:47:13.254605+0000","last_clean":"2024-09-16T04:47:13.254605+0000","last_became_active":"2024-09-16T04:38:58.756778+0000","last_became_peered":"2024-09-16T04:38:58.756778+0000","last_unstale":"2024-09-16T04:47:13.254605+0000","last_undegraded":"2024-09-16T04:47:13.254605+0000","last_fullsized":"2024-09-16T04:47:13.254605+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:24:15.429766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054536700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":591,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:11.265078+0000","last_change":"2024-09-16T04:39:00.532294+0000","last_active":"2024-09-16T04:47:11.265078+0000","last_peered":"2024-09-16T04:47:11.265078+0000","last_clean":"2024-09-16T04:47:11.265078+0000","last_became_active":"2024-09-16T04:38:58.520473+0000","last_became_peered":"2024-09-16T04:38:58.520473+0000","last_unstale":"2024-09-16T04:47:11.265078+0000","last_undegraded":"2024-09-16T04:47:11.265078+0000","last_fullsized":"2024-09-16T04:47:11.265078+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:15:41.548834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053532500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":598,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.254532+0000","last_change":"2024-09-16T04:39:00.523783+0000","last_active":"2024-09-16T04:47:13.254532+0000","last_peered":"2024-09-16T04:47:13.254532+0000","last_clean":"2024-09-16T04:47:13.254532+0000","last_became_active":"2024-09-16T04:38:58.521000+0000","last_became_peered":"2024-09-16T04:38:58.521000+0000","last_unstale":"2024-09-16T04:47:13.254532+0000","last_undegraded":"2024-09-16T04:47:13.254532+0000","last_fullsized":"2024-09-16T04:47:13.254532+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:42.837022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035484999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":591,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:11.263646+0000","last_change":"2024-09-16T04:39:00.532400+0000","last_active":"2024-09-16T04:47:11.263646+0000","last_peered":"2024-09-16T04:47:11.263646+0000","last_clean":"2024-09-16T04:47:11.263646+0000","last_became_active":"2024-09-16T04:38:58.521627+0000","last_became_peered":"2024-09-16T04:38:58.521627+0000","last_unstale":"2024-09-16T04:47:11.263646+0000","last_undegraded":"2024-09-16T04:47:11.263646+0000","last_fullsized":"2024-09-16T04:47:11.263646+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:41.804897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062256500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":594,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:11.805224+0000","last_change":"2024-09-16T04:39:00.530578+0000","last_active":"2024-09-16T04:47:11.805224+0000","last_peered":"2024-09-16T04:47:11.805224+0000","last_clean":"2024-09-16T04:47:11.805224+0000","last_became_active":"2024-09-16T04:38:58.522744+0000","last_became_peered":"2024-09-16T04:38:58.522744+0000","last_unstale":"2024-09-16T04:47:11.805224+0000","last_undegraded":"2024-09-16T04:47:11.805224+0000","last_fullsized":"2024-09-16T04:47:11.805224+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:04:47.450053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000966258,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":601,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.255101+0000","last_change":"2024-09-16T04:39:00.523661+0000","last_active":"2024-09-16T04:47:13.255101+0000","last_peered":"2024-09-16T04:47:13.255101+0000","last_clean":"2024-09-16T04:47:13.255101+0000","last_became_active":"2024-09-16T04:38:58.524372+0000","last_became_peered":"2024-09-16T04:38:58.524372+0000","last_unstale":"2024-09-16T04:47:13.255101+0000","last_undegraded":"2024-09-16T04:47:13.255101+0000","last_fullsized":"2024-09-16T04:47:13.255101+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:46.534727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047533599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":598,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.254361+0000","last_change":"2024-09-16T04:39:00.523838+0000","last_active":"2024-09-16T04:47:13.254361+0000","last_peered":"2024-09-16T04:47:13.254361+0000","last_clean":"2024-09-16T04:47:13.254361+0000","last_became_active":"2024-09-16T04:38:58.522142+0000","last_became_peered":"2024-09-16T04:38:58.522142+0000","last_unstale":"2024-09-16T04:47:13.254361+0000","last_undegraded":"2024-09-16T04:47:13.254361+0000","last_fullsized":"2024-09-16T04:47:13.254361+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:35.411495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039704200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":593,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:11.805254+0000","last_change":"2024-09-16T04:39:00.530064+0000","last_active":"2024-09-16T04:47:11.805254+0000","last_peered":"2024-09-16T04:47:11.805254+0000","last_clean":"2024-09-16T04:47:11.805254+0000","last_became_active":"2024-09-16T04:38:58.523268+0000","last_became_peered":"2024-09-16T04:38:58.523268+0000","last_unstale":"2024-09-16T04:47:11.805254+0000","last_undegraded":"2024-09-16T04:47:11.805254+0000","last_fullsized":"2024-09-16T04:47:11.805254+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:12:33.958554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043992499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":749,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.254488+0000","last_change":"2024-09-16T04:38:52.501054+0000","last_active":"2024-09-16T04:47:13.254488+0000","last_peered":"2024-09-16T04:47:13.254488+0000","last_clean":"2024-09-16T04:47:13.254488+0000","last_became_active":"2024-09-16T04:38:52.500796+0000","last_became_peered":"2024-09-16T04:38:52.500796+0000","last_unstale":"2024-09-16T04:47:13.254488+0000","last_undegraded":"2024-09-16T04:47:13.254488+0000","last_fullsized":"2024-09-16T04:47:13.254488+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:51.492404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:51.492404+0000","last_clean_scrub_stamp":"2024-09-16T04:38:51.492404+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:36.122039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607661,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31148,"kb_used_data":4120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340692,"statfs":{"total":96636764160,"available":96604868608,"internally_reserved":0,"allocated":4218880,"data_stored":1567893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607661,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31668,"kb_used_data":4640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340172,"statfs":{"total":96636764160,"available":96604336128,"internally_reserved":0,"allocated":4751360,"data_stored":2096821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607661,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39988,"kb_used_data":4640,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94331852,"statfs":{"total":96636764160,"available":96595816448,"internally_reserved":0,"allocated":4751360,"data_stored":2096821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640366,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43632,"kb_used_data":4124,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94328208,"statfs":{"total":96636764160,"available":96592084992,"internally_reserved":0,"allocated":4222976,"data_stored":1567912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640367,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31148,"kb_used_data":4120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340692,"statfs":{"total":96636764160,"available":96604868608,"internally_reserved":0,"allocated":4218880,"data_stored":1567893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640365,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39472,"kb_used_data":4124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332368,"statfs":{"total":96636764160,"available":96596344832,"internally_reserved":0,"allocated":4222976,"data_stored":1567912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:42.144 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T04:47:42.144 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:42.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.243+0000 7ffa76cb7640 10 monclient: get_monmap_and_config 2024-09-16T04:47:42.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.243+0000 7ffa76cb7640 10 monclient: build_initial_monmap 2024-09-16T04:47:42.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.243+0000 7ffa76cb7640 10 monclient: monmap: 2024-09-16T04:47:42.245 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:42.245 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:42.245 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:42.244533+0000 2024-09-16T04:47:42.245 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:42.244533+0000 2024-09-16T04:47:42.245 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:42.246 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:42.246 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:42.246 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:42.246 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:42.246 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:42.246 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:42.246 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:42.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.243+0000 7ffa76cb7640 1 Processor -- start 2024-09-16T04:47:42.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.243+0000 7ffa76cb7640 1 -- start start 2024-09-16T04:47:42.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.243+0000 7ffa76cb7640 10 monclient: init 2024-09-16T04:47:42.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa76cb7640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:42.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa76cb7640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-16T04:47:42.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa76cb7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffa700a3c10 0x7ffa7009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:42.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa76cb7640 10 monclient(hunting): picked mon.noname-a con 0x7ffa700a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:47:42.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa76cb7640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ffa700a3ff0 0x7ffa700a4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:42.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa76cb7640 10 monclient(hunting): picked mon.noname-c con 0x7ffa700a3ff0 addr v2:172.21.15.114:3300/0 2024-09-16T04:47:42.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa76cb7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa701369d0 addr v1:172.21.15.46:6789/0 2024-09-16T04:47:42.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa76cb7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa7009ab70 con 0x7ffa701369d0 2024-09-16T04:47:42.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa76cb7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:42.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa76cb7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ffa700a4d60 con 0x7ffa700a3c10 2024-09-16T04:47:42.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa76cb7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:42.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa76cb7640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ffa7009b030 con 0x7ffa700a3ff0 2024-09-16T04:47:42.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa76cb7640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:42.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa76cb7640 10 monclient(hunting): authenticate will time out at 1260.660378s 2024-09-16T04:47:42.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa74a53640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffa700a3c10 0x7ffa7009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:42.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa75254640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ffa701369d0 0x7ffa70136e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33306/0 (socket says 172.21.15.46:33306) 2024-09-16T04:47:42.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa75254640 1 -- 172.21.15.46:0/3782505558 learned_addr learned my addr 172.21.15.46:0/3782505558 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:47:42.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa74a53640 10 monclient(hunting): get_auth_request con 0x7ffa700a3c10 auth_method 0 2024-09-16T04:47:42.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa74a53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:42.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa74a53640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:42.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa74a53640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:42.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa6ffff640 1 --2- 172.21.15.46:0/3782505558 >> v2:172.21.15.114:3300/0 conn(0x7ffa700a3ff0 0x7ffa700a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:42.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa6ffff640 10 monclient(hunting): get_auth_request con 0x7ffa700a3ff0 auth_method 0 2024-09-16T04:47:42.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:42.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:42.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:42.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa74a53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:42.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa74a53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:42.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa74a53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:42.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:42.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:42.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:42.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa74a53640 10 monclient(hunting): handle_auth_done global_id 104307 payload 274 2024-09-16T04:47:42.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa74a53640 1 -- 172.21.15.46:0/3782505558 >> v2:172.21.15.114:3300/0 conn(0x7ffa700a3ff0 msgr2=0x7ffa700a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:42.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa74a53640 1 --2- 172.21.15.46:0/3782505558 >> v2:172.21.15.114:3300/0 conn(0x7ffa700a3ff0 0x7ffa700a4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa74a53640 1 -- 172.21.15.46:0/3782505558 >> v1:172.21.15.46:6789/0 conn(0x7ffa701369d0 legacy=0x7ffa70136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:42.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa74a53640 10 monclient: _finish_hunting 0 2024-09-16T04:47:42.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa74a53640 1 monclient: found mon.noname-a 2024-09-16T04:47:42.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa6f7fe640 1 -- 172.21.15.46:0/3782505558 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1534238521 0 0) 0x7ffa7009ab70 con 0x7ffa701369d0 2024-09-16T04:47:42.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa74a53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:42.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa74a53640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:42.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa74a53640 1 -- 172.21.15.46:0/3782505558 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa7009b310 con 0x7ffa700a3c10 2024-09-16T04:47:42.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.244+0000 7ffa6f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:47:42.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.245+0000 7ffa74a53640 1 --2- 172.21.15.46:0/3782505558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa700a3c10 0x7ffa7009bcf0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ffa7006edc0 tx=0x7ffa6002a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a47716527af1b0b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:42.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.245+0000 7ffa6f7fe640 1 -- 172.21.15.46:0/3782505558 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa60037070 con 0x7ffa700a3c10 2024-09-16T04:47:42.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.245+0000 7ffa6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:42.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.245+0000 7ffa6f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:42.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.245+0000 7ffa6f7fe640 10 monclient: dump: 2024-09-16T04:47:42.261 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:42.261 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:42.261 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:42.261 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:42.261 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:42.261 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:42.262 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.245+0000 7ffa6f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:47:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.245+0000 7ffa6f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:47:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.245+0000 7ffa6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.245+0000 7ffa6f7fe640 1 -- 172.21.15.46:0/3782505558 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa6002add0 con 0x7ffa700a3c10 2024-09-16T04:47:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.245+0000 7ffa76cb7640 5 monclient: authenticate success, global_id 104307 2024-09-16T04:47:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.245+0000 7ffa76cb7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.245+0000 7ffa6f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.245+0000 7ffa76cb7640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.245+0000 7ffa6f7fe640 1 -- 172.21.15.46:0/3782505558 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa60032660 con 0x7ffa700a3c10 2024-09-16T04:47:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.245+0000 7ffa76cb7640 4 set_mon_vals no callback set 2024-09-16T04:47:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.245+0000 7ffa6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.245+0000 7ffa6f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.245+0000 7ffa6f7fe640 10 monclient: dump: 2024-09-16T04:47:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:42.264 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.246+0000 7ffa76cb7640 10 monclient: shutdown 2024-09-16T04:47:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.246+0000 7ffa76cb7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.246+0000 7ffa76cb7640 1 -- 172.21.15.46:0/3782505558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa700a3c10 msgr2=0x7ffa7009bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.246+0000 7ffa76cb7640 1 --2- 172.21.15.46:0/3782505558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa700a3c10 0x7ffa7009bcf0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ffa7006edc0 tx=0x7ffa6002a080 comp rx=0 tx=0).stop 2024-09-16T04:47:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.246+0000 7ffa76cb7640 1 -- 172.21.15.46:0/3782505558 shutdown_connections 2024-09-16T04:47:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.246+0000 7ffa76cb7640 1 --2- 172.21.15.46:0/3782505558 >> v2:172.21.15.114:3300/0 conn(0x7ffa700a3ff0 0x7ffa700a4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.246+0000 7ffa76cb7640 1 --2- 172.21.15.46:0/3782505558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa700a3c10 0x7ffa7009bcf0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.246+0000 7ffa76cb7640 1 -- 172.21.15.46:0/3782505558 >> 172.21.15.46:0/3782505558 conn(0x7ffa700a5d10 msgr2=0x7ffa700a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.246+0000 7ffa76cb7640 1 -- 172.21.15.46:0/3782505558 shutdown_connections 2024-09-16T04:47:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.246+0000 7ffa76cb7640 1 -- 172.21.15.46:0/3782505558 wait complete. 2024-09-16T04:47:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.246+0000 7ffa76cb7640 10 monclient: build_initial_monmap 2024-09-16T04:47:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.246+0000 7ffa76cb7640 10 monclient: monmap: 2024-09-16T04:47:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:42.265 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.247+0000 7ffa76cb7640 1 Processor -- start 2024-09-16T04:47:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.247+0000 7ffa76cb7640 1 -- start start 2024-09-16T04:47:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.247+0000 7ffa76cb7640 10 monclient: init 2024-09-16T04:47:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.247+0000 7ffa76cb7640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.247+0000 7ffa76cb7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:47:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.247+0000 7ffa76cb7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa700a3c10 0x7ffa701cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.247+0000 7ffa76cb7640 10 monclient(hunting): picked mon.noname-a con 0x7ffa700a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa76cb7640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ffa700a3ff0 0x7ffa701cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa76cb7640 10 monclient(hunting): picked mon.noname-c con 0x7ffa700a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa76cb7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa701369d0 0x7ffa701231f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa76cb7640 10 monclient(hunting): picked mon.noname-b con 0x7ffa701369d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa76cb7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa76cb7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7006e3d0 con 0x7ffa700a3c10 2024-09-16T04:47:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa76cb7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa76cb7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffa700a4940 con 0x7ffa701369d0 2024-09-16T04:47:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa76cb7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa76cb7640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7ffa7006e550 con 0x7ffa700a3ff0 2024-09-16T04:47:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa76cb7640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa76cb7640 10 monclient(hunting): authenticate will time out at 1260.664058s 2024-09-16T04:47:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa74a53640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa700a3c10 0x7ffa701cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa74a53640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa700a3c10 0x7ffa701cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41866/0 (socket says 172.21.15.46:41866) 2024-09-16T04:47:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa74a53640 1 -- 172.21.15.46:0/3366946606 learned_addr learned my addr 172.21.15.46:0/3366946606 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa74a53640 10 monclient(hunting): get_auth_request con 0x7ffa700a3c10 auth_method 0 2024-09-16T04:47:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa74a53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa74a53640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa74a53640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6ffff640 1 --2- 172.21.15.46:0/3366946606 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ffa700a3ff0 0x7ffa701cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6ffff640 10 monclient(hunting): get_auth_request con 0x7ffa700a3ff0 auth_method 0 2024-09-16T04:47:42.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:42.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:42.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:42.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa74a53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:42.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa74a53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:42.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa74a53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:42.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:42.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:42.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:42.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa75254640 1 --2- 172.21.15.46:0/3366946606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa701369d0 0x7ffa701231f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:42.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa75254640 10 monclient(hunting): get_auth_request con 0x7ffa701369d0 auth_method 0 2024-09-16T04:47:42.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa75254640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:42.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa75254640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa75254640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa74a53640 10 monclient(hunting): handle_auth_done global_id 104313 payload 824 2024-09-16T04:47:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa74a53640 1 -- 172.21.15.46:0/3366946606 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ffa700a3ff0 msgr2=0x7ffa701cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa74a53640 1 --2- 172.21.15.46:0/3366946606 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ffa700a3ff0 0x7ffa701cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa74a53640 1 -- 172.21.15.46:0/3366946606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa701369d0 msgr2=0x7ffa701231f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa74a53640 1 --2- 172.21.15.46:0/3366946606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa701369d0 0x7ffa701231f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa74a53640 10 monclient: _finish_hunting 0 2024-09-16T04:47:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa74a53640 1 monclient: found mon.noname-a 2024-09-16T04:47:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa74a53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa74a53640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa74a53640 1 -- 172.21.15.46:0/3366946606 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa70125860 con 0x7ffa700a3c10 2024-09-16T04:47:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa74a53640 1 --2- 172.21.15.46:0/3366946606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa700a3c10 0x7ffa701cbee0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffa60000c00 tx=0x7ffa600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6d7fa640 1 -- 172.21.15.46:0/3366946606 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa60037040 con 0x7ffa700a3c10 2024-09-16T04:47:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6d7fa640 10 monclient: dump: 2024-09-16T04:47:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:42.271 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:47:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:47:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa76cb7640 5 monclient: authenticate success, global_id 104313 2024-09-16T04:47:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6d7fa640 1 -- 172.21.15.46:0/3366946606 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa60042070 con 0x7ffa700a3c10 2024-09-16T04:47:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa76cb7640 10 monclient: _renew_subs 2024-09-16T04:47:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa76cb7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa76cb7640 1 -- 172.21.15.46:0/3366946606 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa70126d80 con 0x7ffa700a3c10 2024-09-16T04:47:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6d7fa640 1 -- 172.21.15.46:0/3366946606 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6003d660 con 0x7ffa700a3c10 2024-09-16T04:47:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6d7fa640 10 monclient: dump: 2024-09-16T04:47:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:42.272 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa76cb7640 10 monclient: _renew_subs 2024-09-16T04:47:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa6effd640 4 set_mon_vals no callback set 2024-09-16T04:47:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa76cb7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.248+0000 7ffa76cb7640 1 -- 172.21.15.46:0/3366946606 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa701267c0 con 0x7ffa700a3c10 2024-09-16T04:47:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.249+0000 7ffa6d7fa640 1 -- 172.21.15.46:0/3366946606 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7ffa6003d800 con 0x7ffa700a3c10 2024-09-16T04:47:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.249+0000 7ffa76cb7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.249+0000 7ffa76cb7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.249+0000 7ffa76cb7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.249+0000 7ffa6d7fa640 1 --2- 172.21.15.46:0/3366946606 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ffa5c0383b0 0x7ffa5c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.249+0000 7ffa76cb7640 1 -- 172.21.15.46:0/3366946606 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa7009bcf0 con 0x7ffa700a3c10 2024-09-16T04:47:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.249+0000 7ffa6d7fa640 1 -- 172.21.15.46:0/3366946606 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7ffa6006bc80 con 0x7ffa700a3c10 2024-09-16T04:47:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.250+0000 7ffa6ffff640 1 --2- 172.21.15.46:0/3366946606 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ffa5c0383b0 0x7ffa5c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.250+0000 7ffa6ffff640 10 monclient: get_auth_request con 0x7ffa5c0383b0 auth_method 0 2024-09-16T04:47:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.250+0000 7ffa6ffff640 1 --2- 172.21.15.46:0/3366946606 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ffa5c0383b0 0x7ffa5c03a870 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ffa700a6d00 tx=0x7ffa64009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.254+0000 7ffa6d7fa640 1 -- 172.21.15.46:0/3366946606 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ffa600477e0 con 0x7ffa700a3c10 2024-09-16T04:47:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.254+0000 7ffa6d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:42.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.254+0000 7ffa6d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:42.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.427+0000 7ffa76cb7640 1 -- 172.21.15.46:0/3366946606 --> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ffa70125150 con 0x7ffa5c0383b0 2024-09-16T04:47:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.429+0000 7ffa6d7fa640 1 -- 172.21.15.46:0/3366946606 <== mgr.4099 v2:172.21.15.114:6800/3521677055 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33951 (secure 0 0 0) 0x7ffa70125150 con 0x7ffa5c0383b0 2024-09-16T04:47:42.432 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:47:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-09-16T04:47:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.431+0000 7ffa76cb7640 1 -- 172.21.15.46:0/3366946606 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ffa5c0383b0 msgr2=0x7ffa5c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.431+0000 7ffa76cb7640 1 --2- 172.21.15.46:0/3366946606 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ffa5c0383b0 0x7ffa5c03a870 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ffa700a6d00 tx=0x7ffa64009190 comp rx=0 tx=0).stop 2024-09-16T04:47:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.431+0000 7ffa76cb7640 10 monclient: shutdown 2024-09-16T04:47:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.431+0000 7ffa76cb7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.431+0000 7ffa76cb7640 1 -- 172.21.15.46:0/3366946606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa700a3c10 msgr2=0x7ffa701cbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.431+0000 7ffa76cb7640 1 --2- 172.21.15.46:0/3366946606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa700a3c10 0x7ffa701cbee0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffa60000c00 tx=0x7ffa600046c0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.432+0000 7ffa76cb7640 1 -- 172.21.15.46:0/3366946606 shutdown_connections 2024-09-16T04:47:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.432+0000 7ffa76cb7640 1 --2- 172.21.15.46:0/3366946606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa701369d0 0x7ffa701231f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.432+0000 7ffa76cb7640 1 --2- 172.21.15.46:0/3366946606 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7ffa700a3ff0 0x7ffa701cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.432+0000 7ffa76cb7640 1 --2- 172.21.15.46:0/3366946606 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7ffa5c0383b0 0x7ffa5c03a870 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.432+0000 7ffa76cb7640 1 --2- 172.21.15.46:0/3366946606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa700a3c10 0x7ffa701cbee0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.432+0000 7ffa76cb7640 1 -- 172.21.15.46:0/3366946606 >> 172.21.15.46:0/3366946606 conn(0x7ffa700a5d10 msgr2=0x7ffa700a1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.432+0000 7ffa76cb7640 1 -- 172.21.15.46:0/3366946606 shutdown_connections 2024-09-16T04:47:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.432+0000 7ffa76cb7640 1 -- 172.21.15.46:0/3366946606 wait complete. 2024-09-16T04:47:42.444 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":576,"stamp":"2024-09-16T04:47:41.459924+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":217056,"kb_used_data":25768,"kb_used_omap":9,"kb_used_meta":191158,"kb_avail":566013984,"statfs":{"total":579820584960,"available":579598319616,"internally_reserved":0,"allocated":26386432,"data_stored":10465252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":195746464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002314"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":597,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.254605+0000","last_change":"2024-09-16T04:39:00.524013+0000","last_active":"2024-09-16T04:47:13.254605+0000","last_peered":"2024-09-16T04:47:13.254605+0000","last_clean":"2024-09-16T04:47:13.254605+0000","last_became_active":"2024-09-16T04:38:58.756778+0000","last_became_peered":"2024-09-16T04:38:58.756778+0000","last_unstale":"2024-09-16T04:47:13.254605+0000","last_undegraded":"2024-09-16T04:47:13.254605+0000","last_fullsized":"2024-09-16T04:47:13.254605+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:24:15.429766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054536700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":591,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:11.265078+0000","last_change":"2024-09-16T04:39:00.532294+0000","last_active":"2024-09-16T04:47:11.265078+0000","last_peered":"2024-09-16T04:47:11.265078+0000","last_clean":"2024-09-16T04:47:11.265078+0000","last_became_active":"2024-09-16T04:38:58.520473+0000","last_became_peered":"2024-09-16T04:38:58.520473+0000","last_unstale":"2024-09-16T04:47:11.265078+0000","last_undegraded":"2024-09-16T04:47:11.265078+0000","last_fullsized":"2024-09-16T04:47:11.265078+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:15:41.548834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053532500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":598,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.254532+0000","last_change":"2024-09-16T04:39:00.523783+0000","last_active":"2024-09-16T04:47:13.254532+0000","last_peered":"2024-09-16T04:47:13.254532+0000","last_clean":"2024-09-16T04:47:13.254532+0000","last_became_active":"2024-09-16T04:38:58.521000+0000","last_became_peered":"2024-09-16T04:38:58.521000+0000","last_unstale":"2024-09-16T04:47:13.254532+0000","last_undegraded":"2024-09-16T04:47:13.254532+0000","last_fullsized":"2024-09-16T04:47:13.254532+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:42.837022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035484999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":591,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:11.263646+0000","last_change":"2024-09-16T04:39:00.532400+0000","last_active":"2024-09-16T04:47:11.263646+0000","last_peered":"2024-09-16T04:47:11.263646+0000","last_clean":"2024-09-16T04:47:11.263646+0000","last_became_active":"2024-09-16T04:38:58.521627+0000","last_became_peered":"2024-09-16T04:38:58.521627+0000","last_unstale":"2024-09-16T04:47:11.263646+0000","last_undegraded":"2024-09-16T04:47:11.263646+0000","last_fullsized":"2024-09-16T04:47:11.263646+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:41.804897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062256500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":594,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:11.805224+0000","last_change":"2024-09-16T04:39:00.530578+0000","last_active":"2024-09-16T04:47:11.805224+0000","last_peered":"2024-09-16T04:47:11.805224+0000","last_clean":"2024-09-16T04:47:11.805224+0000","last_became_active":"2024-09-16T04:38:58.522744+0000","last_became_peered":"2024-09-16T04:38:58.522744+0000","last_unstale":"2024-09-16T04:47:11.805224+0000","last_undegraded":"2024-09-16T04:47:11.805224+0000","last_fullsized":"2024-09-16T04:47:11.805224+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:04:47.450053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000966258,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":601,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.255101+0000","last_change":"2024-09-16T04:39:00.523661+0000","last_active":"2024-09-16T04:47:13.255101+0000","last_peered":"2024-09-16T04:47:13.255101+0000","last_clean":"2024-09-16T04:47:13.255101+0000","last_became_active":"2024-09-16T04:38:58.524372+0000","last_became_peered":"2024-09-16T04:38:58.524372+0000","last_unstale":"2024-09-16T04:47:13.255101+0000","last_undegraded":"2024-09-16T04:47:13.255101+0000","last_fullsized":"2024-09-16T04:47:13.255101+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:46.534727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047533599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":598,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.254361+0000","last_change":"2024-09-16T04:39:00.523838+0000","last_active":"2024-09-16T04:47:13.254361+0000","last_peered":"2024-09-16T04:47:13.254361+0000","last_clean":"2024-09-16T04:47:13.254361+0000","last_became_active":"2024-09-16T04:38:58.522142+0000","last_became_peered":"2024-09-16T04:38:58.522142+0000","last_unstale":"2024-09-16T04:47:13.254361+0000","last_undegraded":"2024-09-16T04:47:13.254361+0000","last_fullsized":"2024-09-16T04:47:13.254361+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:35.411495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039704200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":593,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:11.805254+0000","last_change":"2024-09-16T04:39:00.530064+0000","last_active":"2024-09-16T04:47:11.805254+0000","last_peered":"2024-09-16T04:47:11.805254+0000","last_clean":"2024-09-16T04:47:11.805254+0000","last_became_active":"2024-09-16T04:38:58.523268+0000","last_became_peered":"2024-09-16T04:38:58.523268+0000","last_unstale":"2024-09-16T04:47:11.805254+0000","last_undegraded":"2024-09-16T04:47:11.805254+0000","last_fullsized":"2024-09-16T04:47:11.805254+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:12:33.958554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043992499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":749,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.254488+0000","last_change":"2024-09-16T04:38:52.501054+0000","last_active":"2024-09-16T04:47:13.254488+0000","last_peered":"2024-09-16T04:47:13.254488+0000","last_clean":"2024-09-16T04:47:13.254488+0000","last_became_active":"2024-09-16T04:38:52.500796+0000","last_became_peered":"2024-09-16T04:38:52.500796+0000","last_unstale":"2024-09-16T04:47:13.254488+0000","last_undegraded":"2024-09-16T04:47:13.254488+0000","last_fullsized":"2024-09-16T04:47:13.254488+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:51.492404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:51.492404+0000","last_clean_scrub_stamp":"2024-09-16T04:38:51.492404+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:36.122039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607661,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31148,"kb_used_data":4120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340692,"statfs":{"total":96636764160,"available":96604868608,"internally_reserved":0,"allocated":4218880,"data_stored":1567893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607661,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31668,"kb_used_data":4640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340172,"statfs":{"total":96636764160,"available":96604336128,"internally_reserved":0,"allocated":4751360,"data_stored":2096821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607661,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39988,"kb_used_data":4640,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94331852,"statfs":{"total":96636764160,"available":96595816448,"internally_reserved":0,"allocated":4751360,"data_stored":2096821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640366,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43632,"kb_used_data":4124,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94328208,"statfs":{"total":96636764160,"available":96592084992,"internally_reserved":0,"allocated":4222976,"data_stored":1567912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640367,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31148,"kb_used_data":4120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340692,"statfs":{"total":96636764160,"available":96604868608,"internally_reserved":0,"allocated":4218880,"data_stored":1567893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640365,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39472,"kb_used_data":4124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332368,"statfs":{"total":96636764160,"available":96596344832,"internally_reserved":0,"allocated":4222976,"data_stored":1567912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:42.445 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:47:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.543+0000 7f7c4b79a640 10 monclient: get_monmap_and_config 2024-09-16T04:47:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.543+0000 7f7c4b79a640 10 monclient: build_initial_monmap 2024-09-16T04:47:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.544+0000 7f7c4b79a640 10 monclient: monmap: 2024-09-16T04:47:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:42.545450+0000 2024-09-16T04:47:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:42.545450+0000 2024-09-16T04:47:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:42.547 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:42.547 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:42.547 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:42.547 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:42.547 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:42.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.544+0000 7f7c4b79a640 1 Processor -- start 2024-09-16T04:47:42.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.544+0000 7f7c4b79a640 1 -- start start 2024-09-16T04:47:42.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.544+0000 7f7c4b79a640 10 monclient: init 2024-09-16T04:47:42.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c4b79a640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:42.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c4b79a640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-16T04:47:42.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c4b79a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7c440a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:47:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c4b79a640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7c440a3ff0 0x7f7c440a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c4b79a640 10 monclient(hunting): picked mon.noname-b con 0x7f7c440a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c4b79a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7c4413be90 0x7f7c4413e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c4b79a640 10 monclient(hunting): picked mon.noname-a con 0x7f7c4413be90 addr v2:172.21.15.46:3300/0 2024-09-16T04:47:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c4b79a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c4409b940 con 0x7f7c440a3c10 2024-09-16T04:47:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c4b79a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c4b79a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f7c4409ab00 con 0x7f7c4413be90 2024-09-16T04:47:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c4b79a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c4b79a640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f7c4409b030 con 0x7f7c440a3ff0 2024-09-16T04:47:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c4b79a640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c4b79a640 10 monclient(hunting): authenticate will time out at 1260.961390s 2024-09-16T04:47:42.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c48d35640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7c440a3ff0 0x7f7c440a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:42.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c49d37640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7c4413be90 0x7f7c4413e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:42.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c48d35640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7c440a3ff0 0x7f7c440a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38810/0 (socket says 172.21.15.46:38810) 2024-09-16T04:47:42.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c48d35640 1 -- 172.21.15.46:0/990245675 learned_addr learned my addr 172.21.15.46:0/990245675 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:42.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c48d35640 10 monclient(hunting): get_auth_request con 0x7f7c440a3ff0 auth_method 0 2024-09-16T04:47:42.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c49d37640 1 --2- 172.21.15.46:0/990245675 >> v2:172.21.15.46:3300/0 conn(0x7f7c4413be90 0x7f7c4413e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41880/0 (socket says 172.21.15.46:41880) 2024-09-16T04:47:42.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c48d35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:42.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c48d35640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:42.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c48d35640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:42.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c49d37640 10 monclient(hunting): get_auth_request con 0x7f7c4413be90 auth_method 0 2024-09-16T04:47:42.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c49d37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:42.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c49d37640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:42.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c49d37640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:42.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c48d35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:42.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c48d35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:42.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c48d35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:42.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c49d37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:42.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c49d37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:42.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c49d37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:42.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c48d35640 10 monclient(hunting): handle_auth_done global_id 94466 payload 274 2024-09-16T04:47:42.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c48d35640 1 -- 172.21.15.46:0/990245675 >> v2:172.21.15.46:3300/0 conn(0x7f7c4413be90 msgr2=0x7f7c4413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:42.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c48d35640 1 --2- 172.21.15.46:0/990245675 >> v2:172.21.15.46:3300/0 conn(0x7f7c4413be90 0x7f7c4413e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c48d35640 1 -- 172.21.15.46:0/990245675 >> v1:172.21.15.114:6789/0 conn(0x7f7c440a3c10 legacy=0x7f7c4409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:42.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c48d35640 10 monclient: _finish_hunting 0 2024-09-16T04:47:42.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c48d35640 1 monclient: found mon.noname-b 2024-09-16T04:47:42.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c48d35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:42.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c48d35640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:47:42.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.545+0000 7f7c48d35640 1 -- 172.21.15.46:0/990245675 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4409b310 con 0x7f7c440a3ff0 2024-09-16T04:47:42.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.546+0000 7f7c48d35640 1 --2- 172.21.15.46:0/990245675 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7c440a3ff0 0x7f7c440a43d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7c38009870 tx=0x7f7c3802a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea241952a7684ba7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:42.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.546+0000 7f7c37fff640 1 -- 172.21.15.46:0/990245675 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c38037070 con 0x7f7c440a3ff0 2024-09-16T04:47:42.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.546+0000 7f7c37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:42.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.546+0000 7f7c37fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:47:42.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.546+0000 7f7c37fff640 10 monclient: dump: 2024-09-16T04:47:42.560 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:42.560 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:42.560 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:42.560 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:42.560 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:42.560 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:42.561 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:42.561 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:42.561 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:42.561 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:42.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.546+0000 7f7c37fff640 10 monclient: _finish_auth 0 2024-09-16T04:47:42.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.546+0000 7f7c37fff640 10 monclient: _check_auth_tickets 2024-09-16T04:47:42.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.546+0000 7f7c37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:42.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.546+0000 7f7c4b79a640 5 monclient: authenticate success, global_id 94466 2024-09-16T04:47:42.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.546+0000 7f7c37fff640 1 -- 172.21.15.46:0/990245675 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c38032440 con 0x7f7c440a3ff0 2024-09-16T04:47:42.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.546+0000 7f7c37fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:42.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.546+0000 7f7c4b79a640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:42.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.546+0000 7f7c37fff640 1 -- 172.21.15.46:0/990245675 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c38032810 con 0x7f7c440a3ff0 2024-09-16T04:47:42.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.546+0000 7f7c4b79a640 4 set_mon_vals no callback set 2024-09-16T04:47:42.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.546+0000 7f7c37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:42.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.546+0000 7f7c37fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:42.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.546+0000 7f7c37fff640 10 monclient: dump: 2024-09-16T04:47:42.562 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:42.562 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:42.562 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:42.562 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:42.562 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:42.562 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:42.562 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:42.562 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:42.562 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:42.563 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:42.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.547+0000 7f7c4b79a640 10 monclient: shutdown 2024-09-16T04:47:42.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.547+0000 7f7c4b79a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:42.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.547+0000 7f7c4b79a640 1 -- 172.21.15.46:0/990245675 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7c440a3ff0 msgr2=0x7f7c440a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:42.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.547+0000 7f7c4b79a640 1 --2- 172.21.15.46:0/990245675 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7c440a3ff0 0x7f7c440a43d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7c38009870 tx=0x7f7c3802a1c0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.547+0000 7f7c4b79a640 1 -- 172.21.15.46:0/990245675 shutdown_connections 2024-09-16T04:47:42.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.547+0000 7f7c4b79a640 1 --2- 172.21.15.46:0/990245675 >> v2:172.21.15.46:3300/0 conn(0x7f7c4413be90 0x7f7c4413e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.547+0000 7f7c4b79a640 1 --2- 172.21.15.46:0/990245675 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7c440a3ff0 0x7f7c440a43d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.547+0000 7f7c4b79a640 1 -- 172.21.15.46:0/990245675 >> 172.21.15.46:0/990245675 conn(0x7f7c440a6340 msgr2=0x7f7c4406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:42.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.547+0000 7f7c4b79a640 1 -- 172.21.15.46:0/990245675 shutdown_connections 2024-09-16T04:47:42.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.547+0000 7f7c4b79a640 1 -- 172.21.15.46:0/990245675 wait complete. 2024-09-16T04:47:42.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.547+0000 7f7c4b79a640 10 monclient: build_initial_monmap 2024-09-16T04:47:42.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.547+0000 7f7c4b79a640 10 monclient: monmap: 2024-09-16T04:47:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:42.564 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c4b79a640 1 Processor -- start 2024-09-16T04:47:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c4b79a640 1 -- start start 2024-09-16T04:47:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c4b79a640 10 monclient: init 2024-09-16T04:47:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c4b79a640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c4b79a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:47:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c4b79a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c440a3c10 0x7f7c44133d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c4b79a640 10 monclient(hunting): picked mon.noname-a con 0x7f7c440a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c4b79a640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7c440a3ff0 0x7f7c441342a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c4b79a640 10 monclient(hunting): picked mon.noname-c con 0x7f7c440a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c4b79a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c4413be90 0x7f7c44137e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c4b79a640 10 monclient(hunting): picked mon.noname-b con 0x7f7c4413be90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c4b79a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c4b79a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7c440a6b40 con 0x7f7c440a3c10 2024-09-16T04:47:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c4b79a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c4b79a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7c440a69c0 con 0x7f7c4413be90 2024-09-16T04:47:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c4b79a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c4b79a640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f7c4413e7c0 con 0x7f7c440a3ff0 2024-09-16T04:47:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c4b79a640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c4b79a640 10 monclient(hunting): authenticate will time out at 1260.964814s 2024-09-16T04:47:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c49536640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c440a3c10 0x7f7c44133d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c48d35640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7c440a3ff0 0x7f7c441342a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c49536640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c440a3c10 0x7f7c44133d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41886/0 (socket says 172.21.15.46:41886) 2024-09-16T04:47:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c49536640 1 -- 172.21.15.46:0/1191870261 learned_addr learned my addr 172.21.15.46:0/1191870261 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c49536640 10 monclient(hunting): get_auth_request con 0x7f7c440a3c10 auth_method 0 2024-09-16T04:47:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c49536640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c49536640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c49536640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c48d35640 10 monclient(hunting): get_auth_request con 0x7f7c440a3ff0 auth_method 0 2024-09-16T04:47:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c48d35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c48d35640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c48d35640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c49536640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.548+0000 7f7c49536640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:42.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c49d37640 1 --2- 172.21.15.46:0/1191870261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c4413be90 0x7f7c44137e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:42.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c49536640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:42.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c49d37640 10 monclient(hunting): get_auth_request con 0x7f7c4413be90 auth_method 0 2024-09-16T04:47:42.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c49d37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:42.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c49d37640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:42.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c49d37640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:42.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c48d35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:42.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c48d35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:42.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c48d35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:42.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c49d37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:42.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c49d37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:42.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c49d37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:42.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c49536640 10 monclient(hunting): handle_auth_done global_id 104319 payload 824 2024-09-16T04:47:42.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c49536640 1 -- 172.21.15.46:0/1191870261 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7c440a3ff0 msgr2=0x7f7c441342a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:42.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c49536640 1 --2- 172.21.15.46:0/1191870261 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7c440a3ff0 0x7f7c441342a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c49536640 1 -- 172.21.15.46:0/1191870261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c4413be90 msgr2=0x7f7c44137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:42.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c49536640 1 --2- 172.21.15.46:0/1191870261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c4413be90 0x7f7c44137e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c49536640 10 monclient: _finish_hunting 0 2024-09-16T04:47:42.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c49536640 1 monclient: found mon.noname-a 2024-09-16T04:47:42.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c49536640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:42.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c49536640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:42.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c49536640 1 -- 172.21.15.46:0/1191870261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c441347e0 con 0x7f7c440a3c10 2024-09-16T04:47:42.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c48d35640 1 --2- 172.21.15.46:0/1191870261 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7c440a3ff0 0x7f7c441342a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:42.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c49536640 1 --2- 172.21.15.46:0/1191870261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c440a3c10 0x7f7c44133d60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000c430 tx=0x7f7c4000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:42.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c35ffb640 1 -- 172.21.15.46:0/1191870261 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40017070 con 0x7f7c440a3c10 2024-09-16T04:47:42.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:42.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c35ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:42.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c35ffb640 10 monclient: dump: 2024-09-16T04:47:42.569 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:42.569 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:42.569 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:42.569 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:42.569 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:42.569 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:42.569 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:42.569 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:42.570 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:42.570 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:42.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c35ffb640 10 monclient: _finish_auth 0 2024-09-16T04:47:42.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c35ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:47:42.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:42.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c35ffb640 1 -- 172.21.15.46:0/1191870261 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c400052d0 con 0x7f7c440a3c10 2024-09-16T04:47:42.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c4b79a640 5 monclient: authenticate success, global_id 104319 2024-09-16T04:47:42.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c4b79a640 10 monclient: _renew_subs 2024-09-16T04:47:42.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c4b79a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:42.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c4b79a640 1 -- 172.21.15.46:0/1191870261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c4409fa90 con 0x7f7c440a3c10 2024-09-16T04:47:42.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c35ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:42.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c35ffb640 1 -- 172.21.15.46:0/1191870261 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40003830 con 0x7f7c440a3c10 2024-09-16T04:47:42.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:42.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c35ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:42.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c35ffb640 10 monclient: dump: 2024-09-16T04:47:42.571 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:42.571 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:42.571 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:42.571 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:42.571 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:42.571 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:42.571 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:42.571 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:42.571 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:42.572 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:42.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c4b79a640 10 monclient: _renew_subs 2024-09-16T04:47:42.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c377fe640 4 set_mon_vals no callback set 2024-09-16T04:47:42.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c4b79a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:42.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.549+0000 7f7c4b79a640 1 -- 172.21.15.46:0/1191870261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c441391c0 con 0x7f7c440a3c10 2024-09-16T04:47:42.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.550+0000 7f7c35ffb640 1 -- 172.21.15.46:0/1191870261 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f7c4001a070 con 0x7f7c440a3c10 2024-09-16T04:47:42.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.550+0000 7f7c4b79a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:42.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.550+0000 7f7c4b79a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:42.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.550+0000 7f7c4b79a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:42.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.550+0000 7f7c4b79a640 1 -- 172.21.15.46:0/1191870261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c440a43d0 con 0x7f7c440a3c10 2024-09-16T04:47:42.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.550+0000 7f7c35ffb640 1 --2- 172.21.15.46:0/1191870261 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7c280383b0 0x7f7c2803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:42.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.550+0000 7f7c35ffb640 1 -- 172.21.15.46:0/1191870261 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7f7c4004aa10 con 0x7f7c440a3c10 2024-09-16T04:47:42.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.551+0000 7f7c48d35640 1 --2- 172.21.15.46:0/1191870261 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7c280383b0 0x7f7c2803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:42.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.551+0000 7f7c48d35640 10 monclient: get_auth_request con 0x7f7c280383b0 auth_method 0 2024-09-16T04:47:42.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.551+0000 7f7c48d35640 1 --2- 172.21.15.46:0/1191870261 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7c280383b0 0x7f7c2803a870 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7c38034000 tx=0x7f7c3802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:42.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.555+0000 7f7c35ffb640 1 -- 172.21.15.46:0/1191870261 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7c4001ac60 con 0x7f7c440a3c10 2024-09-16T04:47:42.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.555+0000 7f7c35ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:42.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.555+0000 7f7c35ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:42.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.735+0000 7f7c4b79a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:47:42.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.735+0000 7f7c4b79a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:47:42.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.735+0000 7f7c4b79a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:42.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.735+0000 7f7c4b79a640 1 -- 172.21.15.46:0/1191870261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c440a1580 con 0x7f7c440a3c10 2024-09-16T04:47:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.736+0000 7f7c35ffb640 1 -- 172.21.15.46:0/1191870261 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v315) ==== 74+0+12302 (secure 0 0 0) 0x7f7c4001a9a0 con 0x7f7c440a3c10 2024-09-16T04:47:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.736+0000 7f7c35ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:47:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.736+0000 7f7c35ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:47:42.738 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:47:42.738 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":315,"fsid":"01e423f6-9e74-4519-981b-340ad8faba4f","created":"2024-09-16T04:38:39.286708+0000","modified":"2024-09-16T04:46:56.948385+0000","last_up_change":"2024-09-16T04:38:50.487347+0000","last_in_change":"2024-09-16T04:38:42.117316+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":100,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:38:51.392177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:38:57.324878+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ff88780-0630-4bb1-9f5a-5331d7724d17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3454452571},{"type":"v1","addr":"172.21.15.46:6809","nonce":3454452571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3454452571},{"type":"v1","addr":"172.21.15.46:6811","nonce":3454452571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3454452571},{"type":"v1","addr":"172.21.15.46:6815","nonce":3454452571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3454452571},{"type":"v1","addr":"172.21.15.46:6813","nonce":3454452571}]},"public_addr":"172.21.15.46:6809/3454452571","cluster_addr":"172.21.15.46:6811/3454452571","heartbeat_back_addr":"172.21.15.46:6815/3454452571","heartbeat_front_addr":"172.21.15.46:6813/3454452571","state":["exists","up"]},{"osd":1,"uuid":"37be2e88-32d0-4a30-8979-899b34f3be17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6817","nonce":4217009477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6819","nonce":4217009477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6823","nonce":4217009477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":4217009477},{"type":"v1","addr":"172.21.15.46:6821","nonce":4217009477}]},"public_addr":"172.21.15.46:6817/4217009477","cluster_addr":"172.21.15.46:6819/4217009477","heartbeat_back_addr":"172.21.15.46:6823/4217009477","heartbeat_front_addr":"172.21.15.46:6821/4217009477","state":["exists","up"]},{"osd":2,"uuid":"73b9a838-944e-48cd-afab-5260608d1c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6801","nonce":3101080311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6803","nonce":3101080311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6807","nonce":3101080311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3101080311},{"type":"v1","addr":"172.21.15.46:6805","nonce":3101080311}]},"public_addr":"172.21.15.46:6801/3101080311","cluster_addr":"172.21.15.46:6803/3101080311","heartbeat_back_addr":"172.21.15.46:6807/3101080311","heartbeat_front_addr":"172.21.15.46:6805/3101080311","state":["exists","up"]},{"osd":3,"uuid":"d0c22455-5685-4ade-b540-3d5428a7cb46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":890769734},{"type":"v1","addr":"172.21.15.114:6811","nonce":890769734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":890769734},{"type":"v1","addr":"172.21.15.114:6813","nonce":890769734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":890769734},{"type":"v1","addr":"172.21.15.114:6817","nonce":890769734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":890769734},{"type":"v1","addr":"172.21.15.114:6815","nonce":890769734}]},"public_addr":"172.21.15.114:6811/890769734","cluster_addr":"172.21.15.114:6813/890769734","heartbeat_back_addr":"172.21.15.114:6817/890769734","heartbeat_front_addr":"172.21.15.114:6815/890769734","state":["exists","up"]},{"osd":4,"uuid":"f37724e4-2796-43c8-975b-4458986c4ea6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":936316557},{"type":"v1","addr":"172.21.15.114:6803","nonce":936316557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":936316557},{"type":"v1","addr":"172.21.15.114:6805","nonce":936316557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":936316557},{"type":"v1","addr":"172.21.15.114:6809","nonce":936316557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":936316557},{"type":"v1","addr":"172.21.15.114:6807","nonce":936316557}]},"public_addr":"172.21.15.114:6803/936316557","cluster_addr":"172.21.15.114:6805/936316557","heartbeat_back_addr":"172.21.15.114:6809/936316557","heartbeat_front_addr":"172.21.15.114:6807/936316557","state":["exists","up"]},{"osd":5,"uuid":"979362db-e173-47a0-93d1-187742938a85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6819","nonce":938868100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6821","nonce":938868100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6825","nonce":938868100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":938868100},{"type":"v1","addr":"172.21.15.114:6823","nonce":938868100}]},"public_addr":"172.21.15.114:6819/938868100","cluster_addr":"172.21.15.114:6821/938868100","heartbeat_back_addr":"172.21.15.114:6825/938868100","heartbeat_front_addr":"172.21.15.114:6823/938868100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.647737+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.394775+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.463096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.655270+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.652725+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:38:48.431803+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:47:42.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.738+0000 7f7c4b79a640 1 -- 172.21.15.46:0/1191870261 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7c280383b0 msgr2=0x7f7c2803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.738+0000 7f7c4b79a640 1 --2- 172.21.15.46:0/1191870261 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7c280383b0 0x7f7c2803a870 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7c38034000 tx=0x7f7c3802e000 comp rx=0 tx=0).stop 2024-09-16T04:47:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.738+0000 7f7c4b79a640 10 monclient: shutdown 2024-09-16T04:47:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.738+0000 7f7c4b79a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.738+0000 7f7c4b79a640 1 -- 172.21.15.46:0/1191870261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c440a3c10 msgr2=0x7f7c44133d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.738+0000 7f7c4b79a640 1 --2- 172.21.15.46:0/1191870261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c440a3c10 0x7f7c44133d60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000c430 tx=0x7f7c4000c8f0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.738+0000 7f7c4b79a640 1 -- 172.21.15.46:0/1191870261 shutdown_connections 2024-09-16T04:47:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.738+0000 7f7c4b79a640 1 --2- 172.21.15.46:0/1191870261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c4413be90 0x7f7c44137e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.738+0000 7f7c4b79a640 1 --2- 172.21.15.46:0/1191870261 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7c440a3ff0 0x7f7c441342a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.738+0000 7f7c4b79a640 1 --2- 172.21.15.46:0/1191870261 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7c280383b0 0x7f7c2803a870 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.738+0000 7f7c4b79a640 1 --2- 172.21.15.46:0/1191870261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c440a3c10 0x7f7c44133d60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.738+0000 7f7c4b79a640 1 -- 172.21.15.46:0/1191870261 >> 172.21.15.46:0/1191870261 conn(0x7f7c440a6340 msgr2=0x7f7c4413dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.738+0000 7f7c4b79a640 1 -- 172.21.15.46:0/1191870261 shutdown_connections 2024-09-16T04:47:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:42.738+0000 7f7c4b79a640 1 -- 172.21.15.46:0/1191870261 wait complete. 2024-09-16T04:47:43.752 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T04:47:43.752 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:47:43.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.869+0000 7f7aac58d640 10 monclient: get_monmap_and_config 2024-09-16T04:47:43.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.869+0000 7f7aac58d640 10 monclient: build_initial_monmap 2024-09-16T04:47:43.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.869+0000 7f7aac58d640 10 monclient: monmap: 2024-09-16T04:47:43.871 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:43.872 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:43.872 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:43.871081+0000 2024-09-16T04:47:43.872 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:43.871081+0000 2024-09-16T04:47:43.872 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:43.872 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:43.872 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:43.872 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:43.872 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:43.872 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:43.872 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:43.872 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:43.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.870+0000 7f7aac58d640 1 Processor -- start 2024-09-16T04:47:43.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.870+0000 7f7aac58d640 1 -- start start 2024-09-16T04:47:43.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.870+0000 7f7aac58d640 10 monclient: init 2024-09-16T04:47:43.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.870+0000 7f7aac58d640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:43.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.870+0000 7f7aac58d640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-16T04:47:43.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.870+0000 7f7aac58d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7aa40a3c10 0x7f7aa409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:43.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.870+0000 7f7aac58d640 10 monclient(hunting): picked mon.noname-a con 0x7f7aa40a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:47:43.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.870+0000 7f7aac58d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7aa40a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:47:43.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.870+0000 7f7aac58d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7aa413be90 addr v1:172.21.15.114:6789/0 2024-09-16T04:47:43.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.870+0000 7f7aac58d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7aa409b940 con 0x7f7aa40a3ff0 2024-09-16T04:47:43.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aac58d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7aa409ab00 con 0x7f7aa413be90 2024-09-16T04:47:43.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aac58d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:43.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aac58d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f7aa409b030 con 0x7f7aa40a3c10 2024-09-16T04:47:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aac58d640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aac58d640 10 monclient(hunting): authenticate will time out at 1262.287038s 2024-09-16T04:47:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aaa329640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7aa40a3c10 0x7f7aa409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aa9b28640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7aa40a3ff0 0x7f7aa40a43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33320/0 (socket says 172.21.15.46:33320) 2024-09-16T04:47:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aa9b28640 1 -- 172.21.15.46:0/2660993865 learned_addr learned my addr 172.21.15.46:0/2660993865 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:47:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aaa329640 10 monclient(hunting): get_auth_request con 0x7f7aa40a3c10 auth_method 0 2024-09-16T04:47:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aaa329640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aaa329640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aaa329640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aaa329640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aaa329640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aaa329640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aaa329640 10 monclient(hunting): handle_auth_done global_id 104322 payload 274 2024-09-16T04:47:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aaa329640 1 -- 172.21.15.46:0/2660993865 >> v1:172.21.15.114:6789/0 conn(0x7f7aa413be90 legacy=0x7f7aa413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aaa329640 1 -- 172.21.15.46:0/2660993865 >> v1:172.21.15.46:6789/0 conn(0x7f7aa40a3ff0 legacy=0x7f7aa40a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aaa329640 10 monclient: _finish_hunting 0 2024-09-16T04:47:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aaa329640 1 monclient: found mon.noname-a 2024-09-16T04:47:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aaa329640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aaa329640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aaa329640 1 -- 172.21.15.46:0/2660993865 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aa409b310 con 0x7f7aa40a3c10 2024-09-16T04:47:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aa9327640 1 -- 172.21.15.46:0/2660993865 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 430679187 0 0) 0x7f7aa409b940 con 0x7f7aa40a3ff0 2024-09-16T04:47:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.871+0000 7f7aa9327640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:47:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.872+0000 7f7aaa329640 1 --2- 172.21.15.46:0/2660993865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7aa40a3c10 0x7f7aa409bcf0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7aa00097a0 tx=0x7f7aa002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5b38359621d8bf1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.872+0000 7f7aa9327640 1 -- 172.21.15.46:0/2660993865 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0037070 con 0x7f7aa40a3c10 2024-09-16T04:47:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.872+0000 7f7aa9327640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.872+0000 7f7aa9327640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.872+0000 7f7aa9327640 10 monclient: dump: 2024-09-16T04:47:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:43.885 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:43.885 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:43.885 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:43.885 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:43.885 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:43.885 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:43.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.872+0000 7f7aa9327640 10 monclient: _finish_auth 0 2024-09-16T04:47:43.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.872+0000 7f7aa9327640 10 monclient: _check_auth_tickets 2024-09-16T04:47:43.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.872+0000 7f7aa9327640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:43.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.872+0000 7f7aac58d640 5 monclient: authenticate success, global_id 104322 2024-09-16T04:47:43.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.872+0000 7f7aa9327640 1 -- 172.21.15.46:0/2660993865 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7aa0032440 con 0x7f7aa40a3c10 2024-09-16T04:47:43.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.872+0000 7f7aa9327640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:43.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.872+0000 7f7aac58d640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:43.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.872+0000 7f7aac58d640 4 set_mon_vals no callback set 2024-09-16T04:47:43.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.872+0000 7f7aa9327640 1 -- 172.21.15.46:0/2660993865 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa002ae60 con 0x7f7aa40a3c10 2024-09-16T04:47:43.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.872+0000 7f7aa9327640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:43.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.872+0000 7f7aa9327640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:43.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.872+0000 7f7aa9327640 10 monclient: dump: 2024-09-16T04:47:43.886 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:43.886 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:43.886 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:43.887 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.872+0000 7f7aac58d640 10 monclient: shutdown 2024-09-16T04:47:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.872+0000 7f7aac58d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.872+0000 7f7aac58d640 1 -- 172.21.15.46:0/2660993865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7aa40a3c10 msgr2=0x7f7aa409bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.872+0000 7f7aac58d640 1 --2- 172.21.15.46:0/2660993865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7aa40a3c10 0x7f7aa409bcf0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7aa00097a0 tx=0x7f7aa002a110 comp rx=0 tx=0).stop 2024-09-16T04:47:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.873+0000 7f7aac58d640 1 -- 172.21.15.46:0/2660993865 shutdown_connections 2024-09-16T04:47:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.873+0000 7f7aac58d640 1 --2- 172.21.15.46:0/2660993865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7aa40a3c10 0x7f7aa409bcf0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.873+0000 7f7aac58d640 1 -- 172.21.15.46:0/2660993865 >> 172.21.15.46:0/2660993865 conn(0x7f7aa40a6340 msgr2=0x7f7aa406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.873+0000 7f7aac58d640 1 -- 172.21.15.46:0/2660993865 shutdown_connections 2024-09-16T04:47:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.873+0000 7f7aac58d640 1 -- 172.21.15.46:0/2660993865 wait complete. 2024-09-16T04:47:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.873+0000 7f7aac58d640 10 monclient: build_initial_monmap 2024-09-16T04:47:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.873+0000 7f7aac58d640 10 monclient: monmap: 2024-09-16T04:47:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:43.889 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aac58d640 1 Processor -- start 2024-09-16T04:47:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aac58d640 1 -- start start 2024-09-16T04:47:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aac58d640 10 monclient: init 2024-09-16T04:47:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aac58d640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aac58d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:47:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aac58d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7aa40a3c10 0x7f7aa41cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aac58d640 10 monclient(hunting): picked mon.noname-c con 0x7f7aa40a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aac58d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7aa40a3ff0 0x7f7aa4136e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aac58d640 10 monclient(hunting): picked mon.noname-b con 0x7f7aa40a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aac58d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7aa413be90 0x7f7aa413c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aac58d640 10 monclient(hunting): picked mon.noname-a con 0x7f7aa413be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aac58d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aac58d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa40a6b40 con 0x7f7aa413be90 2024-09-16T04:47:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aac58d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aac58d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa40a69c0 con 0x7f7aa40a3ff0 2024-09-16T04:47:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aac58d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aac58d640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f7aa413f9a0 con 0x7f7aa40a3c10 2024-09-16T04:47:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aac58d640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aac58d640 10 monclient(hunting): authenticate will time out at 1262.290622s 2024-09-16T04:47:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aaa329640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7aa40a3c10 0x7f7aa41cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aaab2a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7aa413be90 0x7f7aa413c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aaa329640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7aa40a3c10 0x7f7aa41cbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38828/0 (socket says 172.21.15.46:38828) 2024-09-16T04:47:43.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aaa329640 1 -- 172.21.15.46:0/4266820784 learned_addr learned my addr 172.21.15.46:0/4266820784 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:43.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aaa329640 10 monclient(hunting): get_auth_request con 0x7f7aa40a3c10 auth_method 0 2024-09-16T04:47:43.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aaa329640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:43.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aaa329640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:43.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aaa329640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:43.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aaab2a640 10 monclient(hunting): get_auth_request con 0x7f7aa413be90 auth_method 0 2024-09-16T04:47:43.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aaab2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:43.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aaab2a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:43.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aaab2a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:43.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aaa329640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:43.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aaa329640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:43.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aa9b28640 1 --2- 172.21.15.46:0/4266820784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7aa40a3ff0 0x7f7aa4136e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:43.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aaa329640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:43.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aaab2a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:43.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aaab2a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:43.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aaab2a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:43.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aa9b28640 10 monclient(hunting): get_auth_request con 0x7f7aa40a3ff0 auth_method 0 2024-09-16T04:47:43.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aa9b28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:43.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aa9b28640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:43.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aa9b28640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:43.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aa9b28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:43.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aa9b28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:43.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.874+0000 7f7aa9b28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:43.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7aaa329640 10 monclient(hunting): handle_auth_done global_id 94472 payload 824 2024-09-16T04:47:43.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7aaa329640 1 -- 172.21.15.46:0/4266820784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7aa40a3ff0 msgr2=0x7f7aa4136e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:43.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7aaa329640 1 --2- 172.21.15.46:0/4266820784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7aa40a3ff0 0x7f7aa4136e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:43.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7aaa329640 1 -- 172.21.15.46:0/4266820784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7aa413be90 msgr2=0x7f7aa413c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:43.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7aaa329640 1 --2- 172.21.15.46:0/4266820784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7aa413be90 0x7f7aa413c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:43.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7aaa329640 10 monclient: _finish_hunting 0 2024-09-16T04:47:43.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7aaa329640 1 monclient: found mon.noname-c 2024-09-16T04:47:43.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7aaa329640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:43.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7aaa329640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:47:43.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7aaa329640 1 -- 172.21.15.46:0/4266820784 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aa4120510 con 0x7f7aa40a3c10 2024-09-16T04:47:43.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7aaa329640 1 --2- 172.21.15.46:0/4266820784 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7aa40a3c10 0x7f7aa41cbba0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0000c00 tx=0x7f7aa00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:43.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7a9affd640 1 -- 172.21.15.46:0/4266820784 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0037040 con 0x7f7aa40a3c10 2024-09-16T04:47:43.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7a9affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:43.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7a9affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:47:43.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7a9affd640 10 monclient: dump: 2024-09-16T04:47:43.893 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:43.893 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:43.894 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7a9affd640 10 monclient: _finish_auth 0 2024-09-16T04:47:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7a9affd640 10 monclient: _check_auth_tickets 2024-09-16T04:47:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7a9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7a9affd640 1 -- 172.21.15.46:0/4266820784 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7aa0042070 con 0x7f7aa40a3c10 2024-09-16T04:47:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7aac58d640 5 monclient: authenticate success, global_id 94472 2024-09-16T04:47:43.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7a9affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:43.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7a9affd640 1 -- 172.21.15.46:0/4266820784 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0030a70 con 0x7f7aa40a3c10 2024-09-16T04:47:43.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7aac58d640 10 monclient: _renew_subs 2024-09-16T04:47:43.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7aac58d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:43.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7aac58d640 1 -- 172.21.15.46:0/4266820784 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aa40a09a0 con 0x7f7aa40a3c10 2024-09-16T04:47:43.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7a9affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:43.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7a9affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:43.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7a9affd640 10 monclient: dump: 2024-09-16T04:47:43.895 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:43.895 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:43.895 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:43.895 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:43.895 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:43.895 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:43.896 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:43.896 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:43.896 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:43.896 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:43.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7aa8b26640 4 set_mon_vals no callback set 2024-09-16T04:47:43.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7aac58d640 10 monclient: _renew_subs 2024-09-16T04:47:43.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7aac58d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:43.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.875+0000 7f7aac58d640 1 -- 172.21.15.46:0/4266820784 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aa413f6a0 con 0x7f7aa40a3c10 2024-09-16T04:47:43.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.876+0000 7f7aac58d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:43.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.876+0000 7f7aac58d640 1 -- 172.21.15.46:0/4266820784 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7aa40a5730 con 0x7f7aa40a3c10 2024-09-16T04:47:43.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.876+0000 7f7a9affd640 1 -- 172.21.15.46:0/4266820784 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f7aa0030c10 con 0x7f7aa40a3c10 2024-09-16T04:47:43.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.876+0000 7f7a9affd640 1 --2- 172.21.15.46:0/4266820784 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7a880383b0 0x7f7a8803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:43.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.876+0000 7f7a9affd640 1 -- 172.21.15.46:0/4266820784 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7f7aa006bf60 con 0x7f7aa40a3c10 2024-09-16T04:47:43.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.876+0000 7f7a9affd640 1 --2- 172.21.15.46:0/4266820784 >> [v2:172.21.15.46:6808/3454452571,v1:172.21.15.46:6809/3454452571] conn(0x7f7a8803e220 0x7f7a88040640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:43.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.876+0000 7f7a9affd640 1 -- 172.21.15.46:0/4266820784 --> [v2:172.21.15.46:6808/3454452571,v1:172.21.15.46:6809/3454452571] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7aa002be50 con 0x7f7a8803e220 2024-09-16T04:47:43.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.876+0000 7f7a9affd640 1 -- 172.21.15.46:0/4266820784 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_get_version_reply(handle=1 version=315) ==== 24+0+0 (secure 0 0 0) 0x7f7aa006e630 con 0x7f7aa40a3c10 2024-09-16T04:47:43.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.876+0000 7f7a9affd640 10 monclient: handle_get_version_reply finishing 1 version 315 2024-09-16T04:47:43.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.877+0000 7f7aaab2a640 1 --2- 172.21.15.46:0/4266820784 >> [v2:172.21.15.46:6808/3454452571,v1:172.21.15.46:6809/3454452571] conn(0x7f7a8803e220 0x7f7a88040640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:43.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.877+0000 7f7aaab2a640 10 monclient: get_auth_request con 0x7f7a8803e220 auth_method 0 2024-09-16T04:47:43.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.877+0000 7f7aa9b28640 1 --2- 172.21.15.46:0/4266820784 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7a880383b0 0x7f7a8803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:43.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.877+0000 7f7aa9b28640 10 monclient: get_auth_request con 0x7f7a880383b0 auth_method 0 2024-09-16T04:47:43.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.877+0000 7f7aaab2a640 1 --2- 172.21.15.46:0/4266820784 >> [v2:172.21.15.46:6808/3454452571,v1:172.21.15.46:6809/3454452571] conn(0x7f7a8803e220 0x7f7a88040640 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:43.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.877+0000 7f7aa9b28640 1 --2- 172.21.15.46:0/4266820784 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7a880383b0 0x7f7a8803a870 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4120720 tx=0x7f7a94007e40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:43.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.878+0000 7f7a9affd640 1 -- 172.21.15.46:0/4266820784 <== osd.0 v2:172.21.15.46:6808/3454452571 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7aa002be50 con 0x7f7a8803e220 2024-09-16T04:47:43.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.890+0000 7f7aac58d640 1 -- 172.21.15.46:0/4266820784 --> [v2:172.21.15.46:6808/3454452571,v1:172.21.15.46:6809/3454452571] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7aa409bd70 con 0x7f7a8803e220 2024-09-16T04:47:43.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.891+0000 7f7a9affd640 1 -- 172.21.15.46:0/4266820784 <== osd.0 v2:172.21.15.46:6808/3454452571 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7aa409bd70 con 0x7f7a8803e220 2024-09-16T04:47:43.899 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:47:43.899 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:47:43.899 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:47:43.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.891+0000 7f7aac58d640 1 -- 172.21.15.46:0/4266820784 >> [v2:172.21.15.46:6808/3454452571,v1:172.21.15.46:6809/3454452571] conn(0x7f7a8803e220 msgr2=0x7f7a88040640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:43.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.891+0000 7f7aac58d640 1 --2- 172.21.15.46:0/4266820784 >> [v2:172.21.15.46:6808/3454452571,v1:172.21.15.46:6809/3454452571] conn(0x7f7a8803e220 0x7f7a88040640 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:43.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.891+0000 7f7aac58d640 1 -- 172.21.15.46:0/4266820784 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7a880383b0 msgr2=0x7f7a8803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:43.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.891+0000 7f7aac58d640 1 --2- 172.21.15.46:0/4266820784 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f7a880383b0 0x7f7a8803a870 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4120720 tx=0x7f7a94007e40 comp rx=0 tx=0).stop 2024-09-16T04:47:43.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.891+0000 7f7aac58d640 10 monclient: shutdown 2024-09-16T04:47:43.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.891+0000 7f7aac58d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:43.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.891+0000 7f7aac58d640 1 -- 172.21.15.46:0/4266820784 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7aa40a3c10 msgr2=0x7f7aa41cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:43.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.891+0000 7f7aac58d640 1 --2- 172.21.15.46:0/4266820784 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f7aa40a3c10 0x7f7aa41cbba0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0000c00 tx=0x7f7aa00046c0 comp rx=0 tx=0).stop 2024-09-16T04:47:43.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.891+0000 7f7aaab2a640 1 -- 172.21.15.46:0/4266820784 reap_dead start 2024-09-16T04:47:43.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.891+0000 7f7aac58d640 1 -- 172.21.15.46:0/4266820784 shutdown_connections 2024-09-16T04:47:43.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.891+0000 7f7aac58d640 1 -- 172.21.15.46:0/4266820784 >> 172.21.15.46:0/4266820784 conn(0x7f7aa40a6340 msgr2=0x7f7aa40a1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:43.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.891+0000 7f7aac58d640 1 -- 172.21.15.46:0/4266820784 shutdown_connections 2024-09-16T04:47:43.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:43.891+0000 7f7aac58d640 1 -- 172.21.15.46:0/4266820784 wait complete. 2024-09-16T04:47:43.904 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T04:47:44.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.005+0000 7f2216ae6640 10 monclient: get_monmap_and_config 2024-09-16T04:47:44.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.005+0000 7f2216ae6640 10 monclient: build_initial_monmap 2024-09-16T04:47:44.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.005+0000 7f2216ae6640 10 monclient: monmap: 2024-09-16T04:47:44.007 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:44.007 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.007 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:44.006986+0000 2024-09-16T04:47:44.008 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:44.006986+0000 2024-09-16T04:47:44.008 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:44.008 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.008 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:44.008 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:44.008 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:44.008 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:44.008 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:44.008 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.006+0000 7f2216ae6640 1 Processor -- start 2024-09-16T04:47:44.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.006+0000 7f2216ae6640 1 -- start start 2024-09-16T04:47:44.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.006+0000 7f2216ae6640 10 monclient: init 2024-09-16T04:47:44.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.006+0000 7f2216ae6640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:44.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.006+0000 7f2216ae6640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-16T04:47:44.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.006+0000 7f2216ae6640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f22100a3c10 0x7f221009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.006+0000 7f2216ae6640 10 monclient(hunting): picked mon.noname-c con 0x7f22100a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:47:44.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.006+0000 7f2216ae6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f22100a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:47:44.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.006+0000 7f2216ae6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f221013be90 addr v1:172.21.15.46:6789/0 2024-09-16T04:47:44.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.006+0000 7f2216ae6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f221009b940 con 0x7f221013be90 2024-09-16T04:47:44.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.006+0000 7f2216ae6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f221009ab00 con 0x7f22100a3ff0 2024-09-16T04:47:44.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.006+0000 7f2216ae6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.006+0000 7f2216ae6640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f221009b030 con 0x7f22100a3c10 2024-09-16T04:47:44.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.006+0000 7f2216ae6640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:44.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.006+0000 7f2216ae6640 10 monclient(hunting): authenticate will time out at 1262.422811s 2024-09-16T04:47:44.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.006+0000 7f2215083640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f221013be90 0x7f221013e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33326/0 (socket says 172.21.15.46:33326) 2024-09-16T04:47:44.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.006+0000 7f2215083640 1 -- 172.21.15.46:0/884822833 learned_addr learned my addr 172.21.15.46:0/884822833 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:47:44.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f2214882640 1 --2- 172.21.15.46:0/884822833 >> v2:172.21.15.114:3300/0 conn(0x7f22100a3c10 0x7f221009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f2214882640 10 monclient(hunting): get_auth_request con 0x7f22100a3c10 auth_method 0 2024-09-16T04:47:44.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f2214882640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f2214882640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f2214882640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f220f7fe640 1 -- 172.21.15.46:0/884822833 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3239306419 0 0) 0x7f221009b940 con 0x7f221013be90 2024-09-16T04:47:44.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f220f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:47:44.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f220f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f220f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f220f7fe640 10 monclient(hunting): my global_id is 104331 2024-09-16T04:47:44.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f220f7fe640 1 -- 172.21.15.46:0/884822833 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2200003520 con 0x7f221013be90 2024-09-16T04:47:44.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f2214882640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:44.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f2214882640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:44.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f2214882640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:44.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f2214882640 10 monclient(hunting): handle_auth_done global_id 114154 payload 274 2024-09-16T04:47:44.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f2214882640 1 -- 172.21.15.46:0/884822833 >> v1:172.21.15.114:6789/0 conn(0x7f22100a3ff0 legacy=0x7f22100a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f220f7fe640 1 -- 172.21.15.46:0/884822833 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2367403628 0 0) 0x7f2200003520 con 0x7f221013be90 2024-09-16T04:47:44.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f2214882640 1 -- 172.21.15.46:0/884822833 >> v1:172.21.15.46:6789/0 conn(0x7f221013be90 legacy=0x7f221013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f2214882640 10 monclient: _finish_hunting 0 2024-09-16T04:47:44.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f2214882640 1 monclient: found mon.noname-c 2024-09-16T04:47:44.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f2214882640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:44.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f2214882640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:47:44.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f2214882640 1 -- 172.21.15.46:0/884822833 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f221009b310 con 0x7f22100a3c10 2024-09-16T04:47:44.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.007+0000 7f220f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:47:44.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f2214882640 1 --2- 172.21.15.46:0/884822833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22100a3c10 0x7f221009bcf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f22040097a0 tx=0x7f220402a0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db87bf5a6c2c1155 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:44.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f220f7fe640 1 -- 172.21.15.46:0/884822833 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204037070 con 0x7f22100a3c10 2024-09-16T04:47:44.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f220f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:44.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f220f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:47:44.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f220f7fe640 10 monclient: dump: 2024-09-16T04:47:44.019 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:44.019 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.019 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:44.020 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f220f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:47:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f220f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:47:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f220f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f2216ae6640 5 monclient: authenticate success, global_id 114154 2024-09-16T04:47:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f2216ae6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f220f7fe640 1 -- 172.21.15.46:0/884822833 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f220402ab30 con 0x7f22100a3c10 2024-09-16T04:47:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f220f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f2216ae6640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f2216ae6640 4 set_mon_vals no callback set 2024-09-16T04:47:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f220f7fe640 1 -- 172.21.15.46:0/884822833 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22080019f0 con 0x7f22100a3c10 2024-09-16T04:47:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f220f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f220f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:47:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f220f7fe640 10 monclient: dump: 2024-09-16T04:47:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:44.022 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f2216ae6640 10 monclient: shutdown 2024-09-16T04:47:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f2216ae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f2216ae6640 1 -- 172.21.15.46:0/884822833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22100a3c10 msgr2=0x7f221009bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f2216ae6640 1 --2- 172.21.15.46:0/884822833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22100a3c10 0x7f221009bcf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f22040097a0 tx=0x7f220402a0a0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f2216ae6640 1 -- 172.21.15.46:0/884822833 shutdown_connections 2024-09-16T04:47:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f2216ae6640 1 --2- 172.21.15.46:0/884822833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22100a3c10 0x7f221009bcf0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f2216ae6640 1 -- 172.21.15.46:0/884822833 >> 172.21.15.46:0/884822833 conn(0x7f22100a6340 msgr2=0x7f221006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f2216ae6640 1 -- 172.21.15.46:0/884822833 shutdown_connections 2024-09-16T04:47:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.008+0000 7f2216ae6640 1 -- 172.21.15.46:0/884822833 wait complete. 2024-09-16T04:47:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.009+0000 7f2216ae6640 10 monclient: build_initial_monmap 2024-09-16T04:47:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.009+0000 7f2216ae6640 10 monclient: monmap: 2024-09-16T04:47:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:44.023 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.009+0000 7f2216ae6640 1 Processor -- start 2024-09-16T04:47:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.009+0000 7f2216ae6640 1 -- start start 2024-09-16T04:47:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.009+0000 7f2216ae6640 10 monclient: init 2024-09-16T04:47:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2216ae6640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2216ae6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:47:44.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2216ae6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22100a3c10 0x7f22101cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2216ae6640 10 monclient(hunting): picked mon.noname-a con 0x7f22100a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:44.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2216ae6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22100a3ff0 0x7f2210136e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2216ae6640 10 monclient(hunting): picked mon.noname-b con 0x7f22100a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:44.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2216ae6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f221013be90 0x7f221009d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2216ae6640 10 monclient(hunting): picked mon.noname-c con 0x7f221013be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:44.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2216ae6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2216ae6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f22100a6b40 con 0x7f22100a3c10 2024-09-16T04:47:44.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2216ae6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2216ae6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f22100a69c0 con 0x7f22100a3ff0 2024-09-16T04:47:44.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2216ae6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2216ae6640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f221013f9a0 con 0x7f221013be90 2024-09-16T04:47:44.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2216ae6640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:44.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2216ae6640 10 monclient(hunting): authenticate will time out at 1262.426467s 2024-09-16T04:47:44.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2214882640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22100a3c10 0x7f22101cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2214882640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22100a3c10 0x7f22101cbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41900/0 (socket says 172.21.15.46:41900) 2024-09-16T04:47:44.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2215083640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f221013be90 0x7f221009d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2214882640 1 -- 172.21.15.46:0/3755775472 learned_addr learned my addr 172.21.15.46:0/3755775472 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:44.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2214882640 10 monclient(hunting): get_auth_request con 0x7f22100a3c10 auth_method 0 2024-09-16T04:47:44.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2214882640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2214882640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2214882640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2215083640 10 monclient(hunting): get_auth_request con 0x7f221013be90 auth_method 0 2024-09-16T04:47:44.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2215083640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2215083640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2215083640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2214882640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2214882640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f220ffff640 1 --2- 172.21.15.46:0/3755775472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22100a3ff0 0x7f2210136e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2214882640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f220ffff640 10 monclient(hunting): get_auth_request con 0x7f22100a3ff0 auth_method 0 2024-09-16T04:47:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f220ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f220ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f220ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2215083640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2215083640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2215083640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f220ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f220ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f220ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2214882640 10 monclient(hunting): handle_auth_done global_id 104334 payload 824 2024-09-16T04:47:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2214882640 1 -- 172.21.15.46:0/3755775472 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f221013be90 msgr2=0x7f221009d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2214882640 1 --2- 172.21.15.46:0/3755775472 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f221013be90 0x7f221009d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2214882640 1 -- 172.21.15.46:0/3755775472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22100a3ff0 msgr2=0x7f2210136e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2214882640 1 --2- 172.21.15.46:0/3755775472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22100a3ff0 0x7f2210136e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2214882640 10 monclient: _finish_hunting 0 2024-09-16T04:47:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2214882640 1 monclient: found mon.noname-a 2024-09-16T04:47:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2214882640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2214882640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2214882640 1 -- 172.21.15.46:0/3755775472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2210120510 con 0x7f22100a3c10 2024-09-16T04:47:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.010+0000 7f2215083640 1 --2- 172.21.15.46:0/3755775472 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f221013be90 0x7f221009d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f2214882640 1 --2- 172.21.15.46:0/3755775472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22100a3c10 0x7f22101cbba0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f220402a0e0 tx=0x7f22040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f220d7fa640 1 -- 172.21.15.46:0/3755775472 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204042020 con 0x7f22100a3c10 2024-09-16T04:47:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f220d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f220d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f220d7fa640 10 monclient: dump: 2024-09-16T04:47:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:44.028 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f220d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:47:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f220d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:47:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f220d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f2216ae6640 5 monclient: authenticate success, global_id 104334 2024-09-16T04:47:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f220d7fa640 1 -- 172.21.15.46:0/3755775472 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2204003a40 con 0x7f22100a3c10 2024-09-16T04:47:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f2216ae6640 10 monclient: _renew_subs 2024-09-16T04:47:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f2216ae6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f2216ae6640 1 -- 172.21.15.46:0/3755775472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22101207f0 con 0x7f22100a3c10 2024-09-16T04:47:44.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f220d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f220d7fa640 1 -- 172.21.15.46:0/3755775472 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204037040 con 0x7f22100a3c10 2024-09-16T04:47:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f220d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f220d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f220d7fa640 10 monclient: dump: 2024-09-16T04:47:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:44.032 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f2216ae6640 10 monclient: _renew_subs 2024-09-16T04:47:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f2216ae6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f2216ae6640 1 -- 172.21.15.46:0/3755775472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f221013f580 con 0x7f22100a3c10 2024-09-16T04:47:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.011+0000 7f220effd640 4 set_mon_vals no callback set 2024-09-16T04:47:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.012+0000 7f220d7fa640 1 -- 172.21.15.46:0/3755775472 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f2204045020 con 0x7f22100a3c10 2024-09-16T04:47:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.012+0000 7f220d7fa640 1 --2- 172.21.15.46:0/3755775472 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f21f40383b0 0x7f21f403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.012+0000 7f220d7fa640 1 -- 172.21.15.46:0/3755775472 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7f220406b920 con 0x7f22100a3c10 2024-09-16T04:47:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.012+0000 7f2216ae6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.012+0000 7f2216ae6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.012+0000 7f2216ae6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.012+0000 7f2216ae6640 1 -- 172.21.15.46:0/3755775472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f221009bd70 con 0x7f22100a3c10 2024-09-16T04:47:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.012+0000 7f220ffff640 1 --2- 172.21.15.46:0/3755775472 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f21f40383b0 0x7f21f403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.012+0000 7f220ffff640 10 monclient: get_auth_request con 0x7f21f40383b0 auth_method 0 2024-09-16T04:47:44.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.013+0000 7f220ffff640 1 --2- 172.21.15.46:0/3755775472 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f21f40383b0 0x7f21f403a870 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f221009f6b0 tx=0x7f21fc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:44.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.017+0000 7f220d7fa640 1 -- 172.21.15.46:0/3755775472 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f220404c070 con 0x7f22100a3c10 2024-09-16T04:47:44.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.017+0000 7f220d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:44.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.017+0000 7f220d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:44.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.191+0000 7f2216ae6640 1 -- 172.21.15.46:0/3755775472 --> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f221009f790 con 0x7f21f40383b0 2024-09-16T04:47:44.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.192+0000 7f220d7fa640 1 -- 172.21.15.46:0/3755775472 <== mgr.4099 v2:172.21.15.114:6800/3521677055 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f221009f790 con 0x7f21f40383b0 2024-09-16T04:47:44.194 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T04:47:44.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.194+0000 7f2216ae6640 1 -- 172.21.15.46:0/3755775472 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f21f40383b0 msgr2=0x7f21f403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.194+0000 7f2216ae6640 1 --2- 172.21.15.46:0/3755775472 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f21f40383b0 0x7f21f403a870 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f221009f6b0 tx=0x7f21fc009040 comp rx=0 tx=0).stop 2024-09-16T04:47:44.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.194+0000 7f2216ae6640 10 monclient: shutdown 2024-09-16T04:47:44.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.194+0000 7f2216ae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:44.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.194+0000 7f2216ae6640 1 -- 172.21.15.46:0/3755775472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22100a3c10 msgr2=0x7f22101cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.194+0000 7f2216ae6640 1 --2- 172.21.15.46:0/3755775472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22100a3c10 0x7f22101cbba0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f220402a0e0 tx=0x7f22040046c0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.194+0000 7f2216ae6640 1 -- 172.21.15.46:0/3755775472 shutdown_connections 2024-09-16T04:47:44.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.194+0000 7f2216ae6640 1 --2- 172.21.15.46:0/3755775472 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f221013be90 0x7f221009d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.194+0000 7f2216ae6640 1 --2- 172.21.15.46:0/3755775472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22100a3ff0 0x7f2210136e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.194+0000 7f2216ae6640 1 --2- 172.21.15.46:0/3755775472 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f21f40383b0 0x7f21f403a870 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.194+0000 7f2216ae6640 1 --2- 172.21.15.46:0/3755775472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22100a3c10 0x7f22101cbba0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.194+0000 7f2216ae6640 1 -- 172.21.15.46:0/3755775472 >> 172.21.15.46:0/3755775472 conn(0x7f22100a6340 msgr2=0x7f22100a2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:44.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.194+0000 7f2216ae6640 1 -- 172.21.15.46:0/3755775472 shutdown_connections 2024-09-16T04:47:44.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.194+0000 7f2216ae6640 1 -- 172.21.15.46:0/3755775472 wait complete. 2024-09-16T04:47:44.207 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T04:47:44.207 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:47:44.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.310+0000 7fba1ff28640 10 monclient: get_monmap_and_config 2024-09-16T04:47:44.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.310+0000 7fba1ff28640 10 monclient: build_initial_monmap 2024-09-16T04:47:44.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.310+0000 7fba1ff28640 10 monclient: monmap: 2024-09-16T04:47:44.313 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:44.313 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.313 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:44.312355+0000 2024-09-16T04:47:44.313 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:44.312355+0000 2024-09-16T04:47:44.313 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:44.313 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.314 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:44.314 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:44.314 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:44.314 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:44.314 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:44.314 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.311+0000 7fba1ff28640 1 Processor -- start 2024-09-16T04:47:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.311+0000 7fba1ff28640 1 -- start start 2024-09-16T04:47:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.311+0000 7fba1ff28640 10 monclient: init 2024-09-16T04:47:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1ff28640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1ff28640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-16T04:47:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1ff28640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fba180a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:47:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1ff28640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fba180a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:47:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1ff28640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fba1813cea0 0x7fba1813f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1ff28640 10 monclient(hunting): picked mon.noname-b con 0x7fba1813cea0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1ff28640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba1809b940 con 0x7fba180a3c10 2024-09-16T04:47:44.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1ff28640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba1809ab00 con 0x7fba180a3ff0 2024-09-16T04:47:44.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1ff28640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1ff28640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fba1809b030 con 0x7fba1813cea0 2024-09-16T04:47:44.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1ff28640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:44.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1ff28640 10 monclient(hunting): authenticate will time out at 1262.728204s 2024-09-16T04:47:44.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1dcc4640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fba180a3c10 0x7fba1809bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33332/0 (socket says 172.21.15.46:33332) 2024-09-16T04:47:44.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1dcc4640 1 -- 172.21.15.46:0/4185070147 learned_addr learned my addr 172.21.15.46:0/4185070147 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:47:44.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1e4c5640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fba1813cea0 0x7fba1813f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1e4c5640 10 monclient(hunting): get_auth_request con 0x7fba1813cea0 auth_method 0 2024-09-16T04:47:44.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1e4c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1e4c5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1e4c5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1e4c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1e4c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1e4c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1e4c5640 10 monclient(hunting): handle_auth_done global_id 94478 payload 274 2024-09-16T04:47:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1e4c5640 1 -- 172.21.15.46:0/4185070147 >> v1:172.21.15.114:6789/0 conn(0x7fba180a3ff0 legacy=0x7fba180a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1e4c5640 1 -- 172.21.15.46:0/4185070147 >> v1:172.21.15.46:6789/0 conn(0x7fba180a3c10 legacy=0x7fba1809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1e4c5640 10 monclient: _finish_hunting 0 2024-09-16T04:47:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1ccc2640 1 -- 172.21.15.46:0/4185070147 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2001036969 0 0) 0x7fba1809b940 con 0x7fba180a3c10 2024-09-16T04:47:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1e4c5640 1 monclient: found mon.noname-b 2024-09-16T04:47:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1e4c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1e4c5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:47:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1e4c5640 1 -- 172.21.15.46:0/4185070147 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba1809b310 con 0x7fba1813cea0 2024-09-16T04:47:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.312+0000 7fba1ccc2640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:47:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.313+0000 7fba1e4c5640 1 --2- 172.21.15.46:0/4185070147 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fba1813cea0 0x7fba1813f290 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fba10009dc0 tx=0x7fba1002a640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a75a615f9ff6bcf5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.313+0000 7fba1ccc2640 1 -- 172.21.15.46:0/4185070147 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba1002ae60 con 0x7fba1813cea0 2024-09-16T04:47:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.313+0000 7fba1ccc2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.313+0000 7fba1ccc2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:47:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.313+0000 7fba1ccc2640 10 monclient: dump: 2024-09-16T04:47:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:44.327 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.313+0000 7fba1ccc2640 10 monclient: _finish_auth 0 2024-09-16T04:47:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.313+0000 7fba1ccc2640 10 monclient: _check_auth_tickets 2024-09-16T04:47:44.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.313+0000 7fba1ccc2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:44.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.313+0000 7fba1ccc2640 1 -- 172.21.15.46:0/4185070147 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba10007740 con 0x7fba1813cea0 2024-09-16T04:47:44.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.313+0000 7fba1ff28640 5 monclient: authenticate success, global_id 94478 2024-09-16T04:47:44.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.313+0000 7fba1ff28640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:44.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.313+0000 7fba1ccc2640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:44.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.313+0000 7fba1ccc2640 1 -- 172.21.15.46:0/4185070147 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba10007b10 con 0x7fba1813cea0 2024-09-16T04:47:44.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.313+0000 7fba1ff28640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:44.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.313+0000 7fba1ff28640 4 set_mon_vals no callback set 2024-09-16T04:47:44.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.313+0000 7fba1ccc2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:44.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.313+0000 7fba1ccc2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:44.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.313+0000 7fba1ccc2640 10 monclient: dump: 2024-09-16T04:47:44.328 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:44.328 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:44.329 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.314+0000 7fba1ff28640 10 monclient: shutdown 2024-09-16T04:47:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.314+0000 7fba1ff28640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.314+0000 7fba1ff28640 1 -- 172.21.15.46:0/4185070147 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fba1813cea0 msgr2=0x7fba1813f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.314+0000 7fba1ff28640 1 --2- 172.21.15.46:0/4185070147 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fba1813cea0 0x7fba1813f290 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fba10009dc0 tx=0x7fba1002a640 comp rx=0 tx=0).stop 2024-09-16T04:47:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.314+0000 7fba1ff28640 1 -- 172.21.15.46:0/4185070147 shutdown_connections 2024-09-16T04:47:44.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.314+0000 7fba1ff28640 1 --2- 172.21.15.46:0/4185070147 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fba1813cea0 0x7fba1813f290 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.314+0000 7fba1ff28640 1 -- 172.21.15.46:0/4185070147 >> 172.21.15.46:0/4185070147 conn(0x7fba180a6340 msgr2=0x7fba1806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:44.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.314+0000 7fba1ff28640 1 -- 172.21.15.46:0/4185070147 shutdown_connections 2024-09-16T04:47:44.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.314+0000 7fba1ff28640 1 -- 172.21.15.46:0/4185070147 wait complete. 2024-09-16T04:47:44.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.314+0000 7fba1ff28640 10 monclient: build_initial_monmap 2024-09-16T04:47:44.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.314+0000 7fba1ff28640 10 monclient: monmap: 2024-09-16T04:47:44.330 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:44.330 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:44.330 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:44.330 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:44.330 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:44.330 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.330 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:44.330 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:44.331 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1ff28640 1 Processor -- start 2024-09-16T04:47:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1ff28640 1 -- start start 2024-09-16T04:47:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1ff28640 10 monclient: init 2024-09-16T04:47:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1ff28640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1ff28640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:47:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1ff28640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fba180a3c10 0x7fba181cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1ff28640 10 monclient(hunting): picked mon.noname-c con 0x7fba180a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1ff28640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba180a3ff0 0x7fba18136cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1ff28640 10 monclient(hunting): picked mon.noname-a con 0x7fba180a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1ff28640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba1813cea0 0x7fba18139de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1ff28640 10 monclient(hunting): picked mon.noname-b con 0x7fba1813cea0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1ff28640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1ff28640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fba1813f950 con 0x7fba180a3ff0 2024-09-16T04:47:44.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1ff28640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1ff28640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fba1813f7d0 con 0x7fba1813cea0 2024-09-16T04:47:44.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1ff28640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1ff28640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fba1813fad0 con 0x7fba180a3c10 2024-09-16T04:47:44.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1ff28640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:44.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1ff28640 10 monclient(hunting): authenticate will time out at 1262.731560s 2024-09-16T04:47:44.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1dcc4640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fba180a3c10 0x7fba181cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1d4c3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba180a3ff0 0x7fba18136cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1dcc4640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fba180a3c10 0x7fba181cba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:38858/0 (socket says 172.21.15.46:38858) 2024-09-16T04:47:44.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1dcc4640 1 -- 172.21.15.46:0/1740167822 learned_addr learned my addr 172.21.15.46:0/1740167822 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:44.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1dcc4640 10 monclient(hunting): get_auth_request con 0x7fba180a3c10 auth_method 0 2024-09-16T04:47:44.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1dcc4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1dcc4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1dcc4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1d4c3640 10 monclient(hunting): get_auth_request con 0x7fba180a3ff0 auth_method 0 2024-09-16T04:47:44.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1d4c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1d4c3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1d4c3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1e4c5640 1 --2- 172.21.15.46:0/1740167822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba1813cea0 0x7fba18139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1dcc4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:44.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1dcc4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:44.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1dcc4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:44.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1d4c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:44.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1d4c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:44.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1d4c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:44.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1e4c5640 10 monclient(hunting): get_auth_request con 0x7fba1813cea0 auth_method 0 2024-09-16T04:47:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1e4c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1e4c5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1e4c5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1e4c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1e4c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1e4c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1dcc4640 10 monclient(hunting): handle_auth_done global_id 94481 payload 824 2024-09-16T04:47:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1dcc4640 1 -- 172.21.15.46:0/1740167822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba1813cea0 msgr2=0x7fba18139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1dcc4640 1 --2- 172.21.15.46:0/1740167822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba1813cea0 0x7fba18139de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1dcc4640 1 -- 172.21.15.46:0/1740167822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba180a3ff0 msgr2=0x7fba18136cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1dcc4640 1 --2- 172.21.15.46:0/1740167822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba180a3ff0 0x7fba18136cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1dcc4640 10 monclient: _finish_hunting 0 2024-09-16T04:47:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1dcc4640 1 monclient: found mon.noname-c 2024-09-16T04:47:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.315+0000 7fba1dcc4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba1dcc4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:47:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba1dcc4640 1 -- 172.21.15.46:0/1740167822 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba1813c540 con 0x7fba180a3c10 2024-09-16T04:47:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba1d4c3640 1 --2- 172.21.15.46:0/1740167822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba180a3ff0 0x7fba18136cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba1dcc4640 1 --2- 172.21.15.46:0/1740167822 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fba180a3c10 0x7fba181cba40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fba1400c3f0 tx=0x7fba1400c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba0e7fc640 1 -- 172.21.15.46:0/1740167822 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba140074e0 con 0x7fba180a3c10 2024-09-16T04:47:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba0e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:47:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba0e7fc640 10 monclient: dump: 2024-09-16T04:47:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:44.337 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba0e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:47:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba0e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:47:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba0e7fc640 1 -- 172.21.15.46:0/1740167822 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba140047e0 con 0x7fba180a3c10 2024-09-16T04:47:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba0e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba0e7fc640 1 -- 172.21.15.46:0/1740167822 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba14004c10 con 0x7fba180a3c10 2024-09-16T04:47:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba1ff28640 5 monclient: authenticate success, global_id 94481 2024-09-16T04:47:44.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba1ff28640 10 monclient: _renew_subs 2024-09-16T04:47:44.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba1ff28640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:44.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba1ff28640 1 -- 172.21.15.46:0/1740167822 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba18120670 con 0x7fba180a3c10 2024-09-16T04:47:44.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:44.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba0ffff640 4 set_mon_vals no callback set 2024-09-16T04:47:44.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba0e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:44.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba0e7fc640 10 monclient: dump: 2024-09-16T04:47:44.338 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:44.338 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.338 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.338 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.338 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:44.338 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.338 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:44.339 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba1ff28640 10 monclient: _renew_subs 2024-09-16T04:47:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba1ff28640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.316+0000 7fba1ff28640 1 -- 172.21.15.46:0/1740167822 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fba181200b0 con 0x7fba180a3c10 2024-09-16T04:47:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.317+0000 7fba1ff28640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.317+0000 7fba1ff28640 1 -- 172.21.15.46:0/1740167822 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fba1809bd70 con 0x7fba180a3c10 2024-09-16T04:47:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.317+0000 7fba0e7fc640 1 -- 172.21.15.46:0/1740167822 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fba14011070 con 0x7fba180a3c10 2024-09-16T04:47:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.317+0000 7fba0e7fc640 1 --2- 172.21.15.46:0/1740167822 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb9fc0383e0 0x7fb9fc03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.317+0000 7fba0e7fc640 1 -- 172.21.15.46:0/1740167822 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7fba1404b150 con 0x7fba180a3c10 2024-09-16T04:47:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.317+0000 7fba0e7fc640 1 --2- 172.21.15.46:0/1740167822 >> [v2:172.21.15.46:6816/4217009477,v1:172.21.15.46:6817/4217009477] conn(0x7fb9fc03e250 0x7fb9fc040670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.317+0000 7fba0e7fc640 1 -- 172.21.15.46:0/1740167822 --> [v2:172.21.15.46:6816/4217009477,v1:172.21.15.46:6817/4217009477] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fba1401ce80 con 0x7fb9fc03e250 2024-09-16T04:47:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.317+0000 7fba0e7fc640 1 -- 172.21.15.46:0/1740167822 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_get_version_reply(handle=1 version=315) ==== 24+0+0 (secure 0 0 0) 0x7fba14052050 con 0x7fba180a3c10 2024-09-16T04:47:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.317+0000 7fba0e7fc640 10 monclient: handle_get_version_reply finishing 1 version 315 2024-09-16T04:47:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.317+0000 7fba1d4c3640 1 --2- 172.21.15.46:0/1740167822 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb9fc0383e0 0x7fb9fc03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.317+0000 7fba1d4c3640 10 monclient: get_auth_request con 0x7fb9fc0383e0 auth_method 0 2024-09-16T04:47:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.317+0000 7fba1e4c5640 1 --2- 172.21.15.46:0/1740167822 >> [v2:172.21.15.46:6816/4217009477,v1:172.21.15.46:6817/4217009477] conn(0x7fb9fc03e250 0x7fb9fc040670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.317+0000 7fba1e4c5640 10 monclient: get_auth_request con 0x7fb9fc03e250 auth_method 0 2024-09-16T04:47:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.318+0000 7fba1e4c5640 1 --2- 172.21.15.46:0/1740167822 >> [v2:172.21.15.46:6816/4217009477,v1:172.21.15.46:6817/4217009477] conn(0x7fb9fc03e250 0x7fb9fc040670 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.318+0000 7fba1d4c3640 1 --2- 172.21.15.46:0/1740167822 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb9fc0383e0 0x7fb9fc03a8a0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fba18077090 tx=0x7fba08009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.318+0000 7fba0e7fc640 1 -- 172.21.15.46:0/1740167822 <== osd.1 v2:172.21.15.46:6816/4217009477 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba1401ce80 con 0x7fb9fc03e250 2024-09-16T04:47:44.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.331+0000 7fba1ff28640 1 -- 172.21.15.46:0/1740167822 --> [v2:172.21.15.46:6816/4217009477,v1:172.21.15.46:6817/4217009477] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fba181202f0 con 0x7fb9fc03e250 2024-09-16T04:47:44.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.331+0000 7fba0e7fc640 1 -- 172.21.15.46:0/1740167822 <== osd.1 v2:172.21.15.46:6816/4217009477 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fba181202f0 con 0x7fb9fc03e250 2024-09-16T04:47:44.341 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:47:44.341 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:47:44.341 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:47:44.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.331+0000 7fba1ff28640 1 -- 172.21.15.46:0/1740167822 >> [v2:172.21.15.46:6816/4217009477,v1:172.21.15.46:6817/4217009477] conn(0x7fb9fc03e250 msgr2=0x7fb9fc040670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.331+0000 7fba1ff28640 1 --2- 172.21.15.46:0/1740167822 >> [v2:172.21.15.46:6816/4217009477,v1:172.21.15.46:6817/4217009477] conn(0x7fb9fc03e250 0x7fb9fc040670 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.331+0000 7fba1ff28640 1 -- 172.21.15.46:0/1740167822 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb9fc0383e0 msgr2=0x7fb9fc03a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.331+0000 7fba1ff28640 1 --2- 172.21.15.46:0/1740167822 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fb9fc0383e0 0x7fb9fc03a8a0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fba18077090 tx=0x7fba08009040 comp rx=0 tx=0).stop 2024-09-16T04:47:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.331+0000 7fba1ff28640 10 monclient: shutdown 2024-09-16T04:47:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.331+0000 7fba1ff28640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.331+0000 7fba1ff28640 1 -- 172.21.15.46:0/1740167822 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fba180a3c10 msgr2=0x7fba181cba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.332+0000 7fba1ff28640 1 --2- 172.21.15.46:0/1740167822 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fba180a3c10 0x7fba181cba40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fba1400c3f0 tx=0x7fba1400c8b0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.332+0000 7fba1e4c5640 1 -- 172.21.15.46:0/1740167822 reap_dead start 2024-09-16T04:47:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.332+0000 7fba1ff28640 1 -- 172.21.15.46:0/1740167822 shutdown_connections 2024-09-16T04:47:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.332+0000 7fba1ff28640 1 -- 172.21.15.46:0/1740167822 >> 172.21.15.46:0/1740167822 conn(0x7fba180a6340 msgr2=0x7fba180a2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.332+0000 7fba1ff28640 1 -- 172.21.15.46:0/1740167822 shutdown_connections 2024-09-16T04:47:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.332+0000 7fba1ff28640 1 -- 172.21.15.46:0/1740167822 wait complete. 2024-09-16T04:47:44.344 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T04:47:44.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.445+0000 7f1488023640 10 monclient: get_monmap_and_config 2024-09-16T04:47:44.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.445+0000 7f1488023640 10 monclient: build_initial_monmap 2024-09-16T04:47:44.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.445+0000 7f1488023640 10 monclient: monmap: 2024-09-16T04:47:44.447 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:44.447 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.447 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:44.446641+0000 2024-09-16T04:47:44.447 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:44.446641+0000 2024-09-16T04:47:44.447 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:44.447 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.447 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:44.448 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:44.448 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:44.448 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:44.448 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:44.448 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.445+0000 7f1488023640 1 Processor -- start 2024-09-16T04:47:44.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.445+0000 7f1488023640 1 -- start start 2024-09-16T04:47:44.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.445+0000 7f1488023640 10 monclient: init 2024-09-16T04:47:44.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f1488023640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:44.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f1488023640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-16T04:47:44.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f1488023640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f14800a3c10 0x7f148009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f1488023640 10 monclient(hunting): picked mon.noname-c con 0x7f14800a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:47:44.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f1488023640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f14800a3ff0 0x7f14800a4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f1488023640 10 monclient(hunting): picked mon.noname-a con 0x7f14800a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:47:44.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f1488023640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f14801369d0 addr v1:172.21.15.114:6789/0 2024-09-16T04:47:44.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f1488023640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f148009ab70 con 0x7f14801369d0 2024-09-16T04:47:44.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f1488023640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f1488023640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f14800a4d60 con 0x7f14800a3ff0 2024-09-16T04:47:44.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f1488023640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f1488023640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f148009b030 con 0x7f14800a3c10 2024-09-16T04:47:44.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f1488023640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:44.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f1488023640 10 monclient(hunting): authenticate will time out at 1262.862452s 2024-09-16T04:47:44.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f14855be640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f14800a3ff0 0x7f14800a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f14855be640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f14800a3ff0 0x7f14800a4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35790/0 (socket says 172.21.15.46:35790) 2024-09-16T04:47:44.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f14855be640 1 -- 172.21.15.46:0/2187849503 learned_addr learned my addr 172.21.15.46:0/2187849503 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:44.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f14855be640 10 monclient(hunting): get_auth_request con 0x7f14800a3ff0 auth_method 0 2024-09-16T04:47:44.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f14855be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f14855be640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f14855be640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f14855be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:44.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f14855be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:44.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f14855be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:44.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f1485dbf640 1 --2- 172.21.15.46:0/2187849503 >> v2:172.21.15.114:3300/0 conn(0x7f14800a3c10 0x7f148009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f1485dbf640 10 monclient(hunting): get_auth_request con 0x7f14800a3c10 auth_method 0 2024-09-16T04:47:44.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f1485dbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f1485dbf640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f1485dbf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f14855be640 10 monclient(hunting): handle_auth_done global_id 104343 payload 274 2024-09-16T04:47:44.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f14855be640 1 -- 172.21.15.46:0/2187849503 >> v2:172.21.15.114:3300/0 conn(0x7f14800a3c10 msgr2=0x7f148009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:44.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f14855be640 1 --2- 172.21.15.46:0/2187849503 >> v2:172.21.15.114:3300/0 conn(0x7f14800a3c10 0x7f148009bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f14855be640 1 -- 172.21.15.46:0/2187849503 >> v1:172.21.15.114:6789/0 conn(0x7f14801369d0 legacy=0x7f1480136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:44.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f14855be640 10 monclient: _finish_hunting 0 2024-09-16T04:47:44.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f14855be640 1 monclient: found mon.noname-a 2024-09-16T04:47:44.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f14855be640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:44.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f14855be640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:44.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.446+0000 7f14855be640 1 -- 172.21.15.46:0/2187849503 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f148009b310 con 0x7f14800a3ff0 2024-09-16T04:47:44.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.447+0000 7f14855be640 1 --2- 172.21.15.46:0/2187849503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14800a3ff0 0x7f14800a4400 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f148006be80 tx=0x7f147002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a989aef27666933 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:44.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.447+0000 7f1484dbd640 1 -- 172.21.15.46:0/2187849503 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1470037070 con 0x7f14800a3ff0 2024-09-16T04:47:44.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.447+0000 7f1484dbd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:44.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.447+0000 7f1484dbd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:44.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.447+0000 7f1484dbd640 10 monclient: dump: 2024-09-16T04:47:44.461 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:44.461 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.461 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.461 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.461 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:44.461 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.461 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:44.461 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:44.461 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:44.461 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.447+0000 7f1484dbd640 10 monclient: _finish_auth 0 2024-09-16T04:47:44.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.447+0000 7f1484dbd640 10 monclient: _check_auth_tickets 2024-09-16T04:47:44.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.447+0000 7f1484dbd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:44.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.447+0000 7f1484dbd640 1 -- 172.21.15.46:0/2187849503 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f147002ae20 con 0x7f14800a3ff0 2024-09-16T04:47:44.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.447+0000 7f1488023640 5 monclient: authenticate success, global_id 104343 2024-09-16T04:47:44.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.447+0000 7f1488023640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:44.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.447+0000 7f1484dbd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:44.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.447+0000 7f1488023640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:44.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.447+0000 7f1484dbd640 1 -- 172.21.15.46:0/2187849503 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1470032660 con 0x7f14800a3ff0 2024-09-16T04:47:44.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.447+0000 7f1484dbd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:44.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.447+0000 7f1488023640 4 set_mon_vals no callback set 2024-09-16T04:47:44.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.447+0000 7f1484dbd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:44.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.447+0000 7f1484dbd640 10 monclient: dump: 2024-09-16T04:47:44.463 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:44.463 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.463 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.463 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.463 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:44.463 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.463 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:44.463 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:44.463 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:44.463 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.448+0000 7f1488023640 10 monclient: shutdown 2024-09-16T04:47:44.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.448+0000 7f1488023640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:44.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.448+0000 7f1488023640 1 -- 172.21.15.46:0/2187849503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14800a3ff0 msgr2=0x7f14800a4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.448+0000 7f1488023640 1 --2- 172.21.15.46:0/2187849503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14800a3ff0 0x7f14800a4400 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f148006be80 tx=0x7f147002a110 comp rx=0 tx=0).stop 2024-09-16T04:47:44.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.448+0000 7f1488023640 1 -- 172.21.15.46:0/2187849503 shutdown_connections 2024-09-16T04:47:44.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.448+0000 7f1488023640 1 --2- 172.21.15.46:0/2187849503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14800a3ff0 0x7f14800a4400 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.448+0000 7f1488023640 1 --2- 172.21.15.46:0/2187849503 >> v2:172.21.15.114:3300/0 conn(0x7f14800a3c10 0x7f148009bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.448+0000 7f1488023640 1 -- 172.21.15.46:0/2187849503 >> 172.21.15.46:0/2187849503 conn(0x7f14800a5d10 msgr2=0x7f14800a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:44.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.448+0000 7f1488023640 1 -- 172.21.15.46:0/2187849503 shutdown_connections 2024-09-16T04:47:44.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.448+0000 7f1488023640 1 -- 172.21.15.46:0/2187849503 wait complete. 2024-09-16T04:47:44.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.448+0000 7f1488023640 10 monclient: build_initial_monmap 2024-09-16T04:47:44.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.448+0000 7f1488023640 10 monclient: monmap: 2024-09-16T04:47:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:44.465 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1488023640 1 Processor -- start 2024-09-16T04:47:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1488023640 1 -- start start 2024-09-16T04:47:44.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1488023640 10 monclient: init 2024-09-16T04:47:44.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1488023640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:44.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1488023640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:47:44.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1488023640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f14800a3c10 0x7f14801cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1488023640 10 monclient(hunting): picked mon.noname-c con 0x7f14800a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:44.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1488023640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14800a3ff0 0x7f14801cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1488023640 10 monclient(hunting): picked mon.noname-b con 0x7f14800a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:44.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1488023640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14801369d0 0x7f1480123060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1488023640 10 monclient(hunting): picked mon.noname-a con 0x7f14801369d0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:44.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1488023640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1488023640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f148006e3d0 con 0x7f14801369d0 2024-09-16T04:47:44.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1488023640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1488023640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f14800a4940 con 0x7f14800a3ff0 2024-09-16T04:47:44.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1488023640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1488023640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f148006e550 con 0x7f14800a3c10 2024-09-16T04:47:44.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1488023640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:44.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1488023640 10 monclient(hunting): authenticate will time out at 1262.865795s 2024-09-16T04:47:44.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1485dbf640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f14800a3c10 0x7f14801cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f14865c0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14801369d0 0x7f1480123060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1485dbf640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f14800a3c10 0x7f14801cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:41702/0 (socket says 172.21.15.46:41702) 2024-09-16T04:47:44.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1485dbf640 1 -- 172.21.15.46:0/2828020730 learned_addr learned my addr 172.21.15.46:0/2828020730 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:44.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1485dbf640 10 monclient(hunting): get_auth_request con 0x7f14800a3c10 auth_method 0 2024-09-16T04:47:44.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1485dbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1485dbf640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f1485dbf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f14865c0640 10 monclient(hunting): get_auth_request con 0x7f14801369d0 auth_method 0 2024-09-16T04:47:44.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f14865c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f14865c0640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.449+0000 7f14865c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14855be640 1 --2- 172.21.15.46:0/2828020730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14800a3ff0 0x7f14801cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f1485dbf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:44.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f1485dbf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:44.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f1485dbf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:44.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14865c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:44.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14865c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:44.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14865c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:44.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14855be640 10 monclient(hunting): get_auth_request con 0x7f14800a3ff0 auth_method 0 2024-09-16T04:47:44.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14855be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14855be640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14855be640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14855be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:44.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14855be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:44.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14855be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:44.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f1485dbf640 10 monclient(hunting): handle_auth_done global_id 94484 payload 824 2024-09-16T04:47:44.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f1485dbf640 1 -- 172.21.15.46:0/2828020730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14800a3ff0 msgr2=0x7f14801cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f1485dbf640 1 --2- 172.21.15.46:0/2828020730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14800a3ff0 0x7f14801cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f1485dbf640 1 -- 172.21.15.46:0/2828020730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14801369d0 msgr2=0x7f1480123060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f1485dbf640 1 --2- 172.21.15.46:0/2828020730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14801369d0 0x7f1480123060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f1485dbf640 10 monclient: _finish_hunting 0 2024-09-16T04:47:44.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f1485dbf640 1 monclient: found mon.noname-c 2024-09-16T04:47:44.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f1485dbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:44.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f1485dbf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:47:44.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f1485dbf640 1 -- 172.21.15.46:0/2828020730 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14801269e0 con 0x7f14800a3c10 2024-09-16T04:47:44.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14865c0640 1 --2- 172.21.15.46:0/2828020730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14801369d0 0x7f1480123060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:44.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f1485dbf640 1 --2- 172.21.15.46:0/2828020730 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f14800a3c10 0x7f14801cbee0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f147c00d830 tx=0x7f147c00dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:44.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14767fc640 1 -- 172.21.15.46:0/2828020730 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147c00b6c0 con 0x7f14800a3c10 2024-09-16T04:47:44.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:44.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:47:44.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14767fc640 10 monclient: dump: 2024-09-16T04:47:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:44.471 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14767fc640 10 monclient: _finish_auth 0 2024-09-16T04:47:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14767fc640 10 monclient: _check_auth_tickets 2024-09-16T04:47:44.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:44.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14767fc640 1 -- 172.21.15.46:0/2828020730 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f147c019070 con 0x7f14800a3c10 2024-09-16T04:47:44.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f1488023640 5 monclient: authenticate success, global_id 94484 2024-09-16T04:47:44.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f1488023640 10 monclient: _renew_subs 2024-09-16T04:47:44.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f1488023640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:44.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f1488023640 1 -- 172.21.15.46:0/2828020730 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1480125150 con 0x7f14800a3c10 2024-09-16T04:47:44.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14767fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:44.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14767fc640 1 -- 172.21.15.46:0/2828020730 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147c00ba80 con 0x7f14800a3c10 2024-09-16T04:47:44.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:44.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:44.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14767fc640 10 monclient: dump: 2024-09-16T04:47:44.473 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:44.473 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.473 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.473 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.473 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:44.474 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.474 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:44.474 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:44.474 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:44.474 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f1488023640 10 monclient: _renew_subs 2024-09-16T04:47:44.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f1488023640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:44.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f1488023640 1 -- 172.21.15.46:0/2828020730 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1480126cc0 con 0x7f14800a3c10 2024-09-16T04:47:44.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.450+0000 7f14777fe640 4 set_mon_vals no callback set 2024-09-16T04:47:44.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.451+0000 7f14767fc640 1 -- 172.21.15.46:0/2828020730 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f147c00bc20 con 0x7f14800a3c10 2024-09-16T04:47:44.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.451+0000 7f14767fc640 1 --2- 172.21.15.46:0/2828020730 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f14640383b0 0x7f146403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.451+0000 7f1488023640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:44.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.451+0000 7f14767fc640 1 -- 172.21.15.46:0/2828020730 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7f147c04ba10 con 0x7f14800a3c10 2024-09-16T04:47:44.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.451+0000 7f1488023640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:44.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.451+0000 7f1488023640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:44.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.451+0000 7f1488023640 1 -- 172.21.15.46:0/2828020730 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f148009bcf0 con 0x7f14800a3c10 2024-09-16T04:47:44.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.451+0000 7f14855be640 1 --2- 172.21.15.46:0/2828020730 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f14640383b0 0x7f146403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.451+0000 7f14855be640 10 monclient: get_auth_request con 0x7f14640383b0 auth_method 0 2024-09-16T04:47:44.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.452+0000 7f14855be640 1 --2- 172.21.15.46:0/2828020730 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f14640383b0 0x7f146403a870 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f1470034000 tx=0x7f147002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:44.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.456+0000 7f14767fc640 1 -- 172.21.15.46:0/2828020730 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f147c01e640 con 0x7f14800a3c10 2024-09-16T04:47:44.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.456+0000 7f14767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:44.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.456+0000 7f14767fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:44.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.630+0000 7f1488023640 1 -- 172.21.15.46:0/2828020730 --> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f1480067e40 con 0x7f14640383b0 2024-09-16T04:47:44.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.630+0000 7f14767fc640 1 -- 172.21.15.46:0/2828020730 <== mgr.4099 v2:172.21.15.114:6800/3521677055 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1480067e40 con 0x7f14640383b0 2024-09-16T04:47:44.632 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T04:47:44.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.632+0000 7f1488023640 1 -- 172.21.15.46:0/2828020730 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f14640383b0 msgr2=0x7f146403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.632+0000 7f1488023640 1 --2- 172.21.15.46:0/2828020730 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f14640383b0 0x7f146403a870 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f1470034000 tx=0x7f147002e000 comp rx=0 tx=0).stop 2024-09-16T04:47:44.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.632+0000 7f1488023640 10 monclient: shutdown 2024-09-16T04:47:44.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.632+0000 7f1488023640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:44.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.632+0000 7f1488023640 1 -- 172.21.15.46:0/2828020730 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f14800a3c10 msgr2=0x7f14801cbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.632+0000 7f1488023640 1 --2- 172.21.15.46:0/2828020730 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f14800a3c10 0x7f14801cbee0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f147c00d830 tx=0x7f147c00dcf0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.632+0000 7f1488023640 1 -- 172.21.15.46:0/2828020730 shutdown_connections 2024-09-16T04:47:44.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.632+0000 7f1488023640 1 --2- 172.21.15.46:0/2828020730 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f14640383b0 0x7f146403a870 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.632+0000 7f1488023640 1 --2- 172.21.15.46:0/2828020730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14801369d0 0x7f1480123060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.632+0000 7f1488023640 1 --2- 172.21.15.46:0/2828020730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14800a3ff0 0x7f14801cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.632+0000 7f1488023640 1 --2- 172.21.15.46:0/2828020730 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f14800a3c10 0x7f14801cbee0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.632+0000 7f1488023640 1 -- 172.21.15.46:0/2828020730 >> 172.21.15.46:0/2828020730 conn(0x7f14800a5d10 msgr2=0x7f14800a70c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:44.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.633+0000 7f1488023640 1 -- 172.21.15.46:0/2828020730 shutdown_connections 2024-09-16T04:47:44.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.633+0000 7f1488023640 1 -- 172.21.15.46:0/2828020730 wait complete. 2024-09-16T04:47:44.646 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T04:47:44.647 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:47:44.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.750+0000 7fd30b6ed640 10 monclient: get_monmap_and_config 2024-09-16T04:47:44.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.750+0000 7fd30b6ed640 10 monclient: build_initial_monmap 2024-09-16T04:47:44.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.750+0000 7fd30b6ed640 10 monclient: monmap: 2024-09-16T04:47:44.752 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:44.752 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.752 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:44.751709+0000 2024-09-16T04:47:44.752 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:44.751709+0000 2024-09-16T04:47:44.752 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:44.752 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.752 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:44.752 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:44.752 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:44.753 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:44.753 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:44.753 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.750+0000 7fd30b6ed640 1 Processor -- start 2024-09-16T04:47:44.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd30b6ed640 1 -- start start 2024-09-16T04:47:44.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd30b6ed640 10 monclient: init 2024-09-16T04:47:44.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd30b6ed640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:44.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd30b6ed640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-16T04:47:44.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd30b6ed640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd3040a3c10 0x7fd30409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd30b6ed640 10 monclient(hunting): picked mon.noname-a con 0x7fd3040a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:47:44.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd30b6ed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd3040a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:47:44.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd30b6ed640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd30413be90 addr v1:172.21.15.114:6789/0 2024-09-16T04:47:44.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd30b6ed640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd30409ab70 con 0x7fd3040a3ff0 2024-09-16T04:47:44.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd30b6ed640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3040a4d60 con 0x7fd30413be90 2024-09-16T04:47:44.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd30b6ed640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd30b6ed640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd30409b030 con 0x7fd3040a3c10 2024-09-16T04:47:44.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd30b6ed640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:44.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd30b6ed640 10 monclient(hunting): authenticate will time out at 1263.167579s 2024-09-16T04:47:44.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd309489640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd3040a3c10 0x7fd30409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd308c88640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd3040a3ff0 0x7fd3040a4400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47584/0 (socket says 172.21.15.46:47584) 2024-09-16T04:47:44.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd308c88640 1 -- 172.21.15.46:0/3213115486 learned_addr learned my addr 172.21.15.46:0/3213115486 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:47:44.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd309489640 10 monclient(hunting): get_auth_request con 0x7fd3040a3c10 auth_method 0 2024-09-16T04:47:44.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd309489640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd309489640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd309489640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd309489640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:44.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd309489640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:44.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.751+0000 7fd309489640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:44.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd309489640 10 monclient(hunting): handle_auth_done global_id 104349 payload 274 2024-09-16T04:47:44.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd309489640 1 -- 172.21.15.46:0/3213115486 >> v1:172.21.15.114:6789/0 conn(0x7fd30413be90 legacy=0x7fd30413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd309489640 1 -- 172.21.15.46:0/3213115486 >> v1:172.21.15.46:6789/0 conn(0x7fd3040a3ff0 legacy=0x7fd3040a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd309489640 10 monclient: _finish_hunting 0 2024-09-16T04:47:44.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd309489640 1 monclient: found mon.noname-a 2024-09-16T04:47:44.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd309489640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:44.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd309489640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:44.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd2fbfff640 1 -- 172.21.15.46:0/3213115486 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1066889604 0 0) 0x7fd30409ab70 con 0x7fd3040a3ff0 2024-09-16T04:47:44.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd309489640 1 -- 172.21.15.46:0/3213115486 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd30409b310 con 0x7fd3040a3c10 2024-09-16T04:47:44.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd2fbfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:47:44.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd309489640 1 --2- 172.21.15.46:0/3213115486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3040a3c10 0x7fd30409bcf0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd3000097a0 tx=0x7fd30002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b318f8f20c9e90da server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:44.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd2fbfff640 1 -- 172.21.15.46:0/3213115486 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd300037070 con 0x7fd3040a3c10 2024-09-16T04:47:44.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd2fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:44.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd2fbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:44.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd2fbfff640 10 monclient: dump: 2024-09-16T04:47:44.765 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:44.765 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.765 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.766 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.766 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:44.766 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.766 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:44.766 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:44.766 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:44.766 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd2fbfff640 10 monclient: _finish_auth 0 2024-09-16T04:47:44.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd2fbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:47:44.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd2fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:44.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd30b6ed640 5 monclient: authenticate success, global_id 104349 2024-09-16T04:47:44.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd2fbfff640 1 -- 172.21.15.46:0/3213115486 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd300032440 con 0x7fd3040a3c10 2024-09-16T04:47:44.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd2fbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:44.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd30b6ed640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:44.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd2fbfff640 1 -- 172.21.15.46:0/3213115486 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30002ae60 con 0x7fd3040a3c10 2024-09-16T04:47:44.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd30b6ed640 4 set_mon_vals no callback set 2024-09-16T04:47:44.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd2fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:44.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd2fbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:44.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.752+0000 7fd2fbfff640 10 monclient: dump: 2024-09-16T04:47:44.767 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:44.767 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.767 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.767 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.767 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:44.767 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.767 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:44.767 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:44.768 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:44.768 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.753+0000 7fd30b6ed640 10 monclient: shutdown 2024-09-16T04:47:44.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.753+0000 7fd30b6ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:44.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.753+0000 7fd30b6ed640 1 -- 172.21.15.46:0/3213115486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3040a3c10 msgr2=0x7fd30409bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.753+0000 7fd30b6ed640 1 --2- 172.21.15.46:0/3213115486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3040a3c10 0x7fd30409bcf0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd3000097a0 tx=0x7fd30002a110 comp rx=0 tx=0).stop 2024-09-16T04:47:44.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.753+0000 7fd30b6ed640 1 -- 172.21.15.46:0/3213115486 shutdown_connections 2024-09-16T04:47:44.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.753+0000 7fd30b6ed640 1 --2- 172.21.15.46:0/3213115486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3040a3c10 0x7fd30409bcf0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.753+0000 7fd30b6ed640 1 -- 172.21.15.46:0/3213115486 >> 172.21.15.46:0/3213115486 conn(0x7fd3040a5d10 msgr2=0x7fd3040a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:44.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.753+0000 7fd30b6ed640 1 -- 172.21.15.46:0/3213115486 shutdown_connections 2024-09-16T04:47:44.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.753+0000 7fd30b6ed640 1 -- 172.21.15.46:0/3213115486 wait complete. 2024-09-16T04:47:44.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.754+0000 7fd30b6ed640 10 monclient: build_initial_monmap 2024-09-16T04:47:44.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.754+0000 7fd30b6ed640 10 monclient: monmap: 2024-09-16T04:47:44.769 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:44.769 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:44.769 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:44.769 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:44.769 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:44.769 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.769 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:44.769 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:44.769 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:44.769 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.754+0000 7fd30b6ed640 1 Processor -- start 2024-09-16T04:47:44.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.754+0000 7fd30b6ed640 1 -- start start 2024-09-16T04:47:44.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.754+0000 7fd30b6ed640 10 monclient: init 2024-09-16T04:47:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.754+0000 7fd30b6ed640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.754+0000 7fd30b6ed640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:47:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.754+0000 7fd30b6ed640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd3040a3c10 0x7fd304136a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.754+0000 7fd30b6ed640 10 monclient(hunting): picked mon.noname-c con 0x7fd3040a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.754+0000 7fd30b6ed640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3040a3ff0 0x7fd304138c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.754+0000 7fd30b6ed640 10 monclient(hunting): picked mon.noname-b con 0x7fd3040a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.754+0000 7fd30b6ed640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd30413be90 0x7fd304139160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.754+0000 7fd30b6ed640 10 monclient(hunting): picked mon.noname-a con 0x7fd30413be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.754+0000 7fd30b6ed640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.754+0000 7fd30b6ed640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd30406e550 con 0x7fd30413be90 2024-09-16T04:47:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.754+0000 7fd30b6ed640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.754+0000 7fd30b6ed640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd30406e3d0 con 0x7fd3040a3ff0 2024-09-16T04:47:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.754+0000 7fd30b6ed640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.754+0000 7fd30b6ed640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd30413f9a0 con 0x7fd3040a3c10 2024-09-16T04:47:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.754+0000 7fd30b6ed640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.754+0000 7fd30b6ed640 10 monclient(hunting): authenticate will time out at 1263.170941s 2024-09-16T04:47:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309489640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd3040a3c10 0x7fd304136a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309c8a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd30413be90 0x7fd304139160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309c8a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd30413be90 0x7fd304139160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35808/0 (socket says 172.21.15.46:35808) 2024-09-16T04:47:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309489640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd3040a3c10 0x7fd304136a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:41716/0 (socket says 172.21.15.46:41716) 2024-09-16T04:47:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309c8a640 1 -- 172.21.15.46:0/1058339115 learned_addr learned my addr 172.21.15.46:0/1058339115 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309c8a640 10 monclient(hunting): get_auth_request con 0x7fd30413be90 auth_method 0 2024-09-16T04:47:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309c8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309c8a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309c8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309489640 10 monclient(hunting): get_auth_request con 0x7fd3040a3c10 auth_method 0 2024-09-16T04:47:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309489640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309489640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309489640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309c8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309c8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309c8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309489640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309489640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309489640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd308c88640 1 --2- 172.21.15.46:0/1058339115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3040a3ff0 0x7fd304138c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd308c88640 10 monclient(hunting): get_auth_request con 0x7fd3040a3ff0 auth_method 0 2024-09-16T04:47:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd308c88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd308c88640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd308c88640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309c8a640 10 monclient(hunting): handle_auth_done global_id 104355 payload 824 2024-09-16T04:47:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309c8a640 1 -- 172.21.15.46:0/1058339115 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd3040a3c10 msgr2=0x7fd304136a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309c8a640 1 --2- 172.21.15.46:0/1058339115 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd3040a3c10 0x7fd304136a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309c8a640 1 -- 172.21.15.46:0/1058339115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3040a3ff0 msgr2=0x7fd304138c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309c8a640 1 --2- 172.21.15.46:0/1058339115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3040a3ff0 0x7fd304138c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309c8a640 10 monclient: _finish_hunting 0 2024-09-16T04:47:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309c8a640 1 monclient: found mon.noname-a 2024-09-16T04:47:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309c8a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309c8a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309c8a640 1 -- 172.21.15.46:0/1058339115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd304120250 con 0x7fd30413be90 2024-09-16T04:47:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309489640 1 --2- 172.21.15.46:0/1058339115 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd3040a3c10 0x7fd304136a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd309c8a640 1 --2- 172.21.15.46:0/1058339115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd30413be90 0x7fd304139160 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc00c730 tx=0x7fd2fc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd2f9ffb640 1 -- 172.21.15.46:0/1058339115 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc003b70 con 0x7fd30413be90 2024-09-16T04:47:44.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd2f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:44.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd2f9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:44.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd2f9ffb640 10 monclient: dump: 2024-09-16T04:47:44.774 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:44.774 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.774 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.774 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.774 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:44.774 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.774 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:44.774 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:44.774 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:44.774 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd2f9ffb640 10 monclient: _finish_auth 0 2024-09-16T04:47:44.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd2f9ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:47:44.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd2f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:44.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd30b6ed640 5 monclient: authenticate success, global_id 104355 2024-09-16T04:47:44.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd2f9ffb640 1 -- 172.21.15.46:0/1058339115 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2fc003d30 con 0x7fd30413be90 2024-09-16T04:47:44.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd30b6ed640 10 monclient: _renew_subs 2024-09-16T04:47:44.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd30b6ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:44.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd30b6ed640 1 -- 172.21.15.46:0/1058339115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd304120530 con 0x7fd30413be90 2024-09-16T04:47:44.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd2f9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:44.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd2f9ffb640 1 -- 172.21.15.46:0/1058339115 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc005310 con 0x7fd30413be90 2024-09-16T04:47:44.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd2f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:44.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd2f9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:44.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd2f9ffb640 10 monclient: dump: 2024-09-16T04:47:44.775 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:44.775 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.776 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.776 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.776 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:44.776 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.776 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:44.776 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:44.776 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:44.776 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd2fb7fe640 4 set_mon_vals no callback set 2024-09-16T04:47:44.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.755+0000 7fd30b6ed640 10 monclient: _renew_subs 2024-09-16T04:47:44.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.756+0000 7fd30b6ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:44.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.756+0000 7fd30b6ed640 1 -- 172.21.15.46:0/1058339115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd30409d330 con 0x7fd30413be90 2024-09-16T04:47:44.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.756+0000 7fd2f9ffb640 1 -- 172.21.15.46:0/1058339115 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fd2fc0056c0 con 0x7fd30413be90 2024-09-16T04:47:44.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.756+0000 7fd30b6ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:44.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.756+0000 7fd30b6ed640 1 -- 172.21.15.46:0/1058339115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd30409bcf0 con 0x7fd30413be90 2024-09-16T04:47:44.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.756+0000 7fd2f9ffb640 1 --2- 172.21.15.46:0/1058339115 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd2e80383b0 0x7fd2e803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.756+0000 7fd2f9ffb640 1 -- 172.21.15.46:0/1058339115 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7fd2fc04b6c0 con 0x7fd30413be90 2024-09-16T04:47:44.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.757+0000 7fd2f9ffb640 1 --2- 172.21.15.46:0/1058339115 >> [v2:172.21.15.46:6800/3101080311,v1:172.21.15.46:6801/3101080311] conn(0x7fd2e803e220 0x7fd2e8040640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.757+0000 7fd2f9ffb640 1 -- 172.21.15.46:0/1058339115 --> [v2:172.21.15.46:6800/3101080311,v1:172.21.15.46:6801/3101080311] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd2fc014e50 con 0x7fd2e803e220 2024-09-16T04:47:44.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.757+0000 7fd2f9ffb640 1 -- 172.21.15.46:0/1058339115 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=315) ==== 24+0+0 (secure 0 0 0) 0x7fd2fc052050 con 0x7fd30413be90 2024-09-16T04:47:44.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.757+0000 7fd2f9ffb640 10 monclient: handle_get_version_reply finishing 1 version 315 2024-09-16T04:47:44.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.757+0000 7fd308c88640 1 --2- 172.21.15.46:0/1058339115 >> [v2:172.21.15.46:6800/3101080311,v1:172.21.15.46:6801/3101080311] conn(0x7fd2e803e220 0x7fd2e8040640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.757+0000 7fd308c88640 10 monclient: get_auth_request con 0x7fd2e803e220 auth_method 0 2024-09-16T04:47:44.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.757+0000 7fd309489640 1 --2- 172.21.15.46:0/1058339115 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd2e80383b0 0x7fd2e803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.757+0000 7fd309489640 10 monclient: get_auth_request con 0x7fd2e80383b0 auth_method 0 2024-09-16T04:47:44.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.757+0000 7fd308c88640 1 --2- 172.21.15.46:0/1058339115 >> [v2:172.21.15.46:6800/3101080311,v1:172.21.15.46:6801/3101080311] conn(0x7fd2e803e220 0x7fd2e8040640 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:44.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.757+0000 7fd309489640 1 --2- 172.21.15.46:0/1058339115 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd2e80383b0 0x7fd2e803a870 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd300034000 tx=0x7fd30002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:44.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.758+0000 7fd2f9ffb640 1 -- 172.21.15.46:0/1058339115 <== osd.2 v2:172.21.15.46:6800/3101080311 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd2fc014e50 con 0x7fd2e803e220 2024-09-16T04:47:44.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.770+0000 7fd30b6ed640 1 -- 172.21.15.46:0/1058339115 --> [v2:172.21.15.46:6800/3101080311,v1:172.21.15.46:6801/3101080311] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd30413b150 con 0x7fd2e803e220 2024-09-16T04:47:44.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.770+0000 7fd2f9ffb640 1 -- 172.21.15.46:0/1058339115 <== osd.2 v2:172.21.15.46:6800/3101080311 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd30413b150 con 0x7fd2e803e220 2024-09-16T04:47:44.778 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:47:44.779 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:47:44.779 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:47:44.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.771+0000 7fd30b6ed640 1 -- 172.21.15.46:0/1058339115 >> [v2:172.21.15.46:6800/3101080311,v1:172.21.15.46:6801/3101080311] conn(0x7fd2e803e220 msgr2=0x7fd2e8040640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.771+0000 7fd30b6ed640 1 --2- 172.21.15.46:0/1058339115 >> [v2:172.21.15.46:6800/3101080311,v1:172.21.15.46:6801/3101080311] conn(0x7fd2e803e220 0x7fd2e8040640 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.771+0000 7fd30b6ed640 1 -- 172.21.15.46:0/1058339115 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd2e80383b0 msgr2=0x7fd2e803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.771+0000 7fd30b6ed640 1 --2- 172.21.15.46:0/1058339115 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd2e80383b0 0x7fd2e803a870 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd300034000 tx=0x7fd30002e000 comp rx=0 tx=0).stop 2024-09-16T04:47:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.771+0000 7fd30b6ed640 10 monclient: shutdown 2024-09-16T04:47:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.771+0000 7fd30b6ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.771+0000 7fd30b6ed640 1 -- 172.21.15.46:0/1058339115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd30413be90 msgr2=0x7fd304139160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.771+0000 7fd30b6ed640 1 --2- 172.21.15.46:0/1058339115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd30413be90 0x7fd304139160 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc00c730 tx=0x7fd2fc00cbf0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.771+0000 7fd309c8a640 1 -- 172.21.15.46:0/1058339115 reap_dead start 2024-09-16T04:47:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.771+0000 7fd30b6ed640 1 -- 172.21.15.46:0/1058339115 shutdown_connections 2024-09-16T04:47:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.771+0000 7fd30b6ed640 1 -- 172.21.15.46:0/1058339115 >> 172.21.15.46:0/1058339115 conn(0x7fd3040a5d10 msgr2=0x7fd30409f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.771+0000 7fd30b6ed640 1 -- 172.21.15.46:0/1058339115 shutdown_connections 2024-09-16T04:47:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.771+0000 7fd30b6ed640 1 -- 172.21.15.46:0/1058339115 wait complete. 2024-09-16T04:47:44.783 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T04:47:44.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.883+0000 7f69794ba640 10 monclient: get_monmap_and_config 2024-09-16T04:47:44.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.883+0000 7f69794ba640 10 monclient: build_initial_monmap 2024-09-16T04:47:44.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.883+0000 7f69794ba640 10 monclient: monmap: 2024-09-16T04:47:44.886 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:44.886 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.886 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:44.885146+0000 2024-09-16T04:47:44.886 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:44.885146+0000 2024-09-16T04:47:44.886 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:44.886 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.886 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:44.886 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:44.886 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:44.886 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:44.886 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:44.886 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.884+0000 7f69794ba640 1 Processor -- start 2024-09-16T04:47:44.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.884+0000 7f69794ba640 1 -- start start 2024-09-16T04:47:44.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.884+0000 7f69794ba640 10 monclient: init 2024-09-16T04:47:44.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.884+0000 7f69794ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:44.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.884+0000 7f69794ba640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-16T04:47:44.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.884+0000 7f69794ba640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f69740a3ba0 0x7f697409bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.884+0000 7f69794ba640 10 monclient(hunting): picked mon.noname-c con 0x7f69740a3ba0 addr v2:172.21.15.114:3300/0 2024-09-16T04:47:44.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.884+0000 7f69794ba640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f697409c1c0 0x7f69740a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69794ba640 10 monclient(hunting): picked mon.noname-b con 0x7f697409c1c0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:44.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69794ba640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69740a44c0 0x7f6974135950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69794ba640 10 monclient(hunting): picked mon.noname-a con 0x7f69740a44c0 addr v2:172.21.15.46:3300/0 2024-09-16T04:47:44.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69794ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69794ba640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f697409b860 con 0x7f69740a44c0 2024-09-16T04:47:44.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69794ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69794ba640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f697409b2a0 con 0x7f69740a3ba0 2024-09-16T04:47:44.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69794ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69794ba640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f697409afc0 con 0x7f697409c1c0 2024-09-16T04:47:44.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69794ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:44.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69794ba640 10 monclient(hunting): authenticate will time out at 1263.301067s 2024-09-16T04:47:44.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69737fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69740a44c0 0x7f6974135950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f696bfff640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f697409c1c0 0x7f69740a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69737fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69740a44c0 0x7f6974135950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35824/0 (socket says 172.21.15.46:35824) 2024-09-16T04:47:44.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f696bfff640 10 monclient(hunting): get_auth_request con 0x7f697409c1c0 auth_method 0 2024-09-16T04:47:44.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69737fe640 1 -- 172.21.15.46:0/3608297947 learned_addr learned my addr 172.21.15.46:0/3608297947 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:44.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f696bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f696bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f696bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69737fe640 10 monclient(hunting): get_auth_request con 0x7f69740a44c0 auth_method 0 2024-09-16T04:47:44.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f696bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:44.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f696bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:44.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f696bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:44.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f6972ffd640 1 --2- 172.21.15.46:0/3608297947 >> v2:172.21.15.114:3300/0 conn(0x7f69740a3ba0 0x7f697409bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:44.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:44.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:44.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f6972ffd640 10 monclient(hunting): get_auth_request con 0x7f69740a3ba0 auth_method 0 2024-09-16T04:47:44.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f6972ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f6972ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f6972ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f696bfff640 10 monclient(hunting): handle_auth_done global_id 94490 payload 274 2024-09-16T04:47:44.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f696bfff640 1 -- 172.21.15.46:0/3608297947 >> v2:172.21.15.114:3300/0 conn(0x7f69740a3ba0 msgr2=0x7f697409bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:44.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f696bfff640 1 --2- 172.21.15.46:0/3608297947 >> v2:172.21.15.114:3300/0 conn(0x7f69740a3ba0 0x7f697409bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f696bfff640 1 -- 172.21.15.46:0/3608297947 >> v2:172.21.15.46:3300/0 conn(0x7f69740a44c0 msgr2=0x7f6974135950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:44.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f696bfff640 1 --2- 172.21.15.46:0/3608297947 >> v2:172.21.15.46:3300/0 conn(0x7f69740a44c0 0x7f6974135950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f696bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:47:44.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f696bfff640 1 monclient: found mon.noname-b 2024-09-16T04:47:44.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f696bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:44.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f696bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:47:44.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f696bfff640 1 -- 172.21.15.46:0/3608297947 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f697409b580 con 0x7f697409c1c0 2024-09-16T04:47:44.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f69737fe640 1 --2- 172.21.15.46:0/3608297947 >> v2:172.21.15.46:3300/0 conn(0x7f69740a44c0 0x7f6974135950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:44.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.885+0000 7f6972ffd640 1 --2- 172.21.15.46:0/3608297947 >> v2:172.21.15.114:3300/0 conn(0x7f69740a3ba0 0x7f697409bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:47:44.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f696bfff640 1 --2- 172.21.15.46:0/3608297947 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f697409c1c0 0x7f69740a3f80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6974068440 tx=0x7f695c02a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bc576eb32f057b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:44.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69727fc640 1 -- 172.21.15.46:0/3608297947 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c037070 con 0x7f697409c1c0 2024-09-16T04:47:44.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:44.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69727fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:47:44.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69727fc640 10 monclient: dump: 2024-09-16T04:47:44.899 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:44.899 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.899 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.899 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.899 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:44.899 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.899 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:44.899 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:44.899 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:44.899 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69727fc640 10 monclient: _finish_auth 0 2024-09-16T04:47:44.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69727fc640 10 monclient: _check_auth_tickets 2024-09-16T04:47:44.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:44.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69727fc640 1 -- 172.21.15.46:0/3608297947 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f695c02add0 con 0x7f697409c1c0 2024-09-16T04:47:44.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69794ba640 5 monclient: authenticate success, global_id 94490 2024-09-16T04:47:44.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69794ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:44.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69727fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:44.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69794ba640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:44.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69727fc640 1 -- 172.21.15.46:0/3608297947 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c032660 con 0x7f697409c1c0 2024-09-16T04:47:44.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:44.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69794ba640 4 set_mon_vals no callback set 2024-09-16T04:47:44.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:44.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69727fc640 10 monclient: dump: 2024-09-16T04:47:44.900 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:44.900 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.900 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.900 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.901 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:44.901 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.901 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:44.901 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:44.901 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:44.901 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69794ba640 10 monclient: shutdown 2024-09-16T04:47:44.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69794ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:44.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69794ba640 1 -- 172.21.15.46:0/3608297947 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f697409c1c0 msgr2=0x7f69740a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69794ba640 1 --2- 172.21.15.46:0/3608297947 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f697409c1c0 0x7f69740a3f80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6974068440 tx=0x7f695c02a080 comp rx=0 tx=0).stop 2024-09-16T04:47:44.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69794ba640 1 -- 172.21.15.46:0/3608297947 shutdown_connections 2024-09-16T04:47:44.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69794ba640 1 --2- 172.21.15.46:0/3608297947 >> v2:172.21.15.46:3300/0 conn(0x7f69740a44c0 0x7f6974135950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69794ba640 1 --2- 172.21.15.46:0/3608297947 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f697409c1c0 0x7f69740a3f80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69794ba640 1 --2- 172.21.15.46:0/3608297947 >> v2:172.21.15.114:3300/0 conn(0x7f69740a3ba0 0x7f697409bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.886+0000 7f69794ba640 1 -- 172.21.15.46:0/3608297947 >> 172.21.15.46:0/3608297947 conn(0x7f69740a60c0 msgr2=0x7f69740a64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:44.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.887+0000 7f69794ba640 1 -- 172.21.15.46:0/3608297947 shutdown_connections 2024-09-16T04:47:44.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.887+0000 7f69794ba640 1 -- 172.21.15.46:0/3608297947 wait complete. 2024-09-16T04:47:44.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.887+0000 7f69794ba640 10 monclient: build_initial_monmap 2024-09-16T04:47:44.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.887+0000 7f69794ba640 10 monclient: monmap: 2024-09-16T04:47:44.902 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:44.902 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:44.902 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:44.902 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:44.902 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:44.902 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.902 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:44.902 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:44.902 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:44.902 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.887+0000 7f69794ba640 1 Processor -- start 2024-09-16T04:47:44.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.887+0000 7f69794ba640 1 -- start start 2024-09-16T04:47:44.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.887+0000 7f69794ba640 10 monclient: init 2024-09-16T04:47:44.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69794ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:44.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69794ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:47:44.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69794ba640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f697409c1c0 0x7f69741cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69794ba640 10 monclient(hunting): picked mon.noname-b con 0x7f697409c1c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:44.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69794ba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69740a3ba0 0x7f69741cc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69794ba640 10 monclient(hunting): picked mon.noname-a con 0x7f69740a3ba0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:44.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69794ba640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69740a44c0 0x7f6974122040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69794ba640 10 monclient(hunting): picked mon.noname-c con 0x7f69740a44c0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:44.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69794ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69794ba640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f69740a5540 con 0x7f69740a3ba0 2024-09-16T04:47:44.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69794ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69794ba640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f69740a53c0 con 0x7f697409c1c0 2024-09-16T04:47:44.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69794ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:44.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69794ba640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f6974135e90 con 0x7f69740a44c0 2024-09-16T04:47:44.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69794ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:44.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69794ba640 10 monclient(hunting): authenticate will time out at 1263.304390s 2024-09-16T04:47:44.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f696bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69740a3ba0 0x7f69741cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69737fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69740a44c0 0x7f6974122040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f696bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69740a3ba0 0x7f69741cc490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35836/0 (socket says 172.21.15.46:35836) 2024-09-16T04:47:44.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f696bfff640 1 -- 172.21.15.46:0/3332456460 learned_addr learned my addr 172.21.15.46:0/3332456460 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:44.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69737fe640 10 monclient(hunting): get_auth_request con 0x7f69740a44c0 auth_method 0 2024-09-16T04:47:44.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f696bfff640 10 monclient(hunting): get_auth_request con 0x7f69740a3ba0 auth_method 0 2024-09-16T04:47:44.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f696bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f696bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f696bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:44.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f6972ffd640 1 --2- 172.21.15.46:0/3332456460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f697409c1c0 0x7f69741cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:44.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:44.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f6972ffd640 10 monclient(hunting): get_auth_request con 0x7f697409c1c0 auth_method 0 2024-09-16T04:47:44.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f6972ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:44.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f6972ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:44.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f6972ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:44.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f696bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:44.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f696bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:44.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f696bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:44.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f6972ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:44.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f6972ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:44.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f6972ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:44.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69737fe640 10 monclient(hunting): handle_auth_done global_id 94493 payload 824 2024-09-16T04:47:44.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69737fe640 1 -- 172.21.15.46:0/3332456460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f697409c1c0 msgr2=0x7f69741cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69737fe640 1 --2- 172.21.15.46:0/3332456460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f697409c1c0 0x7f69741cbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69737fe640 1 -- 172.21.15.46:0/3332456460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69740a3ba0 msgr2=0x7f69741cc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:44.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69737fe640 1 --2- 172.21.15.46:0/3332456460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69740a3ba0 0x7f69741cc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:44.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69737fe640 10 monclient: _finish_hunting 0 2024-09-16T04:47:44.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69737fe640 1 monclient: found mon.noname-c 2024-09-16T04:47:44.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:44.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69737fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:47:44.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f69737fe640 1 -- 172.21.15.46:0/3332456460 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69741247f0 con 0x7f69740a44c0 2024-09-16T04:47:44.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.888+0000 7f696bfff640 1 --2- 172.21.15.46:0/3332456460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69740a3ba0 0x7f69741cc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:44.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f69737fe640 1 --2- 172.21.15.46:0/3332456460 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69740a44c0 0x7f6974122040 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f696c009e50 tx=0x7f696c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:44.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f696b7fe640 1 -- 172.21.15.46:0/3332456460 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f696c00cb80 con 0x7f69740a44c0 2024-09-16T04:47:44.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f696b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:44.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f696b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:47:44.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f696b7fe640 10 monclient: dump: 2024-09-16T04:47:44.907 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:44.907 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.907 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.907 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.907 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:44.907 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.908 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:44.908 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:44.908 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:44.908 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f696b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:47:44.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f696b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:47:44.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f696b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:44.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f696b7fe640 1 -- 172.21.15.46:0/3332456460 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f696c00cd40 con 0x7f69740a44c0 2024-09-16T04:47:44.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f69794ba640 5 monclient: authenticate success, global_id 94493 2024-09-16T04:47:44.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f696b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:44.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f696b7fe640 1 -- 172.21.15.46:0/3332456460 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f696c003bd0 con 0x7f69740a44c0 2024-09-16T04:47:44.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f69794ba640 10 monclient: _renew_subs 2024-09-16T04:47:44.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f69794ba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:44.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f69794ba640 1 -- 172.21.15.46:0/3332456460 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6974125ca0 con 0x7f69740a44c0 2024-09-16T04:47:44.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f6971ffb640 4 set_mon_vals no callback set 2024-09-16T04:47:44.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f696b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:44.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f696b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:44.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f696b7fe640 10 monclient: dump: 2024-09-16T04:47:44.909 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:44.909 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:44.909 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.909 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:44.909 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:44.909 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:44.909 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:44.909 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:44.909 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:44.909 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:44.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f69794ba640 10 monclient: _renew_subs 2024-09-16T04:47:44.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f69794ba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:44.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.889+0000 7f69794ba640 1 -- 172.21.15.46:0/3332456460 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f69740a7ba0 con 0x7f69740a44c0 2024-09-16T04:47:44.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.890+0000 7f696b7fe640 1 -- 172.21.15.46:0/3332456460 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f696c01b030 con 0x7f69740a44c0 2024-09-16T04:47:44.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.890+0000 7f69794ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:44.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.890+0000 7f69794ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:44.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.890+0000 7f69794ba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:44.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.890+0000 7f69794ba640 1 -- 172.21.15.46:0/3332456460 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f697409bc80 con 0x7f69740a44c0 2024-09-16T04:47:44.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.890+0000 7f696b7fe640 1 --2- 172.21.15.46:0/3332456460 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f69540383b0 0x7f695403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:44.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.890+0000 7f696b7fe640 1 -- 172.21.15.46:0/3332456460 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7f696c04b870 con 0x7f69740a44c0 2024-09-16T04:47:44.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.890+0000 7f6972ffd640 1 --2- 172.21.15.46:0/3332456460 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f69540383b0 0x7f695403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:44.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.890+0000 7f6972ffd640 10 monclient: get_auth_request con 0x7f69540383b0 auth_method 0 2024-09-16T04:47:44.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.891+0000 7f6972ffd640 1 --2- 172.21.15.46:0/3332456460 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f69540383b0 0x7f695403a870 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6964006e80 tx=0x7f6964008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:44.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.895+0000 7f696b7fe640 1 -- 172.21.15.46:0/3332456460 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f696c022020 con 0x7f69740a44c0 2024-09-16T04:47:44.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.895+0000 7f696b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:44.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:44.895+0000 7f696b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:45.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.069+0000 7f69794ba640 1 -- 172.21.15.46:0/3332456460 --> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f69741240f0 con 0x7f69540383b0 2024-09-16T04:47:45.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.070+0000 7f696b7fe640 1 -- 172.21.15.46:0/3332456460 <== mgr.4099 v2:172.21.15.114:6800/3521677055 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f69741240f0 con 0x7f69540383b0 2024-09-16T04:47:45.072 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T04:47:45.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.072+0000 7f69794ba640 1 -- 172.21.15.46:0/3332456460 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f69540383b0 msgr2=0x7f695403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.072+0000 7f69794ba640 1 --2- 172.21.15.46:0/3332456460 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f69540383b0 0x7f695403a870 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6964006e80 tx=0x7f6964008040 comp rx=0 tx=0).stop 2024-09-16T04:47:45.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.072+0000 7f69794ba640 10 monclient: shutdown 2024-09-16T04:47:45.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.072+0000 7f69794ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:45.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.072+0000 7f69794ba640 1 -- 172.21.15.46:0/3332456460 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69740a44c0 msgr2=0x7f6974122040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.072+0000 7f69794ba640 1 --2- 172.21.15.46:0/3332456460 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69740a44c0 0x7f6974122040 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f696c009e50 tx=0x7f696c00c490 comp rx=0 tx=0).stop 2024-09-16T04:47:45.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.072+0000 7f69794ba640 1 -- 172.21.15.46:0/3332456460 shutdown_connections 2024-09-16T04:47:45.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.072+0000 7f69794ba640 1 --2- 172.21.15.46:0/3332456460 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f69740a44c0 0x7f6974122040 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.072+0000 7f69794ba640 1 --2- 172.21.15.46:0/3332456460 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f69540383b0 0x7f695403a870 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.072+0000 7f69794ba640 1 --2- 172.21.15.46:0/3332456460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69740a3ba0 0x7f69741cc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.072+0000 7f69794ba640 1 --2- 172.21.15.46:0/3332456460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f697409c1c0 0x7f69741cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.072+0000 7f69794ba640 1 -- 172.21.15.46:0/3332456460 >> 172.21.15.46:0/3332456460 conn(0x7f69740a60c0 msgr2=0x7f697409f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:45.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.072+0000 7f69794ba640 1 -- 172.21.15.46:0/3332456460 shutdown_connections 2024-09-16T04:47:45.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.072+0000 7f69794ba640 1 -- 172.21.15.46:0/3332456460 wait complete. 2024-09-16T04:47:45.085 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T04:47:45.085 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:47:45.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.187+0000 7fdc2d926640 10 monclient: get_monmap_and_config 2024-09-16T04:47:45.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.187+0000 7fdc2d926640 10 monclient: build_initial_monmap 2024-09-16T04:47:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.187+0000 7fdc2d926640 10 monclient: monmap: 2024-09-16T04:47:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:45.189364+0000 2024-09-16T04:47:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:45.189364+0000 2024-09-16T04:47:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:45.190 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.188+0000 7fdc2d926640 1 Processor -- start 2024-09-16T04:47:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.188+0000 7fdc2d926640 1 -- start start 2024-09-16T04:47:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.188+0000 7fdc2d926640 10 monclient: init 2024-09-16T04:47:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.188+0000 7fdc2d926640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.188+0000 7fdc2d926640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-16T04:47:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.188+0000 7fdc2d926640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdc28130900 0x7fdc28130ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.188+0000 7fdc2d926640 10 monclient(hunting): picked mon.noname-c con 0x7fdc28130900 addr v2:172.21.15.114:3300/0 2024-09-16T04:47:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.188+0000 7fdc2d926640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdc28133750 0x7fdc28133b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.188+0000 7fdc2d926640 10 monclient(hunting): picked mon.noname-a con 0x7fdc28133750 addr v2:172.21.15.46:3300/0 2024-09-16T04:47:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc2d926640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdc281340d0 addr v1:172.21.15.46:6789/0 2024-09-16T04:47:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc2d926640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc280a16e0 con 0x7fdc281340d0 2024-09-16T04:47:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc2d926640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc2d926640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fdc28134560 con 0x7fdc28133750 2024-09-16T04:47:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc2d926640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc2d926640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fdc2809d5b0 con 0x7fdc28130900 2024-09-16T04:47:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc2d926640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc2d926640 10 monclient(hunting): authenticate will time out at 1263.605035s 2024-09-16T04:47:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc267fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdc28133750 0x7fdc28133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc277fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdc281340d0 0x7fdc2813aa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47590/0 (socket says 172.21.15.46:47590) 2024-09-16T04:47:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc267fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdc28133750 0x7fdc28133b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35842/0 (socket says 172.21.15.46:35842) 2024-09-16T04:47:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc277fe640 1 -- 172.21.15.46:0/1416864358 learned_addr learned my addr 172.21.15.46:0/1416864358 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:47:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc267fc640 10 monclient(hunting): get_auth_request con 0x7fdc28133750 auth_method 0 2024-09-16T04:47:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc267fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:45.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc267fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:45.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc267fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:45.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc26ffd640 1 --2- 172.21.15.46:0/1416864358 >> v2:172.21.15.114:3300/0 conn(0x7fdc28130900 0x7fdc28130ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc26ffd640 10 monclient(hunting): get_auth_request con 0x7fdc28130900 auth_method 0 2024-09-16T04:47:45.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc26ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc26ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:45.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc267fc640 10 monclient(hunting): handle_auth_done global_id 104364 payload 274 2024-09-16T04:47:45.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc267fc640 1 -- 172.21.15.46:0/1416864358 >> v2:172.21.15.114:3300/0 conn(0x7fdc28130900 msgr2=0x7fdc28130ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:45.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc267fc640 1 --2- 172.21.15.46:0/1416864358 >> v2:172.21.15.114:3300/0 conn(0x7fdc28130900 0x7fdc28130ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc267fc640 1 -- 172.21.15.46:0/1416864358 >> v1:172.21.15.46:6789/0 conn(0x7fdc281340d0 legacy=0x7fdc2813aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc267fc640 10 monclient: _finish_hunting 0 2024-09-16T04:47:45.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc267fc640 1 monclient: found mon.noname-a 2024-09-16T04:47:45.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:45.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc267fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:45.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc25ffb640 1 -- 172.21.15.46:0/1416864358 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1620738016 0 0) 0x7fdc280a16e0 con 0x7fdc281340d0 2024-09-16T04:47:45.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc267fc640 1 -- 172.21.15.46:0/1416864358 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc2809dae0 con 0x7fdc28133750 2024-09-16T04:47:45.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.189+0000 7fdc25ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:47:45.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc267fc640 1 --2- 172.21.15.46:0/1416864358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc28133750 0x7fdc28133b90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdc2809bee0 tx=0x7fdc1c02a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965fcbf6fd12c030 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:45.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc25ffb640 1 -- 172.21.15.46:0/1416864358 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c037070 con 0x7fdc28133750 2024-09-16T04:47:45.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:45.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc25ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:45.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc25ffb640 10 monclient: dump: 2024-09-16T04:47:45.201 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:45.201 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:45.201 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.201 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.202 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:45.202 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.202 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:45.202 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:45.202 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:45.202 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc25ffb640 10 monclient: _finish_auth 0 2024-09-16T04:47:45.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc25ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:47:45.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:45.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc25ffb640 1 -- 172.21.15.46:0/1416864358 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc1c02add0 con 0x7fdc28133750 2024-09-16T04:47:45.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc2d926640 5 monclient: authenticate success, global_id 104364 2024-09-16T04:47:45.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc2d926640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:45.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc25ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:45.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc2d926640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:45.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc2d926640 4 set_mon_vals no callback set 2024-09-16T04:47:45.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc25ffb640 1 -- 172.21.15.46:0/1416864358 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c032660 con 0x7fdc28133750 2024-09-16T04:47:45.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:45.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc25ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:45.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc25ffb640 10 monclient: dump: 2024-09-16T04:47:45.203 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:45.203 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:45.203 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.203 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.203 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:45.203 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.203 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:45.203 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:45.203 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:45.204 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc2d926640 10 monclient: shutdown 2024-09-16T04:47:45.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc2d926640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:45.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc2d926640 1 -- 172.21.15.46:0/1416864358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc28133750 msgr2=0x7fdc28133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc2d926640 1 --2- 172.21.15.46:0/1416864358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc28133750 0x7fdc28133b90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdc2809bee0 tx=0x7fdc1c02a080 comp rx=0 tx=0).stop 2024-09-16T04:47:45.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc2d926640 1 -- 172.21.15.46:0/1416864358 shutdown_connections 2024-09-16T04:47:45.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc2d926640 1 --2- 172.21.15.46:0/1416864358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc28133750 0x7fdc28133b90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc2d926640 1 --2- 172.21.15.46:0/1416864358 >> v2:172.21.15.114:3300/0 conn(0x7fdc28130900 0x7fdc28130ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc2d926640 1 -- 172.21.15.46:0/1416864358 >> 172.21.15.46:0/1416864358 conn(0x7fdc280a5960 msgr2=0x7fdc280a1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:45.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc2d926640 1 -- 172.21.15.46:0/1416864358 shutdown_connections 2024-09-16T04:47:45.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.190+0000 7fdc2d926640 1 -- 172.21.15.46:0/1416864358 wait complete. 2024-09-16T04:47:45.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.191+0000 7fdc2d926640 10 monclient: build_initial_monmap 2024-09-16T04:47:45.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.191+0000 7fdc2d926640 10 monclient: monmap: 2024-09-16T04:47:45.205 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:45.205 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:45.205 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:45.205 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:45.205 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:45.205 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.205 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:45.205 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:45.205 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:45.205 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.191+0000 7fdc2d926640 1 Processor -- start 2024-09-16T04:47:45.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc2d926640 1 -- start start 2024-09-16T04:47:45.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc2d926640 10 monclient: init 2024-09-16T04:47:45.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc2d926640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:45.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc2d926640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:47:45.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc2d926640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc28130900 0x7fdc281c9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc2d926640 10 monclient(hunting): picked mon.noname-a con 0x7fdc28130900 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:45.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc2d926640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdc28133750 0x7fdc2812f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc2d926640 10 monclient(hunting): picked mon.noname-c con 0x7fdc28133750 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:45.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc2d926640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc281340d0 0x7fdc2812fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc2d926640 10 monclient(hunting): picked mon.noname-b con 0x7fdc281340d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:45.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc2d926640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc2d926640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdc280a2290 con 0x7fdc28130900 2024-09-16T04:47:45.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc2d926640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc2d926640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2809de80 con 0x7fdc281340d0 2024-09-16T04:47:45.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc2d926640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc2d926640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fdc280a2410 con 0x7fdc28133750 2024-09-16T04:47:45.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc2d926640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:45.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc2d926640 10 monclient(hunting): authenticate will time out at 1263.608542s 2024-09-16T04:47:45.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc26ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc28130900 0x7fdc281c9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc26ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc28130900 0x7fdc281c9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35850/0 (socket says 172.21.15.46:35850) 2024-09-16T04:47:45.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc26ffd640 1 -- 172.21.15.46:0/3442036840 learned_addr learned my addr 172.21.15.46:0/3442036840 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:45.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc26ffd640 10 monclient(hunting): get_auth_request con 0x7fdc28130900 auth_method 0 2024-09-16T04:47:45.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc26ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc26ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:45.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc267fc640 1 --2- 172.21.15.46:0/3442036840 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdc28133750 0x7fdc2812f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc267fc640 10 monclient(hunting): get_auth_request con 0x7fdc28133750 auth_method 0 2024-09-16T04:47:45.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:45.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc26ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:45.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc26ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc26ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc267fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc267fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc267fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc277fe640 1 --2- 172.21.15.46:0/3442036840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc281340d0 0x7fdc2812fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc277fe640 10 monclient(hunting): get_auth_request con 0x7fdc281340d0 auth_method 0 2024-09-16T04:47:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc26ffd640 10 monclient(hunting): handle_auth_done global_id 104370 payload 824 2024-09-16T04:47:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc26ffd640 1 -- 172.21.15.46:0/3442036840 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdc28133750 msgr2=0x7fdc2812f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc26ffd640 1 --2- 172.21.15.46:0/3442036840 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdc28133750 0x7fdc2812f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc26ffd640 1 -- 172.21.15.46:0/3442036840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc281340d0 msgr2=0x7fdc2812fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc26ffd640 1 --2- 172.21.15.46:0/3442036840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc281340d0 0x7fdc2812fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc26ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:47:45.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc26ffd640 1 monclient: found mon.noname-a 2024-09-16T04:47:45.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:45.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc26ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:45.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc26ffd640 1 -- 172.21.15.46:0/3442036840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc28121b70 con 0x7fdc28130900 2024-09-16T04:47:45.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.192+0000 7fdc277fe640 1 --2- 172.21.15.46:0/3442036840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc281340d0 0x7fdc2812fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:47:45.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc26ffd640 1 --2- 172.21.15.46:0/3442036840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc28130900 0x7fdc281c9730 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800d800 tx=0x7fdc1800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:45.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc0ffff640 1 -- 172.21.15.46:0/3442036840 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1800b6c0 con 0x7fdc28130900 2024-09-16T04:47:45.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:45.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc0ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:45.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc0ffff640 10 monclient: dump: 2024-09-16T04:47:45.210 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:45.210 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:45.210 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.210 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.211 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:45.211 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.211 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:45.211 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:45.211 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:45.211 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc0ffff640 10 monclient: _finish_auth 0 2024-09-16T04:47:45.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc0ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:47:45.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:45.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc0ffff640 1 -- 172.21.15.46:0/3442036840 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc18019070 con 0x7fdc28130900 2024-09-16T04:47:45.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc2d926640 5 monclient: authenticate success, global_id 104370 2024-09-16T04:47:45.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc0ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:45.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc0ffff640 1 -- 172.21.15.46:0/3442036840 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1800ba80 con 0x7fdc28130900 2024-09-16T04:47:45.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:45.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc0ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:45.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc0ffff640 10 monclient: dump: 2024-09-16T04:47:45.215 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:45.215 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:45.215 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.215 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.216 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:45.216 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.216 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:45.216 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:45.216 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:45.216 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc2d926640 10 monclient: _renew_subs 2024-09-16T04:47:45.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc2d926640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:45.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc2d926640 1 -- 172.21.15.46:0/3442036840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc281c9c70 con 0x7fdc28130900 2024-09-16T04:47:45.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc257fa640 4 set_mon_vals no callback set 2024-09-16T04:47:45.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc2d926640 10 monclient: _renew_subs 2024-09-16T04:47:45.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc2d926640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:45.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.193+0000 7fdc2d926640 1 -- 172.21.15.46:0/3442036840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc28121e50 con 0x7fdc28130900 2024-09-16T04:47:45.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.194+0000 7fdc2d926640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:45.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.194+0000 7fdc2d926640 1 -- 172.21.15.46:0/3442036840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdc280a4b00 con 0x7fdc28130900 2024-09-16T04:47:45.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.194+0000 7fdc0ffff640 1 -- 172.21.15.46:0/3442036840 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fdc1800bc20 con 0x7fdc28130900 2024-09-16T04:47:45.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.194+0000 7fdc0ffff640 1 --2- 172.21.15.46:0/3442036840 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdc140383b0 0x7fdc1403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.194+0000 7fdc0ffff640 1 -- 172.21.15.46:0/3442036840 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7fdc1804b9f0 con 0x7fdc28130900 2024-09-16T04:47:45.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.194+0000 7fdc0ffff640 1 --2- 172.21.15.46:0/3442036840 >> [v2:172.21.15.114:6810/890769734,v1:172.21.15.114:6811/890769734] conn(0x7fdc1403e220 0x7fdc14040640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.194+0000 7fdc0ffff640 1 -- 172.21.15.46:0/3442036840 --> [v2:172.21.15.114:6810/890769734,v1:172.21.15.114:6811/890769734] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdc1801fe50 con 0x7fdc1403e220 2024-09-16T04:47:45.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.194+0000 7fdc0ffff640 1 -- 172.21.15.46:0/3442036840 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=315) ==== 24+0+0 (secure 0 0 0) 0x7fdc180502c0 con 0x7fdc28130900 2024-09-16T04:47:45.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.194+0000 7fdc0ffff640 10 monclient: handle_get_version_reply finishing 1 version 315 2024-09-16T04:47:45.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.194+0000 7fdc267fc640 1 --2- 172.21.15.46:0/3442036840 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdc140383b0 0x7fdc1403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.194+0000 7fdc267fc640 10 monclient: get_auth_request con 0x7fdc140383b0 auth_method 0 2024-09-16T04:47:45.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.194+0000 7fdc277fe640 1 --2- 172.21.15.46:0/3442036840 >> [v2:172.21.15.114:6810/890769734,v1:172.21.15.114:6811/890769734] conn(0x7fdc1403e220 0x7fdc14040640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.194+0000 7fdc277fe640 10 monclient: get_auth_request con 0x7fdc1403e220 auth_method 0 2024-09-16T04:47:45.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.195+0000 7fdc267fc640 1 --2- 172.21.15.46:0/3442036840 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdc140383b0 0x7fdc1403a870 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c034000 tx=0x7fdc1c02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:45.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.195+0000 7fdc277fe640 1 --2- 172.21.15.46:0/3442036840 >> [v2:172.21.15.114:6810/890769734,v1:172.21.15.114:6811/890769734] conn(0x7fdc1403e220 0x7fdc14040640 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:45.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.195+0000 7fdc0ffff640 1 -- 172.21.15.46:0/3442036840 <== osd.3 v2:172.21.15.114:6810/890769734 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdc1801fe50 con 0x7fdc1403e220 2024-09-16T04:47:45.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.208+0000 7fdc2d926640 1 -- 172.21.15.46:0/3442036840 --> [v2:172.21.15.114:6810/890769734,v1:172.21.15.114:6811/890769734] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fdc280a46b0 con 0x7fdc1403e220 2024-09-16T04:47:45.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.209+0000 7fdc0ffff640 1 -- 172.21.15.46:0/3442036840 <== osd.3 v2:172.21.15.114:6810/890769734 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fdc280a46b0 con 0x7fdc1403e220 2024-09-16T04:47:45.221 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:47:45.221 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:47:45.221 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:47:45.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.209+0000 7fdc2d926640 1 -- 172.21.15.46:0/3442036840 >> [v2:172.21.15.114:6810/890769734,v1:172.21.15.114:6811/890769734] conn(0x7fdc1403e220 msgr2=0x7fdc14040640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.209+0000 7fdc2d926640 1 --2- 172.21.15.46:0/3442036840 >> [v2:172.21.15.114:6810/890769734,v1:172.21.15.114:6811/890769734] conn(0x7fdc1403e220 0x7fdc14040640 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.209+0000 7fdc2d926640 1 -- 172.21.15.46:0/3442036840 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdc140383b0 msgr2=0x7fdc1403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.209+0000 7fdc2d926640 1 --2- 172.21.15.46:0/3442036840 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdc140383b0 0x7fdc1403a870 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c034000 tx=0x7fdc1c02e000 comp rx=0 tx=0).stop 2024-09-16T04:47:45.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.209+0000 7fdc2d926640 10 monclient: shutdown 2024-09-16T04:47:45.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.209+0000 7fdc2d926640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:45.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.209+0000 7fdc2d926640 1 -- 172.21.15.46:0/3442036840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc28130900 msgr2=0x7fdc281c9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.209+0000 7fdc2d926640 1 --2- 172.21.15.46:0/3442036840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc28130900 0x7fdc281c9730 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800d800 tx=0x7fdc1800dcc0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.209+0000 7fdc277fe640 1 -- 172.21.15.46:0/3442036840 reap_dead start 2024-09-16T04:47:45.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.209+0000 7fdc2d926640 1 -- 172.21.15.46:0/3442036840 shutdown_connections 2024-09-16T04:47:45.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.209+0000 7fdc2d926640 1 -- 172.21.15.46:0/3442036840 >> 172.21.15.46:0/3442036840 conn(0x7fdc280a5960 msgr2=0x7fdc2809f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:45.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.209+0000 7fdc2d926640 1 -- 172.21.15.46:0/3442036840 shutdown_connections 2024-09-16T04:47:45.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.209+0000 7fdc2d926640 1 -- 172.21.15.46:0/3442036840 wait complete. 2024-09-16T04:47:45.225 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T04:47:45.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.325+0000 7fda5f52c640 10 monclient: get_monmap_and_config 2024-09-16T04:47:45.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.325+0000 7fda5f52c640 10 monclient: build_initial_monmap 2024-09-16T04:47:45.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.325+0000 7fda5f52c640 10 monclient: monmap: 2024-09-16T04:47:45.327 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:45.327 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:45.327 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:45.326975+0000 2024-09-16T04:47:45.327 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:45.326975+0000 2024-09-16T04:47:45.327 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:45.328 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.328 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:45.328 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:45.328 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:45.328 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:45.328 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:45.328 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.326+0000 7fda5f52c640 1 Processor -- start 2024-09-16T04:47:45.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.326+0000 7fda5f52c640 1 -- start start 2024-09-16T04:47:45.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.326+0000 7fda5f52c640 10 monclient: init 2024-09-16T04:47:45.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.326+0000 7fda5f52c640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:45.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.326+0000 7fda5f52c640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-16T04:47:45.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.326+0000 7fda5f52c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fda580a3c10 0x7fda5809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.326+0000 7fda5f52c640 10 monclient(hunting): picked mon.noname-c con 0x7fda580a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:47:45.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.326+0000 7fda5f52c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fda580a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:47:45.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.326+0000 7fda5f52c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fda5813be90 addr v1:172.21.15.46:6789/0 2024-09-16T04:47:45.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.326+0000 7fda5f52c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda5809b940 con 0x7fda5813be90 2024-09-16T04:47:45.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.326+0000 7fda5f52c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda5809ab00 con 0x7fda580a3ff0 2024-09-16T04:47:45.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.326+0000 7fda5f52c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.326+0000 7fda5f52c640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fda5809b030 con 0x7fda580a3c10 2024-09-16T04:47:45.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.326+0000 7fda5f52c640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:45.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.326+0000 7fda5f52c640 10 monclient(hunting): authenticate will time out at 1263.742833s 2024-09-16T04:47:45.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.326+0000 7fda5dac9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fda5813be90 0x7fda5813e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47604/0 (socket says 172.21.15.46:47604) 2024-09-16T04:47:45.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.326+0000 7fda5dac9640 1 -- 172.21.15.46:0/3743799420 learned_addr learned my addr 172.21.15.46:0/3743799420 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:47:45.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda5d2c8640 1 --2- 172.21.15.46:0/3743799420 >> v2:172.21.15.114:3300/0 conn(0x7fda580a3c10 0x7fda5809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda5d2c8640 10 monclient(hunting): get_auth_request con 0x7fda580a3c10 auth_method 0 2024-09-16T04:47:45.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda5d2c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda5d2c8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda5d2c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:45.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda4ffff640 1 -- 172.21.15.46:0/3743799420 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4033950380 0 0) 0x7fda5809b940 con 0x7fda5813be90 2024-09-16T04:47:45.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda4ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:47:45.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:45.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda4ffff640 10 monclient(hunting): my global_id is 104373 2024-09-16T04:47:45.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda4ffff640 1 -- 172.21.15.46:0/3743799420 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda44003520 con 0x7fda5813be90 2024-09-16T04:47:45.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda5d2c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:45.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda5d2c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:45.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda5d2c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:45.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda4ffff640 1 -- 172.21.15.46:0/3743799420 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3110230860 0 0) 0x7fda5809ab00 con 0x7fda580a3ff0 2024-09-16T04:47:45.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda4ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:47:45.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:45.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda4ffff640 10 monclient(hunting): my global_id is 114196 2024-09-16T04:47:45.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda4ffff640 1 -- 172.21.15.46:0/3743799420 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda5809b940 con 0x7fda580a3ff0 2024-09-16T04:47:45.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda5d2c8640 10 monclient(hunting): handle_auth_done global_id 114193 payload 274 2024-09-16T04:47:45.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda5d2c8640 1 -- 172.21.15.46:0/3743799420 >> v1:172.21.15.114:6789/0 conn(0x7fda580a3ff0 legacy=0x7fda580a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda5d2c8640 1 -- 172.21.15.46:0/3743799420 >> v1:172.21.15.46:6789/0 conn(0x7fda5813be90 legacy=0x7fda5813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda4ffff640 1 -- 172.21.15.46:0/3743799420 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 491523210 0 0) 0x7fda44003520 con 0x7fda5813be90 2024-09-16T04:47:45.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda5d2c8640 10 monclient: _finish_hunting 0 2024-09-16T04:47:45.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda5d2c8640 1 monclient: found mon.noname-c 2024-09-16T04:47:45.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda5d2c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:45.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda5d2c8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3300/0 2024-09-16T04:47:45.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda5d2c8640 1 -- 172.21.15.46:0/3743799420 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda5809b310 con 0x7fda580a3c10 2024-09-16T04:47:45.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.327+0000 7fda4ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:47:45.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda5d2c8640 1 --2- 172.21.15.46:0/3743799420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda580a3c10 0x7fda5809bcf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fda540097a0 tx=0x7fda5402a0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82a6b576930e6950 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:45.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda4ffff640 1 -- 172.21.15.46:0/3743799420 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda54037070 con 0x7fda580a3c10 2024-09-16T04:47:45.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:45.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda4ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:47:45.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda4ffff640 10 monclient: dump: 2024-09-16T04:47:45.340 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:45.340 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:45.341 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda4ffff640 10 monclient: _finish_auth 0 2024-09-16T04:47:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda4ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:47:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda5f52c640 5 monclient: authenticate success, global_id 114193 2024-09-16T04:47:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda5f52c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda4ffff640 1 -- 172.21.15.46:0/3743799420 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda5402ab30 con 0x7fda580a3c10 2024-09-16T04:47:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda4ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda5f52c640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda5f52c640 4 set_mon_vals no callback set 2024-09-16T04:47:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda4ffff640 1 -- 172.21.15.46:0/3743799420 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda500019f0 con 0x7fda580a3c10 2024-09-16T04:47:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda4ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:47:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda4ffff640 10 monclient: dump: 2024-09-16T04:47:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:45.343 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:45.343 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda5f52c640 10 monclient: shutdown 2024-09-16T04:47:45.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda5f52c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:45.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda5f52c640 1 -- 172.21.15.46:0/3743799420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda580a3c10 msgr2=0x7fda5809bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda5f52c640 1 --2- 172.21.15.46:0/3743799420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda580a3c10 0x7fda5809bcf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fda540097a0 tx=0x7fda5402a0a0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda5f52c640 1 -- 172.21.15.46:0/3743799420 shutdown_connections 2024-09-16T04:47:45.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda5f52c640 1 --2- 172.21.15.46:0/3743799420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda580a3c10 0x7fda5809bcf0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.328+0000 7fda5f52c640 1 -- 172.21.15.46:0/3743799420 >> 172.21.15.46:0/3743799420 conn(0x7fda580a6340 msgr2=0x7fda5806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:45.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.329+0000 7fda5f52c640 1 -- 172.21.15.46:0/3743799420 shutdown_connections 2024-09-16T04:47:45.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.329+0000 7fda5f52c640 1 -- 172.21.15.46:0/3743799420 wait complete. 2024-09-16T04:47:45.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.329+0000 7fda5f52c640 10 monclient: build_initial_monmap 2024-09-16T04:47:45.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.329+0000 7fda5f52c640 10 monclient: monmap: 2024-09-16T04:47:45.343 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:45.344 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:45.344 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:45.344 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:45.344 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:45.344 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.344 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:45.344 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:45.344 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:45.344 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.329+0000 7fda5f52c640 1 Processor -- start 2024-09-16T04:47:45.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.329+0000 7fda5f52c640 1 -- start start 2024-09-16T04:47:45.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.329+0000 7fda5f52c640 10 monclient: init 2024-09-16T04:47:45.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5f52c640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:45.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5f52c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:47:45.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5f52c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda580a3c10 0x7fda58136f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5f52c640 10 monclient(hunting): picked mon.noname-b con 0x7fda580a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:45.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5f52c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda580a3ff0 0x7fda581374d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5f52c640 10 monclient(hunting): picked mon.noname-a con 0x7fda580a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:45.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5f52c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5813be90 0x7fda58138fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5f52c640 10 monclient(hunting): picked mon.noname-c con 0x7fda5813be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:45.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5f52c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5f52c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fda580a6b40 con 0x7fda580a3ff0 2024-09-16T04:47:45.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5f52c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5f52c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fda580a69c0 con 0x7fda580a3c10 2024-09-16T04:47:45.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5f52c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5f52c640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fda5813f9a0 con 0x7fda5813be90 2024-09-16T04:47:45.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5f52c640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:45.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5f52c640 10 monclient(hunting): authenticate will time out at 1263.746425s 2024-09-16T04:47:45.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5cac7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda580a3ff0 0x7fda581374d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5cac7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda580a3ff0 0x7fda581374d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35862/0 (socket says 172.21.15.46:35862) 2024-09-16T04:47:45.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5cac7640 1 -- 172.21.15.46:0/2624392540 learned_addr learned my addr 172.21.15.46:0/2624392540 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:45.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5cac7640 10 monclient(hunting): get_auth_request con 0x7fda580a3ff0 auth_method 0 2024-09-16T04:47:45.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5cac7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5cac7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5cac7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:45.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5dac9640 1 --2- 172.21.15.46:0/2624392540 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5813be90 0x7fda58138fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5dac9640 10 monclient(hunting): get_auth_request con 0x7fda5813be90 auth_method 0 2024-09-16T04:47:45.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5dac9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5dac9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5dac9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:45.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5d2c8640 1 --2- 172.21.15.46:0/2624392540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda580a3c10 0x7fda58136f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5d2c8640 10 monclient(hunting): get_auth_request con 0x7fda580a3c10 auth_method 0 2024-09-16T04:47:45.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5d2c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5d2c8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5d2c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:45.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5cac7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:45.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5cac7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:45.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5cac7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:45.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5dac9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:45.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5dac9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:45.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5dac9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:45.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5d2c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:45.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5d2c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:45.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5d2c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:45.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5cac7640 10 monclient(hunting): handle_auth_done global_id 104376 payload 824 2024-09-16T04:47:45.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5cac7640 1 -- 172.21.15.46:0/2624392540 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5813be90 msgr2=0x7fda58138fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5cac7640 1 --2- 172.21.15.46:0/2624392540 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5813be90 0x7fda58138fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5cac7640 1 -- 172.21.15.46:0/2624392540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda580a3c10 msgr2=0x7fda58136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5cac7640 1 --2- 172.21.15.46:0/2624392540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda580a3c10 0x7fda58136f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5cac7640 10 monclient: _finish_hunting 0 2024-09-16T04:47:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5cac7640 1 monclient: found mon.noname-a 2024-09-16T04:47:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5cac7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5cac7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5cac7640 1 -- 172.21.15.46:0/2624392540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda581ccc60 con 0x7fda580a3ff0 2024-09-16T04:47:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5d2c8640 1 --2- 172.21.15.46:0/2624392540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda580a3c10 0x7fda58136f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.330+0000 7fda5dac9640 1 --2- 172.21.15.46:0/2624392540 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5813be90 0x7fda58138fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda5cac7640 1 --2- 172.21.15.46:0/2624392540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda580a3ff0 0x7fda581374d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fda4800c430 tx=0x7fda4800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda4dffb640 1 -- 172.21.15.46:0/2624392540 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda48017070 con 0x7fda580a3ff0 2024-09-16T04:47:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda4dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda4dffb640 10 monclient: dump: 2024-09-16T04:47:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:45.349 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda4dffb640 10 monclient: _finish_auth 0 2024-09-16T04:47:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda4dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:47:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda5f52c640 5 monclient: authenticate success, global_id 104376 2024-09-16T04:47:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda4dffb640 1 -- 172.21.15.46:0/2624392540 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda480052f0 con 0x7fda580a3ff0 2024-09-16T04:47:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda5f52c640 10 monclient: _renew_subs 2024-09-16T04:47:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda5f52c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda5f52c640 1 -- 172.21.15.46:0/2624392540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda58120910 con 0x7fda580a3ff0 2024-09-16T04:47:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda4dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda4dffb640 1 -- 172.21.15.46:0/2624392540 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda48003830 con 0x7fda580a3ff0 2024-09-16T04:47:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda4dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda4dffb640 10 monclient: dump: 2024-09-16T04:47:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:45.351 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda5f52c640 10 monclient: _renew_subs 2024-09-16T04:47:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda4f7fe640 4 set_mon_vals no callback set 2024-09-16T04:47:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda5f52c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.331+0000 7fda5f52c640 1 -- 172.21.15.46:0/2624392540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda5813ac40 con 0x7fda580a3ff0 2024-09-16T04:47:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.332+0000 7fda4dffb640 1 -- 172.21.15.46:0/2624392540 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fda4801a070 con 0x7fda580a3ff0 2024-09-16T04:47:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.332+0000 7fda4dffb640 1 --2- 172.21.15.46:0/2624392540 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda3c0383b0 0x7fda3c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.332+0000 7fda4dffb640 1 -- 172.21.15.46:0/2624392540 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7fda4804aaf0 con 0x7fda580a3ff0 2024-09-16T04:47:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.332+0000 7fda5f52c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.332+0000 7fda5f52c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.332+0000 7fda5f52c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.332+0000 7fda5f52c640 1 -- 172.21.15.46:0/2624392540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda5809bd70 con 0x7fda580a3ff0 2024-09-16T04:47:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.332+0000 7fda5d2c8640 1 --2- 172.21.15.46:0/2624392540 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda3c0383b0 0x7fda3c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.332+0000 7fda5d2c8640 10 monclient: get_auth_request con 0x7fda3c0383b0 auth_method 0 2024-09-16T04:47:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.333+0000 7fda5d2c8640 1 --2- 172.21.15.46:0/2624392540 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda3c0383b0 0x7fda3c03a870 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fda54034000 tx=0x7fda5402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.337+0000 7fda4dffb640 1 -- 172.21.15.46:0/2624392540 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fda4801a9b0 con 0x7fda580a3ff0 2024-09-16T04:47:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.337+0000 7fda4dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.337+0000 7fda4dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:45.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.512+0000 7fda5f52c640 1 -- 172.21.15.46:0/2624392540 --> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fda58120150 con 0x7fda3c0383b0 2024-09-16T04:47:45.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.512+0000 7fda4dffb640 1 -- 172.21.15.46:0/2624392540 <== mgr.4099 v2:172.21.15.114:6800/3521677055 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fda58120150 con 0x7fda3c0383b0 2024-09-16T04:47:45.514 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T04:47:45.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.514+0000 7fda5f52c640 1 -- 172.21.15.46:0/2624392540 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda3c0383b0 msgr2=0x7fda3c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.514+0000 7fda5f52c640 1 --2- 172.21.15.46:0/2624392540 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda3c0383b0 0x7fda3c03a870 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fda54034000 tx=0x7fda5402e000 comp rx=0 tx=0).stop 2024-09-16T04:47:45.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.514+0000 7fda5f52c640 10 monclient: shutdown 2024-09-16T04:47:45.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.514+0000 7fda5f52c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:45.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.514+0000 7fda5f52c640 1 -- 172.21.15.46:0/2624392540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda580a3ff0 msgr2=0x7fda581374d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.514+0000 7fda5f52c640 1 --2- 172.21.15.46:0/2624392540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda580a3ff0 0x7fda581374d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fda4800c430 tx=0x7fda4800c8f0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.514+0000 7fda5f52c640 1 -- 172.21.15.46:0/2624392540 shutdown_connections 2024-09-16T04:47:45.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.514+0000 7fda5f52c640 1 --2- 172.21.15.46:0/2624392540 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fda5813be90 0x7fda58138fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.514+0000 7fda5f52c640 1 --2- 172.21.15.46:0/2624392540 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fda3c0383b0 0x7fda3c03a870 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.514+0000 7fda5f52c640 1 --2- 172.21.15.46:0/2624392540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda580a3ff0 0x7fda581374d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.514+0000 7fda5f52c640 1 --2- 172.21.15.46:0/2624392540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda580a3c10 0x7fda58136f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.514+0000 7fda5f52c640 1 -- 172.21.15.46:0/2624392540 >> 172.21.15.46:0/2624392540 conn(0x7fda580a6340 msgr2=0x7fda580a1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:45.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.514+0000 7fda5f52c640 1 -- 172.21.15.46:0/2624392540 shutdown_connections 2024-09-16T04:47:45.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.514+0000 7fda5f52c640 1 -- 172.21.15.46:0/2624392540 wait complete. 2024-09-16T04:47:45.527 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T04:47:45.527 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:47:45.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.631+0000 7f260e5d8640 10 monclient: get_monmap_and_config 2024-09-16T04:47:45.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.631+0000 7f260e5d8640 10 monclient: build_initial_monmap 2024-09-16T04:47:45.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.631+0000 7f260e5d8640 10 monclient: monmap: 2024-09-16T04:47:45.633 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:45.633 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:45.633 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:45.632695+0000 2024-09-16T04:47:45.634 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:45.632695+0000 2024-09-16T04:47:45.634 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:45.634 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.634 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:45.634 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:45.634 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:45.634 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:45.634 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:45.634 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.631+0000 7f260e5d8640 1 Processor -- start 2024-09-16T04:47:45.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.631+0000 7f260e5d8640 1 -- start start 2024-09-16T04:47:45.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f260e5d8640 10 monclient: init 2024-09-16T04:47:45.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f260e5d8640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:45.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f260e5d8640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-16T04:47:45.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f260e5d8640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f26080a3c10 0x7f260809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f260e5d8640 10 monclient(hunting): picked mon.noname-c con 0x7f26080a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:47:45.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f260e5d8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f26080a3ff0 0x7f26080a4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f260e5d8640 10 monclient(hunting): picked mon.noname-a con 0x7f26080a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:47:45.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f260e5d8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f26081369d0 addr v1:172.21.15.114:6789/0 2024-09-16T04:47:45.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f260e5d8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f260809ab70 con 0x7f26081369d0 2024-09-16T04:47:45.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f260e5d8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f260e5d8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f26080a4d60 con 0x7f26080a3ff0 2024-09-16T04:47:45.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f260e5d8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f260e5d8640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f260809b030 con 0x7f26080a3c10 2024-09-16T04:47:45.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f260e5d8640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:45.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f260e5d8640 10 monclient(hunting): authenticate will time out at 1264.048586s 2024-09-16T04:47:45.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f26077fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f26080a3ff0 0x7f26080a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f26077fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f26080a3ff0 0x7f26080a4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35866/0 (socket says 172.21.15.46:35866) 2024-09-16T04:47:45.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f26077fe640 1 -- 172.21.15.46:0/2266452593 learned_addr learned my addr 172.21.15.46:0/2266452593 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:45.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f26077fe640 10 monclient(hunting): get_auth_request con 0x7f26080a3ff0 auth_method 0 2024-09-16T04:47:45.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f26077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f26077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f26077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:45.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f26077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:45.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f26077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:45.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f26077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:45.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f2607fff640 1 --2- 172.21.15.46:0/2266452593 >> v2:172.21.15.114:3300/0 conn(0x7f26080a3c10 0x7f260809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f2607fff640 10 monclient(hunting): get_auth_request con 0x7f26080a3c10 auth_method 0 2024-09-16T04:47:45.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f2607fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f2607fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f2607fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:45.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f26077fe640 10 monclient(hunting): handle_auth_done global_id 104379 payload 274 2024-09-16T04:47:45.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.632+0000 7f26077fe640 1 -- 172.21.15.46:0/2266452593 >> v2:172.21.15.114:3300/0 conn(0x7f26080a3c10 msgr2=0x7f260809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:45.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.633+0000 7f26077fe640 1 --2- 172.21.15.46:0/2266452593 >> v2:172.21.15.114:3300/0 conn(0x7f26080a3c10 0x7f260809bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.633+0000 7f26077fe640 1 -- 172.21.15.46:0/2266452593 >> v1:172.21.15.114:6789/0 conn(0x7f26081369d0 legacy=0x7f2608136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:45.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.633+0000 7f26077fe640 10 monclient: _finish_hunting 0 2024-09-16T04:47:45.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.633+0000 7f26077fe640 1 monclient: found mon.noname-a 2024-09-16T04:47:45.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.633+0000 7f26077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:45.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.633+0000 7f26077fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:45.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.633+0000 7f26077fe640 1 -- 172.21.15.46:0/2266452593 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f260809b310 con 0x7f26080a3ff0 2024-09-16T04:47:45.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.633+0000 7f2607fff640 1 --2- 172.21.15.46:0/2266452593 >> v2:172.21.15.114:3300/0 conn(0x7f26080a3c10 0x7f260809bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:47:45.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.633+0000 7f26077fe640 1 --2- 172.21.15.46:0/2266452593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26080a3ff0 0x7f26080a4400 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f260806be80 tx=0x7f25fc02a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad838d1cf93f608e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:45.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.633+0000 7f2606ffd640 1 -- 172.21.15.46:0/2266452593 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc037070 con 0x7f26080a3ff0 2024-09-16T04:47:45.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.633+0000 7f2606ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:45.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.633+0000 7f2606ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:45.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.633+0000 7f2606ffd640 10 monclient: dump: 2024-09-16T04:47:45.648 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:45.648 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:45.648 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.648 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.649 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:45.649 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.649 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:45.649 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:45.649 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:45.649 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.633+0000 7f2606ffd640 10 monclient: _finish_auth 0 2024-09-16T04:47:45.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.633+0000 7f2606ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:47:45.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.633+0000 7f2606ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:45.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.633+0000 7f2606ffd640 1 -- 172.21.15.46:0/2266452593 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25fc032440 con 0x7f26080a3ff0 2024-09-16T04:47:45.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.633+0000 7f260e5d8640 5 monclient: authenticate success, global_id 104379 2024-09-16T04:47:45.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.633+0000 7f260e5d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:45.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.634+0000 7f2606ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:45.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.634+0000 7f260e5d8640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:45.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.634+0000 7f2606ffd640 1 -- 172.21.15.46:0/2266452593 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc032810 con 0x7f26080a3ff0 2024-09-16T04:47:45.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.634+0000 7f2606ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:45.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.634+0000 7f260e5d8640 4 set_mon_vals no callback set 2024-09-16T04:47:45.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.634+0000 7f2606ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:45.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.634+0000 7f2606ffd640 10 monclient: dump: 2024-09-16T04:47:45.650 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:45.650 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:45.650 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.650 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.650 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:45.651 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.651 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:45.651 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:45.651 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:45.651 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.634+0000 7f260e5d8640 10 monclient: shutdown 2024-09-16T04:47:45.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.634+0000 7f260e5d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:45.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.634+0000 7f260e5d8640 1 -- 172.21.15.46:0/2266452593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26080a3ff0 msgr2=0x7f26080a4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.634+0000 7f260e5d8640 1 --2- 172.21.15.46:0/2266452593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26080a3ff0 0x7f26080a4400 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f260806be80 tx=0x7f25fc02a150 comp rx=0 tx=0).stop 2024-09-16T04:47:45.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.634+0000 7f260e5d8640 1 -- 172.21.15.46:0/2266452593 shutdown_connections 2024-09-16T04:47:45.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.634+0000 7f260e5d8640 1 --2- 172.21.15.46:0/2266452593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26080a3ff0 0x7f26080a4400 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.634+0000 7f260e5d8640 1 --2- 172.21.15.46:0/2266452593 >> v2:172.21.15.114:3300/0 conn(0x7f26080a3c10 0x7f260809bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.634+0000 7f260e5d8640 1 -- 172.21.15.46:0/2266452593 >> 172.21.15.46:0/2266452593 conn(0x7f26080a5d10 msgr2=0x7f26080a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:45.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.634+0000 7f260e5d8640 1 -- 172.21.15.46:0/2266452593 shutdown_connections 2024-09-16T04:47:45.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.634+0000 7f260e5d8640 1 -- 172.21.15.46:0/2266452593 wait complete. 2024-09-16T04:47:45.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.635+0000 7f260e5d8640 10 monclient: build_initial_monmap 2024-09-16T04:47:45.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.635+0000 7f260e5d8640 10 monclient: monmap: 2024-09-16T04:47:45.652 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:45.652 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:45.652 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:45.652 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:45.652 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:45.652 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.652 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:45.652 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:45.653 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.635+0000 7f260e5d8640 1 Processor -- start 2024-09-16T04:47:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.635+0000 7f260e5d8640 1 -- start start 2024-09-16T04:47:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.635+0000 7f260e5d8640 10 monclient: init 2024-09-16T04:47:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260e5d8640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260e5d8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:47:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260e5d8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26080a3c10 0x7f26081cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260e5d8640 10 monclient(hunting): picked mon.noname-b con 0x7f26080a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260e5d8640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f26080a3ff0 0x7f26081cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260e5d8640 10 monclient(hunting): picked mon.noname-c con 0x7f26080a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260e5d8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26081369d0 0x7f2608123040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260e5d8640 10 monclient(hunting): picked mon.noname-a con 0x7f26081369d0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:45.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260e5d8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260e5d8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f260806e3d0 con 0x7f26081369d0 2024-09-16T04:47:45.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260e5d8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260e5d8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f26080a4940 con 0x7f26080a3c10 2024-09-16T04:47:45.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260e5d8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260e5d8640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f260806e550 con 0x7f26080a3ff0 2024-09-16T04:47:45.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260e5d8640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:45.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260e5d8640 10 monclient(hunting): authenticate will time out at 1264.052262s 2024-09-16T04:47:45.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f2607fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26080a3c10 0x7f26081cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260cb75640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26081369d0 0x7f2608123040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f26077fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f26080a3ff0 0x7f26081cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f2607fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26080a3c10 0x7f26081cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.46:44518/0 (socket says 172.21.15.46:44518) 2024-09-16T04:47:45.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f26077fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f26080a3ff0 0x7f26081cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:41778/0 (socket says 172.21.15.46:41778) 2024-09-16T04:47:45.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f2607fff640 1 -- 172.21.15.46:0/2662668369 learned_addr learned my addr 172.21.15.46:0/2662668369 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:45.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260cb75640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26081369d0 0x7f2608123040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35876/0 (socket says 172.21.15.46:35876) 2024-09-16T04:47:45.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260cb75640 10 monclient(hunting): get_auth_request con 0x7f26081369d0 auth_method 0 2024-09-16T04:47:45.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260cb75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260cb75640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260cb75640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:45.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f2607fff640 10 monclient(hunting): get_auth_request con 0x7f26080a3c10 auth_method 0 2024-09-16T04:47:45.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f2607fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f2607fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f2607fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:45.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f26077fe640 10 monclient(hunting): get_auth_request con 0x7f26080a3ff0 auth_method 0 2024-09-16T04:47:45.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f26077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f26077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f26077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:45.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260cb75640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:45.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260cb75640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:45.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260cb75640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:45.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f2607fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:45.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f2607fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:45.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f2607fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:45.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f26077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:45.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f26077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:45.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f26077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:45.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260cb75640 10 monclient(hunting): handle_auth_done global_id 104382 payload 824 2024-09-16T04:47:45.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260cb75640 1 -- 172.21.15.46:0/2662668369 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f26080a3ff0 msgr2=0x7f26081cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260cb75640 1 --2- 172.21.15.46:0/2662668369 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f26080a3ff0 0x7f26081cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260cb75640 1 -- 172.21.15.46:0/2662668369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26080a3c10 msgr2=0x7f26081cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260cb75640 1 --2- 172.21.15.46:0/2662668369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26080a3c10 0x7f26081cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260cb75640 10 monclient: _finish_hunting 0 2024-09-16T04:47:45.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260cb75640 1 monclient: found mon.noname-a 2024-09-16T04:47:45.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260cb75640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:45.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260cb75640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:45.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260cb75640 1 -- 172.21.15.46:0/2662668369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2608125840 con 0x7f26081369d0 2024-09-16T04:47:45.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f2607fff640 1 --2- 172.21.15.46:0/2662668369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26080a3c10 0x7f26081cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.636+0000 7f260cb75640 1 --2- 172.21.15.46:0/2662668369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26081369d0 0x7f2608123040 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f260000c700 tx=0x7f260000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f2604ff9640 1 -- 172.21.15.46:0/2662668369 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2600003b70 con 0x7f26081369d0 2024-09-16T04:47:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f2604ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f2604ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f2604ff9640 10 monclient: dump: 2024-09-16T04:47:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:45.659 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f2604ff9640 10 monclient: _finish_auth 0 2024-09-16T04:47:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f2604ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:47:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f2604ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f2604ff9640 1 -- 172.21.15.46:0/2662668369 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2600003d10 con 0x7f26081369d0 2024-09-16T04:47:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f260e5d8640 5 monclient: authenticate success, global_id 104382 2024-09-16T04:47:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f260e5d8640 10 monclient: _renew_subs 2024-09-16T04:47:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f260e5d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f260e5d8640 1 -- 172.21.15.46:0/2662668369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2608126cf0 con 0x7f26081369d0 2024-09-16T04:47:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f2604ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:45.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f2604ff9640 1 -- 172.21.15.46:0/2662668369 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2600005310 con 0x7f26081369d0 2024-09-16T04:47:45.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f2604ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:45.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f2604ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:45.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f2604ff9640 10 monclient: dump: 2024-09-16T04:47:45.660 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:45.660 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:45.660 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.660 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.660 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:45.660 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.661 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:45.661 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:45.661 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:45.661 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f260e5d8640 10 monclient: _renew_subs 2024-09-16T04:47:45.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f260e5d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:45.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f260e5d8640 1 -- 172.21.15.46:0/2662668369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2608126a10 con 0x7f26081369d0 2024-09-16T04:47:45.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f26067fc640 4 set_mon_vals no callback set 2024-09-16T04:47:45.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.637+0000 7f260e5d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:45.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.638+0000 7f260e5d8640 1 -- 172.21.15.46:0/2662668369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f260809bcf0 con 0x7f26081369d0 2024-09-16T04:47:45.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.638+0000 7f2604ff9640 1 -- 172.21.15.46:0/2662668369 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f26000054b0 con 0x7f26081369d0 2024-09-16T04:47:45.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.638+0000 7f2604ff9640 1 --2- 172.21.15.46:0/2662668369 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f25f40383e0 0x7f25f403a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.638+0000 7f2604ff9640 1 -- 172.21.15.46:0/2662668369 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7f260004b2d0 con 0x7f26081369d0 2024-09-16T04:47:45.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.638+0000 7f2604ff9640 1 --2- 172.21.15.46:0/2662668369 >> [v2:172.21.15.114:6802/936316557,v1:172.21.15.114:6803/936316557] conn(0x7f25f403e250 0x7f25f4040670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.638+0000 7f2604ff9640 1 -- 172.21.15.46:0/2662668369 --> [v2:172.21.15.114:6802/936316557,v1:172.21.15.114:6803/936316557] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f260001de70 con 0x7f25f403e250 2024-09-16T04:47:45.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.638+0000 7f2604ff9640 1 -- 172.21.15.46:0/2662668369 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=315) ==== 24+0+0 (secure 0 0 0) 0x7f2600052050 con 0x7f26081369d0 2024-09-16T04:47:45.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.638+0000 7f2604ff9640 10 monclient: handle_get_version_reply finishing 1 version 315 2024-09-16T04:47:45.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.638+0000 7f2607fff640 1 --2- 172.21.15.46:0/2662668369 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f25f40383e0 0x7f25f403a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.638+0000 7f2607fff640 10 monclient: get_auth_request con 0x7f25f40383e0 auth_method 0 2024-09-16T04:47:45.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.638+0000 7f26077fe640 1 --2- 172.21.15.46:0/2662668369 >> [v2:172.21.15.114:6802/936316557,v1:172.21.15.114:6803/936316557] conn(0x7f25f403e250 0x7f25f4040670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.638+0000 7f26077fe640 10 monclient: get_auth_request con 0x7f25f403e250 auth_method 0 2024-09-16T04:47:45.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.639+0000 7f2607fff640 1 --2- 172.21.15.46:0/2662668369 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f25f40383e0 0x7f25f403a8a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f25f8006d40 tx=0x7f25f8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:45.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.639+0000 7f26077fe640 1 --2- 172.21.15.46:0/2662668369 >> [v2:172.21.15.114:6802/936316557,v1:172.21.15.114:6803/936316557] conn(0x7f25f403e250 0x7f25f4040670 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:45.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.639+0000 7f2604ff9640 1 -- 172.21.15.46:0/2662668369 <== osd.4 v2:172.21.15.114:6802/936316557 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f260001de70 con 0x7f25f403e250 2024-09-16T04:47:45.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.652+0000 7f260e5d8640 1 -- 172.21.15.46:0/2662668369 --> [v2:172.21.15.114:6802/936316557,v1:172.21.15.114:6803/936316557] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f2608067e40 con 0x7f25f403e250 2024-09-16T04:47:45.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.653+0000 7f2604ff9640 1 -- 172.21.15.46:0/2662668369 <== osd.4 v2:172.21.15.114:6802/936316557 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f2608067e40 con 0x7f25f403e250 2024-09-16T04:47:45.663 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:47:45.664 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:47:45.664 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:47:45.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.653+0000 7f260e5d8640 1 -- 172.21.15.46:0/2662668369 >> [v2:172.21.15.114:6802/936316557,v1:172.21.15.114:6803/936316557] conn(0x7f25f403e250 msgr2=0x7f25f4040670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.653+0000 7f260e5d8640 1 --2- 172.21.15.46:0/2662668369 >> [v2:172.21.15.114:6802/936316557,v1:172.21.15.114:6803/936316557] conn(0x7f25f403e250 0x7f25f4040670 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.653+0000 7f260e5d8640 1 -- 172.21.15.46:0/2662668369 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f25f40383e0 msgr2=0x7f25f403a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.653+0000 7f260e5d8640 1 --2- 172.21.15.46:0/2662668369 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f25f40383e0 0x7f25f403a8a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f25f8006d40 tx=0x7f25f8008040 comp rx=0 tx=0).stop 2024-09-16T04:47:45.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.653+0000 7f260e5d8640 10 monclient: shutdown 2024-09-16T04:47:45.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.653+0000 7f260e5d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:45.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.653+0000 7f260e5d8640 1 -- 172.21.15.46:0/2662668369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26081369d0 msgr2=0x7f2608123040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.653+0000 7f260e5d8640 1 --2- 172.21.15.46:0/2662668369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26081369d0 0x7f2608123040 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f260000c700 tx=0x7f260000cbc0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.653+0000 7f260cb75640 1 -- 172.21.15.46:0/2662668369 reap_dead start 2024-09-16T04:47:45.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.653+0000 7f260e5d8640 1 -- 172.21.15.46:0/2662668369 shutdown_connections 2024-09-16T04:47:45.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.653+0000 7f260e5d8640 1 -- 172.21.15.46:0/2662668369 >> 172.21.15.46:0/2662668369 conn(0x7f26080a5d10 msgr2=0x7f26080a60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:45.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.653+0000 7f260e5d8640 1 -- 172.21.15.46:0/2662668369 shutdown_connections 2024-09-16T04:47:45.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.653+0000 7f260e5d8640 1 -- 172.21.15.46:0/2662668369 wait complete. 2024-09-16T04:47:45.669 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T04:47:45.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.768+0000 7f85064d0640 10 monclient: get_monmap_and_config 2024-09-16T04:47:45.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.768+0000 7f85064d0640 10 monclient: build_initial_monmap 2024-09-16T04:47:45.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.768+0000 7f85064d0640 10 monclient: monmap: 2024-09-16T04:47:45.771 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:45.771 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:45.771 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:45.770394+0000 2024-09-16T04:47:45.771 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:45.770394+0000 2024-09-16T04:47:45.771 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:45.771 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.771 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:45.771 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:45.771 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:45.771 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:45.771 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:45.771 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.769+0000 7f85064d0640 1 Processor -- start 2024-09-16T04:47:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.769+0000 7f85064d0640 1 -- start start 2024-09-16T04:47:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.769+0000 7f85064d0640 10 monclient: init 2024-09-16T04:47:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f85064d0640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f85064d0640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-16T04:47:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f85064d0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f85000a3c10 0x7f850009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f85064d0640 10 monclient(hunting): picked mon.noname-a con 0x7f85000a3c10 addr v2:172.21.15.46:3300/0 2024-09-16T04:47:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f85064d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f85000a3ff0 addr v1:172.21.15.46:6789/0 2024-09-16T04:47:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f85064d0640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f850013be90 0x7f850013e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f85064d0640 10 monclient(hunting): picked mon.noname-b con 0x7f850013be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f85064d0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f850009b940 con 0x7f85000a3ff0 2024-09-16T04:47:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f85064d0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f85064d0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f850009ab00 con 0x7f85000a3c10 2024-09-16T04:47:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f85064d0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f85064d0640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f850009b030 con 0x7f850013be90 2024-09-16T04:47:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f85064d0640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f85064d0640 10 monclient(hunting): authenticate will time out at 1264.186236s 2024-09-16T04:47:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f84fffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f85000a3c10 0x7f850009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f8504a6d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f850013be90 0x7f850013e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f84ff7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f85000a3ff0 0x7f85000a43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47608/0 (socket says 172.21.15.46:47608) 2024-09-16T04:47:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f84ff7fe640 1 -- 172.21.15.46:0/3163784478 learned_addr learned my addr 172.21.15.46:0/3163784478 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:47:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f84fffff640 10 monclient(hunting): get_auth_request con 0x7f85000a3c10 auth_method 0 2024-09-16T04:47:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f84fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f84fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f84fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f8504a6d640 10 monclient(hunting): get_auth_request con 0x7f850013be90 auth_method 0 2024-09-16T04:47:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f8504a6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f8504a6d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f8504a6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f84fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f84fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f84fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f8504a6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f8504a6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f8504a6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f84fffff640 10 monclient(hunting): handle_auth_done global_id 104385 payload 274 2024-09-16T04:47:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f84fffff640 1 -- 172.21.15.46:0/3163784478 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f850013be90 msgr2=0x7f850013e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f84fffff640 1 --2- 172.21.15.46:0/3163784478 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f850013be90 0x7f850013e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f84fffff640 1 -- 172.21.15.46:0/3163784478 >> v1:172.21.15.46:6789/0 conn(0x7f85000a3ff0 legacy=0x7f85000a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f84fffff640 10 monclient: _finish_hunting 0 2024-09-16T04:47:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f84fffff640 1 monclient: found mon.noname-a 2024-09-16T04:47:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f84fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f84fffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f84fffff640 1 -- 172.21.15.46:0/3163784478 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f850009b310 con 0x7f85000a3c10 2024-09-16T04:47:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f84feffd640 1 -- 172.21.15.46:0/3163784478 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 509983478 0 0) 0x7f850009b940 con 0x7f85000a3ff0 2024-09-16T04:47:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f84feffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:47:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.770+0000 7f8504a6d640 1 --2- 172.21.15.46:0/3163784478 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f850013be90 0x7f850013e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.771+0000 7f84fffff640 1 --2- 172.21.15.46:0/3163784478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85000a3c10 0x7f850009bcf0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f84f00097a0 tx=0x7f84f002a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=906c88a4494e0cbd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.771+0000 7f84feffd640 1 -- 172.21.15.46:0/3163784478 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f0037070 con 0x7f85000a3c10 2024-09-16T04:47:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.771+0000 7f84feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.771+0000 7f84feffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.771+0000 7f84feffd640 10 monclient: dump: 2024-09-16T04:47:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:45.783 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.771+0000 7f84feffd640 10 monclient: _finish_auth 0 2024-09-16T04:47:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.771+0000 7f84feffd640 10 monclient: _check_auth_tickets 2024-09-16T04:47:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.771+0000 7f84feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.771+0000 7f84feffd640 1 -- 172.21.15.46:0/3163784478 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84f002adb0 con 0x7f85000a3c10 2024-09-16T04:47:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.771+0000 7f84feffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.771+0000 7f84feffd640 1 -- 172.21.15.46:0/3163784478 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f0032660 con 0x7f85000a3c10 2024-09-16T04:47:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.771+0000 7f85064d0640 5 monclient: authenticate success, global_id 104385 2024-09-16T04:47:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.771+0000 7f85064d0640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.771+0000 7f85064d0640 4 set_mon_vals no callback set 2024-09-16T04:47:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.771+0000 7f84feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.771+0000 7f84feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.771+0000 7f84feffd640 10 monclient: dump: 2024-09-16T04:47:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:45.785 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.772+0000 7f85064d0640 10 monclient: shutdown 2024-09-16T04:47:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.772+0000 7f85064d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.772+0000 7f85064d0640 1 -- 172.21.15.46:0/3163784478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85000a3c10 msgr2=0x7f850009bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.772+0000 7f85064d0640 1 --2- 172.21.15.46:0/3163784478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85000a3c10 0x7f850009bcf0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f84f00097a0 tx=0x7f84f002a060 comp rx=0 tx=0).stop 2024-09-16T04:47:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.772+0000 7f85064d0640 1 -- 172.21.15.46:0/3163784478 shutdown_connections 2024-09-16T04:47:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.772+0000 7f85064d0640 1 --2- 172.21.15.46:0/3163784478 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f850013be90 0x7f850013e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.772+0000 7f85064d0640 1 --2- 172.21.15.46:0/3163784478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85000a3c10 0x7f850009bcf0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.772+0000 7f85064d0640 1 -- 172.21.15.46:0/3163784478 >> 172.21.15.46:0/3163784478 conn(0x7f85000a6340 msgr2=0x7f850006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.772+0000 7f85064d0640 1 -- 172.21.15.46:0/3163784478 shutdown_connections 2024-09-16T04:47:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.772+0000 7f85064d0640 1 -- 172.21.15.46:0/3163784478 wait complete. 2024-09-16T04:47:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.772+0000 7f85064d0640 10 monclient: build_initial_monmap 2024-09-16T04:47:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.772+0000 7f85064d0640 10 monclient: monmap: 2024-09-16T04:47:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:45.787 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:45.787 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.773+0000 7f85064d0640 1 Processor -- start 2024-09-16T04:47:45.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.773+0000 7f85064d0640 1 -- start start 2024-09-16T04:47:45.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.773+0000 7f85064d0640 10 monclient: init 2024-09-16T04:47:45.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.773+0000 7f85064d0640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:45.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.773+0000 7f85064d0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:47:45.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.773+0000 7f85064d0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85000a3c10 0x7f85001cb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.773+0000 7f85064d0640 10 monclient(hunting): picked mon.noname-b con 0x7f85000a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:45.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.773+0000 7f85064d0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85000a3ff0 0x7f8500136ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.773+0000 7f85064d0640 10 monclient(hunting): picked mon.noname-a con 0x7f85000a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:45.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.773+0000 7f85064d0640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f850013be90 0x7f8500139d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.773+0000 7f85064d0640 10 monclient(hunting): picked mon.noname-c con 0x7f850013be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:45.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.773+0000 7f85064d0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.773+0000 7f85064d0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f85000a6b40 con 0x7f85000a3ff0 2024-09-16T04:47:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.773+0000 7f85064d0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.773+0000 7f85064d0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f85000a69c0 con 0x7f85000a3c10 2024-09-16T04:47:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.773+0000 7f85064d0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.773+0000 7f85064d0640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f850013e7c0 con 0x7f850013be90 2024-09-16T04:47:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.773+0000 7f85064d0640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.773+0000 7f85064d0640 10 monclient(hunting): authenticate will time out at 1264.189909s 2024-09-16T04:47:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.773+0000 7f84ff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85000a3ff0 0x7f8500136ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f8504a6d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f850013be90 0x7f8500139d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85000a3ff0 0x7f8500136ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35892/0 (socket says 172.21.15.46:35892) 2024-09-16T04:47:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 1 -- 172.21.15.46:0/257812528 learned_addr learned my addr 172.21.15.46:0/257812528 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 10 monclient(hunting): get_auth_request con 0x7f85000a3ff0 auth_method 0 2024-09-16T04:47:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:45.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f8504a6d640 10 monclient(hunting): get_auth_request con 0x7f850013be90 auth_method 0 2024-09-16T04:47:45.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f8504a6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f8504a6d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f8504a6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:45.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:45.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:45.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:45.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f8504a6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:45.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f8504a6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:45.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f8504a6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:45.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84fffff640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85000a3c10 0x7f85001cb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84fffff640 10 monclient(hunting): get_auth_request con 0x7f85000a3c10 auth_method 0 2024-09-16T04:47:45.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:45.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 10 monclient(hunting): handle_auth_done global_id 104391 payload 824 2024-09-16T04:47:45.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 1 -- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f850013be90 msgr2=0x7f8500139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f850013be90 0x7f8500139d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 1 -- 172.21.15.46:0/257812528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85000a3c10 msgr2=0x7f85001cb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85000a3c10 0x7f85001cb860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:47:45.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 1 monclient: found mon.noname-a 2024-09-16T04:47:45.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:45.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:45.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 1 -- 172.21.15.46:0/257812528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8500121390 con 0x7f85000a3ff0 2024-09-16T04:47:45.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f8504a6d640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f850013be90 0x7f8500139d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:45.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 1 -- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85000a3ff0 msgr2=0x7f8500136ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:47:45.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 1 -- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85000a3ff0 msgr2=0x7f8500136ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:47:45.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85000a3ff0 0x7f8500136ce0 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f84f400c4d0 tx=0x7f84f400c990 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:47:45.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.774+0000 7f84ff7fe640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85000a3ff0 0x7f8500136ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:47:45.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84ff7fe640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85000a3ff0 0x7f8500136ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84ff7fe640 10 monclient: get_auth_request con 0x7f85000a3ff0 auth_method 0 2024-09-16T04:47:45.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84ff7fe640 0 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85000a3ff0 0x7f8500136ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T04:47:45.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84ff7fe640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85000a3ff0 0x7f8500136ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fcff9640 10 monclient: ms_handle_reset current mon [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:45.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fcff9640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:45.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fcff9640 1 -- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85000a3ff0 msgr2=0x7f8500136ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:47:45.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fcff9640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85000a3ff0 0x7f8500136ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fcff9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:47:45.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fcff9640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f850013be90 0x7f84ec0054e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fcff9640 10 monclient(hunting): picked mon.noname-c con 0x7f850013be90 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:45.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fcff9640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85000a3c10 0x7f84ec007f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fcff9640 10 monclient(hunting): picked mon.noname-b con 0x7f85000a3c10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:45.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fcff9640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84ec0089f0 0x7f84ec00ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fcff9640 10 monclient(hunting): picked mon.noname-a con 0x7f84ec0089f0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:45.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fcff9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fcff9640 1 -- 172.21.15.46:0/257812528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec00b980 con 0x7f84ec0089f0 2024-09-16T04:47:45.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fcff9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fcff9640 1 -- 172.21.15.46:0/257812528 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec00bb30 con 0x7f85000a3c10 2024-09-16T04:47:45.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fcff9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:45.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fcff9640 1 -- 172.21.15.46:0/257812528 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f84ec00bd30 con 0x7f850013be90 2024-09-16T04:47:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fcff9640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fffff640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f850013be90 0x7f84ec0054e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fffff640 10 monclient(hunting): get_auth_request con 0x7f850013be90 auth_method 0 2024-09-16T04:47:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f8504a6d640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84ec0089f0 0x7f84ec00ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:47:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f8504a6d640 10 monclient(hunting): get_auth_request con 0x7f84ec0089f0 auth_method 0 2024-09-16T04:47:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f8504a6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f8504a6d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f8504a6d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:47:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:45.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:45.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:47:45.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f8504a6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:45.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f8504a6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:45.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84ff7fe640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85000a3c10 0x7f84ec007f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f8504a6d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:47:45.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84ff7fe640 10 monclient(hunting): get_auth_request con 0x7f85000a3c10 auth_method 0 2024-09-16T04:47:45.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:45.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:45.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84ff7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:47:45.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:45.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:45.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:47:45.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fffff640 10 monclient(hunting): handle_auth_done global_id 104391 payload 293 2024-09-16T04:47:45.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fffff640 1 -- 172.21.15.46:0/257812528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85000a3c10 msgr2=0x7f84ec007f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fffff640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85000a3c10 0x7f84ec007f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fffff640 1 -- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84ec0089f0 msgr2=0x7f84ec00ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:45.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fffff640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84ec0089f0 0x7f84ec00ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:45.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fffff640 10 monclient: _finish_hunting 0 2024-09-16T04:47:45.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fffff640 1 monclient: found mon.noname-c 2024-09-16T04:47:45.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:45.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:47:45.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f84fffff640 1 -- 172.21.15.46:0/257812528 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84ec00bff0 con 0x7f850013be90 2024-09-16T04:47:45.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.975+0000 7f8504a6d640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84ec0089f0 0x7f84ec00ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:45.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f84fffff640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f850013be90 0x7f84ec0054e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f84f0031000 tx=0x7f84f4007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:45.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f84fcff9640 1 -- 172.21.15.46:0/257812528 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f0037040 con 0x7f850013be90 2024-09-16T04:47:45.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f84fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:45.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f84fcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:47:45.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f84fcff9640 10 monclient: dump: 2024-09-16T04:47:45.986 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:45.986 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:45.986 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.986 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.986 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:45.986 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.986 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:45.986 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:45.986 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:45.986 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f84fcff9640 10 monclient: _finish_auth 0 2024-09-16T04:47:45.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f84fcff9640 10 monclient: _check_auth_tickets 2024-09-16T04:47:45.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f84fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:45.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f84fcff9640 1 -- 172.21.15.46:0/257812528 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84f0032800 con 0x7f850013be90 2024-09-16T04:47:45.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f84fcff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:45.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f84fcff9640 1 -- 172.21.15.46:0/257812528 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f0032bd0 con 0x7f850013be90 2024-09-16T04:47:45.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f84fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:45.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f84fcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:45.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f84fcff9640 10 monclient: dump: 2024-09-16T04:47:45.988 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:45.988 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:45.988 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.988 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:45.988 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:45.988 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:45.988 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:45.988 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:45.989 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:45.989 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:45.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f85064d0640 5 monclient: authenticate success, global_id 104391 2024-09-16T04:47:45.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f85064d0640 10 monclient: _renew_subs 2024-09-16T04:47:45.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f85064d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:45.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f85064d0640 1 -- 172.21.15.46:0/257812528 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85001210b0 con 0x7f850013be90 2024-09-16T04:47:45.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f84fe7fc640 4 set_mon_vals no callback set 2024-09-16T04:47:45.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f85064d0640 10 monclient: _renew_subs 2024-09-16T04:47:45.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f85064d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:45.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.976+0000 7f85064d0640 1 -- 172.21.15.46:0/257812528 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f850011fa60 con 0x7f850013be90 2024-09-16T04:47:45.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.977+0000 7f84fcff9640 1 -- 172.21.15.46:0/257812528 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f84f003a040 con 0x7f850013be90 2024-09-16T04:47:45.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.977+0000 7f85064d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:45.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.977+0000 7f85064d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:45.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.977+0000 7f85064d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:45.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.977+0000 7f85064d0640 1 -- 172.21.15.46:0/257812528 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f850009bcf0 con 0x7f850013be90 2024-09-16T04:47:45.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.977+0000 7f84fcff9640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f85000a3ff0 0x7f84ec042a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:45.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.977+0000 7f84fcff9640 1 -- 172.21.15.46:0/257812528 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7f84f006aac0 con 0x7f850013be90 2024-09-16T04:47:45.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.977+0000 7f84ff7fe640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f85000a3ff0 0x7f84ec042a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:45.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.977+0000 7f84ff7fe640 10 monclient: get_auth_request con 0x7f85000a3ff0 auth_method 0 2024-09-16T04:47:45.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.978+0000 7f84ff7fe640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f85000a3ff0 0x7f84ec042a10 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f84f4001bc0 tx=0x7f84f400c990 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:45.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.982+0000 7f84fcff9640 1 -- 172.21.15.46:0/257812528 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f84f004d020 con 0x7f850013be90 2024-09-16T04:47:45.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.982+0000 7f84fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:45.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:45.982+0000 7f84fcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:46.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.156+0000 7f85064d0640 1 -- 172.21.15.46:0/257812528 --> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f8500120dd0 con 0x7f85000a3ff0 2024-09-16T04:47:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.157+0000 7f84fcff9640 1 -- 172.21.15.46:0/257812528 <== mgr.4099 v2:172.21.15.114:6800/3521677055 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f84f4004d20 con 0x7f85000a3ff0 2024-09-16T04:47:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T04:47:46.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.159+0000 7f85064d0640 1 -- 172.21.15.46:0/257812528 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f85000a3ff0 msgr2=0x7f84ec042a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:46.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.159+0000 7f85064d0640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f85000a3ff0 0x7f84ec042a10 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f84f4001bc0 tx=0x7f84f400c990 comp rx=0 tx=0).stop 2024-09-16T04:47:46.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.159+0000 7f85064d0640 10 monclient: shutdown 2024-09-16T04:47:46.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.159+0000 7f85064d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:46.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.159+0000 7f85064d0640 1 -- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f850013be90 msgr2=0x7f84ec0054e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:46.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.159+0000 7f85064d0640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f850013be90 0x7f84ec0054e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f84f0031000 tx=0x7f84f4007e10 comp rx=0 tx=0).stop 2024-09-16T04:47:46.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.159+0000 7f85064d0640 1 -- 172.21.15.46:0/257812528 shutdown_connections 2024-09-16T04:47:46.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.159+0000 7f85064d0640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f85000a3ff0 0x7f84ec042a10 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.159+0000 7f85064d0640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84ec0089f0 0x7f84ec00ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.159+0000 7f85064d0640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85000a3c10 0x7f84ec007f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.159+0000 7f85064d0640 1 --2- 172.21.15.46:0/257812528 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f850013be90 0x7f84ec0054e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.159+0000 7f85064d0640 1 -- 172.21.15.46:0/257812528 >> 172.21.15.46:0/257812528 conn(0x7f85000a6340 msgr2=0x7f85000a2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:46.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.159+0000 7f85064d0640 1 -- 172.21.15.46:0/257812528 shutdown_connections 2024-09-16T04:47:46.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.159+0000 7f85064d0640 1 -- 172.21.15.46:0/257812528 wait complete. 2024-09-16T04:47:46.172 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T04:47:46.172 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:47:46.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.275+0000 7f94ddb31640 10 monclient: get_monmap_and_config 2024-09-16T04:47:46.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.275+0000 7f94ddb31640 10 monclient: build_initial_monmap 2024-09-16T04:47:46.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.275+0000 7f94ddb31640 10 monclient: monmap: 2024-09-16T04:47:46.278 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:46.278 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:46.278 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:46.277361+0000 2024-09-16T04:47:46.278 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:46.277361+0000 2024-09-16T04:47:46.278 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:46.278 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:46.278 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:46.279 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:46.279 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:46.279 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:46.279 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:46.279 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:46.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.276+0000 7f94ddb31640 1 Processor -- start 2024-09-16T04:47:46.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.276+0000 7f94ddb31640 1 -- start start 2024-09-16T04:47:46.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.276+0000 7f94ddb31640 10 monclient: init 2024-09-16T04:47:46.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94ddb31640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:46.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94ddb31640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-16T04:47:46.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94ddb31640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f94d80a3c10 addr v1:172.21.15.114:6789/0 2024-09-16T04:47:46.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94ddb31640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f94d80a3ff0 0x7f94d80a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:46.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94ddb31640 10 monclient(hunting): picked mon.noname-b con 0x7f94d80a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:46.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94ddb31640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f94d813be90 addr v1:172.21.15.46:6789/0 2024-09-16T04:47:46.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94ddb31640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94d809b940 con 0x7f94d813be90 2024-09-16T04:47:46.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94ddb31640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94d809ab00 con 0x7f94d80a3c10 2024-09-16T04:47:46.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94ddb31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:46.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94ddb31640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f94d809b030 con 0x7f94d80a3ff0 2024-09-16T04:47:46.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94ddb31640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:46.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94ddb31640 10 monclient(hunting): authenticate will time out at 1264.693333s 2024-09-16T04:47:46.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94d7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f94d813be90 0x7f94d813e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47610/0 (socket says 172.21.15.46:47610) 2024-09-16T04:47:46.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94d6ffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f94d80a3ff0 0x7f94d80a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94d7fff640 1 -- 172.21.15.46:0/1140574452 learned_addr learned my addr 172.21.15.46:0/1140574452 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:47:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94d6ffd640 10 monclient(hunting): get_auth_request con 0x7f94d80a3ff0 auth_method 0 2024-09-16T04:47:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94d6ffd640 10 monclient(hunting): handle_auth_done global_id 94511 payload 274 2024-09-16T04:47:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94d67fc640 1 -- 172.21.15.46:0/1140574452 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2688358214 0 0) 0x7f94d809b940 con 0x7f94d813be90 2024-09-16T04:47:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94d6ffd640 1 -- 172.21.15.46:0/1140574452 >> v1:172.21.15.114:6789/0 conn(0x7f94d80a3c10 legacy=0x7f94d809bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:46.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94d6ffd640 1 -- 172.21.15.46:0/1140574452 >> v1:172.21.15.46:6789/0 conn(0x7f94d813be90 legacy=0x7f94d813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:46.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94d6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:47:46.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94d6ffd640 1 monclient: found mon.noname-b 2024-09-16T04:47:46.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:46.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94d6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:47:46.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94d6ffd640 1 -- 172.21.15.46:0/1140574452 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94d809b310 con 0x7f94d80a3ff0 2024-09-16T04:47:46.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.277+0000 7f94d67fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:47:46.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.278+0000 7f94d6ffd640 1 --2- 172.21.15.46:0/1140574452 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f94d80a3ff0 0x7f94d80a43d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f94cc0097d0 tx=0x7f94cc02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4298ef2b82d25c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:46.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.278+0000 7f94d67fc640 1 -- 172.21.15.46:0/1140574452 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94cc037070 con 0x7f94d80a3ff0 2024-09-16T04:47:46.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.278+0000 7f94d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:46.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.278+0000 7f94d67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:47:46.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.278+0000 7f94d67fc640 10 monclient: dump: 2024-09-16T04:47:46.293 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:46.294 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:46.294 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.294 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.294 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:46.294 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:46.294 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:46.294 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:46.294 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:46.294 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:46.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.278+0000 7f94d67fc640 10 monclient: _finish_auth 0 2024-09-16T04:47:46.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.278+0000 7f94d67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:47:46.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.278+0000 7f94d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:46.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.278+0000 7f94d67fc640 1 -- 172.21.15.46:0/1140574452 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94cc032440 con 0x7f94d80a3ff0 2024-09-16T04:47:46.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.278+0000 7f94ddb31640 5 monclient: authenticate success, global_id 94511 2024-09-16T04:47:46.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.278+0000 7f94ddb31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:46.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.278+0000 7f94d67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:46.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.278+0000 7f94ddb31640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:46.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.278+0000 7f94ddb31640 4 set_mon_vals no callback set 2024-09-16T04:47:46.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.278+0000 7f94d67fc640 1 -- 172.21.15.46:0/1140574452 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94cc032810 con 0x7f94d80a3ff0 2024-09-16T04:47:46.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.278+0000 7f94d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:46.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.278+0000 7f94d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:46.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.278+0000 7f94d67fc640 10 monclient: dump: 2024-09-16T04:47:46.295 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:46.295 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:46.296 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.296 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.296 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:46.296 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:46.296 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:46.296 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:46.296 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:46.296 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:46.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.279+0000 7f94ddb31640 10 monclient: shutdown 2024-09-16T04:47:46.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.279+0000 7f94ddb31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:46.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.279+0000 7f94ddb31640 1 -- 172.21.15.46:0/1140574452 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f94d80a3ff0 msgr2=0x7f94d80a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:46.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.279+0000 7f94ddb31640 1 --2- 172.21.15.46:0/1140574452 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f94d80a3ff0 0x7f94d80a43d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f94cc0097d0 tx=0x7f94cc02a180 comp rx=0 tx=0).stop 2024-09-16T04:47:46.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.279+0000 7f94ddb31640 1 -- 172.21.15.46:0/1140574452 shutdown_connections 2024-09-16T04:47:46.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.279+0000 7f94ddb31640 1 --2- 172.21.15.46:0/1140574452 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f94d80a3ff0 0x7f94d80a43d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.279+0000 7f94ddb31640 1 -- 172.21.15.46:0/1140574452 >> 172.21.15.46:0/1140574452 conn(0x7f94d80a6340 msgr2=0x7f94d806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:46.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.279+0000 7f94ddb31640 1 -- 172.21.15.46:0/1140574452 shutdown_connections 2024-09-16T04:47:46.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.279+0000 7f94ddb31640 1 -- 172.21.15.46:0/1140574452 wait complete. 2024-09-16T04:47:46.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.279+0000 7f94ddb31640 10 monclient: build_initial_monmap 2024-09-16T04:47:46.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.279+0000 7f94ddb31640 10 monclient: monmap: 2024-09-16T04:47:46.297 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:46.298 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:46.298 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:46.298 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:46.298 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:46.298 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:46.298 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:46.298 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:46.298 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:46.298 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:46.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94ddb31640 1 Processor -- start 2024-09-16T04:47:46.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94ddb31640 1 -- start start 2024-09-16T04:47:46.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94ddb31640 10 monclient: init 2024-09-16T04:47:46.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94ddb31640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:46.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94ddb31640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:47:46.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94ddb31640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f94d80a3c10 0x7f94d8136ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:46.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94ddb31640 10 monclient(hunting): picked mon.noname-c con 0x7f94d80a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:46.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94ddb31640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94d80a3ff0 0x7f94d8137420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:46.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94ddb31640 10 monclient(hunting): picked mon.noname-b con 0x7f94d80a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:46.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94ddb31640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94d813be90 0x7f94d8138ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:46.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94ddb31640 10 monclient(hunting): picked mon.noname-a con 0x7f94d813be90 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:46.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94ddb31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:46.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94ddb31640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f94d80a6b40 con 0x7f94d813be90 2024-09-16T04:47:46.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94ddb31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:46.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94ddb31640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f94d80a69c0 con 0x7f94d80a3ff0 2024-09-16T04:47:46.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94ddb31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:46.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94ddb31640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f94d813f9a0 con 0x7f94d80a3c10 2024-09-16T04:47:46.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94ddb31640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:46.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94ddb31640 10 monclient(hunting): authenticate will time out at 1264.696738s 2024-09-16T04:47:46.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94d77fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f94d80a3c10 0x7f94d8136ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:46.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94d77fe640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f94d80a3c10 0x7f94d8136ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:41816/0 (socket says 172.21.15.46:41816) 2024-09-16T04:47:46.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94d77fe640 1 -- 172.21.15.46:0/3821818277 learned_addr learned my addr 172.21.15.46:0/3821818277 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:46.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94d77fe640 10 monclient(hunting): get_auth_request con 0x7f94d80a3c10 auth_method 0 2024-09-16T04:47:46.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:46.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:46.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:46.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94d7fff640 1 --2- 172.21.15.46:0/3821818277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94d813be90 0x7f94d8138ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:46.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94d7fff640 10 monclient(hunting): get_auth_request con 0x7f94d813be90 auth_method 0 2024-09-16T04:47:46.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:46.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:46.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:46.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:46.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:46.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:46.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:46.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.280+0000 7f94d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:46.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94d6ffd640 1 --2- 172.21.15.46:0/3821818277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94d80a3ff0 0x7f94d8137420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94d77fe640 10 monclient(hunting): handle_auth_done global_id 94514 payload 824 2024-09-16T04:47:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94d77fe640 1 -- 172.21.15.46:0/3821818277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94d80a3ff0 msgr2=0x7f94d8137420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94d77fe640 1 --2- 172.21.15.46:0/3821818277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94d80a3ff0 0x7f94d8137420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94d77fe640 1 -- 172.21.15.46:0/3821818277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94d813be90 msgr2=0x7f94d8138ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94d77fe640 1 --2- 172.21.15.46:0/3821818277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94d813be90 0x7f94d8138ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94d77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:47:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94d77fe640 1 monclient: found mon.noname-c 2024-09-16T04:47:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:46.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94d77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:47:46.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94d77fe640 1 -- 172.21.15.46:0/3821818277 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94d80a08f0 con 0x7f94d80a3c10 2024-09-16T04:47:46.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94d6ffd640 10 monclient: get_auth_request con 0x7f94d80a3ff0 auth_method 0 2024-09-16T04:47:46.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94d6ffd640 1 --2- 172.21.15.46:0/3821818277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94d80a3ff0 0x7f94d8137420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:47:46.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94d77fe640 1 --2- 172.21.15.46:0/3821818277 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f94d80a3c10 0x7f94d8136ee0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f94c800c450 tx=0x7f94c800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:46.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94bffff640 1 -- 172.21.15.46:0/3821818277 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c80052d0 con 0x7f94d80a3c10 2024-09-16T04:47:46.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:46.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94bffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:47:46.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94bffff640 10 monclient: dump: 2024-09-16T04:47:46.303 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:46.303 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:46.303 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.303 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.304 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:46.304 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:46.304 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:46.304 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:46.304 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:46.304 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:46.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94bffff640 10 monclient: _finish_auth 0 2024-09-16T04:47:46.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94bffff640 10 monclient: _check_auth_tickets 2024-09-16T04:47:46.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:46.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94ddb31640 5 monclient: authenticate success, global_id 94514 2024-09-16T04:47:46.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94bffff640 1 -- 172.21.15.46:0/3821818277 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94c8005470 con 0x7f94d80a3c10 2024-09-16T04:47:46.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94ddb31640 10 monclient: _renew_subs 2024-09-16T04:47:46.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94ddb31640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:46.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94ddb31640 1 -- 172.21.15.46:0/3821818277 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94d813ab70 con 0x7f94d80a3c10 2024-09-16T04:47:46.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94bffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:46.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94bffff640 1 -- 172.21.15.46:0/3821818277 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c8012870 con 0x7f94d80a3c10 2024-09-16T04:47:46.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:46.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:46.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94bffff640 10 monclient: dump: 2024-09-16T04:47:46.305 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:46.305 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:46.306 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.306 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.306 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:46.306 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:46.306 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:46.306 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:46.306 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:46.306 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:46.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94ddb31640 10 monclient: _renew_subs 2024-09-16T04:47:46.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94ddb31640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:46.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94ddb31640 1 -- 172.21.15.46:0/3821818277 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f94d813b8f0 con 0x7f94d80a3c10 2024-09-16T04:47:46.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.281+0000 7f94d5ffb640 4 set_mon_vals no callback set 2024-09-16T04:47:46.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.282+0000 7f94ddb31640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:46.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.282+0000 7f94ddb31640 1 -- 172.21.15.46:0/3821818277 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f94d81cc9c0 con 0x7f94d80a3c10 2024-09-16T04:47:46.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.282+0000 7f94bffff640 1 -- 172.21.15.46:0/3821818277 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f94c8012a10 con 0x7f94d80a3c10 2024-09-16T04:47:46.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.282+0000 7f94bffff640 1 --2- 172.21.15.46:0/3821818277 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f94c40383b0 0x7f94c403a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:46.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.282+0000 7f94bffff640 1 -- 172.21.15.46:0/3821818277 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7f94c804b140 con 0x7f94d80a3c10 2024-09-16T04:47:46.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.282+0000 7f94bffff640 1 --2- 172.21.15.46:0/3821818277 >> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] conn(0x7f94c403e220 0x7f94c4040640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:46.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.282+0000 7f94bffff640 1 -- 172.21.15.46:0/3821818277 --> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f94c4040cf0 con 0x7f94c403e220 2024-09-16T04:47:46.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.282+0000 7f94bffff640 1 -- 172.21.15.46:0/3821818277 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_get_version_reply(handle=1 version=315) ==== 24+0+0 (secure 0 0 0) 0x7f94c8052050 con 0x7f94d80a3c10 2024-09-16T04:47:46.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.282+0000 7f94bffff640 10 monclient: handle_get_version_reply finishing 1 version 315 2024-09-16T04:47:46.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.282+0000 7f94d6ffd640 1 --2- 172.21.15.46:0/3821818277 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f94c40383b0 0x7f94c403a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:46.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.283+0000 7f94d6ffd640 10 monclient: get_auth_request con 0x7f94c40383b0 auth_method 0 2024-09-16T04:47:46.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.283+0000 7f94d7fff640 1 --2- 172.21.15.46:0/3821818277 >> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] conn(0x7f94c403e220 0x7f94c4040640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:46.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.283+0000 7f94d7fff640 10 monclient: get_auth_request con 0x7f94c403e220 auth_method 0 2024-09-16T04:47:46.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.283+0000 7f94d6ffd640 1 --2- 172.21.15.46:0/3821818277 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f94c40383b0 0x7f94c403a870 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f94cc034000 tx=0x7f94cc02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:46.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.283+0000 7f94d7fff640 1 --2- 172.21.15.46:0/3821818277 >> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] conn(0x7f94c403e220 0x7f94c4040640 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:46.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.284+0000 7f94bffff640 1 -- 172.21.15.46:0/3821818277 <== osd.5 v2:172.21.15.114:6818/938868100 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f94c4040cf0 con 0x7f94c403e220 2024-09-16T04:47:46.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.296+0000 7f94ddb31640 1 -- 172.21.15.46:0/3821818277 --> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f94d809c6d0 con 0x7f94c403e220 2024-09-16T04:47:46.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.297+0000 7f94bffff640 1 -- 172.21.15.46:0/3821818277 <== osd.5 v2:172.21.15.114:6818/938868100 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f94d809c6d0 con 0x7f94c403e220 2024-09-16T04:47:46.309 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-09-16T04:47:46.309 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:47:46.310 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-09-16T04:47:46.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.297+0000 7f94ddb31640 1 -- 172.21.15.46:0/3821818277 >> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] conn(0x7f94c403e220 msgr2=0x7f94c4040640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:46.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.297+0000 7f94ddb31640 1 --2- 172.21.15.46:0/3821818277 >> [v2:172.21.15.114:6818/938868100,v1:172.21.15.114:6819/938868100] conn(0x7f94c403e220 0x7f94c4040640 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.297+0000 7f94ddb31640 1 -- 172.21.15.46:0/3821818277 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f94c40383b0 msgr2=0x7f94c403a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:46.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.297+0000 7f94ddb31640 1 --2- 172.21.15.46:0/3821818277 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f94c40383b0 0x7f94c403a870 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f94cc034000 tx=0x7f94cc02e000 comp rx=0 tx=0).stop 2024-09-16T04:47:46.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.297+0000 7f94ddb31640 10 monclient: shutdown 2024-09-16T04:47:46.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.297+0000 7f94ddb31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:46.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.297+0000 7f94ddb31640 1 -- 172.21.15.46:0/3821818277 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f94d80a3c10 msgr2=0x7f94d8136ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:46.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.297+0000 7f94ddb31640 1 --2- 172.21.15.46:0/3821818277 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f94d80a3c10 0x7f94d8136ee0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f94c800c450 tx=0x7f94c800c910 comp rx=0 tx=0).stop 2024-09-16T04:47:46.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.297+0000 7f94d7fff640 1 -- 172.21.15.46:0/3821818277 reap_dead start 2024-09-16T04:47:46.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.297+0000 7f94ddb31640 1 -- 172.21.15.46:0/3821818277 shutdown_connections 2024-09-16T04:47:46.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.297+0000 7f94ddb31640 1 -- 172.21.15.46:0/3821818277 >> 172.21.15.46:0/3821818277 conn(0x7f94d80a6340 msgr2=0x7f94d80a2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:46.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.297+0000 7f94ddb31640 1 -- 172.21.15.46:0/3821818277 shutdown_connections 2024-09-16T04:47:46.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.297+0000 7f94ddb31640 1 -- 172.21.15.46:0/3821818277 wait complete. 2024-09-16T04:47:46.312 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T04:47:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.414+0000 7fd5351e4640 10 monclient: get_monmap_and_config 2024-09-16T04:47:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.414+0000 7fd5351e4640 10 monclient: build_initial_monmap 2024-09-16T04:47:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.414+0000 7fd5351e4640 10 monclient: monmap: 2024-09-16T04:47:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:46.415648+0000 2024-09-16T04:47:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:46.415648+0000 2024-09-16T04:47:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:46.417 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:46.417 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:46.417 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:46.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.414+0000 7fd5351e4640 1 Processor -- start 2024-09-16T04:47:46.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.414+0000 7fd5351e4640 1 -- start start 2024-09-16T04:47:46.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.414+0000 7fd5351e4640 10 monclient: init 2024-09-16T04:47:46.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd5351e4640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:46.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd5351e4640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-16T04:47:46.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd5351e4640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd5300a3c10 0x7fd53009bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:46.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd5351e4640 10 monclient(hunting): picked mon.noname-c con 0x7fd5300a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:47:46.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd5351e4640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd5300a3ff0 0x7fd5300a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:46.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd5351e4640 10 monclient(hunting): picked mon.noname-b con 0x7fd5300a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:46.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd5351e4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd5301369d0 addr v1:172.21.15.114:6789/0 2024-09-16T04:47:46.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd5351e4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd53009ab70 con 0x7fd5301369d0 2024-09-16T04:47:46.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd5351e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:46.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd5351e4640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fd5300a4d60 con 0x7fd5300a3c10 2024-09-16T04:47:46.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd5351e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:46.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd5351e4640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd53009b030 con 0x7fd5300a3ff0 2024-09-16T04:47:46.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd5351e4640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:46.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd5351e4640 10 monclient(hunting): authenticate will time out at 1264.831422s 2024-09-16T04:47:46.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52e59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd5300a3ff0 0x7fd5300a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:46.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52e59c640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd5300a3ff0 0x7fd5300a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:41822/0 (socket says 172.21.15.46:41822) 2024-09-16T04:47:46.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52e59c640 1 -- 172.21.15.46:0/3168531569 learned_addr learned my addr 172.21.15.46:0/3168531569 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:46.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52e59c640 10 monclient(hunting): get_auth_request con 0x7fd5300a3ff0 auth_method 0 2024-09-16T04:47:46.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:46.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:46.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:46.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:46.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:46.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:46.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52ed9d640 1 --2- 172.21.15.46:0/3168531569 >> v2:172.21.15.114:3300/0 conn(0x7fd5300a3c10 0x7fd53009bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:46.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52ed9d640 10 monclient(hunting): get_auth_request con 0x7fd5300a3c10 auth_method 0 2024-09-16T04:47:46.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:46.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:46.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:46.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52e59c640 10 monclient(hunting): handle_auth_done global_id 94517 payload 274 2024-09-16T04:47:46.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52e59c640 1 -- 172.21.15.46:0/3168531569 >> v2:172.21.15.114:3300/0 conn(0x7fd5300a3c10 msgr2=0x7fd53009bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:46.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52e59c640 1 --2- 172.21.15.46:0/3168531569 >> v2:172.21.15.114:3300/0 conn(0x7fd5300a3c10 0x7fd53009bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52e59c640 1 -- 172.21.15.46:0/3168531569 >> v1:172.21.15.114:6789/0 conn(0x7fd5301369d0 legacy=0x7fd530136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:46.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52e59c640 10 monclient: _finish_hunting 0 2024-09-16T04:47:46.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52e59c640 1 monclient: found mon.noname-b 2024-09-16T04:47:46.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:46.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:47:46.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52e59c640 1 -- 172.21.15.46:0/3168531569 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd53009b310 con 0x7fd5300a3ff0 2024-09-16T04:47:46.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.415+0000 7fd52ed9d640 1 --2- 172.21.15.46:0/3168531569 >> v2:172.21.15.114:3300/0 conn(0x7fd5300a3c10 0x7fd53009bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:47:46.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.416+0000 7fd52e59c640 1 --2- 172.21.15.46:0/3168531569 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd5300a3ff0 0x7fd5300a43d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd524009870 tx=0x7fd52402a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da6fa5c1461b5efc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:46.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.416+0000 7fd52dd9b640 1 -- 172.21.15.46:0/3168531569 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd524037070 con 0x7fd5300a3ff0 2024-09-16T04:47:46.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.416+0000 7fd52dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:46.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.416+0000 7fd52dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:47:46.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.416+0000 7fd52dd9b640 10 monclient: dump: 2024-09-16T04:47:46.428 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:46.428 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:46.428 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.428 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.429 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:46.429 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:46.430 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:46.430 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:46.430 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:46.430 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:46.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.416+0000 7fd52dd9b640 10 monclient: _finish_auth 0 2024-09-16T04:47:46.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.416+0000 7fd52dd9b640 10 monclient: _check_auth_tickets 2024-09-16T04:47:46.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.416+0000 7fd52dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:46.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.416+0000 7fd5351e4640 5 monclient: authenticate success, global_id 94517 2024-09-16T04:47:46.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.416+0000 7fd5351e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:47:46.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.416+0000 7fd52dd9b640 1 -- 172.21.15.46:0/3168531569 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd524032440 con 0x7fd5300a3ff0 2024-09-16T04:47:46.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.416+0000 7fd52dd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:46.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.416+0000 7fd5351e4640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:46.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.416+0000 7fd5351e4640 4 set_mon_vals no callback set 2024-09-16T04:47:46.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.416+0000 7fd52dd9b640 1 -- 172.21.15.46:0/3168531569 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd524032810 con 0x7fd5300a3ff0 2024-09-16T04:47:46.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.416+0000 7fd52dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:46.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.416+0000 7fd52dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:46.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.416+0000 7fd52dd9b640 10 monclient: dump: 2024-09-16T04:47:46.431 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:46.431 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:46.431 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.431 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.431 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:46.431 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:46.431 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:46.431 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:46.431 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:46.432 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:46.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.417+0000 7fd5351e4640 10 monclient: shutdown 2024-09-16T04:47:46.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.417+0000 7fd5351e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:46.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.417+0000 7fd5351e4640 1 -- 172.21.15.46:0/3168531569 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd5300a3ff0 msgr2=0x7fd5300a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:46.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.417+0000 7fd5351e4640 1 --2- 172.21.15.46:0/3168531569 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd5300a3ff0 0x7fd5300a43d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd524009870 tx=0x7fd52402a1c0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.417+0000 7fd5351e4640 1 -- 172.21.15.46:0/3168531569 shutdown_connections 2024-09-16T04:47:46.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.417+0000 7fd5351e4640 1 --2- 172.21.15.46:0/3168531569 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd5300a3ff0 0x7fd5300a43d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.417+0000 7fd5351e4640 1 --2- 172.21.15.46:0/3168531569 >> v2:172.21.15.114:3300/0 conn(0x7fd5300a3c10 0x7fd53009bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.417+0000 7fd5351e4640 1 -- 172.21.15.46:0/3168531569 >> 172.21.15.46:0/3168531569 conn(0x7fd5300a5d10 msgr2=0x7fd5300a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:46.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.417+0000 7fd5351e4640 1 -- 172.21.15.46:0/3168531569 shutdown_connections 2024-09-16T04:47:46.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.417+0000 7fd5351e4640 1 -- 172.21.15.46:0/3168531569 wait complete. 2024-09-16T04:47:46.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.418+0000 7fd5351e4640 10 monclient: build_initial_monmap 2024-09-16T04:47:46.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.418+0000 7fd5351e4640 10 monclient: monmap: 2024-09-16T04:47:46.432 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:46.433 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:46.433 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:46.433 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:46.433 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:46.433 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:46.433 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:46.433 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:46.433 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:46.433 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:46.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.418+0000 7fd5351e4640 1 Processor -- start 2024-09-16T04:47:46.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.418+0000 7fd5351e4640 1 -- start start 2024-09-16T04:47:46.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.418+0000 7fd5351e4640 10 monclient: init 2024-09-16T04:47:46.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd5351e4640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:46.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd5351e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:47:46.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd5351e4640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd5300a3c10 0x7fd5301cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:46.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd5351e4640 10 monclient(hunting): picked mon.noname-c con 0x7fd5300a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:46.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd5351e4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5300a3ff0 0x7fd530135a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:46.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd5351e4640 10 monclient(hunting): picked mon.noname-b con 0x7fd5300a3ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:46.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd5351e4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5301369d0 0x7fd530135fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:46.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd5351e4640 10 monclient(hunting): picked mon.noname-a con 0x7fd5301369d0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:46.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd5351e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:46.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd5351e4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd5300a4910 con 0x7fd5301369d0 2024-09-16T04:47:46.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd5351e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:46.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd5351e4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd53006e3d0 con 0x7fd5300a3ff0 2024-09-16T04:47:46.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd5351e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:46.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd5351e4640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fd5301cc220 con 0x7fd5300a3c10 2024-09-16T04:47:46.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd5351e4640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:46.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd5351e4640 10 monclient(hunting): authenticate will time out at 1264.835190s 2024-09-16T04:47:46.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52ed9d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd5300a3c10 0x7fd5301cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:46.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52f59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5301369d0 0x7fd530135fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:46.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52ed9d640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd5300a3c10 0x7fd5301cbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:41824/0 (socket says 172.21.15.46:41824) 2024-09-16T04:47:46.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52ed9d640 1 -- 172.21.15.46:0/74166742 learned_addr learned my addr 172.21.15.46:0/74166742 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:46.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52f59e640 10 monclient(hunting): get_auth_request con 0x7fd5301369d0 auth_method 0 2024-09-16T04:47:46.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:46.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52f59e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:46.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:46.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52ed9d640 10 monclient(hunting): get_auth_request con 0x7fd5300a3c10 auth_method 0 2024-09-16T04:47:46.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:46.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:46.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:46.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52e59c640 1 --2- 172.21.15.46:0/74166742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5300a3ff0 0x7fd530135a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:46.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52e59c640 10 monclient(hunting): get_auth_request con 0x7fd5300a3ff0 auth_method 0 2024-09-16T04:47:46.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:46.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:46.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:46.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:46.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:46.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:46.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:46.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:46.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:46.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:46.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:46.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:46.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52f59e640 10 monclient(hunting): handle_auth_done global_id 104400 payload 824 2024-09-16T04:47:46.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52f59e640 1 -- 172.21.15.46:0/74166742 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd5300a3c10 msgr2=0x7fd5301cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:46.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52f59e640 1 --2- 172.21.15.46:0/74166742 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd5300a3c10 0x7fd5301cbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52f59e640 1 -- 172.21.15.46:0/74166742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5300a3ff0 msgr2=0x7fd530135a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:46.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52f59e640 1 --2- 172.21.15.46:0/74166742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5300a3ff0 0x7fd530135a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52f59e640 10 monclient: _finish_hunting 0 2024-09-16T04:47:46.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52f59e640 1 monclient: found mon.noname-a 2024-09-16T04:47:46.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:46.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:47:46.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52f59e640 1 -- 172.21.15.46:0/74166742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd530127470 con 0x7fd5301369d0 2024-09-16T04:47:46.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52ed9d640 1 --2- 172.21.15.46:0/74166742 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd5300a3c10 0x7fd5301cbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:46.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd52f59e640 1 --2- 172.21.15.46:0/74166742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5301369d0 0x7fd530135fb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd52800c700 tx=0x7fd52800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:46.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.419+0000 7fd5177fe640 1 -- 172.21.15.46:0/74166742 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd528003b70 con 0x7fd5301369d0 2024-09-16T04:47:46.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.420+0000 7fd5177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:46.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.420+0000 7fd5177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:47:46.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.420+0000 7fd5177fe640 10 monclient: dump: 2024-09-16T04:47:46.437 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:46.437 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:46.437 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.438 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.438 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:46.438 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:46.438 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:46.438 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:46.438 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:46.438 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:46.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.420+0000 7fd5177fe640 10 monclient: _finish_auth 0 2024-09-16T04:47:46.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.420+0000 7fd5177fe640 10 monclient: _check_auth_tickets 2024-09-16T04:47:46.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.420+0000 7fd5177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:46.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.420+0000 7fd5177fe640 1 -- 172.21.15.46:0/74166742 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd528003d10 con 0x7fd5301369d0 2024-09-16T04:47:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.420+0000 7fd5351e4640 5 monclient: authenticate success, global_id 104400 2024-09-16T04:47:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.420+0000 7fd5351e4640 10 monclient: _renew_subs 2024-09-16T04:47:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.420+0000 7fd5351e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.420+0000 7fd5351e4640 1 -- 172.21.15.46:0/74166742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd530125f00 con 0x7fd5301369d0 2024-09-16T04:47:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.420+0000 7fd5177fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.420+0000 7fd5177fe640 1 -- 172.21.15.46:0/74166742 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd528005310 con 0x7fd5301369d0 2024-09-16T04:47:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.420+0000 7fd5177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.420+0000 7fd5177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:47:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.420+0000 7fd5177fe640 10 monclient: dump: 2024-09-16T04:47:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:46.440 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.420+0000 7fd52d59a640 4 set_mon_vals no callback set 2024-09-16T04:47:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.420+0000 7fd5351e4640 10 monclient: _renew_subs 2024-09-16T04:47:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.420+0000 7fd5351e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.420+0000 7fd5351e4640 1 -- 172.21.15.46:0/74166742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5300a0400 con 0x7fd5301369d0 2024-09-16T04:47:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.421+0000 7fd5177fe640 1 -- 172.21.15.46:0/74166742 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fd5280054b0 con 0x7fd5301369d0 2024-09-16T04:47:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.421+0000 7fd5351e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.421+0000 7fd5351e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.421+0000 7fd5351e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:47:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.421+0000 7fd5351e4640 1 -- 172.21.15.46:0/74166742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5300a6280 con 0x7fd5301369d0 2024-09-16T04:47:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.421+0000 7fd5177fe640 1 --2- 172.21.15.46:0/74166742 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd51c0383b0 0x7fd51c03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:46.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.421+0000 7fd5177fe640 1 -- 172.21.15.46:0/74166742 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7fd52804b300 con 0x7fd5301369d0 2024-09-16T04:47:46.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.421+0000 7fd52ed9d640 1 --2- 172.21.15.46:0/74166742 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd51c0383b0 0x7fd51c03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:46.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.421+0000 7fd52ed9d640 10 monclient: get_auth_request con 0x7fd51c0383b0 auth_method 0 2024-09-16T04:47:46.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.422+0000 7fd52ed9d640 1 --2- 172.21.15.46:0/74166742 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd51c0383b0 0x7fd51c03a870 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fd520006d40 tx=0x7fd520008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:46.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.425+0000 7fd5177fe640 1 -- 172.21.15.46:0/74166742 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd5300a6280 con 0x7fd5301369d0 2024-09-16T04:47:46.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.425+0000 7fd5177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:46.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.425+0000 7fd5177fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:46.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.601+0000 7fd5351e4640 1 -- 172.21.15.46:0/74166742 --> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fd530137580 con 0x7fd51c0383b0 2024-09-16T04:47:46.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.602+0000 7fd5177fe640 1 -- 172.21.15.46:0/74166742 <== mgr.4099 v2:172.21.15.114:6800/3521677055 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fd530137580 con 0x7fd51c0383b0 2024-09-16T04:47:46.604 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T04:47:46.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.604+0000 7fd5351e4640 1 -- 172.21.15.46:0/74166742 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd51c0383b0 msgr2=0x7fd51c03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:46.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.604+0000 7fd5351e4640 1 --2- 172.21.15.46:0/74166742 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd51c0383b0 0x7fd51c03a870 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fd520006d40 tx=0x7fd520008040 comp rx=0 tx=0).stop 2024-09-16T04:47:46.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.604+0000 7fd5351e4640 10 monclient: shutdown 2024-09-16T04:47:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.604+0000 7fd5351e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.604+0000 7fd5351e4640 1 -- 172.21.15.46:0/74166742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5301369d0 msgr2=0x7fd530135fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.604+0000 7fd5351e4640 1 --2- 172.21.15.46:0/74166742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5301369d0 0x7fd530135fb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd52800c700 tx=0x7fd52800cbc0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.604+0000 7fd5351e4640 1 -- 172.21.15.46:0/74166742 shutdown_connections 2024-09-16T04:47:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.604+0000 7fd5351e4640 1 --2- 172.21.15.46:0/74166742 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fd51c0383b0 0x7fd51c03a870 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.604+0000 7fd5351e4640 1 --2- 172.21.15.46:0/74166742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5301369d0 0x7fd530135fb0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.604+0000 7fd5351e4640 1 --2- 172.21.15.46:0/74166742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5300a3ff0 0x7fd530135a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.604+0000 7fd5351e4640 1 --2- 172.21.15.46:0/74166742 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fd5300a3c10 0x7fd5301cbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.604+0000 7fd5351e4640 1 -- 172.21.15.46:0/74166742 >> 172.21.15.46:0/74166742 conn(0x7fd5300a5d10 msgr2=0x7fd5300a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:46.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.604+0000 7fd5351e4640 1 -- 172.21.15.46:0/74166742 shutdown_connections 2024-09-16T04:47:46.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.604+0000 7fd5351e4640 1 -- 172.21.15.46:0/74166742 wait complete. 2024-09-16T04:47:46.617 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:47:46.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.716+0000 7f26994f1640 10 monclient: get_monmap_and_config 2024-09-16T04:47:46.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.716+0000 7f26994f1640 10 monclient: build_initial_monmap 2024-09-16T04:47:46.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.716+0000 7f26994f1640 10 monclient: monmap: 2024-09-16T04:47:46.719 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:46.719 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:46.719 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:47:46.718227+0000 2024-09-16T04:47:46.719 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:47:46.718227+0000 2024-09-16T04:47:46.719 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:46.719 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:46.720 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:47:46.720 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:47:46.720 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:47:46.720 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:47:46.720 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:47:46.720 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:46.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.717+0000 7f26994f1640 1 Processor -- start 2024-09-16T04:47:46.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.717+0000 7f26994f1640 1 -- start start 2024-09-16T04:47:46.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.717+0000 7f26994f1640 10 monclient: init 2024-09-16T04:47:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.717+0000 7f26994f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.717+0000 7f26994f1640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-16T04:47:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.717+0000 7f26994f1640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f26940a3c10 0x7f269409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.717+0000 7f26994f1640 10 monclient(hunting): picked mon.noname-b con 0x7f26940a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.717+0000 7f26994f1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f26940a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:47:46.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.717+0000 7f26994f1640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f269413be90 0x7f269413e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:46.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.717+0000 7f26994f1640 10 monclient(hunting): picked mon.noname-c con 0x7f269413be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:47:46.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f26994f1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f269409b940 con 0x7f26940a3ff0 2024-09-16T04:47:46.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f26994f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:46.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f26994f1640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f269409ab00 con 0x7f269413be90 2024-09-16T04:47:46.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f26994f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:46.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f26994f1640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f269409b030 con 0x7f26940a3c10 2024-09-16T04:47:46.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f26994f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:46.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f26994f1640 10 monclient(hunting): authenticate will time out at 1265.134056s 2024-09-16T04:47:46.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f2692ffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f26940a3c10 0x7f269409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:46.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f2692ffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f26940a3c10 0x7f269409bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:41826/0 (socket says 172.21.15.46:41826) 2024-09-16T04:47:46.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f2692ffd640 1 -- 172.21.15.46:0/1189839643 learned_addr learned my addr 172.21.15.46:0/1189839643 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:46.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f2692ffd640 10 monclient(hunting): get_auth_request con 0x7f26940a3c10 auth_method 0 2024-09-16T04:47:46.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f2692ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:46.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f2692ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:46.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f2692ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:46.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f2692ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:46.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f2692ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:46.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f2692ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:46.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f26937fe640 1 --2- 172.21.15.46:0/1189839643 >> v2:172.21.15.114:3300/0 conn(0x7f269413be90 0x7f269413e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:46.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f26937fe640 10 monclient(hunting): get_auth_request con 0x7f269413be90 auth_method 0 2024-09-16T04:47:46.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f26937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:46.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f26937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:46.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f26937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:46.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f26937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:46.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f26937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:46.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f26937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:46.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f2692ffd640 10 monclient(hunting): handle_auth_done global_id 94523 payload 274 2024-09-16T04:47:46.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f2692ffd640 1 -- 172.21.15.46:0/1189839643 >> v2:172.21.15.114:3300/0 conn(0x7f269413be90 msgr2=0x7f269413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:47:46.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f2692ffd640 1 --2- 172.21.15.46:0/1189839643 >> v2:172.21.15.114:3300/0 conn(0x7f269413be90 0x7f269413e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f2692ffd640 1 -- 172.21.15.46:0/1189839643 >> v1:172.21.15.114:6789/0 conn(0x7f26940a3ff0 legacy=0x7f26940a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:46.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f2692ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:47:46.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f2692ffd640 1 monclient: found mon.noname-b 2024-09-16T04:47:46.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f2692ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:46.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f2692ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.46:3301/0 2024-09-16T04:47:46.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.718+0000 7f2692ffd640 1 -- 172.21.15.46:0/1189839643 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f269409b310 con 0x7f26940a3c10 2024-09-16T04:47:46.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.719+0000 7f2692ffd640 1 --2- 172.21.15.46:0/1189839643 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f26940a3c10 0x7f269409bcf0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2684009870 tx=0x7f268402a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3eac5589cd971b7a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:46.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.719+0000 7f2691ffb640 1 -- 172.21.15.46:0/1189839643 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2684037070 con 0x7f26940a3c10 2024-09-16T04:47:46.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.719+0000 7f2691ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:46.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.719+0000 7f2691ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:47:46.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.719+0000 7f2691ffb640 10 monclient: dump: 2024-09-16T04:47:46.738 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:46.738 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:46.738 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.738 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.738 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:46.738 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:46.738 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:46.738 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:46.739 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:46.739 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:46.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.719+0000 7f2691ffb640 10 monclient: _finish_auth 0 2024-09-16T04:47:46.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.719+0000 7f2691ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:47:46.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.719+0000 7f2691ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:46.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.719+0000 7f2691ffb640 1 -- 172.21.15.46:0/1189839643 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2684032440 con 0x7f26940a3c10 2024-09-16T04:47:46.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.719+0000 7f2691ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:46.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.719+0000 7f2691ffb640 1 -- 172.21.15.46:0/1189839643 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2684032810 con 0x7f26940a3c10 2024-09-16T04:47:46.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.719+0000 7f26994f1640 5 monclient: authenticate success, global_id 94523 2024-09-16T04:47:46.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.719+0000 7f26994f1640 10 monclient: get_monmap_and_config success 2024-09-16T04:47:46.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.719+0000 7f26994f1640 4 set_mon_vals no callback set 2024-09-16T04:47:46.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.719+0000 7f2691ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:46.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.719+0000 7f2691ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:46.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.719+0000 7f2691ffb640 10 monclient: dump: 2024-09-16T04:47:46.740 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:46.740 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:46.740 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.741 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.741 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:46.741 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:46.741 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:46.741 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:46.741 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:46.741 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:46.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.719+0000 7f26994f1640 10 monclient: shutdown 2024-09-16T04:47:46.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.719+0000 7f26994f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:46.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.719+0000 7f26994f1640 1 -- 172.21.15.46:0/1189839643 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f26940a3c10 msgr2=0x7f269409bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:46.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.719+0000 7f26994f1640 1 --2- 172.21.15.46:0/1189839643 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f26940a3c10 0x7f269409bcf0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2684009870 tx=0x7f268402a130 comp rx=0 tx=0).stop 2024-09-16T04:47:46.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.720+0000 7f26994f1640 1 -- 172.21.15.46:0/1189839643 shutdown_connections 2024-09-16T04:47:46.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.720+0000 7f26994f1640 1 --2- 172.21.15.46:0/1189839643 >> v2:172.21.15.114:3300/0 conn(0x7f269413be90 0x7f269413e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.720+0000 7f26994f1640 1 --2- 172.21.15.46:0/1189839643 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f26940a3c10 0x7f269409bcf0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.720+0000 7f26994f1640 1 -- 172.21.15.46:0/1189839643 >> 172.21.15.46:0/1189839643 conn(0x7f26940a6340 msgr2=0x7f269406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:46.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.720+0000 7f26994f1640 1 -- 172.21.15.46:0/1189839643 shutdown_connections 2024-09-16T04:47:46.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.720+0000 7f26994f1640 1 -- 172.21.15.46:0/1189839643 wait complete. 2024-09-16T04:47:46.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.720+0000 7f26994f1640 10 monclient: build_initial_monmap 2024-09-16T04:47:46.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.720+0000 7f26994f1640 10 monclient: monmap: 2024-09-16T04:47:46.744 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:47:46.744 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:47:46.744 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:47:46.744 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:47:46.744 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:47:46.744 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:46.745 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:47:46.745 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:47:46.745 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:47:46.745 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:46.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26994f1640 1 Processor -- start 2024-09-16T04:47:46.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26994f1640 1 -- start start 2024-09-16T04:47:46.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26994f1640 10 monclient: init 2024-09-16T04:47:46.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26994f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:47:46.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26994f1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:47:46.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26994f1640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f26940a3c10 0x7f26941cc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:46.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26994f1640 10 monclient(hunting): picked mon.noname-c con 0x7f26940a3c10 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:47:46.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26994f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26940a3ff0 0x7f2694138350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:46.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26994f1640 10 monclient(hunting): picked mon.noname-a con 0x7f26940a3ff0 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:47:46.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26994f1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f269413be90 0x7f2694138890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:46.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26994f1640 10 monclient(hunting): picked mon.noname-b con 0x7f269413be90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:47:46.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26994f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:46.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26994f1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f26940a6b40 con 0x7f26940a3ff0 2024-09-16T04:47:46.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26994f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:46.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26994f1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f26940a69c0 con 0x7f269413be90 2024-09-16T04:47:46.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26994f1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:47:46.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26994f1640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f269413e7c0 con 0x7f26940a3c10 2024-09-16T04:47:46.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26994f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:47:46.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26994f1640 10 monclient(hunting): authenticate will time out at 1265.137588s 2024-09-16T04:47:46.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f2692ffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f26940a3c10 0x7f26941cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:46.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f2692ffd640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f26940a3c10 0x7f26941cc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:41840/0 (socket says 172.21.15.46:41840) 2024-09-16T04:47:46.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f2692ffd640 1 -- 172.21.15.46:0/2166790415 learned_addr learned my addr 172.21.15.46:0/2166790415 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:47:46.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26927fc640 1 --2- 172.21.15.46:0/2166790415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26940a3ff0 0x7f2694138350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:46.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f2692ffd640 10 monclient(hunting): get_auth_request con 0x7f26940a3c10 auth_method 0 2024-09-16T04:47:46.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f2692ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:46.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f2692ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:46.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f2692ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:46.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26927fc640 10 monclient(hunting): get_auth_request con 0x7f26940a3ff0 auth_method 0 2024-09-16T04:47:46.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:46.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:46.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:46.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26937fe640 1 --2- 172.21.15.46:0/2166790415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f269413be90 0x7f2694138890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:46.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f2692ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:46.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f2692ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:46.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f2692ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:46.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26937fe640 10 monclient(hunting): get_auth_request con 0x7f269413be90 auth_method 0 2024-09-16T04:47:46.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:47:46.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:47:46.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:47:46.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:46.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:46.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:46.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:47:46.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:47:46.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.721+0000 7f26937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:47:46.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f2692ffd640 10 monclient(hunting): handle_auth_done global_id 94526 payload 824 2024-09-16T04:47:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f2692ffd640 1 -- 172.21.15.46:0/2166790415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f269413be90 msgr2=0x7f2694138890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f2692ffd640 1 --2- 172.21.15.46:0/2166790415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f269413be90 0x7f2694138890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f2692ffd640 1 -- 172.21.15.46:0/2166790415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26940a3ff0 msgr2=0x7f2694138350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f2692ffd640 1 --2- 172.21.15.46:0/2166790415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26940a3ff0 0x7f2694138350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f2692ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:47:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f2692ffd640 1 monclient: found mon.noname-c 2024-09-16T04:47:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f2692ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:47:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f2692ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:47:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f2692ffd640 1 -- 172.21.15.46:0/2166790415 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f269413c400 con 0x7f26940a3c10 2024-09-16T04:47:46.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f26927fc640 1 --2- 172.21.15.46:0/2166790415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26940a3ff0 0x7f2694138350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:47:46.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f2692ffd640 1 --2- 172.21.15.46:0/2166790415 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f26940a3c10 0x7f26941cc190 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2684009870 tx=0x7f26840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:46.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f267b7fe640 1 -- 172.21.15.46:0/2166790415 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2684041020 con 0x7f26940a3c10 2024-09-16T04:47:46.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f267b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:46.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f267b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:47:46.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f267b7fe640 10 monclient: dump: 2024-09-16T04:47:46.752 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:46.753 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:46.753 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.753 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.753 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:46.753 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:46.753 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:46.753 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:46.753 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:46.753 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:46.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f267b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:47:46.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f267b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:47:46.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f267b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:47:46.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f267b7fe640 1 -- 172.21.15.46:0/2166790415 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26840306c0 con 0x7f26940a3c10 2024-09-16T04:47:46.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f267b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:47:46.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f267b7fe640 1 -- 172.21.15.46:0/2166790415 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2684037040 con 0x7f26940a3c10 2024-09-16T04:47:46.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f267b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:47:46.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f267b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:47:46.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f267b7fe640 10 monclient: dump: 2024-09-16T04:47:46.754 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:47:46.754 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:47:46.754 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.754 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:47:46.754 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:47:46.754 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:47:46.754 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:47:46.755 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:47:46.755 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:47:46.755 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:47:46.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f26994f1640 5 monclient: authenticate success, global_id 94526 2024-09-16T04:47:46.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f26994f1640 10 monclient: _renew_subs 2024-09-16T04:47:46.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f26994f1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:46.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f26994f1640 1 -- 172.21.15.46:0/2166790415 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26940a0400 con 0x7f26940a3c10 2024-09-16T04:47:46.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f26917fa640 4 set_mon_vals no callback set 2024-09-16T04:47:46.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f26994f1640 10 monclient: _renew_subs 2024-09-16T04:47:46.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f26994f1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:46.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.722+0000 7f26994f1640 1 -- 172.21.15.46:0/2166790415 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f269409de70 con 0x7f26940a3c10 2024-09-16T04:47:46.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.723+0000 7f267b7fe640 1 -- 172.21.15.46:0/2166790415 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f2684044020 con 0x7f26940a3c10 2024-09-16T04:47:46.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.723+0000 7f26994f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:46.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.723+0000 7f26994f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:46.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.723+0000 7f26994f1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:47:46.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.723+0000 7f267b7fe640 1 --2- 172.21.15.46:0/2166790415 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f26800383e0 0x7f268003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:47:46.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.723+0000 7f26994f1640 1 -- 172.21.15.46:0/2166790415 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f269409bcf0 con 0x7f26940a3c10 2024-09-16T04:47:46.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.723+0000 7f267b7fe640 1 -- 172.21.15.46:0/2166790415 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7f268406bc90 con 0x7f26940a3c10 2024-09-16T04:47:46.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.723+0000 7f26927fc640 1 --2- 172.21.15.46:0/2166790415 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f26800383e0 0x7f268003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:47:46.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.723+0000 7f26927fc640 10 monclient: get_auth_request con 0x7f26800383e0 auth_method 0 2024-09-16T04:47:46.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.724+0000 7f26927fc640 1 --2- 172.21.15.46:0/2166790415 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f26800383e0 0x7f268003a8a0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f2694139b70 tx=0x7f2688009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:47:46.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.728+0000 7f267b7fe640 1 -- 172.21.15.46:0/2166790415 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f268404b030 con 0x7f26940a3c10 2024-09-16T04:47:46.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.728+0000 7f267b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:47:46.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.728+0000 7f267b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:47:46.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.898+0000 7f26994f1640 1 -- 172.21.15.46:0/2166790415 --> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f269409d650 con 0x7f26800383e0 2024-09-16T04:47:46.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.900+0000 7f267b7fe640 1 -- 172.21.15.46:0/2166790415 <== mgr.4099 v2:172.21.15.114:6800/3521677055 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33951 (secure 0 0 0) 0x7f2688001d10 con 0x7f26800383e0 2024-09-16T04:47:46.903 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:47:46.903 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-09-16T04:47:46.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.902+0000 7f26994f1640 1 -- 172.21.15.46:0/2166790415 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f26800383e0 msgr2=0x7f268003a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:46.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.902+0000 7f26994f1640 1 --2- 172.21.15.46:0/2166790415 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f26800383e0 0x7f268003a8a0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f2694139b70 tx=0x7f2688009040 comp rx=0 tx=0).stop 2024-09-16T04:47:46.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.902+0000 7f26994f1640 10 monclient: shutdown 2024-09-16T04:47:46.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.902+0000 7f26994f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:47:46.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.902+0000 7f26994f1640 1 -- 172.21.15.46:0/2166790415 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f26940a3c10 msgr2=0x7f26941cc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:47:46.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.902+0000 7f26994f1640 1 --2- 172.21.15.46:0/2166790415 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f26940a3c10 0x7f26941cc190 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2684009870 tx=0x7f26840046c0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.902+0000 7f26994f1640 1 -- 172.21.15.46:0/2166790415 shutdown_connections 2024-09-16T04:47:46.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.902+0000 7f26994f1640 1 --2- 172.21.15.46:0/2166790415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f269413be90 0x7f2694138890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.902+0000 7f26994f1640 1 --2- 172.21.15.46:0/2166790415 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f26800383e0 0x7f268003a8a0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.902+0000 7f26994f1640 1 --2- 172.21.15.46:0/2166790415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26940a3ff0 0x7f2694138350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.902+0000 7f26994f1640 1 --2- 172.21.15.46:0/2166790415 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f26940a3c10 0x7f26941cc190 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:47:46.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.902+0000 7f26994f1640 1 -- 172.21.15.46:0/2166790415 >> 172.21.15.46:0/2166790415 conn(0x7f26940a6340 msgr2=0x7f269413b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:47:46.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.902+0000 7f26994f1640 1 -- 172.21.15.46:0/2166790415 shutdown_connections 2024-09-16T04:47:46.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:47:46.902+0000 7f26994f1640 1 -- 172.21.15.46:0/2166790415 wait complete. 2024-09-16T04:47:46.915 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":578,"stamp":"2024-09-16T04:47:45.460613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":217056,"kb_used_data":25768,"kb_used_omap":9,"kb_used_meta":191158,"kb_avail":566013984,"statfs":{"total":579820584960,"available":579598319616,"internally_reserved":0,"allocated":26386432,"data_stored":10465252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":195746464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002297"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":598,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.254660+0000","last_change":"2024-09-16T04:39:00.524013+0000","last_active":"2024-09-16T04:47:13.254660+0000","last_peered":"2024-09-16T04:47:13.254660+0000","last_clean":"2024-09-16T04:47:13.254660+0000","last_became_active":"2024-09-16T04:38:58.756778+0000","last_became_peered":"2024-09-16T04:38:58.756778+0000","last_unstale":"2024-09-16T04:47:13.254660+0000","last_undegraded":"2024-09-16T04:47:13.254660+0000","last_fullsized":"2024-09-16T04:47:13.254660+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:24:15.429766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054536700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":591,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:11.265078+0000","last_change":"2024-09-16T04:39:00.532294+0000","last_active":"2024-09-16T04:47:11.265078+0000","last_peered":"2024-09-16T04:47:11.265078+0000","last_clean":"2024-09-16T04:47:11.265078+0000","last_became_active":"2024-09-16T04:38:58.520473+0000","last_became_peered":"2024-09-16T04:38:58.520473+0000","last_unstale":"2024-09-16T04:47:11.265078+0000","last_undegraded":"2024-09-16T04:47:11.265078+0000","last_fullsized":"2024-09-16T04:47:11.265078+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:15:41.548834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053532500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":598,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.254532+0000","last_change":"2024-09-16T04:39:00.523783+0000","last_active":"2024-09-16T04:47:13.254532+0000","last_peered":"2024-09-16T04:47:13.254532+0000","last_clean":"2024-09-16T04:47:13.254532+0000","last_became_active":"2024-09-16T04:38:58.521000+0000","last_became_peered":"2024-09-16T04:38:58.521000+0000","last_unstale":"2024-09-16T04:47:13.254532+0000","last_undegraded":"2024-09-16T04:47:13.254532+0000","last_fullsized":"2024-09-16T04:47:13.254532+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:42.837022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035484999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":591,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:11.263646+0000","last_change":"2024-09-16T04:39:00.532400+0000","last_active":"2024-09-16T04:47:11.263646+0000","last_peered":"2024-09-16T04:47:11.263646+0000","last_clean":"2024-09-16T04:47:11.263646+0000","last_became_active":"2024-09-16T04:38:58.521627+0000","last_became_peered":"2024-09-16T04:38:58.521627+0000","last_unstale":"2024-09-16T04:47:11.263646+0000","last_undegraded":"2024-09-16T04:47:11.263646+0000","last_fullsized":"2024-09-16T04:47:11.263646+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:41.804897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062256500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":594,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:11.805224+0000","last_change":"2024-09-16T04:39:00.530578+0000","last_active":"2024-09-16T04:47:11.805224+0000","last_peered":"2024-09-16T04:47:11.805224+0000","last_clean":"2024-09-16T04:47:11.805224+0000","last_became_active":"2024-09-16T04:38:58.522744+0000","last_became_peered":"2024-09-16T04:38:58.522744+0000","last_unstale":"2024-09-16T04:47:11.805224+0000","last_undegraded":"2024-09-16T04:47:11.805224+0000","last_fullsized":"2024-09-16T04:47:11.805224+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:04:47.450053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000966258,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":601,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.255101+0000","last_change":"2024-09-16T04:39:00.523661+0000","last_active":"2024-09-16T04:47:13.255101+0000","last_peered":"2024-09-16T04:47:13.255101+0000","last_clean":"2024-09-16T04:47:13.255101+0000","last_became_active":"2024-09-16T04:38:58.524372+0000","last_became_peered":"2024-09-16T04:38:58.524372+0000","last_unstale":"2024-09-16T04:47:13.255101+0000","last_undegraded":"2024-09-16T04:47:13.255101+0000","last_fullsized":"2024-09-16T04:47:13.255101+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:46.534727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047533599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":598,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.254361+0000","last_change":"2024-09-16T04:39:00.523838+0000","last_active":"2024-09-16T04:47:13.254361+0000","last_peered":"2024-09-16T04:47:13.254361+0000","last_clean":"2024-09-16T04:47:13.254361+0000","last_became_active":"2024-09-16T04:38:58.522142+0000","last_became_peered":"2024-09-16T04:38:58.522142+0000","last_unstale":"2024-09-16T04:47:13.254361+0000","last_undegraded":"2024-09-16T04:47:13.254361+0000","last_fullsized":"2024-09-16T04:47:13.254361+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:35.411495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039704200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":593,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:11.805254+0000","last_change":"2024-09-16T04:39:00.530064+0000","last_active":"2024-09-16T04:47:11.805254+0000","last_peered":"2024-09-16T04:47:11.805254+0000","last_clean":"2024-09-16T04:47:11.805254+0000","last_became_active":"2024-09-16T04:38:58.523268+0000","last_became_peered":"2024-09-16T04:38:58.523268+0000","last_unstale":"2024-09-16T04:47:11.805254+0000","last_undegraded":"2024-09-16T04:47:11.805254+0000","last_fullsized":"2024-09-16T04:47:11.805254+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:57.508067+0000","last_clean_scrub_stamp":"2024-09-16T04:38:57.508067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:12:33.958554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043992499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":749,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:13.254488+0000","last_change":"2024-09-16T04:38:52.501054+0000","last_active":"2024-09-16T04:47:13.254488+0000","last_peered":"2024-09-16T04:47:13.254488+0000","last_clean":"2024-09-16T04:47:13.254488+0000","last_became_active":"2024-09-16T04:38:52.500796+0000","last_became_peered":"2024-09-16T04:38:52.500796+0000","last_unstale":"2024-09-16T04:47:13.254488+0000","last_undegraded":"2024-09-16T04:47:13.254488+0000","last_fullsized":"2024-09-16T04:47:13.254488+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:38:51.492404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:38:51.492404+0000","last_clean_scrub_stamp":"2024-09-16T04:38:51.492404+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:36.122039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607662,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31148,"kb_used_data":4120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340692,"statfs":{"total":96636764160,"available":96604868608,"internally_reserved":0,"allocated":4218880,"data_stored":1567893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607662,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31668,"kb_used_data":4640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340172,"statfs":{"total":96636764160,"available":96604336128,"internally_reserved":0,"allocated":4751360,"data_stored":2096821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607662,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39988,"kb_used_data":4640,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94331852,"statfs":{"total":96636764160,"available":96595816448,"internally_reserved":0,"allocated":4751360,"data_stored":2096821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640366,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43632,"kb_used_data":4124,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94328208,"statfs":{"total":96636764160,"available":96592084992,"internally_reserved":0,"allocated":4222976,"data_stored":1567912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640367,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31148,"kb_used_data":4120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340692,"statfs":{"total":96636764160,"available":96604868608,"internally_reserved":0,"allocated":4218880,"data_stored":1567893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640365,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39472,"kb_used_data":4124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332368,"statfs":{"total":96636764160,"available":96596344832,"internally_reserved":0,"allocated":4222976,"data_stored":1567912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:47:46.921 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T04:38:57.508067+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=38, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=47, tm_sec=42, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:47:46.921 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T04:38:57.508067+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=38, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=47, tm_sec=42, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:47:46.921 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T04:38:57.508067+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=38, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=47, tm_sec=42, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:47:46.922 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T04:38:57.508067+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=38, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=47, tm_sec=42, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:47:46.922 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T04:38:57.508067+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=38, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=47, tm_sec=42, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:47:46.923 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T04:38:57.508067+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=38, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=47, tm_sec=42, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:47:46.923 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T04:38:57.508067+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=38, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=47, tm_sec=42, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:47:46.923 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T04:38:57.508067+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=38, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=47, tm_sec=42, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:47:46.924 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T04:38:51.492404+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=38, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=47, tm_sec=42, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:47:46.924 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T04:48:06.925 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:48:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.043+0000 7fdd21ee6640 10 monclient: get_monmap_and_config 2024-09-16T04:48:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.043+0000 7fdd21ee6640 10 monclient: build_initial_monmap 2024-09-16T04:48:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.043+0000 7fdd21ee6640 10 monclient: monmap: 2024-09-16T04:48:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:48:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:48:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:48:07.044538+0000 2024-09-16T04:48:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:48:07.044538+0000 2024-09-16T04:48:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:48:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:48:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:48:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:48:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:48:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:48:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:48:07.046 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:48:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.043+0000 7fdd21ee6640 1 Processor -- start 2024-09-16T04:48:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.043+0000 7fdd21ee6640 1 -- start start 2024-09-16T04:48:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.043+0000 7fdd21ee6640 10 monclient: init 2024-09-16T04:48:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd21ee6640 10 monclient: _reopen_session rank -1 2024-09-16T04:48:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd21ee6640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-16T04:48:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd21ee6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdd1c0a3c10 addr v1:172.21.15.46:6789/0 2024-09-16T04:48:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd21ee6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdd1c0a3ff0 0x7fdd1c0a4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:48:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd21ee6640 10 monclient(hunting): picked mon.noname-a con 0x7fdd1c0a3ff0 addr v2:172.21.15.46:3300/0 2024-09-16T04:48:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd21ee6640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdd1c13be90 0x7fdd1c13e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:48:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd21ee6640 10 monclient(hunting): picked mon.noname-c con 0x7fdd1c13be90 addr v2:172.21.15.114:3300/0 2024-09-16T04:48:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd21ee6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd1c09ab70 con 0x7fdd1c0a3c10 2024-09-16T04:48:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd21ee6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:48:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd21ee6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fdd1c0a4d60 con 0x7fdd1c0a3ff0 2024-09-16T04:48:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd21ee6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:48:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd21ee6640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fdd1c09b030 con 0x7fdd1c13be90 2024-09-16T04:48:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd21ee6640 10 monclient(hunting): _renew_subs 2024-09-16T04:48:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd21ee6640 10 monclient(hunting): authenticate will time out at 1285.460441s 2024-09-16T04:48:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdd1c0a3ff0 0x7fdd1c0a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:48:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdd1c0a3c10 0x7fdd1c09bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49798/0 (socket says 172.21.15.46:49798) 2024-09-16T04:48:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1b7fe640 1 -- 172.21.15.46:0/1457313056 learned_addr learned my addr 172.21.15.46:0/1457313056 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:48:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1affd640 10 monclient(hunting): get_auth_request con 0x7fdd1c0a3ff0 auth_method 0 2024-09-16T04:48:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:48:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:48:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:48:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1bfff640 1 --2- 172.21.15.46:0/1457313056 >> v2:172.21.15.114:3300/0 conn(0x7fdd1c13be90 0x7fdd1c13e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:48:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:48:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:48:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:48:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1bfff640 10 monclient(hunting): get_auth_request con 0x7fdd1c13be90 auth_method 0 2024-09-16T04:48:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:48:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:48:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:48:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:48:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:48:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:48:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1affd640 10 monclient(hunting): handle_auth_done global_id 104406 payload 274 2024-09-16T04:48:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1affd640 1 -- 172.21.15.46:0/1457313056 >> v2:172.21.15.114:3300/0 conn(0x7fdd1c13be90 msgr2=0x7fdd1c13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:48:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1affd640 1 --2- 172.21.15.46:0/1457313056 >> v2:172.21.15.114:3300/0 conn(0x7fdd1c13be90 0x7fdd1c13e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:48:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1affd640 1 -- 172.21.15.46:0/1457313056 >> v1:172.21.15.46:6789/0 conn(0x7fdd1c0a3c10 legacy=0x7fdd1c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:48:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1affd640 10 monclient: _finish_hunting 0 2024-09-16T04:48:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1affd640 1 monclient: found mon.noname-a 2024-09-16T04:48:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:48:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:48:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1a7fc640 1 -- 172.21.15.46:0/1457313056 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 897066389 0 0) 0x7fdd1c09ab70 con 0x7fdd1c0a3c10 2024-09-16T04:48:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1affd640 1 -- 172.21.15.46:0/1457313056 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd1c09b310 con 0x7fdd1c0a3ff0 2024-09-16T04:48:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.044+0000 7fdd1a7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:48:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.045+0000 7fdd1affd640 1 --2- 172.21.15.46:0/1457313056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a3ff0 0x7fdd1c0a4400 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c09c400 tx=0x7fdd1002a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c35d57173c8f4a88 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:48:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.045+0000 7fdd1a7fc640 1 -- 172.21.15.46:0/1457313056 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd10037070 con 0x7fdd1c0a3ff0 2024-09-16T04:48:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.045+0000 7fdd1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:48:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.045+0000 7fdd1a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:48:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.045+0000 7fdd1a7fc640 10 monclient: dump: 2024-09-16T04:48:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:48:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:48:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:48:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:48:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:48:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:48:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:48:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:48:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:48:07.061 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:48:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.045+0000 7fdd1a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:48:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.045+0000 7fdd1a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:48:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.045+0000 7fdd1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:48:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.045+0000 7fdd1a7fc640 1 -- 172.21.15.46:0/1457313056 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd1002add0 con 0x7fdd1c0a3ff0 2024-09-16T04:48:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.045+0000 7fdd21ee6640 5 monclient: authenticate success, global_id 104406 2024-09-16T04:48:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.045+0000 7fdd21ee6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:48:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.045+0000 7fdd1a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:48:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.045+0000 7fdd1a7fc640 1 -- 172.21.15.46:0/1457313056 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd10032660 con 0x7fdd1c0a3ff0 2024-09-16T04:48:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.045+0000 7fdd1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:48:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.045+0000 7fdd1a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:48:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.045+0000 7fdd1a7fc640 10 monclient: dump: 2024-09-16T04:48:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:48:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:48:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:48:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:48:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:48:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:48:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:48:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:48:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:48:07.062 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:48:07.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.045+0000 7fdd21ee6640 10 monclient: get_monmap_and_config success 2024-09-16T04:48:07.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.045+0000 7fdd21ee6640 4 set_mon_vals no callback set 2024-09-16T04:48:07.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.046+0000 7fdd21ee6640 10 monclient: shutdown 2024-09-16T04:48:07.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.046+0000 7fdd21ee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:48:07.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.046+0000 7fdd21ee6640 1 -- 172.21.15.46:0/1457313056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a3ff0 msgr2=0x7fdd1c0a4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:48:07.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.046+0000 7fdd21ee6640 1 --2- 172.21.15.46:0/1457313056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a3ff0 0x7fdd1c0a4400 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c09c400 tx=0x7fdd1002a080 comp rx=0 tx=0).stop 2024-09-16T04:48:07.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.046+0000 7fdd21ee6640 1 -- 172.21.15.46:0/1457313056 shutdown_connections 2024-09-16T04:48:07.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.046+0000 7fdd21ee6640 1 --2- 172.21.15.46:0/1457313056 >> v2:172.21.15.114:3300/0 conn(0x7fdd1c13be90 0x7fdd1c13e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:48:07.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.046+0000 7fdd21ee6640 1 --2- 172.21.15.46:0/1457313056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a3ff0 0x7fdd1c0a4400 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:48:07.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.046+0000 7fdd21ee6640 1 -- 172.21.15.46:0/1457313056 >> 172.21.15.46:0/1457313056 conn(0x7fdd1c0a5d10 msgr2=0x7fdd1c0a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:48:07.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.046+0000 7fdd21ee6640 1 -- 172.21.15.46:0/1457313056 shutdown_connections 2024-09-16T04:48:07.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.046+0000 7fdd21ee6640 1 -- 172.21.15.46:0/1457313056 wait complete. 2024-09-16T04:48:07.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.047+0000 7fdd21ee6640 10 monclient: build_initial_monmap 2024-09-16T04:48:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.047+0000 7fdd21ee6640 10 monclient: monmap: 2024-09-16T04:48:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:48:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:48:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:48:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:48:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:48:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:48:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:48:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:48:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:48:07.064 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:48:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.047+0000 7fdd21ee6640 1 Processor -- start 2024-09-16T04:48:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.047+0000 7fdd21ee6640 1 -- start start 2024-09-16T04:48:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.047+0000 7fdd21ee6640 10 monclient: init 2024-09-16T04:48:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd21ee6640 10 monclient: _reopen_session rank -1 2024-09-16T04:48:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd21ee6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:48:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd21ee6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a3c10 0x7fdd1c1cb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:48:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd21ee6640 10 monclient(hunting): picked mon.noname-a con 0x7fdd1c0a3c10 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:48:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd21ee6640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdd1c0a3ff0 0x7fdd1c136d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:48:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd21ee6640 10 monclient(hunting): picked mon.noname-c con 0x7fdd1c0a3ff0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:48:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd21ee6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd1c13be90 0x7fdd1c0a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:48:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd21ee6640 10 monclient(hunting): picked mon.noname-b con 0x7fdd1c13be90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:48:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd21ee6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:48:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd21ee6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1c06e550 con 0x7fdd1c0a3c10 2024-09-16T04:48:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd21ee6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:48:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd21ee6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1c06e3d0 con 0x7fdd1c13be90 2024-09-16T04:48:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd21ee6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:48:07.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd21ee6640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7fdd1c13e7c0 con 0x7fdd1c0a3ff0 2024-09-16T04:48:07.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd21ee6640 10 monclient(hunting): _renew_subs 2024-09-16T04:48:07.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd21ee6640 10 monclient(hunting): authenticate will time out at 1285.464246s 2024-09-16T04:48:07.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a3c10 0x7fdd1c1cb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:48:07.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a3c10 0x7fdd1c1cb840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48438/0 (socket says 172.21.15.46:48438) 2024-09-16T04:48:07.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1b7fe640 1 -- 172.21.15.46:0/2042455677 learned_addr learned my addr 172.21.15.46:0/2042455677 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:48:07.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1b7fe640 10 monclient(hunting): get_auth_request con 0x7fdd1c0a3c10 auth_method 0 2024-09-16T04:48:07.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1affd640 1 --2- 172.21.15.46:0/2042455677 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdd1c0a3ff0 0x7fdd1c136d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:48:07.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:48:07.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:48:07.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:48:07.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1affd640 10 monclient(hunting): get_auth_request con 0x7fdd1c0a3ff0 auth_method 0 2024-09-16T04:48:07.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:48:07.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:48:07.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:48:07.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:48:07.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:48:07.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:48:07.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:48:07.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:48:07.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:48:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1bfff640 1 --2- 172.21.15.46:0/2042455677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd1c13be90 0x7fdd1c0a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:48:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1bfff640 10 monclient(hunting): get_auth_request con 0x7fdd1c13be90 auth_method 0 2024-09-16T04:48:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:48:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:48:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:48:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1b7fe640 10 monclient(hunting): handle_auth_done global_id 104412 payload 824 2024-09-16T04:48:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1b7fe640 1 -- 172.21.15.46:0/2042455677 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdd1c0a3ff0 msgr2=0x7fdd1c136d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:48:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1b7fe640 1 --2- 172.21.15.46:0/2042455677 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdd1c0a3ff0 0x7fdd1c136d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:48:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1b7fe640 1 -- 172.21.15.46:0/2042455677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd1c13be90 msgr2=0x7fdd1c0a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:48:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1b7fe640 1 --2- 172.21.15.46:0/2042455677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd1c13be90 0x7fdd1c0a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:48:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:48:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1b7fe640 1 monclient: found mon.noname-a 2024-09-16T04:48:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:48:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.46:3300/0 2024-09-16T04:48:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1b7fe640 1 -- 172.21.15.46:0/2042455677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd1c1cc760 con 0x7fdd1c0a3c10 2024-09-16T04:48:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.048+0000 7fdd1b7fe640 1 --2- 172.21.15.46:0/2042455677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a3c10 0x7fdd1c1cb840 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00c4d0 tx=0x7fdd0c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:48:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd03fff640 1 -- 172.21.15.46:0/2042455677 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c018070 con 0x7fdd1c0a3c10 2024-09-16T04:48:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:48:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd03fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:48:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd03fff640 10 monclient: dump: 2024-09-16T04:48:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:48:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:48:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:48:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:48:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:48:07.070 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:48:07.070 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:48:07.070 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:48:07.070 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:48:07.070 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:48:07.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd03fff640 10 monclient: _finish_auth 0 2024-09-16T04:48:07.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd03fff640 10 monclient: _check_auth_tickets 2024-09-16T04:48:07.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:48:07.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd21ee6640 5 monclient: authenticate success, global_id 104412 2024-09-16T04:48:07.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd03fff640 1 -- 172.21.15.46:0/2042455677 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd0c007500 con 0x7fdd1c0a3c10 2024-09-16T04:48:07.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd21ee6640 10 monclient: _renew_subs 2024-09-16T04:48:07.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd21ee6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:48:07.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd21ee6640 1 -- 172.21.15.46:0/2042455677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd1c11ff90 con 0x7fdd1c0a3c10 2024-09-16T04:48:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd03fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:48:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd03fff640 1 -- 172.21.15.46:0/2042455677 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c004a20 con 0x7fdd1c0a3c10 2024-09-16T04:48:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:48:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:48:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd03fff640 10 monclient: dump: 2024-09-16T04:48:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:48:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:48:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:48:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:48:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:48:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:48:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:48:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:48:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:48:07.072 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:48:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd21ee6640 10 monclient: _renew_subs 2024-09-16T04:48:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd21ee6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:48:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd21ee6640 1 -- 172.21.15.46:0/2042455677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd1c13c6a0 con 0x7fdd1c0a3c10 2024-09-16T04:48:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd19ffb640 4 set_mon_vals no callback set 2024-09-16T04:48:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.049+0000 7fdd03fff640 1 -- 172.21.15.46:0/2042455677 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7fdd0c01b070 con 0x7fdd1c0a3c10 2024-09-16T04:48:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.050+0000 7fdd21ee6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:48:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.050+0000 7fdd21ee6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:48:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.050+0000 7fdd21ee6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:48:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.050+0000 7fdd21ee6640 1 -- 172.21.15.46:0/2042455677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd1c11fd10 con 0x7fdd1c0a3c10 2024-09-16T04:48:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.050+0000 7fdd03fff640 1 --2- 172.21.15.46:0/2042455677 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdd080383b0 0x7fdd0803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:48:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.050+0000 7fdd03fff640 1 -- 172.21.15.46:0/2042455677 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7fdd0c04aa00 con 0x7fdd1c0a3c10 2024-09-16T04:48:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.050+0000 7fdd1affd640 1 --2- 172.21.15.46:0/2042455677 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdd080383b0 0x7fdd0803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:48:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.050+0000 7fdd1affd640 10 monclient: get_auth_request con 0x7fdd080383b0 auth_method 0 2024-09-16T04:48:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.051+0000 7fdd1affd640 1 --2- 172.21.15.46:0/2042455677 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdd080383b0 0x7fdd0803a870 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fdd10034000 tx=0x7fdd1002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:48:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.055+0000 7fdd03fff640 1 -- 172.21.15.46:0/2042455677 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fdd0c01ae10 con 0x7fdd1c0a3c10 2024-09-16T04:48:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.055+0000 7fdd03fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:48:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.055+0000 7fdd03fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:48:07.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.229+0000 7fdd21ee6640 1 -- 172.21.15.46:0/2042455677 --> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdd1c067e40 con 0x7fdd080383b0 2024-09-16T04:48:07.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.231+0000 7fdd03fff640 1 -- 172.21.15.46:0/2042455677 <== mgr.4099 v2:172.21.15.114:6800/3521677055 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33970 (secure 0 0 0) 0x7fdd1c067e40 con 0x7fdd080383b0 2024-09-16T04:48:07.233 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-09-16T04:48:07.233 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-09-16T04:48:07.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.233+0000 7fdd21ee6640 1 -- 172.21.15.46:0/2042455677 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdd080383b0 msgr2=0x7fdd0803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:48:07.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.233+0000 7fdd21ee6640 1 --2- 172.21.15.46:0/2042455677 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdd080383b0 0x7fdd0803a870 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fdd10034000 tx=0x7fdd1002e000 comp rx=0 tx=0).stop 2024-09-16T04:48:07.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.233+0000 7fdd21ee6640 10 monclient: shutdown 2024-09-16T04:48:07.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.233+0000 7fdd21ee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:48:07.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.233+0000 7fdd21ee6640 1 -- 172.21.15.46:0/2042455677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a3c10 msgr2=0x7fdd1c1cb840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:48:07.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.233+0000 7fdd21ee6640 1 --2- 172.21.15.46:0/2042455677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a3c10 0x7fdd1c1cb840 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00c4d0 tx=0x7fdd0c00c990 comp rx=0 tx=0).stop 2024-09-16T04:48:07.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.233+0000 7fdd21ee6640 1 -- 172.21.15.46:0/2042455677 shutdown_connections 2024-09-16T04:48:07.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.233+0000 7fdd21ee6640 1 --2- 172.21.15.46:0/2042455677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd1c13be90 0x7fdd1c0a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:48:07.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.233+0000 7fdd21ee6640 1 --2- 172.21.15.46:0/2042455677 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7fdd1c0a3ff0 0x7fdd1c136d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:48:07.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.233+0000 7fdd21ee6640 1 --2- 172.21.15.46:0/2042455677 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7fdd080383b0 0x7fdd0803a870 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:48:07.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.233+0000 7fdd21ee6640 1 --2- 172.21.15.46:0/2042455677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1c0a3c10 0x7fdd1c1cb840 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:48:07.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.233+0000 7fdd21ee6640 1 -- 172.21.15.46:0/2042455677 >> 172.21.15.46:0/2042455677 conn(0x7fdd1c0a5d10 msgr2=0x7fdd1c09f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:48:07.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.233+0000 7fdd21ee6640 1 -- 172.21.15.46:0/2042455677 shutdown_connections 2024-09-16T04:48:07.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.233+0000 7fdd21ee6640 1 -- 172.21.15.46:0/2042455677 wait complete. 2024-09-16T04:48:07.245 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":588,"stamp":"2024-09-16T04:48:05.464969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124,"ondisk_log_size":124,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":217036,"kb_used_data":25768,"kb_used_omap":9,"kb_used_meta":191158,"kb_avail":566014004,"statfs":{"total":579820584960,"available":579598340096,"internally_reserved":0,"allocated":26386432,"data_stored":10465252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":195746464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002574"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":607,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:49.156160+0000","last_change":"2024-09-16T04:47:49.156184+0000","last_active":"2024-09-16T04:47:49.156160+0000","last_peered":"2024-09-16T04:47:49.156160+0000","last_clean":"2024-09-16T04:47:49.156160+0000","last_became_active":"2024-09-16T04:38:58.756778+0000","last_became_peered":"2024-09-16T04:38:58.756778+0000","last_unstale":"2024-09-16T04:47:49.156160+0000","last_undegraded":"2024-09-16T04:47:49.156160+0000","last_fullsized":"2024-09-16T04:47:49.156160+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:47:49.156065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:47:49.156065+0000","last_clean_scrub_stamp":"2024-09-16T04:47:49.156065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:22:09.221736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054536700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":600,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:44.929583+0000","last_change":"2024-09-16T04:47:44.929625+0000","last_active":"2024-09-16T04:47:44.929583+0000","last_peered":"2024-09-16T04:47:44.929583+0000","last_clean":"2024-09-16T04:47:44.929583+0000","last_became_active":"2024-09-16T04:38:58.520473+0000","last_became_peered":"2024-09-16T04:38:58.520473+0000","last_unstale":"2024-09-16T04:47:44.929583+0000","last_undegraded":"2024-09-16T04:47:44.929583+0000","last_fullsized":"2024-09-16T04:47:44.929583+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:47:44.929503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:47:44.929503+0000","last_clean_scrub_stamp":"2024-09-16T04:47:44.929503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:27:33.549754+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00053532500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":607,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:47.146513+0000","last_change":"2024-09-16T04:47:47.146537+0000","last_active":"2024-09-16T04:47:47.146513+0000","last_peered":"2024-09-16T04:47:47.146513+0000","last_clean":"2024-09-16T04:47:47.146513+0000","last_became_active":"2024-09-16T04:38:58.521000+0000","last_became_peered":"2024-09-16T04:38:58.521000+0000","last_unstale":"2024-09-16T04:47:47.146513+0000","last_undegraded":"2024-09-16T04:47:47.146513+0000","last_fullsized":"2024-09-16T04:47:47.146513+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:47:47.146403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:47:47.146403+0000","last_clean_scrub_stamp":"2024-09-16T04:47:47.146403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:18.169073+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00035484999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":600,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:45.909142+0000","last_change":"2024-09-16T04:47:45.909163+0000","last_active":"2024-09-16T04:47:45.909142+0000","last_peered":"2024-09-16T04:47:45.909142+0000","last_clean":"2024-09-16T04:47:45.909142+0000","last_became_active":"2024-09-16T04:38:58.521627+0000","last_became_peered":"2024-09-16T04:38:58.521627+0000","last_unstale":"2024-09-16T04:47:45.909142+0000","last_undegraded":"2024-09-16T04:47:45.909142+0000","last_fullsized":"2024-09-16T04:47:45.909142+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:47:45.909066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:47:45.909066+0000","last_clean_scrub_stamp":"2024-09-16T04:47:45.909066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:28.341131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062256500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"315'3","reported_seq":604,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:46.894662+0000","last_change":"2024-09-16T04:47:46.894690+0000","last_active":"2024-09-16T04:47:46.894662+0000","last_peered":"2024-09-16T04:47:46.894662+0000","last_clean":"2024-09-16T04:47:46.894662+0000","last_became_active":"2024-09-16T04:38:58.522744+0000","last_became_peered":"2024-09-16T04:38:58.522744+0000","last_unstale":"2024-09-16T04:47:46.894662+0000","last_undegraded":"2024-09-16T04:47:46.894662+0000","last_fullsized":"2024-09-16T04:47:46.894662+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'3","last_scrub_stamp":"2024-09-16T04:47:46.894588+0000","last_deep_scrub":"315'3","last_deep_scrub_stamp":"2024-09-16T04:47:46.894588+0000","last_clean_scrub_stamp":"2024-09-16T04:47:46.894588+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:14:45.893579+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000966258,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":610,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:45.669685+0000","last_change":"2024-09-16T04:47:45.669721+0000","last_active":"2024-09-16T04:47:45.669685+0000","last_peered":"2024-09-16T04:47:45.669685+0000","last_clean":"2024-09-16T04:47:45.669685+0000","last_became_active":"2024-09-16T04:38:58.524372+0000","last_became_peered":"2024-09-16T04:38:58.524372+0000","last_unstale":"2024-09-16T04:47:45.669685+0000","last_undegraded":"2024-09-16T04:47:45.669685+0000","last_fullsized":"2024-09-16T04:47:45.669685+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:47:45.669597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:47:45.669597+0000","last_clean_scrub_stamp":"2024-09-16T04:47:45.669597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:25:46.829107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047533599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":607,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:48.173902+0000","last_change":"2024-09-16T04:47:48.173927+0000","last_active":"2024-09-16T04:47:48.173902+0000","last_peered":"2024-09-16T04:47:48.173902+0000","last_clean":"2024-09-16T04:47:48.173902+0000","last_became_active":"2024-09-16T04:38:58.522142+0000","last_became_peered":"2024-09-16T04:38:58.522142+0000","last_unstale":"2024-09-16T04:47:48.173902+0000","last_undegraded":"2024-09-16T04:47:48.173902+0000","last_fullsized":"2024-09-16T04:47:48.173902+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:47:48.173809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:47:48.173809+0000","last_clean_scrub_stamp":"2024-09-16T04:47:48.173809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:42:23.670084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039704200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"315'2","reported_seq":603,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:47.907284+0000","last_change":"2024-09-16T04:47:47.907335+0000","last_active":"2024-09-16T04:47:47.907284+0000","last_peered":"2024-09-16T04:47:47.907284+0000","last_clean":"2024-09-16T04:47:47.907284+0000","last_became_active":"2024-09-16T04:38:58.523268+0000","last_became_peered":"2024-09-16T04:38:58.523268+0000","last_unstale":"2024-09-16T04:47:47.907284+0000","last_undegraded":"2024-09-16T04:47:47.907284+0000","last_fullsized":"2024-09-16T04:47:47.907284+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'2","last_scrub_stamp":"2024-09-16T04:47:47.907167+0000","last_deep_scrub":"315'2","last_deep_scrub_stamp":"2024-09-16T04:47:47.907167+0000","last_clean_scrub_stamp":"2024-09-16T04:47:47.907167+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:59:50.770669+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00043992499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"315'119","reported_seq":760,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T04:47:46.140181+0000","last_change":"2024-09-16T04:47:46.140215+0000","last_active":"2024-09-16T04:47:46.140181+0000","last_peered":"2024-09-16T04:47:46.140181+0000","last_clean":"2024-09-16T04:47:46.140181+0000","last_became_active":"2024-09-16T04:38:52.500796+0000","last_became_peered":"2024-09-16T04:38:52.500796+0000","last_unstale":"2024-09-16T04:47:46.140181+0000","last_undegraded":"2024-09-16T04:47:46.140181+0000","last_fullsized":"2024-09-16T04:47:46.140181+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'119","last_scrub_stamp":"2024-09-16T04:47:46.140099+0000","last_deep_scrub":"315'119","last_deep_scrub_stamp":"2024-09-16T04:47:46.140099+0000","last_clean_scrub_stamp":"2024-09-16T04:47:46.140099+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:50.017593+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607666,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31148,"kb_used_data":4120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340692,"statfs":{"total":96636764160,"available":96604868608,"internally_reserved":0,"allocated":4218880,"data_stored":1567893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607666,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":4640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":4751360,"data_stored":2096821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607666,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39984,"kb_used_data":4640,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94331856,"statfs":{"total":96636764160,"available":96595820544,"internally_reserved":0,"allocated":4751360,"data_stored":2096821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640370,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43628,"kb_used_data":4124,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94328212,"statfs":{"total":96636764160,"available":96592089088,"internally_reserved":0,"allocated":4222976,"data_stored":1567912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31144,"kb_used_data":4120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340696,"statfs":{"total":96636764160,"available":96604872704,"internally_reserved":0,"allocated":4218880,"data_stored":1567893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640369,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39468,"kb_used_data":4124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332372,"statfs":{"total":96636764160,"available":96596348928,"internally_reserved":0,"allocated":4222976,"data_stored":1567912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:48:07.247 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T04:48:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.346+0000 7f24ab903640 10 monclient: get_monmap_and_config 2024-09-16T04:48:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.346+0000 7f24ab903640 10 monclient: build_initial_monmap 2024-09-16T04:48:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.346+0000 7f24ab903640 10 monclient: monmap: 2024-09-16T04:48:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:48:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:48:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:48:07.347467+0000 2024-09-16T04:48:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:48:07.347467+0000 2024-09-16T04:48:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:48:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:48:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a-legacy 2024-09-16T04:48:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.114:6789/0 mon.noname-c-legacy 2024-09-16T04:48:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-b 2024-09-16T04:48:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:3: v2:172.21.15.46:3300/0 mon.noname-a 2024-09-16T04:48:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:4: v2:172.21.15.114:3300/0 mon.noname-c 2024-09-16T04:48:07.349 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:48:07.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.346+0000 7f24ab903640 1 Processor -- start 2024-09-16T04:48:07.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.346+0000 7f24ab903640 1 -- start start 2024-09-16T04:48:07.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.346+0000 7f24ab903640 10 monclient: init 2024-09-16T04:48:07.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24ab903640 10 monclient: _reopen_session rank -1 2024-09-16T04:48:07.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24ab903640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-16T04:48:07.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24ab903640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f24a40a3c10 0x7f24a409bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:48:07.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24ab903640 10 monclient(hunting): picked mon.noname-c con 0x7f24a40a3c10 addr v2:172.21.15.114:3300/0 2024-09-16T04:48:07.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24ab903640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f24a40a3ff0 addr v1:172.21.15.114:6789/0 2024-09-16T04:48:07.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24ab903640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f24a413be90 addr v1:172.21.15.46:6789/0 2024-09-16T04:48:07.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24ab903640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24a409b940 con 0x7f24a413be90 2024-09-16T04:48:07.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24ab903640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24a409ab00 con 0x7f24a40a3ff0 2024-09-16T04:48:07.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24ab903640 10 monclient(hunting): start opening mon connection 2024-09-16T04:48:07.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24ab903640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f24a409b030 con 0x7f24a40a3c10 2024-09-16T04:48:07.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24ab903640 10 monclient(hunting): _renew_subs 2024-09-16T04:48:07.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24ab903640 10 monclient(hunting): authenticate will time out at 1285.763405s 2024-09-16T04:48:07.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24a9ea0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f24a413be90 0x7f24a413e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49810/0 (socket says 172.21.15.46:49810) 2024-09-16T04:48:07.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24a9ea0640 1 -- 172.21.15.46:0/3220524717 learned_addr learned my addr 172.21.15.46:0/3220524717 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-09-16T04:48:07.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24a969f640 1 --2- 172.21.15.46:0/3220524717 >> v2:172.21.15.114:3300/0 conn(0x7f24a40a3c10 0x7f24a409bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:48:07.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24a969f640 10 monclient(hunting): get_auth_request con 0x7f24a40a3c10 auth_method 0 2024-09-16T04:48:07.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24a969f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:48:07.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24a969f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:48:07.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24a969f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:48:07.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f249bfff640 1 -- 172.21.15.46:0/3220524717 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1984756073 0 0) 0x7f24a409b940 con 0x7f24a413be90 2024-09-16T04:48:07.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f249bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:48:07.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f249bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:48:07.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f249bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:48:07.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f249bfff640 10 monclient(hunting): my global_id is 104415 2024-09-16T04:48:07.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f249bfff640 1 -- 172.21.15.46:0/3220524717 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2490003520 con 0x7f24a413be90 2024-09-16T04:48:07.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24a969f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:48:07.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24a969f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:48:07.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f24a969f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:48:07.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f249bfff640 1 -- 172.21.15.46:0/3220524717 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1707082655 0 0) 0x7f2490003520 con 0x7f24a413be90 2024-09-16T04:48:07.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f249bfff640 1 -- 172.21.15.46:0/3220524717 >> v2:172.21.15.114:3300/0 conn(0x7f24a40a3c10 msgr2=0x7f24a409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:48:07.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.347+0000 7f249bfff640 1 --2- 172.21.15.46:0/3220524717 >> v2:172.21.15.114:3300/0 conn(0x7f24a40a3c10 0x7f24a409bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:48:07.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 1 -- 172.21.15.46:0/3220524717 >> v1:172.21.15.114:6789/0 conn(0x7f24a40a3ff0 legacy=0x7f24a40a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:48:07.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:48:07.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:48:07.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:48:07.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.46:6789/0 2024-09-16T04:48:07.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 1 -- 172.21.15.46:0/3220524717 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a409b310 con 0x7f24a413be90 2024-09-16T04:48:07.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 10 monclient: _finish_auth 0 2024-09-16T04:48:07.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:48:07.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:48:07.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 1 -- 172.21.15.46:0/3220524717 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3110049130 0 0) 0x7f249c0019f0 con 0x7f24a413be90 2024-09-16T04:48:07.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24ab903640 5 monclient: authenticate success, global_id 104415 2024-09-16T04:48:07.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24ab903640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:48:07.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:48:07.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a969f640 1 --2- 172.21.15.46:0/3220524717 >> v2:172.21.15.114:3300/0 conn(0x7f24a40a3c10 0x7f24a409bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:48:07.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:48:07.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 10 monclient: dump: 2024-09-16T04:48:07.362 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:48:07.362 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:48:07.362 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:48:07.362 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:48:07.362 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:48:07.362 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:48:07.362 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:48:07.362 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:48:07.363 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:48:07.363 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:48:07.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] but i'm connected to v1:172.21.15.46:6789/0, reconnecting 2024-09-16T04:48:07.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 10 monclient: _reopen_session rank -1 2024-09-16T04:48:07.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 1 -- 172.21.15.46:0/3220524717 >> v1:172.21.15.46:6789/0 conn(0x7f24a413be90 legacy=0x7f24a413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:48:07.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:48:07.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 1 --2- 172.21.15.46:0/3220524717 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f24900039b0 0x7f2490003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:48:07.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 10 monclient(hunting): picked mon.c con 0x7f24900039b0 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:48:07.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 1 --2- 172.21.15.46:0/3220524717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2490004800 0x7f2490004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:48:07.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 10 monclient(hunting): picked mon.a con 0x7f2490004800 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:48:07.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 1 --2- 172.21.15.46:0/3220524717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24900052e0 0x7f249000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:48:07.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 10 monclient(hunting): picked mon.b con 0x7f24900052e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:48:07.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:48:07.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 1 -- 172.21.15.46:0/3220524717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f24a409b030 con 0x7f2490004800 2024-09-16T04:48:07.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:48:07.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 1 -- 172.21.15.46:0/3220524717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f249000c1b0 con 0x7f24900052e0 2024-09-16T04:48:07.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:48:07.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a8e9e640 1 --2- 172.21.15.46:0/3220524717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2490004800 0x7f2490004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:48:07.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 1 -- 172.21.15.46:0/3220524717 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f249000c3b0 con 0x7f24900039b0 2024-09-16T04:48:07.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 10 monclient(hunting): _renew_subs 2024-09-16T04:48:07.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f249bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:48:07.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a969f640 1 --2- 172.21.15.46:0/3220524717 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f24900039b0 0x7f2490003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:48:07.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24ab903640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:48:07.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a8e9e640 10 monclient(hunting): get_auth_request con 0x7f2490004800 auth_method 0 2024-09-16T04:48:07.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a8e9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:48:07.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a8e9e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:48:07.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a8e9e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:48:07.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a969f640 10 monclient(hunting): get_auth_request con 0x7f24900039b0 auth_method 0 2024-09-16T04:48:07.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a969f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:48:07.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a969f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:48:07.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a969f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:48:07.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a8e9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:48:07.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a8e9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:48:07.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a8e9e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:48:07.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a9ea0640 1 --2- 172.21.15.46:0/3220524717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24900052e0 0x7f249000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:48:07.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a9ea0640 10 monclient(hunting): get_auth_request con 0x7f24900052e0 auth_method 0 2024-09-16T04:48:07.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a9ea0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:48:07.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a9ea0640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:48:07.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a9ea0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:48:07.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a969f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:48:07.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a969f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:48:07.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a969f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:48:07.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a9ea0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:48:07.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a9ea0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:48:07.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a9ea0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:48:07.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a8e9e640 10 monclient(hunting): handle_auth_done global_id 104415 payload 293 2024-09-16T04:48:07.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a8e9e640 1 -- 172.21.15.46:0/3220524717 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f24900039b0 msgr2=0x7f2490003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:48:07.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a8e9e640 1 --2- 172.21.15.46:0/3220524717 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f24900039b0 0x7f2490003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:48:07.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a8e9e640 1 -- 172.21.15.46:0/3220524717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24900052e0 msgr2=0x7f249000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:48:07.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a8e9e640 1 --2- 172.21.15.46:0/3220524717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24900052e0 0x7f249000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:48:07.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a8e9e640 10 monclient: _finish_hunting 0 2024-09-16T04:48:07.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a8e9e640 1 monclient: found mon.a 2024-09-16T04:48:07.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a8e9e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:48:07.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a8e9e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.46:3300/0 2024-09-16T04:48:07.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a8e9e640 1 -- 172.21.15.46:0/3220524717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2490003520 con 0x7f2490004800 2024-09-16T04:48:07.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a9ea0640 1 -- 172.21.15.46:0/3220524717 reap_dead start 2024-09-16T04:48:07.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a8e9e640 10 monclient: _finish_auth 0 2024-09-16T04:48:07.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a8e9e640 10 monclient: _check_auth_tickets 2024-09-16T04:48:07.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.348+0000 7f24a8e9e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:48:07.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.349+0000 7f24a8e9e640 1 --2- 172.21.15.46:0/3220524717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2490004800 0x7f2490004c80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2494001720 tx=0x7f2490005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf64d57c8c67d0b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:48:07.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.349+0000 7f249bfff640 1 -- 172.21.15.46:0/3220524717 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2494001370 con 0x7f2490004800 2024-09-16T04:48:07.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.349+0000 7f249bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:48:07.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.349+0000 7f249bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:48:07.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.349+0000 7f249bfff640 10 monclient: dump: 2024-09-16T04:48:07.367 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:48:07.367 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:48:07.367 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:48:07.367 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:48:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:48:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:48:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:48:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:48:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:48:07.368 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:48:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.349+0000 7f249bfff640 1 -- 172.21.15.46:0/3220524717 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24940052d0 con 0x7f2490004800 2024-09-16T04:48:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.349+0000 7f24ab903640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:48:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.349+0000 7f249bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:48:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.349+0000 7f24ab903640 10 monclient: get_monmap_and_config success 2024-09-16T04:48:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.349+0000 7f249bfff640 1 -- 172.21.15.46:0/3220524717 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249402ab90 con 0x7f2490004800 2024-09-16T04:48:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.349+0000 7f249bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:48:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.349+0000 7f24ab903640 4 set_mon_vals no callback set 2024-09-16T04:48:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.349+0000 7f249bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:48:07.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.349+0000 7f249bfff640 10 monclient: dump: 2024-09-16T04:48:07.369 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:48:07.369 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:48:07.369 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:48:07.369 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:48:07.369 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:48:07.369 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:48:07.369 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:48:07.369 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:48:07.369 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:48:07.369 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:48:07.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.349+0000 7f24ab903640 10 monclient: shutdown 2024-09-16T04:48:07.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.349+0000 7f24ab903640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:48:07.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.349+0000 7f24ab903640 1 -- 172.21.15.46:0/3220524717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2490004800 msgr2=0x7f2490004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:48:07.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.349+0000 7f24ab903640 1 --2- 172.21.15.46:0/3220524717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2490004800 0x7f2490004c80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2494001720 tx=0x7f2490005830 comp rx=0 tx=0).stop 2024-09-16T04:48:07.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.349+0000 7f24ab903640 1 -- 172.21.15.46:0/3220524717 shutdown_connections 2024-09-16T04:48:07.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.349+0000 7f24ab903640 1 --2- 172.21.15.46:0/3220524717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2490004800 0x7f2490004c80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:48:07.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.349+0000 7f24ab903640 1 -- 172.21.15.46:0/3220524717 >> 172.21.15.46:0/3220524717 conn(0x7f24a40a6340 msgr2=0x7f24a406d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:48:07.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.350+0000 7f24ab903640 1 -- 172.21.15.46:0/3220524717 shutdown_connections 2024-09-16T04:48:07.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.350+0000 7f24ab903640 1 -- 172.21.15.46:0/3220524717 wait complete. 2024-09-16T04:48:07.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.350+0000 7f24ab903640 10 monclient: build_initial_monmap 2024-09-16T04:48:07.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.350+0000 7f24ab903640 10 monclient: monmap: 2024-09-16T04:48:07.370 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-09-16T04:48:07.370 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:48:07.370 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-09-16T04:48:07.370 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-09-16T04:48:07.370 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-09-16T04:48:07.370 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:48:07.371 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.noname-a 2024-09-16T04:48:07.371 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-b 2024-09-16T04:48:07.371 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.noname-c 2024-09-16T04:48:07.371 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:48:07.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.350+0000 7f24ab903640 1 Processor -- start 2024-09-16T04:48:07.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.350+0000 7f24ab903640 1 -- start start 2024-09-16T04:48:07.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.350+0000 7f24ab903640 10 monclient: init 2024-09-16T04:48:07.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24ab903640 10 monclient: _reopen_session rank -1 2024-09-16T04:48:07.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24ab903640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:48:07.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24ab903640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2490004800 0x7f24a4120000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:48:07.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24ab903640 10 monclient(hunting): picked mon.noname-c con 0x7f2490004800 addr [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] 2024-09-16T04:48:07.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24ab903640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a4120540 0x7f24a41209a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:48:07.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24ab903640 10 monclient(hunting): picked mon.noname-a con 0x7f24a4120540 addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] 2024-09-16T04:48:07.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24ab903640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24a4120ee0 0x7f24a41d2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:48:07.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24ab903640 10 monclient(hunting): picked mon.noname-b con 0x7f24a4120ee0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T04:48:07.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24ab903640 10 monclient(hunting): start opening mon connection 2024-09-16T04:48:07.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24ab903640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f24a409be70 con 0x7f24a4120540 2024-09-16T04:48:07.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24ab903640 10 monclient(hunting): start opening mon connection 2024-09-16T04:48:07.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24ab903640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f24a409bcf0 con 0x7f24a4120ee0 2024-09-16T04:48:07.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24ab903640 10 monclient(hunting): start opening mon connection 2024-09-16T04:48:07.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24ab903640 1 -- --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_getmap magic: 0 -- 0x7f24a409bff0 con 0x7f2490004800 2024-09-16T04:48:07.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24ab903640 10 monclient(hunting): _renew_subs 2024-09-16T04:48:07.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24ab903640 10 monclient(hunting): authenticate will time out at 1285.767414s 2024-09-16T04:48:07.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a969f640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2490004800 0x7f24a4120000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:48:07.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a8e9e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a4120540 0x7f24a41209a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:48:07.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a969f640 1 --2- >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2490004800 0x7f24a4120000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3301/0 says I am v2:172.21.15.46:45672/0 (socket says 172.21.15.46:45672) 2024-09-16T04:48:07.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a969f640 1 -- 172.21.15.46:0/245231167 learned_addr learned my addr 172.21.15.46:0/245231167 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-09-16T04:48:07.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a969f640 10 monclient(hunting): get_auth_request con 0x7f2490004800 auth_method 0 2024-09-16T04:48:07.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a969f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:48:07.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a969f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:48:07.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a969f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:48:07.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a8e9e640 10 monclient(hunting): get_auth_request con 0x7f24a4120540 auth_method 0 2024-09-16T04:48:07.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a8e9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:48:07.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a8e9e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:48:07.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a8e9e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:48:07.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a969f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:48:07.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a969f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:48:07.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a969f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:48:07.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a8e9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:48:07.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a8e9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:48:07.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a8e9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:48:07.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a9ea0640 1 --2- 172.21.15.46:0/245231167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24a4120ee0 0x7f24a41d2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:48:07.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a9ea0640 10 monclient(hunting): get_auth_request con 0x7f24a4120ee0 auth_method 0 2024-09-16T04:48:07.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a9ea0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:48:07.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a9ea0640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:48:07.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a9ea0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:48:07.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a969f640 10 monclient(hunting): handle_auth_done global_id 94532 payload 824 2024-09-16T04:48:07.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a969f640 1 -- 172.21.15.46:0/245231167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24a4120ee0 msgr2=0x7f24a41d2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:48:07.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a969f640 1 --2- 172.21.15.46:0/245231167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24a4120ee0 0x7f24a41d2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:48:07.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a969f640 1 -- 172.21.15.46:0/245231167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a4120540 msgr2=0x7f24a41209a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:48:07.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a969f640 1 --2- 172.21.15.46:0/245231167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a4120540 0x7f24a41209a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:48:07.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a969f640 10 monclient: _finish_hunting 0 2024-09-16T04:48:07.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a969f640 1 monclient: found mon.noname-c 2024-09-16T04:48:07.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a969f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:48:07.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a969f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.46:3301/0 2024-09-16T04:48:07.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a969f640 1 -- 172.21.15.46:0/245231167 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a4123a40 con 0x7f2490004800 2024-09-16T04:48:07.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.351+0000 7f24a8e9e640 1 --2- 172.21.15.46:0/245231167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a4120540 0x7f24a41209a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:48:07.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f24a969f640 1 --2- 172.21.15.46:0/245231167 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2490004800 0x7f24a4120000 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f24a0011d00 tx=0x7f24a000f610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:48:07.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f2499ffb640 1 -- 172.21.15.46:0/245231167 <== mon.2 v2:172.21.15.46:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a000fd20 con 0x7f2490004800 2024-09-16T04:48:07.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f2499ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:48:07.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f2499ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:48:07.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f2499ffb640 10 monclient: dump: 2024-09-16T04:48:07.375 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:48:07.375 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:48:07.375 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:48:07.375 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:48:07.375 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:48:07.375 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:48:07.375 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:48:07.375 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:48:07.375 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:48:07.376 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:48:07.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f2499ffb640 10 monclient: _finish_auth 0 2024-09-16T04:48:07.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f2499ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:48:07.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f2499ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:48:07.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f2499ffb640 1 -- 172.21.15.46:0/245231167 <== mon.2 v2:172.21.15.46:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24a0004030 con 0x7f2490004800 2024-09-16T04:48:07.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f24ab903640 5 monclient: authenticate success, global_id 94532 2024-09-16T04:48:07.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f2499ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:48:07.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f2499ffb640 1 -- 172.21.15.46:0/245231167 <== mon.2 v2:172.21.15.46:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a0004420 con 0x7f2490004800 2024-09-16T04:48:07.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f2499ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:48:07.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f2499ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:48:07.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f2499ffb640 10 monclient: dump: 2024-09-16T04:48:07.376 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-09-16T04:48:07.376 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 01e423f6-9e74-4519-981b-340ad8faba4f 2024-09-16T04:48:07.376 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-09-16T04:38:10.269597+0000 2024-09-16T04:48:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-09-16T04:38:10.269597+0000 2024-09-16T04:48:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-09-16T04:48:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-09-16T04:48:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-09-16T04:48:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T04:48:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2: [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] mon.c 2024-09-16T04:48:07.377 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:48:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f24ab903640 10 monclient: _renew_subs 2024-09-16T04:48:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f24ab903640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:48:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f24ab903640 1 -- 172.21.15.46:0/245231167 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24a4122310 con 0x7f2490004800 2024-09-16T04:48:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f249b7fe640 4 set_mon_vals no callback set 2024-09-16T04:48:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f24ab903640 10 monclient: _renew_subs 2024-09-16T04:48:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f24ab903640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:48:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.352+0000 7f24ab903640 1 -- 172.21.15.46:0/245231167 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f24a40a1040 con 0x7f2490004800 2024-09-16T04:48:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.353+0000 7f2499ffb640 1 -- 172.21.15.46:0/245231167 <== mon.2 v2:172.21.15.46:3301/0 4 ==== mgrmap(e 5) ==== 45737+0+0 (secure 0 0 0) 0x7f24a0012040 con 0x7f2490004800 2024-09-16T04:48:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.353+0000 7f24ab903640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:48:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.353+0000 7f24ab903640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:48:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.353+0000 7f24ab903640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:48:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.353+0000 7f24ab903640 1 -- 172.21.15.46:0/245231167 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24a40a0b10 con 0x7f2490004800 2024-09-16T04:48:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.353+0000 7f2499ffb640 1 --2- 172.21.15.46:0/245231167 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f24880383b0 0x7f248803a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:48:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.353+0000 7f2499ffb640 1 -- 172.21.15.46:0/245231167 <== mon.2 v2:172.21.15.46:3301/0 5 ==== osd_map(315..315 src has 1..315) ==== 4346+0+0 (secure 0 0 0) 0x7f24a004bc40 con 0x7f2490004800 2024-09-16T04:48:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.354+0000 7f24a8e9e640 1 --2- 172.21.15.46:0/245231167 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f24880383b0 0x7f248803a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:48:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.354+0000 7f24a8e9e640 10 monclient: get_auth_request con 0x7f24880383b0 auth_method 0 2024-09-16T04:48:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.354+0000 7f24a8e9e640 1 --2- 172.21.15.46:0/245231167 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f24880383b0 0x7f248803a870 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2494032000 tx=0x7f249402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:48:07.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.358+0000 7f2499ffb640 1 -- 172.21.15.46:0/245231167 <== mon.2 v2:172.21.15.46:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f24a0019070 con 0x7f2490004800 2024-09-16T04:48:07.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.358+0000 7f2499ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:48:07.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.358+0000 7f2499ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:48:07.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.534+0000 7f24ab903640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T04:48:07.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.534+0000 7f24ab903640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T04:48:07.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.534+0000 7f24ab903640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.46:3301/0 2024-09-16T04:48:07.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.534+0000 7f24ab903640 1 -- 172.21.15.46:0/245231167 --> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f24a40a43d0 con 0x7f2490004800 2024-09-16T04:48:07.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.538+0000 7f2499ffb640 1 -- 172.21.15.46:0/245231167 <== mon.2 v2:172.21.15.46:3301/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f24a0051080 con 0x7f2490004800 2024-09-16T04:48:07.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.538+0000 7f2499ffb640 10 monclient: handle_config config(1 keys) 2024-09-16T04:48:07.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.538+0000 7f249b7fe640 4 set_mon_vals no callback set 2024-09-16T04:48:07.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.538+0000 7f249b7fe640 10 set_mon_vals mon_health_to_clog = false 2024-09-16T04:48:07.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.538+0000 7f2499ffb640 1 -- 172.21.15.46:0/245231167 <== mon.2 v2:172.21.15.46:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v5)=0 v5) ==== 123+0+0 (secure 0 0 0) 0x7f24a0003e10 con 0x7f2490004800 2024-09-16T04:48:07.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.538+0000 7f2499ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T04:48:07.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.538+0000 7f2499ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:48:07.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.540+0000 7f24ab903640 1 -- 172.21.15.46:0/245231167 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f24880383b0 msgr2=0x7f248803a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:48:07.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.540+0000 7f24ab903640 1 --2- 172.21.15.46:0/245231167 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f24880383b0 0x7f248803a870 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2494032000 tx=0x7f249402e000 comp rx=0 tx=0).stop 2024-09-16T04:48:07.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.540+0000 7f24ab903640 10 monclient: shutdown 2024-09-16T04:48:07.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.540+0000 7f24ab903640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:48:07.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.540+0000 7f24ab903640 1 -- 172.21.15.46:0/245231167 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2490004800 msgr2=0x7f24a4120000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:48:07.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.540+0000 7f24ab903640 1 --2- 172.21.15.46:0/245231167 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2490004800 0x7f24a4120000 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f24a0011d00 tx=0x7f24a000f610 comp rx=0 tx=0).stop 2024-09-16T04:48:07.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.542+0000 7f24ab903640 1 -- 172.21.15.46:0/245231167 shutdown_connections 2024-09-16T04:48:07.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.542+0000 7f24ab903640 1 --2- 172.21.15.46:0/245231167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24a4120ee0 0x7f24a41d2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:48:07.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.542+0000 7f24ab903640 1 --2- 172.21.15.46:0/245231167 >> [v2:172.21.15.114:6800/3521677055,v1:172.21.15.114:6801/3521677055] conn(0x7f24880383b0 0x7f248803a870 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:48:07.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.542+0000 7f24ab903640 1 --2- 172.21.15.46:0/245231167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a4120540 0x7f24a41209a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:48:07.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.542+0000 7f24ab903640 1 --2- 172.21.15.46:0/245231167 >> [v2:172.21.15.46:3301/0,v1:172.21.15.46:6790/0] conn(0x7f2490004800 0x7f24a4120000 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:48:07.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.542+0000 7f24ab903640 1 -- 172.21.15.46:0/245231167 >> 172.21.15.46:0/245231167 conn(0x7f24a40a6340 msgr2=0x7f24a413b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:48:07.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.542+0000 7f24ab903640 1 -- 172.21.15.46:0/245231167 shutdown_connections 2024-09-16T04:48:07.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-09-16T04:48:07.542+0000 7f24ab903640 1 -- 172.21.15.46:0/245231167 wait complete. 2024-09-16T04:48:07.554 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T04:48:07.554 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T04:48:07.554 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T04:48:07.554 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:48:07.588 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T04:48:07.588 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T04:48:07.588 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:48:07.657 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T04:48:07.657 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T04:48:07.657 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:48:07.727 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T04:48:07.727 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T04:48:07.727 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:48:07.766 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T04:48:07.767 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T04:48:07.767 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:48:07.846 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T04:48:07.846 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T04:48:07.846 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:48:07.916 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T04:48:07.916 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T04:48:07.916 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T04:48:07.916 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:48:07.986 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T04:48:07.987 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T04:48:07.987 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T04:48:07.987 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:48:07.998 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T04:48:07.999 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:48:07.999 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:48:08.051 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:48:08.051 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T04:48:08.051 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:48:08.064 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T04:48:08.064 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T04:48:08.064 DEBUG:teuthology.orchestra.run.smithi046:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | head -n 1 2024-09-16T04:48:08.135 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi046.front.sepia.ceph.com 2024-09-16T04:48:08.135 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T04:48:08.386 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi046.front.sepia.ceph.com 2024-09-16T04:48:08.386 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T04:48:08.477 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi046.front.sepia.ceph.com 2024-09-16T04:48:08.477 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T04:48:08.543 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T04:48:08.543 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T04:48:08.678 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T04:48:08.678 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T04:48:08.782 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T04:48:08.783 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T04:48:08.841 INFO:tasks.ceph:Archiving mon data... 2024-09-16T04:48:08.842 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906031/data/mon.a.tgz 2024-09-16T04:48:08.842 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-09-16T04:48:08.859 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.rdNb3Qidsa 2024-09-16T04:48:08.859 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.rdNb3Qidsa 2024-09-16T04:48:09.036 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.rdNb3Qidsa 2024-09-16T04:48:09.121 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.rdNb3Qidsa is 516KB 2024-09-16T04:48:09.221 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.rdNb3Qidsa 2024-09-16T04:48:09.236 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906031/data/mon.c.tgz 2024-09-16T04:48:09.237 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-09-16T04:48:09.291 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.cMqYEJdA07 2024-09-16T04:48:09.291 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.cMqYEJdA07 2024-09-16T04:48:09.456 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.cMqYEJdA07 2024-09-16T04:48:09.541 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.cMqYEJdA07 is 519KB 2024-09-16T04:48:09.639 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.cMqYEJdA07 2024-09-16T04:48:09.654 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906031/data/mon.b.tgz 2024-09-16T04:48:09.654 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-09-16T04:48:09.673 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.YfZUcrvP7m 2024-09-16T04:48:09.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.YfZUcrvP7m 2024-09-16T04:48:09.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.YfZUcrvP7m 2024-09-16T04:48:09.936 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.YfZUcrvP7m is 516KB 2024-09-16T04:48:10.035 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.YfZUcrvP7m 2024-09-16T04:48:10.050 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T04:48:10.050 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T04:48:10.052 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T04:48:10.133 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T04:48:10.169 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T04:48:10.169 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T04:48:10.171 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906031/remote/smithi046/crash 2024-09-16T04:48:10.171 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T04:48:10.198 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906031/remote/smithi114/crash 2024-09-16T04:48:10.199 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T04:48:10.234 INFO:tasks.ceph:Compressing logs... 2024-09-16T04:48:10.234 DEBUG:teuthology.orchestra.run.smithi046:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T04:48:10.243 DEBUG:teuthology.orchestra.run.smithi114:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T04:48:10.266 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T04:48:10.266 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42677.log 2024-09-16T04:48:10.426 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37379.log 2024-09-16T04:48:10.427 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.42677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43630.log 2024-09-16T04:48:10.427 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.37379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41878.log 2024-09-16T04:48:10.427 INFO:teuthology.orchestra.run.smithi046.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.42677.log.gz 2024-09-16T04:48:10.427 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41996.log 2024-09-16T04:48:10.427 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43630.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37379.log.gz 2024-09-16T04:48:10.427 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38483.log 2024-09-16T04:48:10.427 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43630.log.gz 2024-09-16T04:48:10.427 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35533.log 2024-09-16T04:48:10.427 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41878.log.gz 2024-09-16T04:48:10.427 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.38483.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41996.log.gz 2024-09-16T04:48:10.427 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35294.log 2024-09-16T04:48:10.427 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43672.log 2024-09-16T04:48:10.427 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38483.log.gz 2024-09-16T04:48:10.428 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36998.log 2024-09-16T04:48:10.428 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35533.log.gz 2024-09-16T04:48:10.428 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43672.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35294.log.gz 2024-09-16T04:48:10.428 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40053.log 2024-09-16T04:48:10.428 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38044.log 2024-09-16T04:48:10.428 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36998.log: /var/log/ceph/ceph-client.admin.40053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40974.log 2024-09-16T04:48:10.428 INFO:teuthology.orchestra.run.smithi046.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.43672.log.gz 2024-09-16T04:48:10.428 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.38044.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36998.log.gz 2024-09-16T04:48:10.428 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37664.log 2024-09-16T04:48:10.428 INFO:teuthology.orchestra.run.smithi046.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.40053.log.gz 2024-09-16T04:48:10.428 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36330.log 2024-09-16T04:48:10.428 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40974.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.38044.log.gz 2024-09-16T04:48:10.429 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.37664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44557.log 2024-09-16T04:48:10.429 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40974.log.gz 2024-09-16T04:48:10.429 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45229.log 2024-09-16T04:48:10.429 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37664.log.gz 2024-09-16T04:48:10.429 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36118.log 2024-09-16T04:48:10.429 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36330.log.gz 2024-09-16T04:48:10.429 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45229.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44557.log.gz 2024-09-16T04:48:10.429 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40011.log 2024-09-16T04:48:10.429 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35546.log 2024-09-16T04:48:10.429 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45229.log.gz 2024-09-16T04:48:10.429 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39090.log 2024-09-16T04:48:10.429 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36118.log.gz 2024-09-16T04:48:10.429 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41836.log 2024-09-16T04:48:10.430 INFO:teuthology.orchestra.run.smithi046.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.40011.log.gz 2024-09-16T04:48:10.430 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37534.log 2024-09-16T04:48:10.430 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35546.log.gz 2024-09-16T04:48:10.430 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35992.log 2024-09-16T04:48:10.430 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39090.log.gz 2024-09-16T04:48:10.430 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.37534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44137.log 2024-09-16T04:48:10.430 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35992.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.41836.log.gz 2024-09-16T04:48:10.430 INFO:teuthology.orchestra.run.smithi046.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.37534.log.gz 2024-09-16T04:48:10.430 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44389.log 2024-09-16T04:48:10.430 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42549.log 2024-09-16T04:48:10.431 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44137.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35992.log.gz 2024-09-16T04:48:10.431 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43884.log 2024-09-16T04:48:10.431 INFO:teuthology.orchestra.run.smithi046.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.44137.log.gz 2024-09-16T04:48:10.431 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35290.log 2024-09-16T04:48:10.431 INFO:teuthology.orchestra.run.smithi046.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.44389.log.gz 2024-09-16T04:48:10.431 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43884.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T04:48:10.431 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42549.log.gz 2024-09-16T04:48:10.431 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43221.log 2024-09-16T04:48:10.431 INFO:teuthology.orchestra.run.smithi046.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.43884.log.gz 2024-09-16T04:48:10.431 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40890.log 2024-09-16T04:48:10.431 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35290.log.gz 2024-09-16T04:48:10.431 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37750.log 2024-09-16T04:48:10.431 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40890.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43221.log.gz 2024-09-16T04:48:10.432 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37834.log 2024-09-16T04:48:10.432 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42253.log 2024-09-16T04:48:10.432 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.37750.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.40890.log.gz 2024-09-16T04:48:10.432 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36244.log 2024-09-16T04:48:10.432 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.37834.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.37750.log.gz 2024-09-16T04:48:10.432 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45313.log 2024-09-16T04:48:10.432 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36244.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37834.log.gz 2024-09-16T04:48:10.432 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40552.log 2024-09-16T04:48:10.432 INFO:teuthology.orchestra.run.smithi046.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.42253.log.gz 2024-09-16T04:48:10.432 INFO:teuthology.orchestra.run.smithi046.stderr:gzip/var/log/ceph/ceph-client.admin.45313.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42423.log 2024-09-16T04:48:10.432 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36244.log.gz 2024-09-16T04:48:10.432 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41541.log 2024-09-16T04:48:10.432 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45313.log.gz 2024-09-16T04:48:10.432 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44263.log 2024-09-16T04:48:10.433 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42423.log: /var/log/ceph/ceph-client.admin.41541.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.40552.log.gz 2024-09-16T04:48:10.433 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40806.log 2024-09-16T04:48:10.433 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42423.log.gz 2024-09-16T04:48:10.433 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35288.log 2024-09-16T04:48:10.433 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41541.log.gz 2024-09-16T04:48:10.433 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44515.log 2024-09-16T04:48:10.433 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40806.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44263.log.gz 2024-09-16T04:48:10.433 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37792.log 2024-09-16T04:48:10.433 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37622.log 2024-09-16T04:48:10.433 INFO:teuthology.orchestra.run.smithi046.stderr: 89.9%/var/log/ceph/ceph-client.admin.44515.log: -- replaced with /var/log/ceph/ceph-client.admin.40806.log.gz 2024-09-16T04:48:10.433 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.37792.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35288.log.gz 2024-09-16T04:48:10.433 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41751.log 2024-09-16T04:48:10.433 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44515.log.gz 2024-09-16T04:48:10.433 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37337.log 2024-09-16T04:48:10.434 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.37622.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.37792.log.gz 2024-09-16T04:48:10.434 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39300.log 2024-09-16T04:48:10.434 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41751.log: /var/log/ceph/ceph-client.admin.37337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37082.log 2024-09-16T04:48:10.434 INFO:teuthology.orchestra.run.smithi046.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.37622.log.gz 2024-09-16T04:48:10.434 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41751.log.gz 2024-09-16T04:48:10.434 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37293.log 2024-09-16T04:48:10.434 INFO:teuthology.orchestra.run.smithi046.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.37337.log.gz 2024-09-16T04:48:10.434 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.37082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38693.log 2024-09-16T04:48:10.434 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.37293.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.39300.log.gz 2024-09-16T04:48:10.434 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39004.log 2024-09-16T04:48:10.434 INFO:teuthology.orchestra.run.smithi046.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.37082.log.gz 2024-09-16T04:48:10.434 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.38693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38313.log 2024-09-16T04:48:10.434 INFO:teuthology.orchestra.run.smithi046.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.37293.log.gz 2024-09-16T04:48:10.435 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39004.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38693.log.gz 2024-09-16T04:48:10.435 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41101.log 2024-09-16T04:48:10.435 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.38313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35293.log 2024-09-16T04:48:10.435 INFO:teuthology.orchestra.run.smithi046.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.39004.log.gz 2024-09-16T04:48:10.435 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44599.log 2024-09-16T04:48:10.435 INFO:teuthology.orchestra.run.smithi046.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.38313.log.gz 2024-09-16T04:48:10.435 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42465.log 2024-09-16T04:48:10.435 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35293.log: /var/log/ceph/ceph-client.admin.44599.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.41101.log.gz 2024-09-16T04:48:10.435 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35543.log 2024-09-16T04:48:10.435 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35293.log.gz 2024-09-16T04:48:10.435 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42465.log: 89.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.44599.log.gz -5 2024-09-16T04:48:10.435 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39216.log 2024-09-16T04:48:10.435 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44725.log 2024-09-16T04:48:10.436 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35543.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.42465.log.gz 2024-09-16T04:48:10.436 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40720.log 2024-09-16T04:48:10.436 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35543.log.gz 2024-09-16T04:48:10.436 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39384.log 2024-09-16T04:48:10.436 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39216.log.gz 2024-09-16T04:48:10.436 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41583.log 2024-09-16T04:48:10.436 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44725.log.gz 2024-09-16T04:48:10.436 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44893.log 2024-09-16T04:48:10.436 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40720.log.gz 2024-09-16T04:48:10.436 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41583.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39384.log.gz 2024-09-16T04:48:10.436 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43095.log 2024-09-16T04:48:10.436 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43968.log 2024-09-16T04:48:10.436 INFO:teuthology.orchestra.run.smithi046.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.41583.log.gz 2024-09-16T04:48:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40221.log 2024-09-16T04:48:10.437 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44893.log.gz 2024-09-16T04:48:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35068.log 2024-09-16T04:48:10.437 INFO:teuthology.orchestra.run.smithi046.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.43095.log.gz 2024-09-16T04:48:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41143.log 2024-09-16T04:48:10.437 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43968.log.gz 2024-09-16T04:48:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43587.log 2024-09-16T04:48:10.437 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40221.log.gz 2024-09-16T04:48:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44221.log 2024-09-16T04:48:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41143.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35068.log.gz 2024-09-16T04:48:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44851.log 2024-09-16T04:48:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44010.log 2024-09-16T04:48:10.437 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41143.log.gz 2024-09-16T04:48:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44221.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43587.log.gz 2024-09-16T04:48:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40678.log 2024-09-16T04:48:10.438 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44221.log.gz 2024-09-16T04:48:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37208.log 2024-09-16T04:48:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44010.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44851.log.gz 2024-09-16T04:48:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43137.log 2024-09-16T04:48:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40678.log: /var/log/ceph/ceph-client.admin.37208.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44010.log.gz 2024-09-16T04:48:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39174.log 2024-09-16T04:48:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43137.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37208.log.gz 2024-09-16T04:48:10.438 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40678.log.gz 2024-09-16T04:48:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35544.log 2024-09-16T04:48:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39799.log 2024-09-16T04:48:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39174.log: /var/log/ceph/ceph-client.admin.35544.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.43137.log.gz 2024-09-16T04:48:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35545.log 2024-09-16T04:48:10.439 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39174.log.gz 2024-09-16T04:48:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39799.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35544.log.gz 2024-09-16T04:48:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39426.log 2024-09-16T04:48:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35839.log 2024-09-16T04:48:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39426.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.39799.log.gz 2024-09-16T04:48:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35287.log 2024-09-16T04:48:10.439 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35545.log.gz 2024-09-16T04:48:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35839.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39426.log.gz 2024-09-16T04:48:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39132.log 2024-09-16T04:48:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose 87.9% -- -- replaced with /var/log/ceph/ceph-client.admin.35839.log.gz 2024-09-16T04:48:10.439 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/ceph-client.admin.36826.log 2024-09-16T04:48:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35287.log: /var/log/ceph/ceph-client.admin.39132.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T04:48:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36826.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35287.log.gz 2024-09-16T04:48:10.440 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35289.log 2024-09-16T04:48:10.440 INFO:teuthology.orchestra.run.smithi046.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.39132.log.gz 2024-09-16T04:48:10.440 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45271.log 2024-09-16T04:48:10.440 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.audit.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.36826.log.gz 2024-09-16T04:48:10.440 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39841.log 2024-09-16T04:48:10.440 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45271.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35289.log.gz 2024-09-16T04:48:10.440 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42079.log 2024-09-16T04:48:10.440 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44053.log 2024-09-16T04:48:10.440 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39841.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45271.log.gz 2024-09-16T04:48:10.440 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42835.log 2024-09-16T04:48:10.440 INFO:teuthology.orchestra.run.smithi046.stderr: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T04:48:10.440 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41329.log 2024-09-16T04:48:10.440 INFO:teuthology.orchestra.run.smithi046.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.39841.log.gz 2024-09-16T04:48:10.441 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38819.log 2024-09-16T04:48:10.441 INFO:teuthology.orchestra.run.smithi046.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.42079.log.gz 2024-09-16T04:48:10.441 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44053.log.gz 2024-09-16T04:48:10.441 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43053.log 2024-09-16T04:48:10.441 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38228.log 2024-09-16T04:48:10.441 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.38819.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.42835.log.gz 2024-09-16T04:48:10.441 INFO:teuthology.orchestra.run.smithi046.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.41329.log.gz 2024-09-16T04:48:10.441 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37452.log 2024-09-16T04:48:10.441 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42169.log 2024-09-16T04:48:10.441 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38819.log.gz 2024-09-16T04:48:10.441 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.38228.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43053.log.gz 2024-09-16T04:48:10.441 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.37452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45103.log 2024-09-16T04:48:10.441 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38228.log.gz 2024-09-16T04:48:10.442 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36286.log 2024-09-16T04:48:10.442 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45061.log 2024-09-16T04:48:10.442 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37452.log.gz 2024-09-16T04:48:10.442 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45103.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42169.log.gz 2024-09-16T04:48:10.442 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42967.log 2024-09-16T04:48:10.442 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose 88.8% -- /var/log/ceph/ceph-client.admin.41371.log -- replaced with /var/log/ceph/ceph-client.admin.45103.log.gz 2024-09-16T04:48:10.442 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:48:10.442 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45061.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36286.log.gz 2024-09-16T04:48:10.442 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36034.log 2024-09-16T04:48:10.442 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45061.log.gz 2024-09-16T04:48:10.442 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39552.log 2024-09-16T04:48:10.442 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44641.log 2024-09-16T04:48:10.442 INFO:teuthology.orchestra.run.smithi046.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.42967.log.gz 2024-09-16T04:48:10.442 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36034.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.41371.log.gz 2024-09-16T04:48:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32075.log 2024-09-16T04:48:10.443 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36034.log.gz 2024-09-16T04:48:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36540.log 2024-09-16T04:48:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44641.log: /var/log/ceph/ceph-client.admin.32075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43305.log 2024-09-16T04:48:10.443 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39552.log.gz 2024-09-16T04:48:10.443 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44641.log.gz 2024-09-16T04:48:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37708.log 2024-09-16T04:48:10.443 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32075.log.gz 2024-09-16T04:48:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43494.log 2024-09-16T04:48:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43305.log: /var/log/ceph/ceph-client.admin.37708.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36540.log.gz 2024-09-16T04:48:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43179.log 2024-09-16T04:48:10.443 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43305.log.gz 2024-09-16T04:48:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35840.log 2024-09-16T04:48:10.443 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37708.log.gz 2024-09-16T04:48:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35798.log 2024-09-16T04:48:10.444 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43494.log.gz 2024-09-16T04:48:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41016.log 2024-09-16T04:48:10.444 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43179.log.gz 2024-09-16T04:48:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31771.log 2024-09-16T04:48:10.444 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35840.log.gz 2024-09-16T04:48:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41016.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-16T04:48:10.444 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35798.log.gz 2024-09-16T04:48:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.31771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43009.log 2024-09-16T04:48:10.444 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41016.log.gz 2024-09-16T04:48:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:gzip/var/log/ceph/ceph-mon.c.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40422.log 2024-09-16T04:48:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43009.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.31771.log.gz 2024-09-16T04:48:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44935.log 2024-09-16T04:48:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 89.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.43009.log.gz 2024-09-16T04:48:10.445 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/ceph-client.admin.41059.log 2024-09-16T04:48:10.445 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40422.log: /var/log/ceph/ceph-client.admin.44935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42591.log 2024-09-16T04:48:10.445 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40422.log.gz 2024-09-16T04:48:10.445 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41059.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44935.log.gz 2024-09-16T04:48:10.445 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41625.log 2024-09-16T04:48:10.445 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36666.log 2024-09-16T04:48:10.445 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36624.log 91.1% -- replaced with /var/log/ceph/ceph-client.admin.41059.log.gz 2024-09-16T04:48:10.445 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:48:10.445 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36666.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.41625.log.gz 2024-09-16T04:48:10.445 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44347.log 2024-09-16T04:48:10.445 INFO:teuthology.orchestra.run.smithi046.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.42591.log.gz 2024-09-16T04:48:10.445 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37040.log 2024-09-16T04:48:10.445 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36666.log.gz 2024-09-16T04:48:10.446 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39594.log 2024-09-16T04:48:10.446 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.37040.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36624.log.gz 2024-09-16T04:48:10.446 INFO:teuthology.orchestra.run.smithi046.stderr: -5 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44347.log.gz 2024-09-16T04:48:10.446 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph.tmp-client.admin.27388.log 2024-09-16T04:48:10.446 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41497.log 89.9% -- replaced with /var/log/ceph/ceph-client.admin.37040.log.gz 2024-09-16T04:48:10.446 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:48:10.446 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.tmp-client.admin.27388.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.39594.log.gz 2024-09-16T04:48:10.446 INFO:teuthology.orchestra.run.smithi046.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27388.log.gz 2024-09-16T04:48:10.446 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43714.log 2024-09-16T04:48:10.446 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32000.log 2024-09-16T04:48:10.446 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41497.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.43714.log: /var/log/ceph/ceph-client.admin.34855.log 2024-09-16T04:48:10.446 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.32000.log: gzip -5 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41497.log.gz 2024-09-16T04:48:10.446 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40504.log 2024-09-16T04:48:10.446 INFO:teuthology.orchestra.run.smithi046.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.43714.log.gz 2024-09-16T04:48:10.447 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.34855.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32000.log.gz 2024-09-16T04:48:10.447 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38567.log 2024-09-16T04:48:10.447 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34855.log.gz 2024-09-16T04:48:10.447 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44767.log 2024-09-16T04:48:10.447 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40504.log: /var/log/ceph/ceph-client.admin.38567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44305.log 2024-09-16T04:48:10.447 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40095.log 2024-09-16T04:48:10.447 INFO:teuthology.orchestra.run.smithi046.stderr: 90.2% 90.9% -- replaced with /var/log/ceph/ceph-client.admin.40504.log.gz 2024-09-16T04:48:10.447 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38567.log.gz 2024-09-16T04:48:10.447 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40594.log 2024-09-16T04:48:10.447 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44767.log.gz 2024-09-16T04:48:10.447 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40095.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T04:48:10.447 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44305.log.gz 2024-09-16T04:48:10.447 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37166.log 2024-09-16T04:48:10.447 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40095.log.gz 2024-09-16T04:48:10.448 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37960.log 2024-09-16T04:48:10.448 INFO:teuthology.orchestra.run.smithi046.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.40594.log.gz 2024-09-16T04:48:10.448 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.37166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35778.log 2024-09-16T04:48:10.448 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.37960.log: gzip 88.8% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.37166.log.gz -- 2024-09-16T04:48:10.448 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/ceph-client.admin.35950.log 2024-09-16T04:48:10.448 INFO:teuthology.orchestra.run.smithi046.stderr:gzip/var/log/ceph/ceph-client.admin.35778.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39510.log 2024-09-16T04:48:10.448 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37960.log.gz 2024-09-16T04:48:10.448 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35698.log 2024-09-16T04:48:10.448 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35778.log.gz 2024-09-16T04:48:10.448 INFO:teuthology.orchestra.run.smithi046.stderr:gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35950.log.gz -5 2024-09-16T04:48:10.448 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42635.log 2024-09-16T04:48:10.448 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39510.log: gzip/var/log/ceph/ceph-client.admin.35698.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36742.log 2024-09-16T04:48:10.448 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40848.log 2024-09-16T04:48:10.449 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35698.log.gz 2024-09-16T04:48:10.449 INFO:teuthology.orchestra.run.smithi046.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.39510.log.gz 2024-09-16T04:48:10.449 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36742.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42635.log.gz 2024-09-16T04:48:10.449 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43347.log 2024-09-16T04:48:10.449 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38147.log 2024-09-16T04:48:10.449 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38525.log 2024-09-16T04:48:10.449 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36742.log.gz 2024-09-16T04:48:10.449 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40848.log.gz 2024-09-16T04:48:10.449 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.38147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42751.log 2024-09-16T04:48:10.449 INFO:teuthology.orchestra.run.smithi046.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.43347.log.gz 2024-09-16T04:48:10.449 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7%/var/log/ceph/ceph-client.admin.38525.log: -- replaced with /var/log/ceph/ceph-client.admin.38147.log.gz 2024-09-16T04:48:10.449 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-16T04:48:10.449 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38525.log.gz 2024-09-16T04:48:10.450 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44809.log 2024-09-16T04:48:10.450 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36582.log 2024-09-16T04:48:10.450 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42751.log.gz 2024-09-16T04:48:10.450 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41667.log 2024-09-16T04:48:10.450 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38609.log 2024-09-16T04:48:10.450 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36582.log: /var/log/ceph/ceph-client.admin.41667.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44809.log.gz 2024-09-16T04:48:10.450 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38441.log 2024-09-16T04:48:10.450 INFO:teuthology.orchestra.run.smithi046.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.41667.log.gz/var/log/ceph/ceph-client.admin.38609.log: 2024-09-16T04:48:10.450 INFO:teuthology.orchestra.run.smithi046.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.36582.log.gz 2024-09-16T04:48:10.450 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38609.log.gz 2024-09-16T04:48:10.450 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41793.log 2024-09-16T04:48:10.450 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.38441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41286.log 2024-09-16T04:48:10.450 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41793.log: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44095.log 2024-09-16T04:48:10.451 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38441.log.gz 2024-09-16T04:48:10.451 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41793.log.gz/var/log/ceph/ceph-client.admin.41286.log: gzip 2024-09-16T04:48:10.451 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38086.log 2024-09-16T04:48:10.451 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44095.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41286.log.gz 2024-09-16T04:48:10.451 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44095.log.gz 2024-09-16T04:48:10.451 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45187.log 2024-09-16T04:48:10.451 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.38086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35110.log 2024-09-16T04:48:10.451 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39468.log 2024-09-16T04:48:10.451 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38086.log.gz 2024-09-16T04:48:10.451 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35110.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45187.log.gz 2024-09-16T04:48:10.451 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35110.log.gz 2024-09-16T04:48:10.451 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40764.log 2024-09-16T04:48:10.451 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44473.log 2024-09-16T04:48:10.452 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40764.log: gzip -5 88.3% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.39468.log.gz /var/log/ceph/ceph-client.admin.42339.log 2024-09-16T04:48:10.452 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:48:10.452 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40764.log.gz 2024-09-16T04:48:10.452 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44473.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.44473.log.gz 2024-09-16T04:48:10.452 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45145.log 2024-09-16T04:48:10.452 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42339.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T04:48:10.452 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45145.log: 89.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42881.log 2024-09-16T04:48:10.452 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42339.log.gz 2024-09-16T04:48:10.452 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.45145.log.gz 2024-09-16T04:48:10.452 INFO:teuthology.orchestra.run.smithi046.stderr: 92.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T04:48:10.452 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37250.log 2024-09-16T04:48:10.452 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40347.log 2024-09-16T04:48:10.452 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.37250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42211.log 2024-09-16T04:48:10.452 INFO:teuthology.orchestra.run.smithi046.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.42881.log.gz 2024-09-16T04:48:10.453 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37250.log.gz 2024-09-16T04:48:10.453 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40347.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40347.log.gz 2024-09-16T04:48:10.453 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41205.log 2024-09-16T04:48:10.453 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35718.log 2024-09-16T04:48:10.453 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38002.log 2024-09-16T04:48:10.453 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35718.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.42211.log.gz 2024-09-16T04:48:10.453 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41205.log.gz 2024-09-16T04:48:10.453 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35718.log.gz 2024-09-16T04:48:10.453 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38735.log 2024-09-16T04:48:10.453 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.38002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41455.log 2024-09-16T04:48:10.453 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.38735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38861.log 2024-09-16T04:48:10.453 INFO:teuthology.orchestra.run.smithi046.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.38002.log.gz 2024-09-16T04:48:10.453 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41455.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38735.log.gz 2024-09-16T04:48:10.454 INFO:teuthology.orchestra.run.smithi046.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.41455.log.gz 2024-09-16T04:48:10.454 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42507.log 2024-09-16T04:48:10.454 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.38861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39048.log 2024-09-16T04:48:10.454 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38777.log 2024-09-16T04:48:10.454 INFO:teuthology.orchestra.run.smithi046.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.38861.log.gz 2024-09-16T04:48:10.454 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42507.log.gz 2024-09-16T04:48:10.454 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39048.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.39048.log.gz 2024-09-16T04:48:10.454 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45019.log 2024-09-16T04:48:10.454 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.38777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40305.log 2024-09-16T04:48:10.454 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39757.log 2024-09-16T04:48:10.454 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40305.log: 88.3% 92.0% -- replaced with /var/log/ceph/ceph-client.admin.38777.log.gz 2024-09-16T04:48:10.454 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45019.log.gz 2024-09-16T04:48:10.454 INFO:teuthology.orchestra.run.smithi046.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.40305.log.gz 2024-09-16T04:48:10.454 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36414.log 2024-09-16T04:48:10.455 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43800.log 2024-09-16T04:48:10.455 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43842.log 2024-09-16T04:48:10.455 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39757.log.gz 2024-09-16T04:48:10.455 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43800.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36414.log.gz 2024-09-16T04:48:10.455 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43800.log.gz 2024-09-16T04:48:10.455 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35542.log 2024-09-16T04:48:10.455 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43431.log 2024-09-16T04:48:10.455 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43389.log 2024-09-16T04:48:10.455 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43842.log.gz 2024-09-16T04:48:10.455 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43431.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35542.log.gz 2024-09-16T04:48:10.455 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43431.log.gz 2024-09-16T04:48:10.455 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44977.log 2024-09-16T04:48:10.455 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36372.log 2024-09-16T04:48:10.456 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40932.log 2024-09-16T04:48:10.456 INFO:teuthology.orchestra.run.smithi046.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.43389.log.gz 2024-09-16T04:48:10.456 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36372.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44977.log.gz 2024-09-16T04:48:10.456 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36372.log.gz 2024-09-16T04:48:10.456 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42295.log 2024-09-16T04:48:10.456 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44683.log 2024-09-16T04:48:10.456 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41709.log 2024-09-16T04:48:10.456 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40932.log.gz 2024-09-16T04:48:10.456 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42295.log.gz 2024-09-16T04:48:10.456 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44683.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44683.log.gz 2024-09-16T04:48:10.456 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45353.log 2024-09-16T04:48:10.456 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34992.log 2024-09-16T04:48:10.456 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36202.log 2024-09-16T04:48:10.457 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41709.log.gz 2024-09-16T04:48:10.457 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.34992.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45353.log.gz 2024-09-16T04:48:10.457 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34992.log.gz 2024-09-16T04:48:10.457 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42923.log 2024-09-16T04:48:10.457 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44179.log 2024-09-16T04:48:10.457 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38355.log 2024-09-16T04:48:10.457 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36202.log.gz 2024-09-16T04:48:10.457 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44179.log: gzip -5 91.3% --verbose -- /var/log/ceph/ceph-client.admin.39883.log 2024-09-16T04:48:10.457 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42923.log.gz 2024-09-16T04:48:10.457 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.38355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39258.log 2024-09-16T04:48:10.457 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44179.log.gz 2024-09-16T04:48:10.457 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39883.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.38355.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42127.log 2024-09-16T04:48:10.457 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:48:10.458 INFO:teuthology.orchestra.run.smithi046.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.39883.log.gz 2024-09-16T04:48:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39258.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39258.log.gz 2024-09-16T04:48:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34936.log 2024-09-16T04:48:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43263.log 2024-09-16T04:48:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.34936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35152.log 2024-09-16T04:48:10.460 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42127.log.gz 2024-09-16T04:48:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43263.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34936.log.gz 2024-09-16T04:48:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36868.log 2024-09-16T04:48:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36498.log 2024-09-16T04:48:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36868.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43263.log.gz 2024-09-16T04:48:10.461 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35152.log.gz 2024-09-16T04:48:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41413.log 2024-09-16T04:48:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36498.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.36868.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40137.log 2024-09-16T04:48:10.461 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:48:10.461 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42381.log 2024-09-16T04:48:10.461 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36498.log.gz 2024-09-16T04:48:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41413.log: /var/log/ceph/ceph-client.admin.40137.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41413.log.gz 2024-09-16T04:48:10.461 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40137.log.gz 2024-09-16T04:48:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40263.log 2024-09-16T04:48:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37918.log 2024-09-16T04:48:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36956.log 2024-09-16T04:48:10.461 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42381.log.gz 2024-09-16T04:48:10.462 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.37918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39669.log 90.7% -- replaced with /var/log/ceph/ceph-client.admin.40263.log.gz 2024-09-16T04:48:10.462 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:48:10.462 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39925.log 2024-09-16T04:48:10.462 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37918.log.gz 2024-09-16T04:48:10.462 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5%/var/log/ceph/ceph-client.admin.39669.log: -- replaced with /var/log/ceph/ceph-client.admin.36956.log.gz 2024-09-16T04:48:10.462 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38399.log 2024-09-16T04:48:10.462 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39669.log.gz/var/log/ceph/ceph-client.admin.39925.log: 2024-09-16T04:48:10.462 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39342.log 2024-09-16T04:48:10.462 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8%/var/log/ceph/ceph-client.admin.38399.log: -- replaced with /var/log/ceph/ceph-client.admin.39925.log.gz 2024-09-16T04:48:10.462 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37876.log 2024-09-16T04:48:10.462 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38651.log 2024-09-16T04:48:10.462 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38399.log.gz/var/log/ceph/ceph-client.admin.39342.log: 2024-09-16T04:48:10.462 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.37876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43926.log 2024-09-16T04:48:10.463 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.38651.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.39342.log.gz 2024-09-16T04:48:10.463 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37876.log.gz 2024-09-16T04:48:10.463 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37578.log 2024-09-16T04:48:10.463 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38651.log.gz 2024-09-16T04:48:10.463 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38271.log 2024-09-16T04:48:10.463 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43926.log: /var/log/ceph/ceph-client.admin.37578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36914.log 2024-09-16T04:48:10.463 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.38271.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.43926.log.gz 2024-09-16T04:48:10.463 INFO:teuthology.orchestra.run.smithi046.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.37578.log.gz 2024-09-16T04:48:10.463 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36076.log 2024-09-16T04:48:10.463 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38271.log.gz 2024-09-16T04:48:10.464 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38922.log 2024-09-16T04:48:10.464 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36914.log: /var/log/ceph/ceph-client.admin.36076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39969.log 2024-09-16T04:48:10.464 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.38922.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36076.log.gz 2024-09-16T04:48:10.464 INFO:teuthology.orchestra.run.smithi046.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.36914.log.gz 2024-09-16T04:48:10.464 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38922.log.gz 2024-09-16T04:48:10.464 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40636.log 2024-09-16T04:48:10.464 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36160.log 2024-09-16T04:48:10.464 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40636.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39969.log.gz 2024-09-16T04:48:10.464 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44431.log 2024-09-16T04:48:10.464 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36456.log 2024-09-16T04:48:10.464 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36160.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.40636.log.gz 2024-09-16T04:48:10.464 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37124.log 2024-09-16T04:48:10.464 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36160.log.gz 2024-09-16T04:48:10.464 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36456.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44431.log.gz 2024-09-16T04:48:10.465 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41920.log 2024-09-16T04:48:10.465 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.37124.log: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40179.log 2024-09-16T04:48:10.465 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36456.log.gz 2024-09-16T04:48:10.465 INFO:teuthology.orchestra.run.smithi046.stderr:gzip 89.9% -- replaced with /var/log/ceph/ceph-client.admin.37124.log.gz 2024-09-16T04:48:10.465 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43758.log 2024-09-16T04:48:10.465 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40179.log: /var/log/ceph/ceph-client.admin.41920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35017.log 2024-09-16T04:48:10.465 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43758.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40179.log.gz 2024-09-16T04:48:10.465 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41920.log.gz 2024-09-16T04:48:10.465 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43758.log.gz 2024-09-16T04:48:10.465 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35017.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35017.log.gz 2024-09-16T04:48:10.503 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37217.log 2024-09-16T04:48:10.503 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40390.log 2024-09-16T04:48:10.503 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38715.log 2024-09-16T04:48:10.503 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36828.log 2024-09-16T04:48:10.504 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39854.log 2024-09-16T04:48:10.504 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37465.log 89.3% 2024-09-16T04:48:10.504 INFO:teuthology.orchestra.run.smithi114.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37217.log.gz 2024-09-16T04:48:10.504 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36828.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.40390.log.gz 2024-09-16T04:48:10.504 INFO:teuthology.orchestra.run.smithi114.stderr: 90.7%/var/log/ceph/ceph-client.admin.39854.log: -- replaced with /var/log/ceph/ceph-client.admin.38715.log.gz 2024-09-16T04:48:10.504 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40176.log 2024-09-16T04:48:10.504 INFO:teuthology.orchestra.run.smithi114.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36828.log.gz 2024-09-16T04:48:10.504 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39337.log 2024-09-16T04:48:10.504 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39854.log.gz 2024-09-16T04:48:10.504 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36720.log 2024-09-16T04:48:10.504 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31493.log 2024-09-16T04:48:10.504 INFO:teuthology.orchestra.run.smithi114.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.37465.log.gz 2024-09-16T04:48:10.505 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36720.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.40176.log.gz 2024-09-16T04:48:10.505 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36898.log 2024-09-16T04:48:10.505 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37463.log 2024-09-16T04:48:10.505 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36898.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.39337.log.gz 2024-09-16T04:48:10.505 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35642.log 2024-09-16T04:48:10.505 INFO:teuthology.orchestra.run.smithi114.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36720.log.gz 2024-09-16T04:48:10.505 INFO:teuthology.orchestra.run.smithi114.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31493.log.gz 2024-09-16T04:48:10.505 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35365.log 2024-09-16T04:48:10.505 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36684.log 2024-09-16T04:48:10.505 INFO:teuthology.orchestra.run.smithi114.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36898.log.gz 2024-09-16T04:48:10.505 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38607.log 2024-09-16T04:48:10.505 INFO:teuthology.orchestra.run.smithi114.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35642.log.gz 2024-09-16T04:48:10.505 INFO:teuthology.orchestra.run.smithi114.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37463.log.gz 2024-09-16T04:48:10.505 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35436.log 2024-09-16T04:48:10.506 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39962.log 2024-09-16T04:48:10.506 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35365.log.gz 2024-09-16T04:48:10.506 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35436.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38607.log.gz 2024-09-16T04:48:10.506 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39442.log 2024-09-16T04:48:10.506 INFO:teuthology.orchestra.run.smithi114.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36684.log.gz 2024-09-16T04:48:10.506 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37146.log 2024-09-16T04:48:10.506 INFO:teuthology.orchestra.run.smithi114.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35436.log.gz 2024-09-16T04:48:10.506 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39962.log: /var/log/ceph/ceph-client.admin.39442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36685.log 2024-09-16T04:48:10.506 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37360.log 2024-09-16T04:48:10.506 INFO:teuthology.orchestra.run.smithi114.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.39962.log.gz 2024-09-16T04:48:10.506 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36685.log: 90.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.39442.log.gz 2024-09-16T04:48:10.506 INFO:teuthology.orchestra.run.smithi114.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39750.log 2024-09-16T04:48:10.506 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36158.log 2024-09-16T04:48:10.507 INFO:teuthology.orchestra.run.smithi114.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.36685.log.gz 2024-09-16T04:48:10.507 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39750.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37146.log.gz 2024-09-16T04:48:10.507 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37568.log 2024-09-16T04:48:10.507 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38287.log 2024-09-16T04:48:10.507 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39750.log.gz 2024-09-16T04:48:10.507 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36158.log: /var/log/ceph/ceph-client.admin.37568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39406.log 2024-09-16T04:48:10.507 INFO:teuthology.orchestra.run.smithi114.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37360.log.gz 2024-09-16T04:48:10.507 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38287.log: gzip -5 88.4% --verbose -- replaced with /var/log/ceph/ceph-client.admin.36158.log.gz -- 2024-09-16T04:48:10.507 INFO:teuthology.orchestra.run.smithi114.stderr: /var/log/ceph/ceph-client.admin.40068.log 2024-09-16T04:48:10.507 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40282.log 2024-09-16T04:48:10.507 INFO:teuthology.orchestra.run.smithi114.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.37568.log.gz 2024-09-16T04:48:10.507 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37635.log 2024-09-16T04:48:10.507 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40282.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38287.log.gz 2024-09-16T04:48:10.507 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35367.log 2024-09-16T04:48:10.508 INFO:teuthology.orchestra.run.smithi114.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40068.log.gz 2024-09-16T04:48:10.508 INFO:teuthology.orchestra.run.smithi114.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.39406.log.gz 2024-09-16T04:48:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38081.log 2024-09-16T04:48:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37635.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40282.log.gz 2024-09-16T04:48:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35319.log 2024-09-16T04:48:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40283.log 2024-09-16T04:48:10.508 INFO:teuthology.orchestra.run.smithi114.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.37635.log.gz 2024-09-16T04:48:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35319.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.35367.log.gz 2024-09-16T04:48:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36157.log 2024-09-16T04:48:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37039.log 2024-09-16T04:48:10.508 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38081.log.gz 2024-09-16T04:48:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37464.log 2024-09-16T04:48:10.508 INFO:teuthology.orchestra.run.smithi114.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.35319.log.gz 2024-09-16T04:48:10.509 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37039.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.40283.log.gz 2024-09-16T04:48:10.509 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38357.log 2024-09-16T04:48:10.509 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36296.log 2024-09-16T04:48:10.509 INFO:teuthology.orchestra.run.smithi114.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36157.log.gz 2024-09-16T04:48:10.509 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38357.log: gzip -5 --verbose -- 89.5% /var/log/ceph/ceph-client.admin.36792.log -- replaced with /var/log/ceph/ceph-client.admin.37039.log.gz 2024-09-16T04:48:10.509 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:48:10.509 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37464.log.gz 2024-09-16T04:48:10.509 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39441.log 2024-09-16T04:48:10.509 INFO:teuthology.orchestra.run.smithi114.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38357.log.gz 2024-09-16T04:48:10.509 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36399.log 2024-09-16T04:48:10.509 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36506.log 2024-09-16T04:48:10.509 INFO:teuthology.orchestra.run.smithi114.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.36296.log.gz 2024-09-16T04:48:10.509 INFO:teuthology.orchestra.run.smithi114.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36792.log.gz 2024-09-16T04:48:10.510 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37532.log 2024-09-16T04:48:10.510 INFO:teuthology.orchestra.run.smithi114.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39441.log.gz 2024-09-16T04:48:10.510 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38820.log 2024-09-16T04:48:10.510 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40388.log 2024-09-16T04:48:10.510 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36399.log.gz 2024-09-16T04:48:10.510 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35641.log 2024-09-16T04:48:10.510 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36506.log.gz 2024-09-16T04:48:10.510 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31442.log 2024-09-16T04:48:10.510 INFO:teuthology.orchestra.run.smithi114.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37532.log.gz 2024-09-16T04:48:10.510 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37980.log 2024-09-16T04:48:10.510 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35641.log: /var/log/ceph/ceph-client.admin.31442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40138.log 2024-09-16T04:48:10.510 INFO:teuthology.orchestra.run.smithi114.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38820.log.gz 2024-09-16T04:48:10.510 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40388.log.gz 2024-09-16T04:48:10.511 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38570.log 2024-09-16T04:48:10.511 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35641.log.gz 2024-09-16T04:48:10.511 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31442.log.gz 2024-09-16T04:48:10.511 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37041.log 2024-09-16T04:48:10.511 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40138.log: /var/log/ceph/ceph-client.admin.38570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35849.log 2024-09-16T04:48:10.511 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37041.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.37980.log.gz 2024-09-16T04:48:10.511 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40067.log 2024-09-16T04:48:10.511 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39236.log 2024-09-16T04:48:10.511 INFO:teuthology.orchestra.run.smithi114.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40138.log.gz 2024-09-16T04:48:10.511 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38570.log.gz 2024-09-16T04:48:10.511 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37877.log 2024-09-16T04:48:10.511 INFO:teuthology.orchestra.run.smithi114.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35849.log.gz 2024-09-16T04:48:10.511 INFO:teuthology.orchestra.run.smithi114.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.37041.log.gz 2024-09-16T04:48:10.512 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31653.log 2024-09-16T04:48:10.512 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39236.log: /var/log/ceph/ceph-client.admin.37877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36364.log 2024-09-16T04:48:10.512 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39028.log 2024-09-16T04:48:10.512 INFO:teuthology.orchestra.run.smithi114.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.39236.log.gz 2024-09-16T04:48:10.512 INFO:teuthology.orchestra.run.smithi114.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40067.log.gz 2024-09-16T04:48:10.512 INFO:teuthology.orchestra.run.smithi114.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31653.log.gz 2024-09-16T04:48:10.512 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36364.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.37877.log.gz 2024-09-16T04:48:10.512 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36401.log 2024-09-16T04:48:10.512 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38288.log 2024-09-16T04:48:10.512 INFO:teuthology.orchestra.run.smithi114.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36364.log.gz 2024-09-16T04:48:10.512 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38995.log 2024-09-16T04:48:10.512 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38462.log 2024-09-16T04:48:10.512 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39028.log.gz 2024-09-16T04:48:10.512 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36400.log 2024-09-16T04:48:10.513 INFO:teuthology.orchestra.run.smithi114.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.36401.log.gz 2024-09-16T04:48:10.513 INFO:teuthology.orchestra.run.smithi114.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38288.log.gz 2024-09-16T04:48:10.513 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35471.log 2024-09-16T04:48:10.513 INFO:teuthology.orchestra.run.smithi114.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38995.log.gz 2024-09-16T04:48:10.513 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36507.log 2024-09-16T04:48:10.513 INFO:teuthology.orchestra.run.smithi114.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38462.log.gz 2024-09-16T04:48:10.513 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35884.log 2024-09-16T04:48:10.513 INFO:teuthology.orchestra.run.smithi114.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36400.log.gz 2024-09-16T04:48:10.513 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40175.log 2024-09-16T04:48:10.513 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37429.log 2024-09-16T04:48:10.513 INFO:teuthology.orchestra.run.smithi114.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36507.log.gz 2024-09-16T04:48:10.513 INFO:teuthology.orchestra.run.smithi114.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.35471.log.gz 2024-09-16T04:48:10.513 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38082.log 2024-09-16T04:48:10.514 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36193.log 2024-09-16T04:48:10.514 INFO:teuthology.orchestra.run.smithi114.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.35884.log.gz 2024-09-16T04:48:10.514 INFO:teuthology.orchestra.run.smithi114.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40175.log.gz 2024-09-16T04:48:10.514 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36260.log 2024-09-16T04:48:10.514 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38289.log 2024-09-16T04:48:10.514 INFO:teuthology.orchestra.run.smithi114.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38082.log.gz 2024-09-16T04:48:10.514 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37429.log.gz 2024-09-16T04:48:10.514 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36722.log 2024-09-16T04:48:10.514 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38289.log: gzip -5 90.5% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.36193.log.gz /var/log/ceph/ceph-client.admin.38185.log 2024-09-16T04:48:10.514 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:48:10.514 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39339.log 2024-09-16T04:48:10.514 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36260.log.gz 2024-09-16T04:48:10.514 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37255.log 2024-09-16T04:48:10.514 INFO:teuthology.orchestra.run.smithi114.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.38289.log.gz 2024-09-16T04:48:10.515 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39339.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.36722.log.gz 2024-09-16T04:48:10.515 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37944.log 2024-09-16T04:48:10.515 INFO:teuthology.orchestra.run.smithi114.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38185.log.gz 2024-09-16T04:48:10.515 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39545.log 2024-09-16T04:48:10.515 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35883.log 2024-09-16T04:48:10.515 INFO:teuthology.orchestra.run.smithi114.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.39339.log.gz 2024-09-16T04:48:10.515 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39545.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-16T04:48:10.515 INFO:teuthology.orchestra.run.smithi114.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.37255.log.gz 2024-09-16T04:48:10.515 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38821.log 2024-09-16T04:48:10.515 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37944.log.gz 2024-09-16T04:48:10.515 INFO:teuthology.orchestra.run.smithi114.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.39545.log.gz 2024-09-16T04:48:10.515 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35470.log 2024-09-16T04:48:10.515 INFO:teuthology.orchestra.run.smithi114.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35883.log.gz 2024-09-16T04:48:10.516 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39234.log 2024-09-16T04:48:10.516 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38821.log: /var/log/ceph/ceph-client.admin.35470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35985.log 2024-09-16T04:48:10.516 INFO:teuthology.orchestra.run.smithi114.stderr: 89.0%/var/log/ceph/ceph-client.admin.39234.log: -- replaced with /var/log/ceph/ceph-client.admin.38821.log.gz 2024-09-16T04:48:10.516 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36191.log 2024-09-16T04:48:10.516 INFO:teuthology.orchestra.run.smithi114.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35470.log.gz 2024-09-16T04:48:10.516 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.35985.log: /var/log/ceph/ceph-client.admin.38822.log 2024-09-16T04:48:10.516 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39132.log 2024-09-16T04:48:10.516 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39234.log.gz 2024-09-16T04:48:10.516 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37774.log 2024-09-16T04:48:10.516 INFO:teuthology.orchestra.run.smithi114.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35985.log.gz 2024-09-16T04:48:10.516 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39132.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T04:48:10.516 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37945.log 2024-09-16T04:48:10.516 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36191.log.gz 2024-09-16T04:48:10.517 INFO:teuthology.orchestra.run.smithi114.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39132.log.gz 2024-09-16T04:48:10.517 INFO:teuthology.orchestra.run.smithi114.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.38822.log.gz 2024-09-16T04:48:10.517 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38713.log 2024-09-16T04:48:10.517 INFO:teuthology.orchestra.run.smithi114.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.37774.log.gz 2024-09-16T04:48:10.517 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40069.log 2024-09-16T04:48:10.517 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37945.log: /var/log/ceph/ceph-client.admin.38713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37978.log 2024-09-16T04:48:10.517 INFO:teuthology.orchestra.run.smithi114.stderr: 89.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T04:48:10.517 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40069.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37945.log.gz 2024-09-16T04:48:10.517 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31613.log 2024-09-16T04:48:10.517 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37361.log 2024-09-16T04:48:10.517 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37978.log: /var/log/ceph/ceph-client.admin.31613.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.40069.log.gz 2024-09-16T04:48:10.517 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38713.log.gz 2024-09-16T04:48:10.517 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39647.log 2024-09-16T04:48:10.517 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36508.log 2024-09-16T04:48:10.518 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37361.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.31613.log.gz 2024-09-16T04:48:10.518 INFO:teuthology.orchestra.run.smithi114.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37978.log.gz 2024-09-16T04:48:10.518 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39201.log 2024-09-16T04:48:10.518 INFO:teuthology.orchestra.run.smithi114.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37361.log.gz 2024-09-16T04:48:10.518 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31533.log 2024-09-16T04:48:10.518 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36508.log: /var/log/ceph/ceph-client.admin.39201.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39647.log.gz 2024-09-16T04:48:10.518 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38391.log 2024-09-16T04:48:10.518 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39440.log 2024-09-16T04:48:10.518 INFO:teuthology.orchestra.run.smithi114.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.36508.log.gz 2024-09-16T04:48:10.518 INFO:teuthology.orchestra.run.smithi114.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39201.log.gz 2024-09-16T04:48:10.518 INFO:teuthology.orchestra.run.smithi114.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31533.log.gz 2024-09-16T04:48:10.518 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36791.log 2024-09-16T04:48:10.518 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37430.log 2024-09-16T04:48:10.519 INFO:teuthology.orchestra.run.smithi114.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38391.log.gz 2024-09-16T04:48:10.519 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39646.log 2024-09-16T04:48:10.519 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40352.log 2024-09-16T04:48:10.519 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37430.log: /var/log/ceph/ceph-client.admin.39646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35745.log 2024-09-16T04:48:10.519 INFO:teuthology.orchestra.run.smithi114.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36791.log.gz 2024-09-16T04:48:10.519 INFO:teuthology.orchestra.run.smithi114.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37430.log.gz 2024-09-16T04:48:10.519 INFO:teuthology.orchestra.run.smithi114.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.39440.log.gz 2024-09-16T04:48:10.519 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38606.log 2024-09-16T04:48:10.519 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36055.log 2024-09-16T04:48:10.519 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39819.log 2024-09-16T04:48:10.519 INFO:teuthology.orchestra.run.smithi114.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39646.log.gz 2024-09-16T04:48:10.519 INFO:teuthology.orchestra.run.smithi114.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35745.log.gz 2024-09-16T04:48:10.519 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37003.log 2024-09-16T04:48:10.520 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40352.log.gz 2024-09-16T04:48:10.520 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35779.log 2024-09-16T04:48:10.520 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38606.log.gz 2024-09-16T04:48:10.520 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37003.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.36055.log.gz 2024-09-16T04:48:10.520 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35951.log 2024-09-16T04:48:10.520 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35779.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39819.log.gz 2024-09-16T04:48:10.520 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38184.log 2024-09-16T04:48:10.520 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37670.log 2024-09-16T04:48:10.520 INFO:teuthology.orchestra.run.smithi114.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35779.log.gz 89.7% 2024-09-16T04:48:10.520 INFO:teuthology.orchestra.run.smithi114.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37003.log.gz 2024-09-16T04:48:10.521 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37979.log 2024-09-16T04:48:10.521 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38892.log 2024-09-16T04:48:10.521 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35951.log.gz 2024-09-16T04:48:10.521 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38186.log 2024-09-16T04:48:10.521 INFO:teuthology.orchestra.run.smithi114.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37670.log.gz 2024-09-16T04:48:10.521 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38184.log.gz 2024-09-16T04:48:10.521 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35539.log 2024-09-16T04:48:10.521 INFO:teuthology.orchestra.run.smithi114.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37979.log.gz 2024-09-16T04:48:10.521 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37669.log 2024-09-16T04:48:10.521 INFO:teuthology.orchestra.run.smithi114.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38892.log.gz 2024-09-16T04:48:10.521 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39200.log 2024-09-16T04:48:10.521 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37669.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.38186.log.gz 2024-09-16T04:48:10.521 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38151.log 2024-09-16T04:48:10.522 INFO:teuthology.orchestra.run.smithi114.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35539.log.gz 2024-09-16T04:48:10.522 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36089.log 2024-09-16T04:48:10.522 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39200.log: /var/log/ceph/ceph-client.admin.38151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38571.log 2024-09-16T04:48:10.522 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37669.log.gz 2024-09-16T04:48:10.522 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36089.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38151.log.gz 2024-09-16T04:48:10.522 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39098.log 2024-09-16T04:48:10.522 INFO:teuthology.orchestra.run.smithi114.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39200.log.gz 2024-09-16T04:48:10.522 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35952.log 2024-09-16T04:48:10.522 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38571.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36089.log.gz 2024-09-16T04:48:10.522 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35573.log 2024-09-16T04:48:10.522 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35952.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38571.log.gz 2024-09-16T04:48:10.522 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38784.log 2024-09-16T04:48:10.522 INFO:teuthology.orchestra.run.smithi114.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39098.log.gz 2024-09-16T04:48:10.522 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39716.log 2024-09-16T04:48:10.523 INFO:teuthology.orchestra.run.smithi114.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35952.log.gz 2024-09-16T04:48:10.523 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35882.log 2024-09-16T04:48:10.523 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38784.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35573.log.gz 2024-09-16T04:48:10.523 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35538.log 2024-09-16T04:48:10.523 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39543.log 2024-09-16T04:48:10.523 INFO:teuthology.orchestra.run.smithi114.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39716.log.gz 2024-09-16T04:48:10.523 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38784.log.gz 2024-09-16T04:48:10.523 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39853.log 2024-09-16T04:48:10.523 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35538.log: /var/log/ceph/ceph-client.admin.39543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37040.log 2024-09-16T04:48:10.523 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39853.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35882.log.gz 2024-09-16T04:48:10.523 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37110.log 2024-09-16T04:48:10.523 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38927.log 2024-09-16T04:48:10.523 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37040.log: /var/log/ceph/ceph-client.admin.37110.log: 89.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36721.log 2024-09-16T04:48:10.523 INFO:teuthology.orchestra.run.smithi114.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35538.log.gz 2024-09-16T04:48:10.524 INFO:teuthology.orchestra.run.smithi114.stderr: 88.1% 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37040.log.gz -- replaced with /var/log/ceph/ceph-client.admin.39543.log.gz 2024-09-16T04:48:10.524 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:48:10.524 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37218.log 2024-09-16T04:48:10.524 INFO:teuthology.orchestra.run.smithi114.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.39853.log.gz 2024-09-16T04:48:10.524 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37111.log 2024-09-16T04:48:10.524 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36721.log: /var/log/ceph/ceph-client.admin.37218.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37110.log.gz 2024-09-16T04:48:10.524 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35366.log 90.6% -- replaced with /var/log/ceph/ceph-client.admin.38927.log.gz 2024-09-16T04:48:10.524 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:48:10.524 INFO:teuthology.orchestra.run.smithi114.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37218.log.gz 2024-09-16T04:48:10.524 INFO:teuthology.orchestra.run.smithi114.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36721.log.gz 2024-09-16T04:48:10.524 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39612.log 2024-09-16T04:48:10.524 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35366.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37111.log.gz 2024-09-16T04:48:10.524 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40389.log 2024-09-16T04:48:10.524 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36577.log 2024-09-16T04:48:10.525 INFO:teuthology.orchestra.run.smithi114.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35366.log.gz 2024-09-16T04:48:10.525 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39924.log 2024-09-16T04:48:10.525 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35986.log 2024-09-16T04:48:10.525 INFO:teuthology.orchestra.run.smithi114.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39612.log.gz 2024-09-16T04:48:10.525 INFO:teuthology.orchestra.run.smithi114.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40389.log.gz 2024-09-16T04:48:10.525 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37841.log 2024-09-16T04:48:10.525 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35744.log 2024-09-16T04:48:10.525 INFO:teuthology.orchestra.run.smithi114.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39924.log.gz 2024-09-16T04:48:10.525 INFO:teuthology.orchestra.run.smithi114.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36577.log.gz 2024-09-16T04:48:10.525 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38253.log 2024-09-16T04:48:10.525 INFO:teuthology.orchestra.run.smithi114.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35986.log.gz 2024-09-16T04:48:10.525 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38083.log 2024-09-16T04:48:10.525 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36578.log 2024-09-16T04:48:10.526 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38253.log: /var/log/ceph/ceph-client.admin.38083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38926.log 2024-09-16T04:48:10.526 INFO:teuthology.orchestra.run.smithi114.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37841.log.gz 2024-09-16T04:48:10.526 INFO:teuthology.orchestra.run.smithi114.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35744.log.gz 2024-09-16T04:48:10.526 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36471.log 2024-09-16T04:48:10.526 INFO:teuthology.orchestra.run.smithi114.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.38083.log.gz 2024-09-16T04:48:10.526 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38926.log: gzip -5 --verbose 88.4% -- -- replaced with /var/log/ceph/ceph-client.admin.36578.log.gz /var/log/ceph/ceph-client.admin.40031.log 2024-09-16T04:48:10.526 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:48:10.526 INFO:teuthology.orchestra.run.smithi114.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.38253.log.gz 2024-09-16T04:48:10.526 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38500.log 2024-09-16T04:48:10.526 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36471.log: 87.7%/var/log/ceph/ceph-client.admin.40031.log: -- replaced with /var/log/ceph/ceph-client.admin.38926.log.gz 2024-09-16T04:48:10.526 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38925.log 2024-09-16T04:48:10.526 INFO:teuthology.orchestra.run.smithi114.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36471.log.gz 2024-09-16T04:48:10.526 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36363.log 2024-09-16T04:48:10.526 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35677.log 2024-09-16T04:48:10.527 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40031.log.gz 2024-09-16T04:48:10.527 INFO:teuthology.orchestra.run.smithi114.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.38500.log.gz 2024-09-16T04:48:10.527 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38048.log 2024-09-16T04:48:10.527 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35778.log 2024-09-16T04:48:10.527 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35987.log 2024-09-16T04:48:10.527 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38925.log.gz 2024-09-16T04:48:10.527 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35778.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36363.log.gz 2024-09-16T04:48:10.527 INFO:teuthology.orchestra.run.smithi114.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38048.log.gz 2024-09-16T04:48:10.527 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39510.log 2024-09-16T04:48:10.527 INFO:teuthology.orchestra.run.smithi114.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.35677.log.gz 2024-09-16T04:48:10.527 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5/var/log/ceph/ceph-client.admin.35987.log: --verbose -- /var/log/ceph/ceph-client.admin.31573.log 2024-09-16T04:48:10.527 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37772.log 2024-09-16T04:48:10.527 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31573.log: gzip -5 --verbose -- 89.3% /var/log/ceph/ceph-client.admin.40353.log 2024-09-16T04:48:10.528 INFO:teuthology.orchestra.run.smithi114.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35778.log.gz 2024-09-16T04:48:10.528 INFO:teuthology.orchestra.run.smithi114.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39510.log.gz 2024-09-16T04:48:10.528 INFO:teuthology.orchestra.run.smithi114.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.35987.log.gz 2024-09-16T04:48:10.528 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36192.log 2024-09-16T04:48:10.528 INFO:teuthology.orchestra.run.smithi114.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31573.log.gz 2024-09-16T04:48:10.528 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40245.log 2024-09-16T04:48:10.528 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37253.log 2024-09-16T04:48:10.528 INFO:teuthology.orchestra.run.smithi114.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40353.log.gz 2024-09-16T04:48:10.528 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37772.log.gz 2024-09-16T04:48:10.528 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36934.log 2024-09-16T04:48:10.528 INFO:teuthology.orchestra.run.smithi114.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36192.log.gz 2024-09-16T04:48:10.528 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36933.log 2024-09-16T04:48:10.528 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37147.log 2024-09-16T04:48:10.529 INFO:teuthology.orchestra.run.smithi114.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40245.log.gz 2024-09-16T04:48:10.529 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37148.log 2024-09-16T04:48:10.529 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37147.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.36934.log.gz 2024-09-16T04:48:10.529 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35676.log 2024-09-16T04:48:10.529 INFO:teuthology.orchestra.run.smithi114.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37253.log.gz 2024-09-16T04:48:10.529 INFO:teuthology.orchestra.run.smithi114.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36933.log.gz 2024-09-16T04:48:10.529 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36294.log 2024-09-16T04:48:10.529 INFO:teuthology.orchestra.run.smithi114.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37147.log.gz 2024-09-16T04:48:10.529 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38994.log 2024-09-16T04:48:10.529 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38461.log 2024-09-16T04:48:10.529 INFO:teuthology.orchestra.run.smithi114.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35676.log.gz 2024-09-16T04:48:10.529 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37324.log 2024-09-16T04:48:10.529 INFO:teuthology.orchestra.run.smithi114.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.37148.log.gz 2024-09-16T04:48:10.530 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35780.log 2024-09-16T04:48:10.530 INFO:teuthology.orchestra.run.smithi114.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.36294.log.gz 2024-09-16T04:48:10.530 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39648.log 2024-09-16T04:48:10.530 INFO:teuthology.orchestra.run.smithi114.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38994.log.gz 2024-09-16T04:48:10.530 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39338.log 2024-09-16T04:48:10.530 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39648.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38461.log.gz 2024-09-16T04:48:10.530 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39235.log 2024-09-16T04:48:10.530 INFO:teuthology.orchestra.run.smithi114.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.35780.log.gz 2024-09-16T04:48:10.530 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39338.log: 89.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.37324.log.gz -5 2024-09-16T04:48:10.530 INFO:teuthology.orchestra.run.smithi114.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38499.log 2024-09-16T04:48:10.530 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36088.log 2024-09-16T04:48:10.530 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39235.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.39648.log.gz 2024-09-16T04:48:10.530 INFO:teuthology.orchestra.run.smithi114.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39338.log.gz 2024-09-16T04:48:10.531 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38356.log 2024-09-16T04:48:10.531 INFO:teuthology.orchestra.run.smithi114.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39235.log.gz 2024-09-16T04:48:10.531 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38392.log 2024-09-16T04:48:10.531 INFO:teuthology.orchestra.run.smithi114.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38499.log.gz 2024-09-16T04:48:10.531 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37004.log 2024-09-16T04:48:10.531 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38356.log: /var/log/ceph/ceph-client.admin.38392.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-16T04:48:10.531 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36088.log.gz 2024-09-16T04:48:10.531 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36090.log 2024-09-16T04:48:10.531 INFO:teuthology.orchestra.run.smithi114.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.38392.log.gz 2024-09-16T04:48:10.531 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T04:48:10.531 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.5.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.38356.log.gz 2024-09-16T04:48:10.531 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36090.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37004.log.gz 2024-09-16T04:48:10.531 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35469.log 2024-09-16T04:48:10.531 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35574.log 2024-09-16T04:48:10.532 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35469.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.36090.log.gz 2024-09-16T04:48:10.532 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39097.log 2024-09-16T04:48:10.532 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35675.log 2024-09-16T04:48:10.532 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35574.log: /var/log/ceph/ceph-client.admin.39097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35572.log 2024-09-16T04:48:10.532 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35675.log: 89.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.35469.log.gz 2024-09-16T04:48:10.532 INFO:teuthology.orchestra.run.smithi114.stderr: -5 90.6% -- replaced with /var/log/ceph/ceph-client.admin.35574.log.gz 2024-09-16T04:48:10.532 INFO:teuthology.orchestra.run.smithi114.stderr: --verbose -- /var/log/ceph/ceph-client.admin.37566.log 2024-09-16T04:48:10.532 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39544.log 2024-09-16T04:48:10.532 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39097.log.gz 2024-09-16T04:48:10.532 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35572.log: /var/log/ceph/ceph-client.admin.37566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36932.log 2024-09-16T04:48:10.532 INFO:teuthology.orchestra.run.smithi114.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35675.log.gz 2024-09-16T04:48:10.532 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39544.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T04:48:10.532 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35320.log 2024-09-16T04:48:10.533 INFO:teuthology.orchestra.run.smithi114.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39544.log.gz 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37566.log.gz 2024-09-16T04:48:10.533 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:48:10.533 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-mgr.x.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35572.log.gz 2024-09-16T04:48:10.533 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36829.log 2024-09-16T04:48:10.533 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35320.log: gzip -5 --verbose 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36932.log.gz 2024-09-16T04:48:10.533 INFO:teuthology.orchestra.run.smithi114.stderr: -- /var/log/ceph/ceph-client.admin.39509.log 2024-09-16T04:48:10.533 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35320.log.gz 2024-09-16T04:48:10.533 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39923.log 2024-09-16T04:48:10.533 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38891.log 2024-09-16T04:48:10.533 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39923.log: 88.0% 90.7% -- replaced with /var/log/ceph/ceph-client.admin.36829.log.gz -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T04:48:10.533 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:48:10.533 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39303.log 2024-09-16T04:48:10.533 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5/var/log/ceph/ceph-client.admin.38891.log: --verbose -- /var/log/ceph/ceph-client.admin.37773.log 2024-09-16T04:48:10.534 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39509.log.gz/var/log/ceph/ceph-client.admin.39303.log: 2024-09-16T04:48:10.534 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37671.log 2024-09-16T04:48:10.534 INFO:teuthology.orchestra.run.smithi114.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39923.log.gz 2024-09-16T04:48:10.534 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36827.log 2024-09-16T04:48:10.534 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37773.log: 89.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39715.log 2024-09-16T04:48:10.534 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37671.log: -- replaced with /var/log/ceph/ceph-client.admin.38891.log.gz 2024-09-16T04:48:10.534 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36827.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.39303.log.gz 2024-09-16T04:48:10.534 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40032.log 2024-09-16T04:48:10.534 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36615.log 2024-09-16T04:48:10.534 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37773.log.gz 2024-09-16T04:48:10.534 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39029.log 2024-09-16T04:48:10.534 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40139.log 2024-09-16T04:48:10.534 INFO:teuthology.orchestra.run.smithi114.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36827.log.gz 2024-09-16T04:48:10.534 INFO:teuthology.orchestra.run.smithi114.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40032.log.gz 2024-09-16T04:48:10.535 INFO:teuthology.orchestra.run.smithi114.stderr: 89.6% 90.8% -- replaced with /var/log/ceph/ceph-client.admin.37671.log.gz/var/log/ceph/ceph-client.admin.39029.log: 2024-09-16T04:48:10.535 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.39715.log.gz 2024-09-16T04:48:10.535 INFO:teuthology.orchestra.run.smithi114.stderr: -- /var/log/ceph/ceph-mon.b.log 2024-09-16T04:48:10.535 INFO:teuthology.orchestra.run.smithi114.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.36615.log.gz 2024-09-16T04:48:10.535 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40139.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39029.log.gzgzip 2024-09-16T04:48:10.535 INFO:teuthology.orchestra.run.smithi114.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40246.log 2024-09-16T04:48:10.535 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-mon.b.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40139.log.gz 2024-09-16T04:48:10.535 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39304.log 2024-09-16T04:48:10.535 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38047.log 2024-09-16T04:48:10.535 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40246.log: /var/log/ceph/ceph-client.admin.39304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37636.log 2024-09-16T04:48:10.535 INFO:teuthology.orchestra.run.smithi114.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40246.log.gz 2024-09-16T04:48:10.535 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38047.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39304.log.gz 2024-09-16T04:48:10.535 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40281.log 2024-09-16T04:48:10.535 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39030.log 2024-09-16T04:48:10.536 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40281.log: 89.3%gzip -5 --verbose 88.0% -- -- replaced with /var/log/ceph/ceph-client.admin.37636.log.gz /var/log/ceph/ceph-client.admin.40174.log 2024-09-16T04:48:10.536 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:48:10.536 INFO:teuthology.orchestra.run.smithi114.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38047.log.gz 2024-09-16T04:48:10.536 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.39030.log: -- /var/log/ceph/ceph-client.admin.37254.log 2024-09-16T04:48:10.536 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36295.log 2024-09-16T04:48:10.536 INFO:teuthology.orchestra.run.smithi114.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40281.log.gz 2024-09-16T04:48:10.536 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37254.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.39030.log.gz 2024-09-16T04:48:10.536 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40174.log.gz 2024-09-16T04:48:10.536 INFO:teuthology.orchestra.run.smithi114.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37254.log.gz 2024-09-16T04:48:10.536 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38677.log 2024-09-16T04:48:10.536 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35848.log 2024-09-16T04:48:10.536 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37875.log 2024-09-16T04:48:10.536 INFO:teuthology.orchestra.run.smithi114.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36295.log.gz 2024-09-16T04:48:10.536 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37739.log 2024-09-16T04:48:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37875.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38677.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38498.log 2024-09-16T04:48:10.537 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:48:10.537 INFO:teuthology.orchestra.run.smithi114.stderr: 89.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38390.log 2024-09-16T04:48:10.537 INFO:teuthology.orchestra.run.smithi114.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35848.log.gz 2024-09-16T04:48:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37739.log: /var/log/ceph/ceph-client.admin.38498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36261.log 2024-09-16T04:48:10.537 INFO:teuthology.orchestra.run.smithi114.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37875.log.gz 2024-09-16T04:48:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38390.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.37739.log.gz 2024-09-16T04:48:10.537 INFO:teuthology.orchestra.run.smithi114.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39749.log 2024-09-16T04:48:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38714.log 2024-09-16T04:48:10.537 INFO:teuthology.orchestra.run.smithi114.stderr: 89.4%/var/log/ceph/ceph-client.admin.36261.log: -- replaced with /var/log/ceph/ceph-client.admin.38498.log.gz 2024-09-16T04:48:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39749.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38390.log.gz 2024-09-16T04:48:10.537 INFO:teuthology.orchestra.run.smithi114.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36261.log.gz 2024-09-16T04:48:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39961.log 2024-09-16T04:48:10.537 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39749.log.gz 2024-09-16T04:48:10.538 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37876.log 2024-09-16T04:48:10.538 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5/var/log/ceph/ceph-client.admin.39961.log: --verbose -- /var/log/ceph/ceph-client.admin.39818.log 2024-09-16T04:48:10.538 INFO:teuthology.orchestra.run.smithi114.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38714.log.gz 2024-09-16T04:48:10.538 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37876.log: gzip -5 --verbose -- 88.2% /var/log/ceph/ceph-client.admin.39751.log 2024-09-16T04:48:10.538 INFO:teuthology.orchestra.run.smithi114.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39961.log.gz 2024-09-16T04:48:10.538 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39818.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37876.log.gz 2024-09-16T04:48:10.538 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39407.log 2024-09-16T04:48:10.538 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38150.log 2024-09-16T04:48:10.538 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36470.log 2024-09-16T04:48:10.538 INFO:teuthology.orchestra.run.smithi114.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39818.log.gz 2024-09-16T04:48:10.538 INFO:teuthology.orchestra.run.smithi114.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.39751.log.gz/var/log/ceph/ceph-client.admin.38150.log: 2024-09-16T04:48:10.538 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- 89.2% /var/log/ceph/ceph-osd.3.log 2024-09-16T04:48:10.538 INFO:teuthology.orchestra.run.smithi114.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39407.log.gz 2024-09-16T04:48:10.538 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38254.log 2024-09-16T04:48:10.539 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38678.log 2024-09-16T04:48:10.539 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38254.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.38150.log.gz 2024-09-16T04:48:10.539 INFO:teuthology.orchestra.run.smithi114.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36470.log.gz 2024-09-16T04:48:10.539 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37325.log 2024-09-16T04:48:10.539 INFO:teuthology.orchestra.run.smithi114.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38254.log.gz 2024-09-16T04:48:10.539 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39133.log 2024-09-16T04:48:10.539 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35435.log 2024-09-16T04:48:10.539 INFO:teuthology.orchestra.run.smithi114.stderr: 88.5%/var/log/ceph/ceph-client.admin.37325.log: -- replaced with /var/log/ceph/ceph-client.admin.38678.log.gz 2024-09-16T04:48:10.539 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36614.log 2024-09-16T04:48:10.539 INFO:teuthology.orchestra.run.smithi114.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37325.log.gz 2024-09-16T04:48:10.539 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35435.log: 90.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36613.log 2024-09-16T04:48:10.539 INFO:teuthology.orchestra.run.smithi114.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39133.log.gz 2024-09-16T04:48:10.539 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38785.log 2024-09-16T04:48:10.539 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36614.log: /var/log/ceph/ceph-client.admin.36613.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35435.log.gz 2024-09-16T04:48:10.540 INFO:teuthology.orchestra.run.smithi114.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36614.log.gz 2024-09-16T04:48:10.540 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37738.log 2024-09-16T04:48:10.540 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38785.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36613.log.gz 2024-09-16T04:48:10.540 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37533.log 2024-09-16T04:48:10.540 INFO:teuthology.orchestra.run.smithi114.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39855.log 2024-09-16T04:48:10.540 INFO:teuthology.orchestra.run.smithi114.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38785.log.gz 2024-09-16T04:48:10.540 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37738.log: /var/log/ceph/ceph-client.admin.37533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37567.log 2024-09-16T04:48:10.540 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39855.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37533.log.gz 2024-09-16T04:48:10.540 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39960.log 2024-09-16T04:48:10.540 INFO:teuthology.orchestra.run.smithi114.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.39855.log.gz/var/log/ceph/ceph-client.admin.37567.log: 2024-09-16T04:48:10.540 INFO:teuthology.orchestra.run.smithi114.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37738.log.gz 2024-09-16T04:48:10.540 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37567.log.gz 2024-09-16T04:48:10.540 INFO:teuthology.orchestra.run.smithi114.stderr: -- /var/log/ceph/ceph-client.admin.38608.log 2024-09-16T04:48:10.540 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36899.log 2024-09-16T04:48:10.541 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39131.log 2024-09-16T04:48:10.541 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36899.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39960.log.gz 2024-09-16T04:48:10.541 INFO:teuthology.orchestra.run.smithi114.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.38608.log.gz 2024-09-16T04:48:10.541 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37362.log 2024-09-16T04:48:10.541 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39613.log 2024-09-16T04:48:10.541 INFO:teuthology.orchestra.run.smithi114.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.36899.log.gz 2024-09-16T04:48:10.541 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.37362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36054.log 2024-09-16T04:48:10.541 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39613.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39131.log.gz 2024-09-16T04:48:10.541 INFO:teuthology.orchestra.run.smithi114.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.37362.log.gz 2024-09-16T04:48:10.541 INFO:teuthology.orchestra.run.smithi114.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39613.log.gz 2024-09-16T04:48:10.541 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37842.log 2024-09-16T04:48:10.541 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.36054.log: /var/log/ceph/ceph-client.admin.37842.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37842.log.gz 2024-09-16T04:48:10.541 INFO:teuthology.orchestra.run.smithi114.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36054.log.gz 2024-09-16T04:48:10.541 INFO:teuthology.orchestra.run.smithi114.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T04:48:10.566 INFO:teuthology.orchestra.run.smithi046.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-16T04:48:10.771 INFO:teuthology.orchestra.run.smithi046.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T04:48:10.813 INFO:teuthology.orchestra.run.smithi114.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-16T04:48:12.448 INFO:teuthology.orchestra.run.smithi114.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-16T04:48:12.541 INFO:teuthology.orchestra.run.smithi046.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T04:48:12.578 INFO:teuthology.orchestra.run.smithi114.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-16T04:48:12.841 INFO:teuthology.orchestra.run.smithi114.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-16T04:48:12.843 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T04:48:12.843 INFO:teuthology.orchestra.run.smithi114.stderr:real 0m2.549s 2024-09-16T04:48:12.843 INFO:teuthology.orchestra.run.smithi114.stderr:user 0m7.300s 2024-09-16T04:48:12.843 INFO:teuthology.orchestra.run.smithi114.stderr:sys 0m0.410s 2024-09-16T04:48:13.029 INFO:teuthology.orchestra.run.smithi046.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T04:48:13.130 INFO:teuthology.orchestra.run.smithi046.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T04:48:13.133 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-09-16T04:48:13.133 INFO:teuthology.orchestra.run.smithi046.stderr:real 0m2.876s 2024-09-16T04:48:13.133 INFO:teuthology.orchestra.run.smithi046.stderr:user 0m8.464s 2024-09-16T04:48:13.133 INFO:teuthology.orchestra.run.smithi046.stderr:sys 0m0.420s 2024-09-16T04:48:13.133 INFO:tasks.ceph:Archiving logs... 2024-09-16T04:48:13.134 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906031/remote/smithi046/log 2024-09-16T04:48:13.135 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T04:48:14.735 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906031/remote/smithi114/log 2024-09-16T04:48:14.737 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T04:48:16.165 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T04:48:16.175 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T04:48:16.175 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T04:48:16.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T04:48:16.261 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T04:48:16.269 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T04:48:16.269 DEBUG:teuthology.orchestra.run.smithi046:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:48:16.272 DEBUG:teuthology.orchestra.run.smithi114:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:48:16.287 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-09-16T04:48:16.289 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-09-16T04:48:16.291 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T04:48:16.292 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-09-16T04:48:16.292 INFO:teuthology.orchestra.run.smithi046.stdout:^* hv01.front.sepia.ceph.com 3 6 377 26 +90us[ +128us] +/- 47ms 2024-09-16T04:48:16.292 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 23 +1074us[+1074us] +/- 60ms 2024-09-16T04:48:16.292 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 -1471us[-1471us] +/- 73ms 2024-09-16T04:48:16.292 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:48:16.294 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T04:48:16.294 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-09-16T04:48:16.294 INFO:teuthology.orchestra.run.smithi114.stdout:^* hv01.front.sepia.ceph.com 3 6 377 24 +89us[ +96us] +/- 47ms 2024-09-16T04:48:16.294 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 23 +1092us[+1092us] +/- 60ms 2024-09-16T04:48:16.294 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 -1466us[-1466us] +/- 73ms 2024-09-16T04:48:16.294 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:48:16.294 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T04:48:16.303 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T04:48:16.303 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T04:48:16.351 DEBUG:teuthology.orchestra.run.smithi046:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T04:48:16.427 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T04:48:16.496 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T04:48:16.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:48:16.528 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T04:48:16.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:48:16.558 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T04:48:16.567 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T04:48:16.593 INFO:teuthology.task.internal:Duration was 972.528363 seconds 2024-09-16T04:48:16.593 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T04:48:16.643 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T04:48:16.643 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T04:48:16.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T04:48:16.687 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T04:48:16.688 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T04:48:16.982 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T04:48:16.982 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com 2024-09-16T04:48:16.982 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T04:48:17.045 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T04:48:17.046 DEBUG:teuthology.orchestra.run.smithi114:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T04:48:17.072 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T04:48:17.072 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T04:48:17.088 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T04:48:17.140 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T04:48:17.140 DEBUG:teuthology.orchestra.run.smithi046:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T04:48:17.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T04:48:17.284 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T04:48:17.293 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T04:48:17.294 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T04:48:17.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T04:48:17.343 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T04:48:17.352 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:48:17.362 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:48:17.383 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = core 2024-09-16T04:48:17.407 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2024-09-16T04:48:17.425 DEBUG:teuthology.orchestra.run.smithi046:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:48:17.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:48:17.479 DEBUG:teuthology.orchestra.run.smithi114:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:48:17.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:48:17.493 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T04:48:17.503 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T04:48:17.503 DEBUG:teuthology.misc:Transferring archived files from smithi046:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906031/remote/smithi046 2024-09-16T04:48:17.504 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T04:48:17.555 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906031/remote/smithi114 2024-09-16T04:48:17.555 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T04:48:17.588 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T04:48:17.588 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T04:48:17.591 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T04:48:17.638 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T04:48:17.649 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T04:48:17.650 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T04:48:17.676 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T04:48:17.676 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T04:48:17.679 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T04:48:17.692 INFO:teuthology.orchestra.run.smithi046.stdout: 265336 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 04:48 /home/ubuntu/cephtest 2024-09-16T04:48:17.694 INFO:teuthology.orchestra.run.smithi114.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 04:48 /home/ubuntu/cephtest 2024-09-16T04:48:17.695 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T04:48:17.727 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T04:48:17.862 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/many workloads/pool-create-delete} duration: 972.5283625125885 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T04:48:17.863 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T04:48:17.951 INFO:teuthology.run:pass